tag:blogger.com,1999:blog-3192075693503486662024-02-19T02:13:00.733+00:00Circle TheoryLiving, working and dreaming in <strike>London</strike> <strike>Manchester</strike> LondonFractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.comBlogger118125tag:blogger.com,1999:blog-319207569350348666.post-6372667287738604302019-03-04T14:54:00.001+00:002019-03-04T14:54:46.815+00:00Feedback on Article 3(3)<p>The following is my feedback regarding the proposed legislation around uploading of software to radio devices.</p>
<p>See this link for more details:<br>
<a href="https://blog.mehl.mx/2019/protect-freedom-on-radio-devices-raise-your-voice-today/">https://blog.mehl.mx/2019/protect-freedom-on-radio-devices-raise-your-voice-today/</a></p>
<p>“Upload of software on radio equipment” initiative direct link:<br>
<a href="https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2018-6621038_en">https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2018-6621038_en</a></p>
<p>Feedback direct link:<br>
<a href="https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2018-6621038/feedback/F238237_en?p_id=380919">https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2018-6621038/feedback/F238237_en?p_id=380919</a></p>
<p>This is an extremely concerning piece of proposed legislation and I urge you to read my words here and hopefully understand why this is such a bad idea.</p>
<p>By limiting the type of software that can be loaded onto these devices it will unequivocally lower the overall level of security in the ecosystem. There are countless examples of insecure software which has been left for <em>years</em> on devices that are actively in use, with manufacturers who are unwilling to devote resources to fixing security holes. Open Source has traditionally stepped up to fix these kinds of problems, led by people who want to avoid waste, to be more secure, and to have individual choice.</p>
<p>If users cannot load e.g. open source software onto a device, then this blows up the whole idea of sustainability, up-cycling and re-use. It would be a shameful thing to do as it would render many devices that could be re-used as completely useless and just ending up in landfill. That may as well be an environmental crime that this article would enable.</p>
<p>For example, many SSL attacks have been discovered in the past 5 years, and if the security in some software on a device was found to be vulnerable but no remedy was forthcoming from the manufacturer (as they had gone out of business or they were focusing on latest released devices), then that renders the device an active security risk, potentially compromised, and could cost an enterprise millions in staff time, losses from being attacked, etc. The same kind of situation can exist on phones, on other networked, radio devices such as WiFi routers, etc. Preventing a user to update them in this way is dangerous and actively helps/supports attackers who could otherwise be thwarted.</p>
<p>Allowing open source software on devices (e.g. phones, routers) means that the life, security and performance of them can be hugely extended, in a way that the user is in control of. For routers, it means that a device can be protected from new and active threats / vulnerabilities - these are literally being discovered each week. How quickly are approved updates from a manufacturer likely to arrive? Nowhere near that, obviously.</p>
<p>This article must be removed. This tendency toward centralised control of software on devices owned by individuals, charities, and companies both small and large must be examined carefully. There is simply no way that this can work in a beneficial way to users, because trust has so frequently been broken, and this is essentially guaranteed regardless of legislation.</p>
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-62979268189691985402018-03-17T23:09:00.002+00:002018-03-17T23:09:24.200+00:00Porting Inversion part 2<p><img src="http://www.fractos.com/images/circle-theory/Porting%20Inversion-2.jpg" alt="Looking towards TCR from Euston"><br>
Things have progressed and I think it’s time to write up and highlight a few items.</p>
<h3 id="choices-of-what-to-port">Choices of what to port</h3>
<p>There are some things which I have chosen to abandon for now. These include my Inversion.Web.Razor extensions in <code>inversion-razor</code>, and the ConfigurationHelper and Pipeline models for service containers in Inversion.Extensibility.</p>
<p>In the case of the Razor extensions, I think I will need to rewrite this entirely in light of the AspNetCore support for Razor. I’d much rather be integrated with that than remain off-piste with Antaris RazorEngine.</p>
<p>Regarding ConfigurationHelper and Pipeline interfaces, the configuration options are so starkly different in .Net Standard, and so much easier to deal with, that there’s no point having a shim to pull configuration from a database etc when the new methods are so much more accessible.</p>
<h3 id="publishing-to-nuget">Publishing to NuGet</h3>
<p>I’m not sure which sociopath designed the web UI for NuGet, but they need their head examining; they’re clearly insane. It’s not quite as bad as the Visual Studio interface though, at least.</p>
<p>Having fought through the various stuff it puts in your way to prevent you getting a clean package uploaded, the various repos - inversion-dev, inversion-data, inversion-data and inversion-messaging - have been made available. The assemblies that make up those repos are available separately in order to keep project tech hierarchies segregated, e.g. Inversion.Data base library is separate to Inversion.Data.AmazonSQS which is separate to Inversion.Data.Redis etc. Don’t cross the streams unless you have to.</p>
<p>The versions are in the 1.0.x range and the references of child packages to their parents are currently set to 1.0.* as I was pretty shocked how stupid the management of minor versions was in the NuGet CLI. Anyway, working now.</p>
<h3 id="github-organisation">GitHub organisation</h3>
<p>The main libraries are now under the newly formed <code>inversion-org</code> GitHub organisation - <a href="https://github.com/inversion-org">https://github.com/inversion-org</a></p>
<p>Guy, Rob and myself are owners.</p>
<p>Here you will find:</p>
<ul>
<li><a href="https://github.com/inversion-org/inversion-data">Inversion.Data</a></li>
<li><a href="https://github.com/inversion-org/inversion-extensibility">Inversion.Extensibity</a></li>
<li><a href="https://github.com/inversion-org/inversion-messaging">Inversion.Messaging</a></li>
</ul>
<p>The main Inversion library hasn’t moved yet but this will be its home in the future. You can still find it here:</p>
<ul>
<li><a href="https://github.com/guy-murphy/inversion-dev">Inversion</a></li>
</ul>
<p>Other libraries, such as <a href="https://github.com/torquemaya/inversion-ultrastructure">Inversion.Ultrastructure</a> will also be moved into the organisation shortly.</p>
<h3 id="adding-travis-automation">Adding Travis automation</h3>
<p>Some Travis automation has been added to build and publish the NuGet packages, which it does sort of blindly as it doesn’t check first if the version already exists and so seeks forgiveness rather than permission when it fails to upload. I suppose there might be ways to automate incrementing the patch number on the .csproj file, but at the moment it is manual.</p>
<p>Things to do next:</p>
<ul>
<li>add unit tests (with automation for commit status updates)</li>
<li>test the libraries work!</li>
<li>create basic application that uses the libraries via NuGet and can be deployed as a container</li>
</ul>
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-43297868849510570572018-03-04T20:04:00.002+00:002018-03-04T20:04:43.677+00:005 - 8 Rooms - Angina P<h3 id="idm-minimalist-dnb">IDM, minimalist d’n’b</h3>
<p><img src="https://i.scdn.co/image/031afec5723a0ed6831a5a86aa1c796f641fcbc6" alt="8 Rooms - Angina P"></p>
<ul>
<li><a href="http://www.anginap.com">http://www.anginap.com</a></li>
<li><a href="https://soundcloud.com/anginap">https://soundcloud.com/anginap</a></li>
</ul>
<p>I think I first came across Ulla’s work on MySpace, a very long time ago. It was during the period when musicians in the same network would find you and then in response you would place hastily photoshopped “thanks for the add!” messages on their boards, carrying band and EP names, and it was all one nice music party, plus a bunch of emos and the worst CSS in the world. I fucking miss MySpace.</p>
<p>Anyway, I found some of her tracks via MySpace and some via <a href="http://MP3.com">MP3.com</a> (yes, really) which carried “Tokyo 6pm” into my ears and I was lost instantly.</p>
<p>In truth, I’ve already reviewed this album, although I can’t remember exactly who it was for (maybe for her label, Notochord?) It was providing a soundbite synopsis that pretty much said that this is crystal-sharp IDM and complete headphone-fuel. I meant it back then and I still mean it today.</p>
<p>There’s space here - open and modern like stark and hard white-grey architecture - but it’s a space filled with details that are reminiscent of Japanese technical design and wonderful emotive sweeps. Framing this is a real perfectionist focus on complex layers of accurately scattered percussion. These combine to make this one of my favourite albums, let alone IDM, providing what I generally term “coding music”. This describes tracks that for years I hadn’t identified a genre for, but eventually worked out were something along the lines of tech-step jungle and complex, minimalist drum’n’bass.</p>
<p><em>“Destroy you, with my robots”</em></p>
<p>Three of the tracks are remixes and are extremely high quality, making wonderful companions of the originals and blending right on in with the whole album. Semiomime’s remix of “Known Issues” makes me think we might be related as they know exactly what sort of drums I like.</p>
<h3 id="highlights">Highlights</h3>
<ul>
<li>Glitter - this was my ringtone for ages until I hit the Mirror’s Edge soundtrack by Solar Fields</li>
<li>Known Issues (Semiomime remix)</li>
</ul>
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-1699372795224215512018-03-01T21:18:00.001+00:002018-03-27T22:55:06.299+01:00Porting Inversion<p><img src="http://www.fractos.com/images/circle-theory/Porting%20Inversion.jpg" alt="Aldgate somewhere"></p>
<p>Just recently I have been porting the Inversion libraries over to .Net Standard, such that they will be able to support the existing set of behavioural applications that make up the image delivery part of the Digital Library Cloud Service (see <a href="https://dlcs.info">https://dlcs.info</a>).</p>
<p>To do this I received some initial help from Robert Stiff (<a href="http://twitter.com/UatecUK">@UatecUK</a>) to prove that the basics would work in the new execution environment, since it is vaguely exotic, having been based around the Reactive Extensions for .Net. Once we had ported the OWIN-based hello-world website across, I was convinced that I would be able to port across almost the entire application.</p>
<p>Last weekend I set about taking the ported versions of the libraries in <a href="https://github.com/guy-murphy/inversion-dev">https://github.com/guy-murphy/inversion-dev</a>, giving them some basic metadata and publishing them to NuGet as .Net Standard 2.0 packages (and yes, I have Guy’s explicit permission to do this). This included the new web code that Robert and myself tested the proof of concept with which has been added as the Inversion.Web.AspNetCore package.</p>
<p>Next, I ported and published the libraries in Inversion.Data (<a href="https://github.com/fractos/inversion-data">https://github.com/fractos/inversion-data</a>) into separate NuGet packages. This means that the different storage dependencies will remain separate in a target application.</p>
<p>Inversion.Extensibility (<a href="https://github.com/fractos/inversion-extensibility">https://github.com/fractos/inversion-extensibility</a>) was ported and published next, with a separate package for <a href="http://Inversion.Extensibility.Web">Inversion.Extensibility.Web</a>.</p>
<p>Finally, I ported Inversion.Messaging (<a href="https://github.com/fractos/inversion-messaging">https://github.com/fractos/inversion-messaging</a>) and packaged it separately, just as I had done for Inversion.Data.</p>
<p>It should be noted that each of these repositories has a separate "dotnetcore" branch for this code.</p>
<p>Things to do next:</p>
<ul>
<li>add unit tests</li>
<li>add Travis CI scripting to build, pack and publish to NuGet</li>
<li>test the libraries work when imported!</li>
</ul>
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-88122839740170038382018-02-21T23:14:00.002+00:002018-02-21T23:14:38.302+00:004 - 3 - Peter Gabriel<h3 id="progressive...-stuff">progressive… stuff</h3>
<p><img src="https://upload.wikimedia.org/wikipedia/en/8/89/Peter_Gabriel_%28self-titled_album%2C_1980_-_cover_art%29.jpg" alt="3"></p>
<p>Look, I’ll be honest; there’s a lot of pretty weird stuff on this album, and I am aware that Peter Gabriel could be described as massively up themselves. However, when I first came across this album (and <a href="https://en.m.wikipedia.org/wiki/Peter_Gabriel_(1982_album)">Security</a>, I was an impressionable, slightly obsessive youth with a head full of magic and sci-fi. This was fuel. This fed that whirlwind. At the time, I didn’t know just how much heroin Gabriel was doing. Seems quite obvious now…</p>
<p>The backing tracks and effects on this album date from the dawn of synth technology, and also the beginning of some techniques like Phil Collins’ noise-gate clipped drum tracks. No cymbals were allowed during the recordimg, according to the <a href="https://en.m.wikipedia.org/wiki/Peter_Gabriel_(1980_album)">wiki</a>, on the basis that an “artist given complete freedom dies a horrible death,” so restrictions forming part of the creative process - a rebellion of sorts.</p>
<p>There’s a ridiculous roll-call of musicians on the album, including Tony Levin, Kate Bush, the aforementioned Phil Collins, Paul Weller and Robert Fripp, it’s no wonder that there are some good sounds going on. I’ve always loved Tony Levin’s bass techniques; they’re just so out there.</p>
<h3 id="highlights">Highlights</h3>
<ul>
<li>Intruder - for its gated drums and creepy atmos.</li>
<li>I Don’t Remember - I’m totally in love with the bass guitar sound on this.</li>
<li>Not One Of Us - The intro and break are just beautiful.</li>
</ul>
<h3 id="trivia">Trivia</h3>
<p>The version I first heard was my brother’s cassette copy of the vinyl record which had skips in places which made me think some friend of his, whacked into God-mode on acid, had put it together in a fit of genius, or it was an authentic home-run of a happy accident:</p>
<p><em>Peak-time viewing born in a flash</em><br>
<em>as I burn into your memory-</em> <strong>skip</strong><br>
<em>I burn into your memory-</em> <strong>skip</strong><br>
<em>I burn into your memory-</em> <strong>click</strong><br>
<em>-cells</em></p>
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-65508973580250784732018-02-18T00:04:00.001+00:002018-02-18T00:04:35.160+00:003 - 10,000 days - Tool<h3 id="progressive-psychedelic-dark-rock">progressive, psychedelic dark rock</h3>
<p><img src="https://upload.wikimedia.org/wikipedia/en/f/f8/10000Days.jpg" alt="10,000 days"></p>
<p><a href="https://www.toolband.com">Tool homepage</a></p>
<p>Quite late in the Tool canon, “10,000 days” sees them in full psychedelic rock flow, with quiet, noodly, introspective sections blossoming into complex, stoned soundscapes.</p>
<p>I’ve only had the stand-out track “Right In Two” on my phone for years, having given the full album a listen only a handful of times in the past. However, giving it another audition yields some different bits that I’m picking up on and I have restored the full track list to my working collection. I think maybe it was the rambling intro for “Lost Keys (blame Hoffman)” that originally did it, where a Doctor tries to get a patient to talk after presenting at an Emergency Room in a silent, troubled state. I guess it doesn’t feel quite so close to the bone now, so that’s progress, right?</p>
<p><em>Sunkist and Sudafed, gyroscopes and infrared</em></p>
<p>The sound they produce is immense; signature guitar setup pervades, the drums display various complex-timing trickery, and the chunky, intricate basslines, thick with harmonics, bounce around the low- and mid- range. Get a decent copy and bathe your ears in it.</p>
<h3 id="trivia">Trivia</h3>
<p>Two of the tracks - “Wings for Marie (Pt 1)” and “10,000 days (Wings Pt 2)” can be layered together to form a single song (see <a href="https://m.youtube.com/watch?v=EFjEp79zaNw">https://m.youtube.com/watch?v=EFjEp79zaNw</a>) which has an interesting call-response thing going on.</p>
<h3 id="highlights">Highlights</h3>
<ul>
<li>Right In Two</li>
<li>The Pot</li>
<li>Vicarious</li>
</ul>
<h3 id="massive-update">MASSIVE UPDATE</h3>
<p>It turns out that the packaging for the CD is EPIC - with a built-in stereogram viewer and about 15 illustrations.</p>
<p><img src="http://www.fractos.com/images/circle-theory/albums/ToolAlbumCover.jpg" alt="Showing off the cover"></p>
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-56355492810650863392018-01-17T18:31:00.002+00:002018-01-17T18:31:15.882+00:002 - 1/f - immune<h3 id="progressive-hard-rock">progressive hard rock?</h3>
<p><a href="https://immune.bandcamp.com/album/1-f">Bandcamp link</a></p>
<p><img src="https://f4.bcbits.com/img/a2688509579_10.jpg" alt="1/f"></p>
<p>Once upon a stumbling around Bandcamp, I found <code>immune</code> linked from somewhere, referenced as an interesting, Tool-esque, British band to explore. Was not disappointed!</p>
<p>I don’t really know that much about them aside from that they went through a name change at some point - they are now known as <a href="http://masterandthemule.bandcamp.com/">Master & The Mule</a> - but they have a talent for melodic, dark, progressive rock. It’s quite thoughtful, introspective, and only shouty when it really needs to be.</p>
<h2 id="highlights">Highlights</h2>
<ul>
<li>Monkey</li>
<li>Selling Screen</li>
<li>Consume</li>
</ul>
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-74960198382292492252018-01-17T10:27:00.002+00:002018-01-17T10:27:38.601+00:001 - Brave enough - Lindsey Stirling<h3 id="primarily-instrumental-violin">primarily instrumental violin</h3>
<p><a href="https://www.youtube.com/channel/UCyC_4jvPzLiSkJkLIkA7B8g">YouTube channel</a></p>
<p><img src="https://upload.wikimedia.org/wikipedia/en/4/44/Brave_Enough_-_Lindsey_Sterling.jpg" alt="Brave Enough"></p>
<p>Note: <em>This is first in my album list due to a naming quirk (the ID3 tags all have “Brave Enough” surrounded by quote marks).</em></p>
<p>I’ve followed Lindsey Stirling’s career for a few albums now, and hearing that she had a new one out last year was an automatic decision to buy and add to the collection. There are a few more vocal collaborations this time, but still the same rich, unpretentious melody lines and beats. Her music, like her, is perky, fun and can catch you emotionally with a set of heartfelt and brilliant tunes. And when she’s got an idea in her head she really lands it simply because she’s insanely good at expression.</p>
<p>The album is a tribute and celebration of a friend of hers who, tragically, died young, and the songs are weaved around memories of good and sad times, aspirations and regrets. The emotional tug of the songs is strong and conveys a great deal, though I think I preferred the big, instrumental tunes on her previous album <a href="https://en.m.wikipedia.org/wiki/Shatter_Me_(album)">“Shatter Me”</a>.</p>
<p>Overall it’s bittersweet and that’s a wonderful thing. If you don’t know her work then please check it out. She’s a delight and a wonderful talent who may surprise you. Further, the relationship she has with her fans at live shows and via social media is infectious and commendable.</p>
<h2 id="highlights">Highlights</h2>
<ul>
<li>Brave Enough ft. Christina Perri</li>
<li>Something Wild ft. Andrew McMahon</li>
<li>Gavi’s Song</li>
</ul>
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-67799576306809151052018-01-16T23:28:00.002+00:002018-01-16T23:28:44.685+00:000 - Intro<p>And then I decided that I wanted to review every album that I have on my phone, in alphabetical order, as they play. “Good excuse to write!” says the aspirational bit of my brain which really does need a good kicking every so often. God help us all; here we go.</p>
<p><img src="http://www.fractos.com/images/circle-theory/albums/album-list2.jpg" alt="Album list"></p>
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-89680110831771289972017-03-02T12:31:00.001+00:002017-03-02T12:31:42.603+00:00Regarding Dependency Injection<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
A colleague of mine recently posted on an internal Slack channel about this blog article: <a href="http://www.devtrends.co.uk/blog/how-not-to-do-dependency-injection-the-static-or-singleton-container" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer;">How not to do dependency injection - the static or singleton container</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Reading the article, I became aware of something that <a href="http://guy-murphy.github.io/2014/11/24/service-locator-vs-dependency-injection/" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer;">a friend of mine had written about</a>, namely the absolute vitriol that many coding blogs and generally good books seem to have against Service Location. For example, I’m reading <a href="https://www.amazon.co.uk/gp/product/0735683204/ref=as_li_qf_sp_asin_il_tl?ie=UTF8&camp=1634&creative=6738&creativeASIN=0735683204&linkCode=as2&tag=readingpile-21" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer;">Adaptive Code via C#</a> at the moment and in that Gary McLean Hall has a small <em style="box-sizing: border-box;">meltdown</em> about it, which is a shame because it’s a pretty good book for people to learn about refactoring and use of basic patterns. However it also commits the cardinal sin of saying that the ‘D’ in SOLID is for Dependency <em style="box-sizing: border-box;">Injection</em> and not Dependency <em style="box-sizing: border-box;">Inversion</em>, and that betrays the architectural bias of the author.</div>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
The upshot being:</div>
<ul style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; margin-bottom: 0.83999rem; padding-top: 0.16001rem;">
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;">Anyone who tells you service location is an anti-pattern isn’t fully aware of the problems that it is supposed to be an answer for.</li>
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;">Dependency Injection moves away from the original point - separating the configuration of services from their use.</li>
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;">Dependency Injection vastly increases the surface area of an object via abuse of the constructor which isn’t bound by interface contract, thereby avoiding abstracting dependencies fully and leading to constructors being the medium by which relationships between objects are communicated - (this is not a virtue)</li>
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;">By making assumptions about state, Dependency Injection turns architectural <em style="box-sizing: border-box;">uses-a</em> relationships into <em style="box-sizing: border-box;">has-a</em> - which blocks the use of singletons and a bunch of other architectural patterns where they would be appropriate.</li>
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;">Further, it means that although relationships between data entities are modelled, behavioural relationships between objects are not because they become one great big dependency ball.</li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
<a href="http://guy-murphy.github.io/2014/11/24/service-locator-vs-dependency-injection/" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer;">Guy Murphy - Service Locator vs Dependency Injection</a></div>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Another in-depth article Guy wrote about Inversion of Control can be found here:</div>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
<a href="http://guy-murphy.github.io/2014/11/27/I-come-not-to-bury-IoC-but-to-praise-it/" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer;">Guy Murphy - I come not to bury IoC, but to praise it</a></div>
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-86564271972051821342017-01-27T15:17:00.002+00:002017-01-27T15:17:57.661+00:00Microscaling follow-up<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Hello. Long time no follow-up blog about this.</div>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
My <a href="http://circle-theory.blogspot.com/2015/11/microscaling-with-load-balancing-in.html" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer;">original post</a> detailed the problem that I was seeing with using Amazon’s EC2 Container Service to manage a cluster of identical containers that required to have traffic routed to them individually from a load balancer. At the time, this couldn’t be performed without extra tooling around the system, and I hinted that I had come up with a solution that would allow the process of <a href="http://microscaling.org/" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer;"><em style="box-sizing: border-box;">microscaling</em></a> to be achieved on EC2.</div>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
I had a part two of that article sitting in draft for well over a year; various things happened - we bought our first house, my partner became pregnant and our daughter was born back in October, and part one even got cited by <a href="http://force12.io/" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer;">force12.io</a> on their <a href="http://microscaling.org/" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer;">microscaling.org</a> page in their collection of papers - so, you will perhaps forgive me for being a little distracted and not getting around to finishing that bit off.</div>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Meantime, things moved along in the world of ECS.</div>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
The Application Load Balancer feature arrived and after a couple of months of me studiously ignoring its existence, I can now say that it does exactly what I needed Warden to do in the first place, namely that an ECS Service is now not limited to static port routing and a new Service instance with a dynamically allocated external port on the container host can be routed to automatically by the ALB. (Previously, it had meant that adding a new Service instance would require a new container host to be added to a cluster as the “Classic” Elastic Load Balancer could not route automatically to the dynamic ports.)</div>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
However, I will document what I’d come up with that filled the gap.</div>
<h2 style="background-color: white; box-sizing: border-box; color: #373d49; font-family: "Source Sans Pro", "Helvetica Neue", Helvetica, Arial, sans-serif; font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; font-size: 1.95313rem; line-height: 3rem; margin-bottom: 0.183584rem; margin-top: 0px; padding-top: 0.816416rem;">
<a href="https://www.blogger.com/null" id="Warden_12" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer; text-decoration: underline;"></a>Warden</h2>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Warden (<a href="https://github.com/fractos/warden" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer;">https://github.com/fractos/warden</a>) was my hack for this, which enabled me to run multiple identical containers on an ECS cluster and have them load-balanced. It is the Go version of a set of Perl scripts that I had created which managed the ECS cluster of image servers.</div>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
It was impossible to manage the level (number of instances) of an ECS Service if its definition required a static container port and you wanted to re-use the same host for as many container instances as possible. So, instead of defining an ECS Service, Warden’s Manager would manage the number of Tasks that were programatically being run across the cluster, increasing or decreasing them according to a metric (e.g. number of Elastic Beanstalk instances in an upstream system that calls the image servers I’m working on). ECS would then schedule the new Tasks across the cluster and the Registrar process would detect a change in the running container instances on a host, updating the host’s local Nginx routing setup to match and enrolling or removing the host from an associated Elastic Load Balancer if required.<br style="box-sizing: border-box;" />Each container host in the cluster would have two extra containers running:</div>
<ul style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; margin-bottom: 0.83999rem; padding-top: 0.16001rem;">
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;">Redx (<a href="https://github.com/CBarraford/docker-redx" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer;">https://github.com/CBarraford/docker-redx</a>)<br style="box-sizing: border-box;" />Which is a modified version of Nginx that takes its routing configuration from a live Redis database.</li>
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;">Redis<br style="box-sizing: border-box;" />To serve as Redx’s configuration database. Lua code in the Nginx config allows the routing configuration to be read dynamically from Redis, so front-ends and back-ends can be added or removed without having to restart the Nginx process.</li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
The cluster would need a central Redis instance that would serve as the service database and competition platform for Warden instances.</div>
<ol style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; margin-bottom: 0.83999rem; padding-top: 0.16001rem;">
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;">Synchronised the list of currently active containers on a host with a local nginx configuration held in Redis - “Registrar”</li>
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;">Managed the number of Tasks running across an ECS cluster for a particular Task Definition according to a connected metric - “Manager”</li>
</ol>
<h3 style="background-color: white; box-sizing: border-box; color: #373d49; font-family: "Source Sans Pro", "Helvetica Neue", Helvetica, Arial, sans-serif; font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; font-size: 1.64571rem; line-height: 3rem; margin-bottom: 0.07599rem; margin-top: 0px; padding-top: 0.92401rem;">
<a href="https://www.blogger.com/null" id="The_Registrar_27" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer; text-decoration: underline;"></a>The Registrar</h3>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
The Registrar periodically examines the list of Docker containers that are running on the host and, for those it recognises, it inspects the container, pulling out the local IP address that Docker has assigned. The IP addresses and the exposed port numbers are used to update the Nginx configuration held in a local Redis. Each Service has an Nginx front-end which is on a well-known port, unique for that Service. The separate container instances are added as Nginx back-ends associated with the Service front-end. Arriving traffic is therefore load-balanced internally across all the matching containers on the container host. If a Service has any container instances on the host then Warden ensures the EC2 instance is enrolled in the assigned Elastic Load Balancer, and removes it from the ELB if there are zero container instances.</div>
<h3 style="background-color: white; box-sizing: border-box; color: #373d49; font-family: "Source Sans Pro", "Helvetica Neue", Helvetica, Arial, sans-serif; font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; font-size: 1.64571rem; line-height: 3rem; margin-bottom: 0.07599rem; margin-top: 0px; padding-top: 0.92401rem;">
<a href="https://www.blogger.com/null" id="The_Manager_30" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer; text-decoration: underline;"></a>The Manager</h3>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
This process would also run on each container host, but the instances would hold a centralised competition for a leader. Each instance detects if the currently presiding leader’s Availability Zone is listed as currently active and checks for a recent heartbeat message from the leader. If there is a problem, then a competition is held and instances roll random numbers as their entry. The winner is picked from a Redis sorted set and they become the leader. A kill message is lodged for the previous leader to pick up if they return from whatever disaster befell their AZ.</div>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Meanwhile, the new leader emits heartbeats and measures how many Tasks for a Service need to be running by using a specified metric. I’d hard-coded this in the Perl version to look at the number of Elastic Beanstalk instances were running for a particular application and then using a set of files in S3 to map between the metric and the number of Tasks that should run, like a very simple static database. Manager uses the AWS SDK to increase or decrease the running number of Tasks for a particular Task Definition across the ECS cluster before waiting and looping its lifecycle.</div>
<h3 style="background-color: white; box-sizing: border-box; color: #373d49; font-family: "Source Sans Pro", "Helvetica Neue", Helvetica, Arial, sans-serif; font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; font-size: 1.64571rem; line-height: 3rem; margin-bottom: 0.07599rem; margin-top: 0px; padding-top: 0.92401rem;">
<a href="https://www.blogger.com/null" id="Afterwards_35" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer; text-decoration: underline;"></a>Afterwards</h3>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
I never got around to writing a clean way to define a Service in such a way that meant Warden could pick up its configuration dynamically. Also, how to abstract away the metrics that Manager would use to decide on Task numbers. I’d like to look into how to produce a plug-in architecture for Go that is friendly to being completely agnostic for both of these factors.</div>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Further, the network topology that the project moved towards made it unnecessary to run Warden’s Manager. Originally there was one ECS cluster that spread across the three Availability Zones in the eu-west-1 Region and there was no notion of an ECS Service to keep a desired number of container instances running across the cluster, so Manager filled that gap. Then there were changes that were made in response to realising that Elastic File System, once it eventually emerged from Preview, was more expensive to run than separate NFS volumes and servers per AZ. This meant splitting up the system into verticals - one per AZ - so that each layer of the system would only deal with one NFS volume and one ECS cluster per AZ. Warden’s Registrar still ran on each container host to synchronise the load balancing across the image server containers, but the number of containers running was managed by an ECS Service.</div>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
It was a good exploration of what was possible with a bit of scripting and glue. Experience of producing tooling in Go that was reactive to system conditions has also been invaluable.</div>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
For comparison, this is the original version of Warden which was written in Perl and effectively is just glue between the Redis, Docker and AWS CLIs: <a href="https://github.com/fractos/warden-perl" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer;">https://github.com/fractos/warden-perl</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Go was a natural choice for making a better, more solid version of Warden, mainly because of its Redis and AWS SDK access via libraries.</div>
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-72004851506343932582016-12-07T17:39:00.001+00:002017-01-21T09:58:09.346+00:00Configuring a containerised system<div class="MsoNormal" style="background: #FDFDFD; line-height: normal; margin-bottom: 11.25pt;">
<span style="color: #111111; font-family: "helvetica" , sans-serif; font-size: 12.0pt;">(from <a href="http://fractos.github.io/containers/2016/12/07/configuring-a-containerised-system.html">http://fractos.github.io/containers/2016/12/07/configuring-a-containerised-system.html</a>)</span><br />
<span style="color: #111111; font-family: "helvetica" , sans-serif; font-size: 12.0pt;"><br /></span>
<div style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 16px; margin-bottom: 15px; padding: 0px;">
These are some basic methods of passing configuration into a container.</div>
<h2 id="building-in-configuration-files" style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 28px; font-weight: 300; margin: 0px 0px 15px; padding: 0px;">
1. Building in configuration files</h2>
<div style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 16px; margin-bottom: 15px; padding: 0px;">
At build time, copy configuration files into the container image.</div>
<ul style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 16px; margin: 0px 0px 15px 30px; padding: 0px;">
<li>inflexible</li>
<li>insecure</li>
</ul>
<h2 id="configuration-fetched-and-applied-during-transient-build-step" style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 28px; font-weight: 300; margin: 0px 0px 15px; padding: 0px;">
2. Configuration fetched and applied during transient build step</h2>
<div style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 16px; margin-bottom: 15px; padding: 0px;">
Essentially this means pulling configuration data from somewhere and removing it before the end of the build step.</div>
<ul style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 16px; margin: 0px 0px 15px 30px; padding: 0px;">
<li>fragile</li>
<li>values sit in build cache</li>
</ul>
<h2 id="environment-variables" style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 28px; font-weight: 300; margin: 0px 0px 15px; padding: 0px;">
3. Environment variables</h2>
<div style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 16px; margin-bottom: 15px; padding: 0px;">
Discrete environment variable values injected at container instantiation time.</div>
<ul style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 16px; margin: 0px 0px 15px 30px; padding: 0px;">
<li>flexible - can re-configure per instance</li>
<li>good integration with Docker, ECS, shell</li>
</ul>
<h2 id="holding-configuration-in-s3" style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 28px; font-weight: 300; margin: 0px 0px 15px; padding: 0px;">
4. Holding configuration in S3</h2>
<div style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 16px; margin-bottom: 15px; padding: 0px;">
Pass S3 address to container by either an environment variable or a command parameter. Scripts or apps then retrieve dynamic configuration from the S3 bucket. Can be private, with ambient credentials assumed from an associated EC2 role on the container host, or by AWS credentials that are passed by environment variables.</div>
<ul style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 16px; margin: 0px 0px 15px 30px; padding: 0px;">
<li>allows dynamic configuration during container lifetime</li>
</ul>
<h2 id="attach-a-configuration-volume" style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 28px; font-weight: 300; margin: 0px 0px 15px; padding: 0px;">
5. Attach a configuration volume</h2>
<div style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 16px; margin-bottom: 15px; padding: 0px;">
A specific volume that holds configuration files can be attached to the container during instantiation.</div>
<ul style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 16px; margin: 0px 0px 15px 30px; padding: 0px;">
<li>allows dynamic configuration during container lifetime</li>
<li>have to start managing volumes across container hosts</li>
<li>Kubernetes uses this type of solution</li>
</ul>
<h2 id="use-a-configuration-serverservice" style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 28px; font-weight: 300; margin: 0px 0px 15px; padding: 0px;">
6. Use a configuration server/service</h2>
<div style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 16px; margin-bottom: 15px; padding: 0px;">
Pass server details into the container at instantiation time. Scripts or apps will retrieve configuration from a service such as Vault, etcd, Redis, a database etc.</div>
<span style="color: #111111; font-family: "helvetica" , sans-serif; font-size: 12.0pt;"></span><br />
<ul style="color: #111111; font-family: Helvetica, Arial, sans-serif; font-size: 16px; margin: 0px 0px 15px 30px; padding: 0px;">
<li>client dependencies</li>
<li>container contents may have to be adapted to integrate with a configuration server</li>
<li>allows dynamic configuration during container lifetime</li>
</ul>
</div>
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-14122937207450414832016-09-09T11:04:00.000+01:002016-09-09T11:04:10.644+01:00I can see the sea :)I can see the sea from this train :)<br />
<br />
It's quite far out, with bright orange beached buoys in the middle distance before a platoon of wind power generators that are standing in the eventual water toward the horizon.<br />
<br />
Curlews are striding about, or maybe redshanks still in their summer plumage.<br />
<br />
Always a thrill to catch sight of the shore and the different kind of life that teems there, all under a grey-blue coastal sky.Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0Rhyl, UK53.319140999999988 -3.491633999999976453.281198499999988 -3.5723149999999766 53.357083499999987 -3.4109529999999761tag:blogger.com,1999:blog-319207569350348666.post-44985908698530646982016-08-23T10:44:00.000+01:002016-08-23T10:44:36.845+01:00Canary farm<h2>
<br /></h2>
<h2>
"Canary farm"</h2>
<div>
An IT system connected to various other brittle systems that invariably picks up the blame for them being unavailable.</div>
<div>
<br /></div>
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-61325110998667929452016-08-08T14:56:00.003+01:002016-08-08T15:05:18.343+01:00Fractos-cumuli tipping point<br /><br /><br />Fractos-cumuli tipping point:<br /><br />When the monthly cost of the Cloud project you are working on exceeds your monthly salary.<br /><br /><br />
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-86287042069969977842016-07-03T00:28:00.000+01:002016-07-03T00:39:35.882+01:00Deploying a .Net website to Amazon Elastic Beanstalk with TeamCityNote for posterity.<br />
<br />
Steps I had to take to get a .Net website deployed to Elastic Beanstalk with TeamCity.<br />
<br />
I was originally working from <a href="http://www.tonytruong.net/deploying-a-net-web-application-to-aws-elastic-beanstalk-through-teamcity/" target="_blank">this tutorial</a>, but soon realised that things weren't going to plan at all. This was a pain in the arse.<br />
<br />
Pre-requisites:<br />
<br />
Install <a href="https://www.visualstudio.com/en-us/downloads/download-visual-studio-vs.aspx" target="_blank">Visual Studio Community edition</a> on the TeamCity server.<br />
Install <a href="http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/deploy_NET_standalone_tool.html" target="_blank">AWS Deployment Tool</a> awsdeploy according to the instructions in <a href="http://www.tonytruong.net/deploying-a-net-web-application-to-aws-elastic-beanstalk-through-teamcity/" target="_blank">the tutorial</a>.<br />
<br />
1. Web.config transform<br />
<br />
The version of the website I wanted to deploy needed the transformation for Release build to be performed before it was packaged up.<br />
<br />
I edited the .csproj of the website to include BeforeBuild and AfterBuild steps that would transform the Web.config file to its Release form:<br />
<target condition=" '$(Configuration)' == 'Release' " name="BeforeBuild">
<copy destinationfiles="Web.temp.config" overwritereadonlyfiles="True" sourcefiles="Web.config">
<transformxml destination="Web.config" source="Web.temp.config" transform="Web.$(Configuration).config">
</transformxml></copy></target>
<target name="AfterBuild">
<copy destinationfiles="Web.config" overwritereadonlyfiles="True" sourcefiles="Web.temp.config">
<delete files="Web.temp.config">
</delete></copy></target><br />
<br />
2. Working around MSBuild being retarded about indirect references<br />
<br />
MSBuild tries to be smart about including references in a deployment package. So smart that it doesn't actually include the dependencies of references that it includes - so you will probably end up with a website that doesn't work.<br />
<br />
The indirect dependencies are included as references in the website project but MSBuild still filters them out.<br />
<br />
I found some advice somewhere saying you should set the Copy Local property on the references to True. I dutifully did this, but it turns out that Visual Studio (including 2015 Update 2) has a huge bug in this area as it will not update the .csproj file if you set Copy Local to true. However, if you set it to False then click Save All, it will create the XML element in the .csproj file. Once you've set it to False and saved, ONLY THEN can you set it to True (and hit Save All again). You can select all the references at once to perform this action in Visual Studio.<br />
<br />
3. Release build project in TeamCity must export everything as an artefact<br />
<br />
I was finding it impossible to run MSBuild in the deployment chain without having access to the compiled code. So, my Release build project in TeamCity has all the project and library folders marked as being artefacts, including an umbrella folder for any git submodules that are included.<br />
<br />
Then, in the deployment build chain, I have both a snapshot dependency on the Release build chain (which means that it will only use successful build artefacts) and an artefact dependency which includes all the project and library folders that were exported as artefacts from the Release build chain above.<br />
<br />
4. Deployment build chain copies transformed web.config back into website<br />
<br />
The transformed version of the web.config ends up squirrelled away under the website's obj folder. I had to include an initial build step in the deployment chain to copy the transformed file back into the right folder so the packaging step can find it.<br />
<br />
This looks something like this (obviously replace <website> with the name of the website's folder):<br />
<br />
copy /Y <website>\obj\Release\TransformWebConfig\transformed\Web.config <website>\Web.config<br />
<br />
5. Getting the command line parameters right for the packaging build step<br />
<br />
This particular website is based at the root of the IIS folder on the remote server (i.e. c:\inetpub\wwwroot), so I had to include an instruction to deploy to the "Default Web Site" bare IIS path in the command line parameters.<br />
<br />
Also this includes the Package instruction, the Configuration type, the SolutionDir definition (which is the TeamCity checkout directory - this is set as I use a particular scheme for cutting down on duplication of nested git submodules that involves telling included projects to use $(SolutionDir) as the base for their references), and the PackageTempRootDir property - an empty property which tells the packager not to make a deep hierarchy within the zipped output file.<br />
<br />
/T:Package /P:Configuration=Release /property:SolutionDir="%system.teamcity.build.checkoutDir%"\ /P:PackageTempRootDir= /P:DeployIISAppPath="Default Web Site"<br />
<br />
6. Create a configuration file for awsdeploy for the specific Elastic Beanstalk environment<br />
<br />
For reasons that escaped me, I found it impossible to include certain parameters on the command line for the awsdeploy command. I had to work around this by creating a static configuration file for a particular Elastic Beanstalk environment. Quite bare-bones data:<br />
<br />
AWSProfileName = default<br />
Region = <your aws region><br />
Template = ElasticBeanstalk<br />
UploadBucket = <the s3 bucket that elastic beanstalk uses e.g. elasticbeanstalk-eu-west-1-accountnumber><br />
Application.Name = <elastic beanstalk application name><br />
Environment.Name = <elastic beanstalk environment name><br />
<div>
<br /></div>
I put this file in a well known place on the build server that I knew TeamCity could get at.<br />
<br />
7. Getting the command line parameters right for the awsdeploy build step<br />
<br />
awsdeploy needs a few command line parameters and it was a bit hit and miss to sort it out, but here's what I ended up with:<br />
<br />
/DAWSAccessKey=<api access key> /DAWSSecretKey=<api secret key> /DDeploymentPackage=%teamcity.build.checkoutDir%\<website>\obj\Release\Package\<name of project website>.csproj.zip /v /w /r <configuration file from step 6><br />
<br />
That should do the actual deployment to the Elastic Beanstalk environment. The deployment package is the pathname of the zip that the previous step creates which should be under the obj\Release\Package folder and have the same name as the .csproj that MSBuild was executed against.Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-67983917528282835102016-03-01T23:58:00.003+00:002016-03-01T23:58:58.253+00:00Showing a stream in Immediate Window (for posterity)Recording how to show a stream's contents in Immediate Window for posterity. In this example it's the response stream when a WebClient raises an exception:<br />
<br />
System.Text.Encoding.UTF8.GetString((byte[])$exception.Response.GetResponseStream().GetType().GetMethod("InternalGetBuffer", System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic).Invoke($exception.Response.GetResponseStream(), null))<br />
<br />
(from: <a href="http://stackoverflow.com/questions/7755312/view-content-of-a-stream-in-visual-studios-quickwatch-window">http://stackoverflow.com/questions/7755312/view-content-of-a-stream-in-visual-studios-quickwatch-window</a>)<br />
<br />Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-27593384946501887402016-02-16T17:10:00.001+00:002016-03-07T09:38:43.329+00:00Bookmarks 20160216Various public datasets<br />
<br />
<a href="https://github.com/caesar0301/awesome-public-datasets" target="_blank">https://github.com/caesar0301/awesome-public-datasets</a><br />
<br />
<br />
I <3 postgresql<br />
<a href="http://www.postgresonline.com/journal/archives/228-PostgreSQL-Array-The-ANY-and-Contains-trick.html" target="_blank">http://www.postgresonline.com/journal/archives/228-PostgreSQL-Array-The-ANY-and-Contains-trick.html</a><br />
<br />
<br />
Shocked I missed this - arbitrary linkage to IFTTT with the Maker Channel:<br />
<br />
<a href="http://blog.ifttt.com/post/121786069098/introducing-the-maker-channel" target="_blank">http://blog.ifttt.com/post/121786069098/introducing-the-maker-channel</a><br />
<br /><!--3-->Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-2729482292519062632015-11-07T14:29:00.003+00:002017-01-17T17:09:07.002+00:00Microscaling with Load-balancing in Amazon ECS (part 1)<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; font-family: Georgia, Cambria, serif; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Currently I am involved in the development of a digital imaging platform for libraries and other institutions (the Digital Library Cloud Service, see <a href="http://dlcs.info/" style="background: 0px 0px; box-sizing: border-box; cursor: pointer;">http://dlcs.info</a>) This platform uses a wide palette of Amazon Web Services for the orchestration of moving image assets between storage systems, retrieval of metadata and the presentation of deep-zoom image tiles. It’s sorta-kinda an “elastic image server” which supports the IIIF (<a href="http://iiif.io/" style="background: 0px 0px; box-sizing: border-box; cursor: pointer;">http://iiif.io</a>) standard.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; font-family: Georgia, Cambria, serif; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
The system is split into two layers - Orchestration and Image-Serving. Both of these are intended to auto-scale independently, and this has been achieved with the Orchestration layer by deploying it as an Elastic Beanstalk application. Since this application is (currently) a .Net application that lives in IIS, then this environment suits it well.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; font-family: Georgia, Cambria, serif; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
The Image-Serving layer is Linux-based, comprising of a lightweight HTTP daemon (lighttpd) and Ruven Patel’s IIPSrv (<a href="http://iipimage.sourceforge.net/" style="background: 0px 0px; box-sizing: border-box; cursor: pointer;">http://iipimage.sourceforge.net/</a>) with optimised Jpeg-2000 support. This is quite easily deployed as a Docker container and that is what our current prototype uses.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; font-family: Georgia, Cambria, serif; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Since we are using Docker anyway, I looked into how Amazon’s Elastic Container Service could be used to auto-scale the Image-Serving layer on demand.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; font-family: Georgia, Cambria, serif; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
What I found was that the concept of a Service in ECS did not cover how I thought it could be used.</div>
<h3 style="-webkit-font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; box-sizing: border-box; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.64571rem; line-height: 3rem; margin-bottom: 0.07599rem; margin-top: 0px; padding-top: 0.92401rem;">
<a href="https://www.blogger.com/null" id="Services_under_ECS_10" style="background: 0px 0px; box-sizing: border-box; cursor: pointer; text-decoration: underline;"></a>Services under ECS</h3>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; font-family: Georgia, Cambria, serif; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
(<a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-load-balancing.html" style="background: 0px 0px; box-sizing: border-box; cursor: pointer;">http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-load-balancing.html</a>)</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; font-family: Georgia, Cambria, serif; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
A Service in ECS represents parts of an application that are distributed across various Docker containers. A cluster of EC2 instances form a group of hosts that a Service may be deployed to. Each host may contain a maximum of one instance of a Service. When scaling occurs, the EC2 cluster can expand or contract with Auto-Scaling Group rules and the ECS Scheduler decides where new instances are placed.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; font-family: Georgia, Cambria, serif; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
The limitation of a maximum of a single Service instance per host is one that I was not expecting, but can understand given the way Elastic Load Balancers are configured. In ECS, the port numbers of containers in the Service are static because you’ll only get one instance of the Service per host, and therefore load balancing can occur across the cluster in a static way (from the Elastic Load Balancer’s point of view).</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; font-family: Georgia, Cambria, serif; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
When I saw the setup dialog for a Task in ECS mention the internal and external ports for a Docker container, I had hoped that this implied that the external port on the container could be balanced across if it was left blank, but that turned out not to be the case.</div>
<h3 style="-webkit-font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; box-sizing: border-box; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.64571rem; line-height: 3rem; margin-bottom: 0.07599rem; margin-top: 0px; padding-top: 0.92401rem;">
<a href="https://www.blogger.com/null" id="Microscaling_20" style="background: 0px 0px; box-sizing: border-box; cursor: pointer; text-decoration: underline;"></a>Microscaling</h3>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; font-family: Georgia, Cambria, serif; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
What I wanted was the ability to work with multiple similar Docker containers populating EC2 cluster members dynamically in response to some metric - e.g. a CloudWatch alarm from our orchestration layer’s load balancer. As that Elastic Beanstalk application scales in response to demand, so would the number of deployed containers up to the available space of the hosts in the cluster as defined in ECS.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; font-family: Georgia, Cambria, serif; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
It turns out that this technique has a term - microscaling - and I’ve been researching it recently, including the offering from <a href="http://force12.io/" style="background: 0px 0px; box-sizing: border-box; cursor: pointer;">Force12.io</a> - (<a href="http://force12.io/" style="background: 0px 0px; box-sizing: border-box; cursor: pointer;">http://force12.io/</a>) - see <a href="http://microscaling.org/">http://microscaling.org</a> for more details.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; font-family: Georgia, Cambria, serif; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Once the number of containers is under dynamic control, the cluster itself could then expand or contract by responding to a separate Auto-Scaling Group rule.</div>
<h3 style="-webkit-font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; box-sizing: border-box; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.64571rem; line-height: 3rem; margin-bottom: 0.07599rem; margin-top: 0px; padding-top: 0.92401rem;">
<a href="https://www.blogger.com/null" id="Loadbalancing_28" style="background: 0px 0px; box-sizing: border-box; cursor: pointer; text-decoration: underline;"></a>Load-balancing</h3>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; font-family: Georgia, Cambria, serif; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
The other issue is load balancing across the cluster. In the scenario I want, the ports on the deployed containers would need to be routed to directly by the load balancer, so that means multiple different ports per host. This is something which would bend the ELB architecture out of shape, I think.<br />
<b><br /></b>
<b>UPDATE:</b> Now that Amazon have released their Application Load Balancer feature, then the load balancing problem is more or less solved on AWS. I must have been one of many people who were after this kind of feature as a senior ECS developer talked to me about it on Twitter over a year ago.</div>
<h3 style="-webkit-font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; box-sizing: border-box; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.64571rem; line-height: 3rem; margin-bottom: 0.07599rem; margin-top: 0px; padding-top: 0.92401rem;">
<a href="https://www.blogger.com/null" id="Use_case_and_solution_33" style="background: 0px 0px; box-sizing: border-box; cursor: pointer; text-decoration: underline;"></a>Use case and solution</h3>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; font-family: Georgia, Cambria, serif; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Our use case is that I want to have an auto-scaling group of EC2 instances in an ECS cluster which host multiple similar containers that will listen on the same internal port, but are routed to via their external port that is assigned by Docker (e.g. while still appearing to be port 80 inside the container). They will be available to be load-balanced across all the containers that are running in the cluster. These large EC2 instances could host multiple different ‘services’ comprised of similar containers.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; font-family: Georgia, Cambria, serif; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Essentially, we’d like to have a number of large container hosts and fill them on-demand with web servers or other cookie-cutter services, with load balancing going direct to the containers, across a cluster, across AZs.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; font-family: Georgia, Cambria, serif; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
I have created a system for performing both microscaling and load-balancing within ECS, and I will show the solution in the next blog post.</div>
</div>
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-80626189342643189282015-11-04T22:22:00.001+00:002015-11-04T23:34:30.017+00:00Twitter - We should talk.Hi Twitter.<br />
<br />
Please, come in and sit down. I think it's time we had a little chat.<br />
<br />
I know things haven't been great lately, and I think you're coming undone at the seams a bit.<br />
<br />
I'd like to lay a few things out for discussion.<br />
<br />
<br />
<h3>
Your position.</h3>
<br />
You are in a unique position of being a vital, real-time connection between content producers and interested parties, who may also be content producers. You enjoy a position which is (blessedly) outside Facebook and is not a walled garden. You have made amazing technical advances to be able to support this amazing real-time platform. You are, in general, viewed as a force for good. You have friends.<br />
<br />
I have witnessed more news, events, conversations and opinions on Twitter than almost any other medium I've used in the past twenty years that I've been on the Internet. It's addictive to be part of this huge, global conversation and I love that.<br />
<br />
<h3>
Recent activity.</h3>
<div>
But you keep doing these <i>things</i>.</div>
<div>
<br /></div>
<h4>
Layoffs.</h4>
Maybe these were justified because you had over-hired, but it looked reactionary which, if I was one of your investors, would make me quite nervous and/or irritated. Stop jerking your knees. Be more solid.<br />
<br />
<h4>
Moments.</h4>
Experimental. I think it's good to separate curated items like this, which leaves timelines alone. More on that later. Definitely interesting as a thing for surrounding current affairs and getting people interested in them. The "You're caught up" is very Circa-like (RIP Circa). The Trending items have been kind of filling that purpose already, though, but the presentation in Moments is very clean, so I hope that this can build on that kind of attention into something cool. But I'm really not sure that this is the new user grabber that you think it is.<br />
<br />
Please don't retire trending or hashtags though. That would be uncool.<br />
<br />
<h4>
Polling.</h4>
The polling functionality is a really powerful thing. You can do a lot of good in this area. Keep up with the good work on this. Be a part of online democracy and live reaction. We like.<br />
<br />
<h4>
Periscope.</h4>
Still niche, but an excellent reporting tool. It would be nice to see better integration of that in things like the TweetDeck Chrome extension. Real problems here are bandwidth, video quality, people's data plans, vertical aspect video.<br />
<br />
<h4>
Vine.</h4>
I know Instagram do 15 seconds, but 6 seconds is still cool, right? The integration of this service makes sense because it lends itself to the concise nature of a Tweet. However, the only people I have really seen (in the UK) using Vine are comedians. Think about that for one of those 6 seconds.<br />
<br />
<h4>
Apparent lack of action regarding the tackling of online abuse.</h4>
<div>
You really, REALLY need to sort this shit out. Provide tools, streamline reporting processes, support victims, unleash rogue AI... Literally anything apart from sitting on your hands about bullying and threatening behaviour.</div>
<div>
<br /></div>
<h4>
Favourites vs likes.</h4>
Stars vs hearts. Connotations and implications. Organisation vs emotion. I could go on. In fact ...<br />
<br />
The whole point of Web 2.0 was the ability to customise your web experience. What happened? How can it not be clear that this change makes it means something totally different? It smacks of flag-waving. You must stop doing these kinds of changes. People (read: your user base) get very used to these semantics and symbols, and you'd be amazed what could drive a person away if they felt aggrieved enough about it. As it is, it's now fairly evident that the people who make these kinds of decisions don't appear to use Twitter or "get it" - the blog post from Akarshan Kumar made it clear that you are going for lowest common denominator, and that's extremely patronising.<br />
<br />
The best possible thing you can do here is to bring back the Star icon, and call the collection of things "Starred", because that is completely agnostic to emotional connections. Then this feature can go back to its meaning being implied from context (which includes timing) and also have zero connotations when used as an organisational method. Please do this because what you've done makes no sense to me and many others.<br />
<br />
<h4>
Noise.</h4>
Most of Twitter is noisy. If I step outside of the accounts I follow then I find myself in a horrific sea of chatter. I'm actually cool with that. It supports the notion of 'tuning-in' to certain voices; apart from the Etsy bots - I'm f-ing sick of those. You should be able to spot them and many other types of spammer by now - so why aren't they automatically banned?<br />
<br />
<h4>
Concentration on in-house clients.</h4>
At the cost of pissing off every other development team who ever tried to make a 3rd party add on or client for Twitter. It's like you really don't want an ecosystem at all. You are a PLATFORM, a SERVICE, let people MAKE THINGS THAT USE YOUR PLATFORM. If this is about the whole advertising thing - i.e. that you can't control placement of adverts in 3rd party clients, then you're doing things backwards. FIND ANOTHER MODEL (see below). It's not like you're showing adverts in TweetDeck's chrome extension yet anyway (and please don't start now - I'll cry).<br />
<br />
<h4>
Perverse notion of messing with the chronology of timelines.</h4>
Twitter's utility is that it shows a chronological view of content. I have absolutely no idea why you think that modifying this is in any way a good thing to do. Even the the out-of-order replies reaction (the "blue line" fiasco) was a UX-disaster with people (like me) complaining about the cognitive dissonance it provoked and being the final nail in the coffin for staying the hell away from the web client. The considered plans for "Tweets-you-may-have-missed" is just barmy. If there are things that you think I've missed - PUT THEM INTO A SEPARATE LIST. Seriously, if you cock up the chronology of Tweets in a timeline then you might as well be a randomly ordered collection of things people have said at some point in the past X hours like Facebook. Just stop that line of thinking.<br />
<br />
<h4>
Consideration of Tweets longer than 140 characters.</h4>
No, no, no, no, NO. Twitter is concise. Keep it that way. How am I supposed to catch up on 500+ Tweets that happened overnight if they are a bunch of essays? That's what I have RSS for! If someone wants to post more than 140 characters then they should host content elsewhere which has meta-data on it that allows the previews in clients - just like they do now. CREATE AN ECOSYSTEM, NOT A WALLED GARDEN.<br />
<br />
Also - please support RSS. RSS is your friend.<br />
<br />
<h4>
The web client.</h4>
I mean, seriously. I'm really glad that we got the Bootstrap project out of Twitter (I use it every day at work), but my God you guys really need to completely re-think your UX. Jesus F-ing Christ, what a mess. If I was approaching that project I would absolutely throw the entire template away and hope that no-one ever mentioned it at parties I went to.<br />
<br />
<h4>
Stock price.</h4>
By doubling down on bad ideas to please investors, all it goes to show is that you have collected the wrong investors, with the wrong promises whispered into the wrong ears.<br />
<br />
The knock-on effects of promising swathes of new users is removing focus from what makes your platform great.<br />
<br />
New users are only valuable if you can present advertising to them. And you've seen the current ad-block fuss.<br />
<br />
What you <i>need</i> are <i>customers</i>.<br />
<br />
<h3>
Recommended actions.</h3>
<br />
<h4>
Subscription model.</h4>
<br />
I would absolutely pay $5-10 dollars a month for a subscription to Twitter. That subscription would be ad-free, with a reasonable expectation of timely support for problems with my account, payment, online abuse, and the knowledge that my 3rd party Twitter client could be used without limitation.<br />
<br />
I can envisage you running a subscription programme in parallel with your current operations, the only difference being the ad-free nature of the channel, a better SLA for support and the unlocked API access.<br />
<br />
The more I think about it, (and I've been thinking about this for a few years), the more strongly I believe that this would be hugely successful for you.<br />
<br />
But how much would subscription actually cost you? What change would that have on revenue when a (hopefully significant) percentage of your users are throwing you money each month? What would be the effect upon new user take-up, if they joined knowing that they could upgrade their experience? What would be the effect on API use if it was suddenly opened up to paying users?<br />
<br />
I can't help but feel that these changes would be positive.<br />
<br />
<h4>
Content producer payment.</h4>
<div>
Given that many people join up to be able to follow people with something to say, celebrities, authors, musicians, news outlets, etc, then perhaps that should be rewarded somehow.</div>
<div>
<br /></div>
<div>
There are a few different services which allow users to make payment via Twitter. By taking that in-house this could be like your own version of Patreon, (minus the data leakage), offering content producers payback for being popular.</div>
<div>
<br />
Again, this is about creating an ecosystem. Reward people for taking part.</div>
<br />
<h3>
Signing off.</h3>
Twitter, I don't know if you'll read this. I hope you do and I hope that some of it gets through to you. You are a vital, current thing, and it is in everyone's interest (apart from Facebook's) that you stop these self-harming actions and become something better.<br />
<br />
Good day to you, whomever you are.<br />
<br />
@fractos.<br />
<br />Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-18164511433095112572015-09-21T12:21:00.001+01:002015-09-21T12:21:01.518+01:00Fractos' axiomFractos' axiom:<br />
<br />
There are only three types of software bug:<br />
<br />
<br />
<ol>
<li>Namespace issues</li>
<li>Sequence mistakes</li>
<li>Typos</li>
<li>Out by one errors</li>
</ol>
<div>
<br /></div>
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-21883629760370618142015-07-30T17:06:00.000+01:002015-07-30T17:06:22.996+01:00Books, mostlyHola.<br />
<br />
My last set of posts about <a href="http://circle-theory.blogspot.co.uk/search?q=model">Using the Model-Store pattern in Inversion</a> took a large heap of my blogging-concentration, but as they've been going since March it's good that they are finally out of the way.<br />
<br />
Work's going well and I'm getting my teeth stuck into producing a <a href="http://dlcs.info/">cloud-based platform for digital library services</a> for digitised collections, services like deep-zoom image serving, discovery, managing metadata and work with the emerging <a href="http://iiif.io/">IIIF </a>standard for viewers. Work days, I'm at Wellcome, which is a bit like being on a spaceship, although the food is probably better than NASA serve.<br />
<br />
I'm cycling to work which is moderately terrifying. Drivers are terrifying. Cyclists are terrifying. Especially cyclists, frankly, considering some of the whacked-out shit I've seen them do in just a few weeks of riding.<br />
<br />
At the moment I'm studying for the Microsoft Programming in C# Exam (70-483) which is on Monday afternoon. It helps out <a href="http://www.digirati.com/">Digirati </a>and definitely doesn't hurt me to go for the certification. However, the study has kinda precluded me making much progress with the stuff I'm reading.<br />
<br />
Oh, I bought a <a href="http://www.amazon.co.uk/gp/product/B00UOUFMP4/ref=as_li_qf_sp_asin_il_tl?ie=UTF8&camp=1634&creative=6738&creativeASIN=B00UOUFMP4&linkCode=as2&tag=readingpile-21">Microsoft Band</a> the other day, partly because geek and partly to mitigate an RSI I'm getting in my right index finger. Reaching for my phone in jeans pockets every 30 seconds is really causing a problem, it seems, so the Band picks up the notifications for me. The cycle / run tracking is really good too, although I was tipped over into buying one when I learned that they'd opened up the <a href="http://developer.microsoftband.com/bandSDK">SDK </a>to allow arbitrary RSS feeds to be added to the tiles.<br />
<br />
<a href="http://www.amazon.co.uk/gp/product/B00UOUFMP4/ref=as_li_tl?ie=UTF8&camp=1634&creative=6738&creativeASIN=B00UOUFMP4&linkCode=as2&tag=readingpile-21&linkId=FO2F6B2ZP3X6GFQV"><img border="0" src="http://ws-eu.amazon-adsystem.com/widgets/q?_encoding=UTF8&ASIN=B00UOUFMP4&Format=_SL160_&ID=AsinImage&MarketPlace=GB&ServiceVersion=20070822&WS=1&tag=readingpile-21" /></a><img alt="" border="0" height="1" src="http://ir-uk.amazon-adsystem.com/e/ir?t=readingpile-21&l=as2&o=2&a=B00UOUFMP4" style="border: none !important; margin: 0px !important;" width="1" /><br />
<div>
<br /></div>
<div>
Go get one! They're really cool.</div>
<br />
Currently reading:<br />
<br />
Assassin's Apprentice (Farseer Trilogy part 1)<br />
Robin Hobb<br />
<br />
<a href="http://www.amazon.co.uk/gp/product/000756225X/ref=as_li_tl?ie=UTF8&camp=1634&creative=6738&creativeASIN=000756225X&linkCode=as2&tag=readingpile-21&linkId=OW3XVISUGW43QBFL"><img border="0" src="http://ws-eu.amazon-adsystem.com/widgets/q?_encoding=UTF8&ASIN=000756225X&Format=_SL160_&ID=AsinImage&MarketPlace=GB&ServiceVersion=20070822&WS=1&tag=readingpile-21" /></a><img alt="" border="0" height="1" src="http://ir-uk.amazon-adsystem.com/e/ir?t=readingpile-21&l=as2&o=2&a=000756225X" style="border: none !important; margin: 0px !important;" width="1" /><br />
<div>
<br /></div>
<div>
<br /></div>
<div>
Building Microservices</div>
<div>
Sam Newmann</div>
<div>
<br /></div>
<div>
<div>
<a href="http://www.amazon.co.uk/gp/product/1491950358/ref=as_li_tl?ie=UTF8&camp=1634&creative=6738&creativeASIN=1491950358&linkCode=as2&tag=readingpile-21&linkId=RXXSNQOYGXY2ERRF"><img border="0" src="http://ws-eu.amazon-adsystem.com/widgets/q?_encoding=UTF8&ASIN=1491950358&Format=_SL160_&ID=AsinImage&MarketPlace=GB&ServiceVersion=20070822&WS=1&tag=readingpile-21" /></a><img alt="" border="0" height="1" src="http://ir-uk.amazon-adsystem.com/e/ir?t=readingpile-21&l=as2&o=2&a=1491950358" style="border: none !important; margin: 0px !important;" width="1" /></div>
</div>
<div>
<br /></div>
<div>
(Well, I lie - I just finished it the other day - really bloody good, though.)</div>
<div>
<br /></div>
<div>
And my exam book:</div>
<div>
Programming in C# Exam Ref 70-483</div>
<div>
Wouter de Kort</div>
<div>
<br /></div>
<div>
<div>
<a href="http://www.amazon.co.uk/gp/product/0735676828/ref=as_li_tl?ie=UTF8&camp=1634&creative=6738&creativeASIN=0735676828&linkCode=as2&tag=readingpile-21&linkId=RX6VNSOBUYJGMB4N"><img border="0" src="http://ws-eu.amazon-adsystem.com/widgets/q?_encoding=UTF8&ASIN=0735676828&Format=_SL160_&ID=AsinImage&MarketPlace=GB&ServiceVersion=20070822&WS=1&tag=readingpile-21" /></a><img alt="" border="0" height="1" src="http://ir-uk.amazon-adsystem.com/e/ir?t=readingpile-21&l=as2&o=2&a=0735676828" style="border: none !important; margin: 0px !important;" width="1" /></div>
</div>
<div>
<br /></div>
<div>
Which has got a whole bunch of text inconsistencies (calling things modules, for example), some really bad practice (if you generally mark your classes as sealed then you, whomever you are, are a sociopath), and some answers missing. So that's nice.</div>
<div>
<br /></div>
<div>
<br /></div>
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-57230268521109153772015-07-21T00:10:00.000+01:002015-07-30T10:57:15.863+01:00Using the Model-Store pattern in Inversion (part 5)<h2 style="margin: 0px;">
<div style="font-size: medium; font-weight: normal;">
This post is the fifth in a series about using a Model-Store pattern in the <a href="https://github.com/guy-murphy/inversion-dev" target="_blank">Inversion behavioural micro-framework for .Net</a> (see: <a href="https://github.com/guy-murphy/inversion-dev">https://github.com/guy-murphy/inversion-dev</a>).<br />
</div>
<div style="font-size: medium; font-weight: normal;">The first part can be found <a href="http://circle-theory.blogspot.co.uk/2015/03/using-model-store-pattern-in-inversion.html" target="_blank">here</a>.<br />
</div>
<div style="font-size: medium; font-weight: normal;">The second part can be found <a href="http://circle-theory.blogspot.com/2015/04/using-model-store-pattern-in-inversion.html" target="_blank">here</a>.<br />
</div>
<div style="font-size: medium; font-weight: normal;">The third part can be found <a href="http://circle-theory.blogspot.com/2015/05/using-model-store-pattern-in-inversion.html" target="_blank">here</a>.<br />
</div>
<div style="font-size: medium; font-weight: normal;">The fourth part can be found <a href="http://circle-theory.blogspot.com/2015/07/using-model-store-pattern-in-inversion.html" target="_blank">here</a>.<br /></div>
<br />
</h2>
<h2 style="margin: 0px;">Aftermath</h2>
<br />
<div style="margin: 0px;">
Looking back, I might as well have titled this essay - "I hate ORMs".<br />
<br />
Firstly, because I really, really do - I think they are at the root of many evils in .Net software architecture. That's not to say that its the ORMs fault, because it really isn't. However, time after time we are told to "just use" Entity Framework or N-Hibernate because that's what Microsoft or the big boys have blogged about, and time after time I see people getting into the same mess with a half-cocked Repository pattern, absurdly heavyweight models and bloody DTOs springing up all over the place. What I want is a clean, quick, safe and lightweight pattern of data access, and an ORM has never convinced me that it is any of these things because people tend to abuse the functionality it provides without thinking about the consequences or that there might be other ways to approach data access.<br />
<br />
Secondly, I didn't actually need to say that this was using Inversion, because Model-Store can be done in any .Net framework. However, it's been useful to write the foundations of the Inversion.Data namespace by considering the paths that writing this article has led me down.<br />
<br />
The views in this set of articles almost feel heretical when looking at the current "best practice" of .Net application development. Having an immutable model POCO and a lightweight store (which eschews ORM-like functionality) shifts complexity away from the data access implementation, forcing any business logic back to the application where (in my opinion) it belongs. Data access should be dumb, MUST be dumb, if the application is to be agnostic about the storage technology to be used. This massively increases the platform and language options available for porting and the development of new features. I would counsel avoiding relying upon the features of a particular storage medium, because you never know when that medium could become suddenly unsuitable or unavailable.
</div>
<br />
<div>
Model objects are best designed very closely to the underlying table or document structures. This is for the simplicity of maintenance and the ease of introduction to new development staff. A Model should remain at a level of abstraction such that the properties would be the same across common types of database representation. Therefore, an application that uses this pattern can achieve granular control over the strategies used for reading, writing and updating individual data classes whilst also remaining agnostic to storage technology choices.</div>
<br />
<div>
When developing the <a href="http://myoffers.co.uk/">MyOffers.co.uk</a> data analytics engine, we tended to write Stores for at least two storage mediums, e.g. MS Sql Server and Redis and/or MongoDB. This forced us into keeping data access straightforward while letting us concentrate on what the application actually needed to accomplish. These same patterns helped us again when writing a cloud-based quiz engine that would operate at scale and the new version of the website application. It forms part of a discipline that assists what I believe to be a fast and clean way of considering modern system architecture.<br /></div>
<br />
<h3>Immutable Models and CQRS/Event-Sourcing</h3>
<br />
<div>There are other benefits that immutable Models can bring. If you take the basic tenet that a Model is immutable, then it follows that any change that is required to a Model can be represented as a separate thing. This is the basis of an architecture which gives <a href="http://martinfowler.com/bliki/CQRS.html">CQRS</a>-like benefits to an application that uses immutable Models in combination with read-only, optimised Model Stores, with changes persisted in the form of Events to write-capable Stores which could have a completely different storage medium that is optimised for write, thus separating the reads (a Query) from the writes (a Command).<br />
</div>
<br />
<div>This helps support <a href="http://www.allthingsdistributed.com/2008/12/eventually_consistent.html">eventual consistency</a>, because any changes must be persisted explicitly and the source that the read-optimised Store fetches from can be lazily updated by a mechanism that is in the background. With this architecture, concurrent threads are guaranteed to read the same value when referencing an object, and modifications can be made without having to worry about locking or making a local copy.<br />
</div>
<br />
<h3>Ending</h3>
<br />
<div>It is not necessary to use the whole of Model-Store to get these advantages. However, the Store part of the pattern provides an efficient way of packaging data access which can use the immutable Model objects, and shows us that when implementing a DAL in the .Net ecosystem there are other options than Repository patterns and ORMs like Entity Framework or N-Hibernate.
</div>
<br />
<div>If you haven't agreed with me so far, then perhaps the most compelling thing I can offer is that, most of the time, an application need only read what the values of a Model object are; updates and writes are scarce by comparison. Why weigh your application down with the ability to track changes in an object when you don't use it in most operations?
</div>
<br />
<div>If you design your application to make changes explicitly then you can take advantage of the benefits of lightweight data access.<br />
</div>
<br />
<div>I've wanted to document some of my thoughts about this pattern for a while, and I do hope that some of you have stuck through to the end. There are many things that can be pointed to in this set of arguments which other developers may not agree with, and that's fine. As I said, this style of pattern may not be for you. However - I have seen it work in small, medium and production-scale systems, making them fly, whilst also providing a great deal of freedom as a developer.<br />
</div>
<br />
<div>At the very least - please check out the <a href="https://github.com/guy-murphy/inversion-dev" target="_blank">Inversion</a> framework. Full disclosure being that I'm a major contributor for it and I am using it very happily in the production of a large, automatically scaling cloud-based system for a major institution at the moment, including the bits and pieces you will find in the <a href="https://github.com/fractos/inversion-data">Inversion.Data</a> namespace.<br />
</div>
<br />
<div>Dream large.</div>Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-88418373917479855432015-07-12T18:40:00.001+01:002015-07-12T18:40:19.504+01:00Using the Model-Store pattern in Inversion (part 4)<div>
This post is the fourth in a series about using a Model-Store pattern in the <a href="https://github.com/guy-murphy/inversion-dev" target="_blank">Inversion behavioural micro-framework for .Net</a> (see: <a href="https://github.com/guy-murphy/inversion-dev">https://github.com/guy-murphy/inversion-dev</a>).<br />
<br />
The code that I have been developing as part of this article series can be found at <a href="https://github.com/fractos/inversion-data" target="_blank">https://github.com/fractos/inversion-data</a><br />
<br />
The first part can be found <a href="http://circle-theory.blogspot.co.uk/2015/03/using-model-store-pattern-in-inversion.html" target="_blank">here</a>.<br />
The second part can be found <a href="http://circle-theory.blogspot.com/2015/04/using-model-store-pattern-in-inversion.html" target="_blank">here</a>.<br />
The third part can be found <a href="http://circle-theory.blogspot.com/2015/05/using-model-store-pattern-in-inversion.html" target="_blank">here</a>.
<br />
<br />
<h2>
Using the Store</h2>
<div>
<br /></div>
<div>
In this instalment, I'm going to show you what using the pattern looks like. I'll put together a few example functions that use the IUserStore interface to access and manipulate data stored in an underlying MongoDB instance.<br />
<br />
<br />
<h3>
Configuring and Instantiation</h3>
<br />
<br />
We are going to use the MongoDBUserStore class that I showed you in the previous article to perform operations in MongoDB. To instantiate it, we will need a few details that describe the MongoDB connection-string, the database name and the name of the collection used for users.<br />
<br />
I am using a local MongoDB server, which has a database called 'test' and I want to access model objects in a collection called 'users':<br />
<br />
<script src="https://gist.github.com/fractos/a0e11cf828463b0396e7.js"></script><br />
<br />
Those details can then be passed to the MongoDBUserStore constructor:<br />
<br />
<script src="https://gist.github.com/fractos/0115da512f678c42910b.js"></script><br />
<br />
But since this is an IDisposable and we only actually care about the interface methods, then more correctly we should be calling it like this:<br />
<br />
<script src="https://gist.github.com/fractos/4fea9171e63fb7424f93.js"></script><br />
<br />
At which point we are off to the races.<br />
<br />
However, the Store could be gained by <i>dependency injection</i> behind an interface, or fetched from a service container (such as Inversion.Naiad, which implements Inversion.Process.IServiceContainer). I'll try and cover the use of IServiceContainer in a separate article, but the call might look something like this:<br />
<br />
<script src="https://gist.github.com/fractos/024a05195adb3fa8d6b6.js"></script><br />
<br />
By using interfaces alone, you can keep your code free from references to the assemblies needed for accessing MongoDB, keeping the main application as agnostic as possible. All the code cares about is the IUserStore interface.<br />
<br />
Asking a service container for an object of type IUserStore which has the label "store-user" will cause the container to perform any configuration required according to the instructions that are associated with the service label. The application doesn't have to care about the configuration of the Store because it is handled by the container. This is a form of <i>loose coupling</i> by <i>service location</i>.<br />
<br />
The more agnostic your code is about its dependencies, the more flexible your application can be.<br />
<br />
<h3>
Creating a collection of Users in MongoDB</h3>
<br />
This method assumes that you have a list of User models which are ready to be written to the database. I've included the basic version of instantiating the Store just to keep it simple.<br />
<br />
<script src="https://gist.github.com/fractos/559b7497c0fbb6ed40ea.js"></script></div>
<div>
<br /></div>
</div>
<br />
After this call, you should find that the User models have been written into MongoDB, complete with Object IDs.<br />
<br />
<h3>
Read a list of Users from MongoDB</h3>
<br />
This method will return all the users from MongoDB as a list of User models.<br />
<br />
Note that the IEnumerable<user> is converted to an IList<user> with ToList() before the using statement has been terminated (and therefore the Store and its resources disposed) in order to avoid the enumeration code reading a closed connection.</user></user><br />
<br />
<script src="https://gist.github.com/fractos/b6f45090985965807d77.js"></script><br />
<br />
<h3>
Updating a list of Users</h3>
<br />
This function iterates through a passed list of User models, modifying their Password property and then writing them back into MongoDB. It makes use of the Mutate method on the User object in order to change an immutable object. The ID of the User model will remain unchanged, however, so the act of writing the data into MongoDB will perform an update rather than an insert.<br />
<br />
<br />
<script src="https://gist.github.com/fractos/d89d9c0849478036d208.js"></script><br />
<br />
<h3>
Deleting a list of Users</h3>
<br />
This function will delete each of a list of User models from MongoDB.<br />
<br />
<script src="https://gist.github.com/fractos/a69cd136927add3cdeb2.js"></script><br />
<br />
<h3>
Next time ...</h3>
<br />
In the next (and final, so far ...) instalment, I'll wrap up with a conclusion and some thoughts about how this process went.<br />
<br />Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0tag:blogger.com,1999:blog-319207569350348666.post-29295068878154233892015-05-14T14:29:00.001+01:002015-05-14T14:29:42.258+01:00Reading progress ...Finished William Gibson's "The Peripheral" earlier this week after reaching about 80% of the way through it, realising that I was totally absorbed in it and that I didn't want it to end. So, I restarted the book and I am so glad that I did. It's an amazing, lushly detailed work. Very happy that he has returned to sci-fi with what might possibly be his finest work.<br />
<br />
<a href="http://www.amazon.co.uk/gp/product/0241961009/ref=as_li_tl?ie=UTF8&camp=1634&creative=6738&creativeASIN=0241961009&linkCode=as2&tag=readingpile-21&linkId=YJ2E5JWHHAPETHYL"><img border="0" src="http://ws-eu.amazon-adsystem.com/widgets/q?_encoding=UTF8&ASIN=0241961009&Format=_SL160_&ID=AsinImage&MarketPlace=GB&ServiceVersion=20070822&WS=1&tag=readingpile-21" /></a><img alt="" border="0" height="1" src="http://ir-uk.amazon-adsystem.com/e/ir?t=readingpile-21&l=as2&o=2&a=0241961009" style="border: none !important; margin: 0px !important;" width="1" /><br />
<div>
<br /></div>
<div>
There are so many books queued up on my Kindle. I've recently got back into going to the gym where I tend to get a good hour's worth of reading done if I find a free cross-trainer. Now I can start to chew through the pile of purchases I've made over the past six months since I moved into my fiancé's place from my old flat where the gym was just over the road.</div>
<div>
<br /></div>
<div>
First up, I'm hitting the start of the Farseer Trilogy by Robin Hobb. The whole series were on offer a while back and I was a bit click-happy.</div>
<div>
<br /></div>
<div>
Currently reading:</div>
<div>
<br /></div>
<div>
"Assassin's Apprentice (Book One of the Farseer Trilogy)" by Robin Hobb</div>
<div>
<br /></div>
<div>
<div>
<a href="http://www.amazon.co.uk/gp/product/000756225X/ref=as_li_tl?ie=UTF8&camp=1634&creative=6738&creativeASIN=000756225X&linkCode=as2&tag=readingpile-21&linkId=3NEHFB7UN5IFQMG5"><img border="0" src="http://ws-eu.amazon-adsystem.com/widgets/q?_encoding=UTF8&ASIN=000756225X&Format=_SL160_&ID=AsinImage&MarketPlace=GB&ServiceVersion=20070822&WS=1&tag=readingpile-21" /></a><img alt="" border="0" height="1" src="http://ir-uk.amazon-adsystem.com/e/ir?t=readingpile-21&l=as2&o=2&a=000756225X" style="border: none !important; margin: 0px !important;" width="1" /></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
Fractoshttp://www.blogger.com/profile/03224029111769065049noreply@blogger.com0