Feedback on Article 3(3)

The following is my feedback regarding the proposed legislation around uploading of software to radio devices.

See this link for more details:
https://blog.mehl.mx/2019/protect-freedom-on-radio-devices-raise-your-voice-today/

“Upload of software on radio equipment” initiative direct link:
https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2018-6621038_en

Feedback direct link:
https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2018-6621038/feedback/F238237_en?p_id=380919

This is an extremely concerning piece of proposed legislation and I urge you to read my words here and hopefully understand why this is such a bad idea.

By limiting the type of software that can be loaded onto these devices it will unequivocally lower the overall level of security in the ecosystem. There are countless examples of insecure software which has been left for years on devices that are actively in use, with manufacturers who are unwilling to devote resources to fixing security holes. Open Source has traditionally stepped up to fix these kinds of problems, led by people who want to avoid waste, to be more secure, and to have individual choice.

If users cannot load e.g. open source software onto a device, then this blows up the whole idea of sustainability, up-cycling and re-use. It would be a shameful thing to do as it would render many devices that could be re-used as completely useless and just ending up in landfill. That may as well be an environmental crime that this article would enable.

For example, many SSL attacks have been discovered in the past 5 years, and if the security in some software on a device was found to be vulnerable but no remedy was forthcoming from the manufacturer (as they had gone out of business or they were focusing on latest released devices), then that renders the device an active security risk, potentially compromised, and could cost an enterprise millions in staff time, losses from being attacked, etc. The same kind of situation can exist on phones, on other networked, radio devices such as WiFi routers, etc. Preventing a user to update them in this way is dangerous and actively helps/supports attackers who could otherwise be thwarted.

Allowing open source software on devices (e.g. phones, routers) means that the life, security and performance of them can be hugely extended, in a way that the user is in control of. For routers, it means that a device can be protected from new and active threats / vulnerabilities - these are literally being discovered each week. How quickly are approved updates from a manufacturer likely to arrive? Nowhere near that, obviously.

This article must be removed. This tendency toward centralised control of software on devices owned by individuals, charities, and companies both small and large must be examined carefully. There is simply no way that this can work in a beneficial way to users, because trust has so frequently been broken, and this is essentially guaranteed regardless of legislation.

Porting Inversion part 2

Things have progressed and I think it’s time to write up and highlight a few items.

Choices of what to port

There are some things which I have chosen to abandon for now. These include my Inversion.Web.Razor extensions in inversion-razor, and the ConfigurationHelper and Pipeline models for service containers in Inversion.Extensibility.

In the case of the Razor extensions, I think I will need to rewrite this entirely in light of the AspNetCore support for Razor. I’d much rather be integrated with that than remain off-piste with Antaris RazorEngine.

Regarding ConfigurationHelper and Pipeline interfaces, the configuration options are so starkly different in .Net Standard, and so much easier to deal with, that there’s no point having a shim to pull configuration from a database etc when the new methods are so much more accessible.

Publishing to NuGet

I’m not sure which sociopath designed the web UI for NuGet, but they need their head examining; they’re clearly insane. It’s not quite as bad as the Visual Studio interface though, at least.

Having fought through the various stuff it puts in your way to prevent you getting a clean package uploaded, the various repos - inversion-dev, inversion-data, inversion-data and inversion-messaging - have been made available. The assemblies that make up those repos are available separately in order to keep project tech hierarchies segregated, e.g. Inversion.Data base library is separate to Inversion.Data.AmazonSQS which is separate to Inversion.Data.Redis etc. Don’t cross the streams unless you have to.

The versions are in the 1.0.x range and the references of child packages to their parents are currently set to 1.0.* as I was pretty shocked how stupid the management of minor versions was in the NuGet CLI. Anyway, working now.

GitHub organisation

The main libraries are now under the newly formed inversion-org GitHub organisation - https://github.com/inversion-org

Guy, Rob and myself are owners.

Here you will find:

• Inversion.Data
• Inversion.Extensibity
• Inversion.Messaging

The main Inversion library hasn’t moved yet but this will be its home in the future. You can still find it here:

• Inversion

Other libraries, such as Inversion.Ultrastructure will also be moved into the organisation shortly.

Adding Travis automation

Some Travis automation has been added to build and publish the NuGet packages, which it does sort of blindly as it doesn’t check first if the version already exists and so seeks forgiveness rather than permission when it fails to upload. I suppose there might be ways to automate incrementing the patch number on the .csproj file, but at the moment it is manual.

Things to do next:

• add unit tests (with automation for commit status updates)
• test the libraries work!
• create basic application that uses the libraries via NuGet and can be deployed as a container

5 - 8 Rooms - Angina P

IDM, minimalist d’n’b

• http://www.anginap.com
• https://soundcloud.com/anginap

I think I first came across Ulla’s work on MySpace, a very long time ago. It was during the period when musicians in the same network would find you and then in response you would place hastily photoshopped “thanks for the add!” messages on their boards, carrying band and EP names, and it was all one nice music party, plus a bunch of emos and the worst CSS in the world. I fucking miss MySpace.

Anyway, I found some of her tracks via MySpace and some via MP3.com (yes, really) which carried “Tokyo 6pm” into my ears and I was lost instantly.

In truth, I’ve already reviewed this album, although I can’t remember exactly who it was for (maybe for her label, Notochord?) It was providing a soundbite synopsis that pretty much said that this is crystal-sharp IDM and complete headphone-fuel. I meant it back then and I still mean it today.

There’s space here - open and modern like stark and hard white-grey architecture - but it’s a space filled with details that are reminiscent of Japanese technical design and wonderful emotive sweeps. Framing this is a real perfectionist focus on complex layers of accurately scattered percussion. These combine to make this one of my favourite albums, let alone IDM, providing what I generally term “coding music”. This describes tracks that for years I hadn’t identified a genre for, but eventually worked out were something along the lines of tech-step jungle and complex, minimalist drum’n’bass.

“Destroy you, with my robots”

Three of the tracks are remixes and are extremely high quality, making wonderful companions of the originals and blending right on in with the whole album. Semiomime’s remix of “Known Issues” makes me think we might be related as they know exactly what sort of drums I like.

Highlights

• Glitter - this was my ringtone for ages until I hit the Mirror’s Edge soundtrack by Solar Fields
• Known Issues (Semiomime remix)

Porting Inversion

Just recently I have been porting the Inversion libraries over to .Net Standard, such that they will be able to support the existing set of behavioural applications that make up the image delivery part of the Digital Library Cloud Service (see https://dlcs.info).

To do this I received some initial help from Robert Stiff (@UatecUK) to prove that the basics would work in the new execution environment, since it is vaguely exotic, having been based around the Reactive Extensions for .Net. Once we had ported the OWIN-based hello-world website across, I was convinced that I would be able to port across almost the entire application.

Last weekend I set about taking the ported versions of the libraries in https://github.com/guy-murphy/inversion-dev, giving them some basic metadata and publishing them to NuGet as .Net Standard 2.0 packages (and yes, I have Guy’s explicit permission to do this). This included the new web code that Robert and myself tested the proof of concept with which has been added as the Inversion.Web.AspNetCore package.

Next, I ported and published the libraries in Inversion.Data (https://github.com/fractos/inversion-data) into separate NuGet packages. This means that the different storage dependencies will remain separate in a target application.

Inversion.Extensibility (https://github.com/fractos/inversion-extensibility) was ported and published next, with a separate package for Inversion.Extensibility.Web.

Finally, I ported Inversion.Messaging (https://github.com/fractos/inversion-messaging) and packaged it separately, just as I had done for Inversion.Data.

It should be noted that each of these repositories has a separate "dotnetcore" branch for this code.

Things to do next:

• add unit tests
• add Travis CI scripting to build, pack and publish to NuGet
• test the libraries work when imported!

4 - 3 - Peter Gabriel

progressive… stuff

Look, I’ll be honest; there’s a lot of pretty weird stuff on this album, and I am aware that Peter Gabriel could be described as massively up themselves. However, when I first came across this album (and Security, I was an impressionable, slightly obsessive youth with a head full of magic and sci-fi. This was fuel. This fed that whirlwind. At the time, I didn’t know just how much heroin Gabriel was doing. Seems quite obvious now…

The backing tracks and effects on this album date from the dawn of synth technology, and also the beginning of some techniques like Phil Collins’ noise-gate clipped drum tracks. No cymbals were allowed during the recordimg, according to the wiki, on the basis that an “artist given complete freedom dies a horrible death,” so restrictions forming part of the creative process - a rebellion of sorts.

There’s a ridiculous roll-call of musicians on the album, including Tony Levin, Kate Bush, the aforementioned Phil Collins, Paul Weller and Robert Fripp, it’s no wonder that there are some good sounds going on. I’ve always loved Tony Levin’s bass techniques; they’re just so out there.

Highlights

• Intruder - for its gated drums and creepy atmos.
• I Don’t Remember - I’m totally in love with the bass guitar sound on this.
• Not One Of Us - The intro and break are just beautiful.

Trivia

The version I first heard was my brother’s cassette copy of the vinyl record which had skips in places which made me think some friend of his, whacked into God-mode on acid, had put it together in a fit of genius, or it was an authentic home-run of a happy accident:

Peak-time viewing born in a flash
as I burn into your memory- skip
I burn into your memory- skip
I burn into your memory- click
-cells

3 - 10,000 days - Tool

progressive, psychedelic dark rock

Tool homepage

Quite late in the Tool canon, “10,000 days” sees them in full psychedelic rock flow, with quiet, noodly, introspective sections blossoming into complex, stoned soundscapes.

I’ve only had the stand-out track “Right In Two” on my phone for years, having given the full album a listen only a handful of times in the past. However, giving it another audition yields some different bits that I’m picking up on and I have restored the full track list to my working collection. I think maybe it was the rambling intro for “Lost Keys (blame Hoffman)” that originally did it, where a Doctor tries to get a patient to talk after presenting at an Emergency Room in a silent, troubled state. I guess it doesn’t feel quite so close to the bone now, so that’s progress, right?

Sunkist and Sudafed, gyroscopes and infrared

The sound they produce is immense; signature guitar setup pervades, the drums display various complex-timing trickery, and the chunky, intricate basslines, thick with harmonics, bounce around the low- and mid- range. Get a decent copy and bathe your ears in it.

Trivia

Two of the tracks - “Wings for Marie (Pt 1)” and “10,000 days (Wings Pt 2)” can be layered together to form a single song (see https://m.youtube.com/watch?v=EFjEp79zaNw) which has an interesting call-response thing going on.

Highlights

• Right In Two
• The Pot
• Vicarious

MASSIVE UPDATE

It turns out that the packaging for the CD is EPIC - with a built-in stereogram viewer and about 15 illustrations.

2 - 1/f - immune

progressive hard rock?

Bandcamp link

Once upon a stumbling around Bandcamp, I found immune linked from somewhere, referenced as an interesting, Tool-esque, British band to explore. Was not disappointed!

I don’t really know that much about them aside from that they went through a name change at some point - they are now known as Master & The Mule - but they have a talent for melodic, dark, progressive rock. It’s quite thoughtful, introspective, and only shouty when it really needs to be.

Highlights

• Monkey
• Selling Screen
• Consume

1 - Brave enough - Lindsey Stirling

primarily instrumental violin

YouTube channel

Note: This is first in my album list due to a naming quirk (the ID3 tags all have “Brave Enough” surrounded by quote marks).

I’ve followed Lindsey Stirling’s career for a few albums now, and hearing that she had a new one out last year was an automatic decision to buy and add to the collection. There are a few more vocal collaborations this time, but still the same rich, unpretentious melody lines and beats. Her music, like her, is perky, fun and can catch you emotionally with a set of heartfelt and brilliant tunes. And when she’s got an idea in her head she really lands it simply because she’s insanely good at expression.

The album is a tribute and celebration of a friend of hers who, tragically, died young, and the songs are weaved around memories of good and sad times, aspirations and regrets. The emotional tug of the songs is strong and conveys a great deal, though I think I preferred the big, instrumental tunes on her previous album “Shatter Me”.

Overall it’s bittersweet and that’s a wonderful thing. If you don’t know her work then please check it out. She’s a delight and a wonderful talent who may surprise you. Further, the relationship she has with her fans at live shows and via social media is infectious and commendable.

Highlights

• Brave Enough ft. Christina Perri
• Something Wild ft. Andrew McMahon
• Gavi’s Song

0 - Intro

And then I decided that I wanted to review every album that I have on my phone, in alphabetical order, as they play. “Good excuse to write!” says the aspirational bit of my brain which really does need a good kicking every so often. God help us all; here we go.

Regarding Dependency Injection

A colleague of mine recently posted on an internal Slack channel about this blog article: How not to do dependency injection - the static or singleton container.

Reading the article, I became aware of something that a friend of mine had written about, namely the absolute vitriol that many coding blogs and generally good books seem to have against Service Location. For example, I’m reading Adaptive Code via C# at the moment and in that Gary McLean Hall has a small meltdown about it, which is a shame because it’s a pretty good book for people to learn about refactoring and use of basic patterns. However it also commits the cardinal sin of saying that the ‘D’ in SOLID is for Dependency Injection and not Dependency Inversion, and that betrays the architectural bias of the author.

The upshot being:

• Anyone who tells you service location is an anti-pattern isn’t fully aware of the problems that it is supposed to be an answer for.
• Dependency Injection moves away from the original point - separating the configuration of services from their use.
• Dependency Injection vastly increases the surface area of an object via abuse of the constructor which isn’t bound by interface contract, thereby avoiding abstracting dependencies fully and leading to constructors being the medium by which relationships between objects are communicated - (this is not a virtue)
• By making assumptions about state, Dependency Injection turns architectural uses-a relationships into has-a - which blocks the use of singletons and a bunch of other architectural patterns where they would be appropriate.
• Further, it means that although relationships between data entities are modelled, behavioural relationships between objects are not because they become one great big dependency ball.

Guy Murphy - Service Locator vs Dependency Injection

Another in-depth article Guy wrote about Inversion of Control can be found here:

Guy Murphy - I come not to bury IoC, but to praise it

Microscaling follow-up

Hello. Long time no follow-up blog about this.

My original post detailed the problem that I was seeing with using Amazon’s EC2 Container Service to manage a cluster of identical containers that required to have traffic routed to them individually from a load balancer. At the time, this couldn’t be performed without extra tooling around the system, and I hinted that I had come up with a solution that would allow the process of microscaling to be achieved on EC2.

I had a part two of that article sitting in draft for well over a year; various things happened - we bought our first house, my partner became pregnant and our daughter was born back in October, and part one even got cited by force12.io on their microscaling.org page in their collection of papers - so, you will perhaps forgive me for being a little distracted and not getting around to finishing that bit off.

Meantime, things moved along in the world of ECS.

The Application Load Balancer feature arrived and after a couple of months of me studiously ignoring its existence, I can now say that it does exactly what I needed Warden to do in the first place, namely that an ECS Service is now not limited to static port routing and a new Service instance with a dynamically allocated external port on the container host can be routed to automatically by the ALB. (Previously, it had meant that adding a new Service instance would require a new container host to be added to a cluster as the “Classic” Elastic Load Balancer could not route automatically to the dynamic ports.)

However, I will document what I’d come up with that filled the gap.

Warden

Warden (https://github.com/fractos/warden) was my hack for this, which enabled me to run multiple identical containers on an ECS cluster and have them load-balanced. It is the Go version of a set of Perl scripts that I had created which managed the ECS cluster of image servers.

It was impossible to manage the level (number of instances) of an ECS Service if its definition required a static container port and you wanted to re-use the same host for as many container instances as possible. So, instead of defining an ECS Service, Warden’s Manager would manage the number of Tasks that were programatically being run across the cluster, increasing or decreasing them according to a metric (e.g. number of Elastic Beanstalk instances in an upstream system that calls the image servers I’m working on). ECS would then schedule the new Tasks across the cluster and the Registrar process would detect a change in the running container instances on a host, updating the host’s local Nginx routing setup to match and enrolling or removing the host from an associated Elastic Load Balancer if required.
Each container host in the cluster would have two extra containers running:

• Redx (https://github.com/CBarraford/docker-redx)
  Which is a modified version of Nginx that takes its routing configuration from a live Redis database.
• Redis
  To serve as Redx’s configuration database. Lua code in the Nginx config allows the routing configuration to be read dynamically from Redis, so front-ends and back-ends can be added or removed without having to restart the Nginx process.

The cluster would need a central Redis instance that would serve as the service database and competition platform for Warden instances.

1. Synchronised the list of currently active containers on a host with a local nginx configuration held in Redis - “Registrar”
2. Managed the number of Tasks running across an ECS cluster for a particular Task Definition according to a connected metric - “Manager”

The Registrar

The Registrar periodically examines the list of Docker containers that are running on the host and, for those it recognises, it inspects the container, pulling out the local IP address that Docker has assigned. The IP addresses and the exposed port numbers are used to update the Nginx configuration held in a local Redis. Each Service has an Nginx front-end which is on a well-known port, unique for that Service. The separate container instances are added as Nginx back-ends associated with the Service front-end. Arriving traffic is therefore load-balanced internally across all the matching containers on the container host. If a Service has any container instances on the host then Warden ensures the EC2 instance is enrolled in the assigned Elastic Load Balancer, and removes it from the ELB if there are zero container instances.

The Manager

This process would also run on each container host, but the instances would hold a centralised competition for a leader. Each instance detects if the currently presiding leader’s Availability Zone is listed as currently active and checks for a recent heartbeat message from the leader. If there is a problem, then a competition is held and instances roll random numbers as their entry. The winner is picked from a Redis sorted set and they become the leader. A kill message is lodged for the previous leader to pick up if they return from whatever disaster befell their AZ.

Meanwhile, the new leader emits heartbeats and measures how many Tasks for a Service need to be running by using a specified metric. I’d hard-coded this in the Perl version to look at the number of Elastic Beanstalk instances were running for a particular application and then using a set of files in S3 to map between the metric and the number of Tasks that should run, like a very simple static database. Manager uses the AWS SDK to increase or decrease the running number of Tasks for a particular Task Definition across the ECS cluster before waiting and looping its lifecycle.

Afterwards

I never got around to writing a clean way to define a Service in such a way that meant Warden could pick up its configuration dynamically. Also, how to abstract away the metrics that Manager would use to decide on Task numbers. I’d like to look into how to produce a plug-in architecture for Go that is friendly to being completely agnostic for both of these factors.

Further, the network topology that the project moved towards made it unnecessary to run Warden’s Manager. Originally there was one ECS cluster that spread across the three Availability Zones in the eu-west-1 Region and there was no notion of an ECS Service to keep a desired number of container instances running across the cluster, so Manager filled that gap. Then there were changes that were made in response to realising that Elastic File System, once it eventually emerged from Preview, was more expensive to run than separate NFS volumes and servers per AZ. This meant splitting up the system into verticals - one per AZ - so that each layer of the system would only deal with one NFS volume and one ECS cluster per AZ. Warden’s Registrar still ran on each container host to synchronise the load balancing across the image server containers, but the number of containers running was managed by an ECS Service.

It was a good exploration of what was possible with a bit of scripting and glue. Experience of producing tooling in Go that was reactive to system conditions has also been invaluable.

For comparison, this is the original version of Warden which was written in Perl and effectively is just glue between the Redis, Docker and AWS CLIs: https://github.com/fractos/warden-perl.

Go was a natural choice for making a better, more solid version of Warden, mainly because of its Redis and AWS SDK access via libraries.

Configuring a containerised system

(from http://fractos.github.io/containers/2016/12/07/configuring-a-containerised-system.html)

These are some basic methods of passing configuration into a container.

1. Building in configuration files

At build time, copy configuration files into the container image.

• inflexible
• insecure

2. Configuration fetched and applied during transient build step

Essentially this means pulling configuration data from somewhere and removing it before the end of the build step.

• fragile
• values sit in build cache

3. Environment variables

Discrete environment variable values injected at container instantiation time.

• flexible - can re-configure per instance
• good integration with Docker, ECS, shell

4. Holding configuration in S3

Pass S3 address to container by either an environment variable or a command parameter. Scripts or apps then retrieve dynamic configuration from the S3 bucket. Can be private, with ambient credentials assumed from an associated EC2 role on the container host, or by AWS credentials that are passed by environment variables.

• allows dynamic configuration during container lifetime

5. Attach a configuration volume

A specific volume that holds configuration files can be attached to the container during instantiation.

• allows dynamic configuration during container lifetime
• have to start managing volumes across container hosts
• Kubernetes uses this type of solution

6. Use a configuration server/service

Pass server details into the container at instantiation time. Scripts or apps will retrieve configuration from a service such as Vault, etcd, Redis, a database etc.

• client dependencies
• container contents may have to be adapted to integrate with a configuration server
• allows dynamic configuration during container lifetime

I can see the sea :)

I can see the sea from this train :)

It's quite far out, with bright orange beached buoys in the middle distance before a platoon of wind power generators that are standing in the eventual water toward the horizon.

Curlews are striding about, or maybe redshanks still in their summer plumage.

Always a thrill to catch sight of the shore and the different kind of life that teems there, all under a grey-blue coastal sky.

Canary farm

"Canary farm"

An IT system connected to various other brittle systems that invariably picks up the blame for them being unavailable.

Fractos-cumuli tipping point

Fractos-cumuli tipping point:

When the monthly cost of the Cloud project you are working on exceeds your monthly salary.

Deploying a .Net website to Amazon Elastic Beanstalk with TeamCity

Note for posterity.

Steps I had to take to get a .Net website deployed to Elastic Beanstalk with TeamCity.

I was originally working from this tutorial, but soon realised that things weren't going to plan at all. This was a pain in the arse.

Pre-requisites:

Install Visual Studio Community edition on the TeamCity server.
Install AWS Deployment Tool awsdeploy according to the instructions in the tutorial.

1. Web.config transform

The version of the website I wanted to deploy needed the transformation for Release build to be performed before it was packaged up.

I edited the .csproj of the website to include BeforeBuild and AfterBuild steps that would transform the Web.config file to its Release form:
<target condition=" '$(Configuration)' == 'Release' " name="BeforeBuild"> <copy destinationfiles="Web.temp.config" overwritereadonlyfiles="True" sourcefiles="Web.config"> <transformxml destination="Web.config" source="Web.temp.config" transform="Web.$(Configuration).config"> </transformxml></copy></target> <target name="AfterBuild"> <copy destinationfiles="Web.config" overwritereadonlyfiles="True" sourcefiles="Web.temp.config"> <delete files="Web.temp.config"> </delete></copy></target>

2.  Working around MSBuild being retarded about indirect references

MSBuild tries to be smart about including references in a deployment package. So smart that it doesn't actually include the dependencies of references that it includes - so you will probably end up with a website that doesn't work.

The indirect dependencies are included as references in the website project but MSBuild still filters them out.

I found some advice somewhere saying you should set the Copy Local property on the references to True. I dutifully did this, but it turns out that Visual Studio (including 2015 Update 2) has a huge bug in this area as it will not update the .csproj file if you set Copy Local to true. However, if you set it to False then click Save All, it will create the XML element in the .csproj file. Once you've set it to False and saved, ONLY THEN can you set it to True (and hit Save All again).  You can select all the references at once to perform this action in Visual Studio.

3. Release build project in TeamCity must export everything as an artefact

I was finding it impossible to run MSBuild in the deployment chain without having access to the compiled code. So, my Release build project in TeamCity has all the project and library folders marked as being artefacts, including an umbrella folder for any git submodules that are included.

Then, in the deployment build chain, I have both a snapshot dependency on the Release build chain (which means that it will only use successful build artefacts) and an artefact dependency which includes all the project and library folders that were exported as artefacts from the Release build chain above.

4. Deployment build chain copies transformed web.config back into website

The transformed version of the web.config ends up squirrelled away under the website's obj folder. I had to include an initial build step in the deployment chain to copy the transformed file back into the right folder so the packaging step can find it.

This looks something like this (obviously replace <website> with the name of the website's folder):

copy /Y <website>\obj\Release\TransformWebConfig\transformed\Web.config <website>\Web.config

5. Getting the command line parameters right for the packaging build step

This particular website is based at the root of the IIS folder on the remote server (i.e. c:\inetpub\wwwroot), so I had to include an instruction to deploy to the "Default Web Site" bare IIS path in the command line parameters.

Also this includes the Package instruction, the Configuration type, the SolutionDir definition (which is the TeamCity checkout directory - this is set as I use a particular scheme for cutting down on duplication of nested git submodules that involves telling included projects to use $(SolutionDir) as the base for their references), and the PackageTempRootDir property - an empty property which tells the packager not to make a deep hierarchy within the zipped output file.

/T:Package /P:Configuration=Release /property:SolutionDir="%system.teamcity.build.checkoutDir%"\ /P:PackageTempRootDir= /P:DeployIISAppPath="Default Web Site"

6. Create a configuration file for awsdeploy for the specific Elastic Beanstalk environment

For reasons that escaped me, I found it impossible to include certain parameters on the command line for the awsdeploy command. I had to work around this by creating a static configuration file for a particular Elastic Beanstalk environment. Quite bare-bones data:

AWSProfileName = default
Region = <your aws region>
Template = ElasticBeanstalk
UploadBucket = <the s3 bucket that elastic beanstalk uses e.g. elasticbeanstalk-eu-west-1-accountnumber>
Application.Name = <elastic beanstalk application name>
Environment.Name = <elastic beanstalk environment name>

I put this file in a well known place on the build server that I knew TeamCity could get at.

7. Getting the command line parameters right for the awsdeploy build step

awsdeploy needs a few command line parameters and it was a bit hit and miss to sort it out, but here's what I ended up with:

/DAWSAccessKey=<api access key> /DAWSSecretKey=<api secret key> /DDeploymentPackage=%teamcity.build.checkoutDir%\<website>\obj\Release\Package\<name of project website>.csproj.zip /v /w /r <configuration file from step 6>

That should do the actual deployment to the Elastic Beanstalk environment. The deployment package is the pathname of the zip that the previous step creates which should be under the obj\Release\Package folder and have the same name as the .csproj that MSBuild was executed against.

Showing a stream in Immediate Window (for posterity)

Recording how to show a stream's contents in Immediate Window for posterity. In this example it's the response stream when a WebClient raises an exception:

System.Text.Encoding.UTF8.GetString((byte[])$exception.Response.GetResponseStream().GetType().GetMethod("InternalGetBuffer", System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic).Invoke($exception.Response.GetResponseStream(), null))

(from: http://stackoverflow.com/questions/7755312/view-content-of-a-stream-in-visual-studios-quickwatch-window)

Bookmarks 20160216

Various public datasets

https://github.com/caesar0301/awesome-public-datasets


I <3 postgresql
http://www.postgresonline.com/journal/archives/228-PostgreSQL-Array-The-ANY-and-Contains-trick.html


Shocked I missed this - arbitrary linkage to IFTTT with the Maker Channel:

http://blog.ifttt.com/post/121786069098/introducing-the-maker-channel

Microscaling with Load-balancing in Amazon ECS (part 1)

Currently I am involved in the development of a digital imaging platform for libraries and other institutions (the Digital Library Cloud Service, see http://dlcs.info) This platform uses a wide palette of Amazon Web Services for the orchestration of moving image assets between storage systems, retrieval of metadata and the presentation of deep-zoom image tiles. It’s sorta-kinda an “elastic image server” which supports the IIIF (http://iiif.io) standard.

The system is split into two layers - Orchestration and Image-Serving. Both of these are intended to auto-scale independently, and this has been achieved with the Orchestration layer by deploying it as an Elastic Beanstalk application. Since this application is (currently) a .Net application that lives in IIS, then this environment suits it well.

The Image-Serving layer is Linux-based, comprising of a lightweight HTTP daemon (lighttpd) and Ruven Patel’s IIPSrv (http://iipimage.sourceforge.net/) with optimised Jpeg-2000 support. This is quite easily deployed as a Docker container and that is what our current prototype uses.

Since we are using Docker anyway, I looked into how Amazon’s Elastic Container Service could be used to auto-scale the Image-Serving layer on demand.

What I found was that the concept of a Service in ECS did not cover how I thought it could be used.

Services under ECS

(http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-load-balancing.html)

A Service in ECS represents parts of an application that are distributed across various Docker containers. A cluster of EC2 instances form a group of hosts that a Service may be deployed to. Each host may contain a maximum of one instance of a Service. When scaling occurs, the EC2 cluster can expand or contract with Auto-Scaling Group rules and the ECS Scheduler decides where new instances are placed.

The limitation of a maximum of a single Service instance per host is one that I was not expecting, but can understand given the way Elastic Load Balancers are configured. In ECS, the port numbers of containers in the Service are static because you’ll only get one instance of the Service per host, and therefore load balancing can occur across the cluster in a static way (from the Elastic Load Balancer’s point of view).

When I saw the setup dialog for a Task in ECS mention the internal and external ports for a Docker container, I had hoped that this implied that the external port on the container could be balanced across if it was left blank, but that turned out not to be the case.

Microscaling

What I wanted was the ability to work with multiple similar Docker containers populating EC2 cluster members dynamically in response to some metric - e.g. a CloudWatch alarm from our orchestration layer’s load balancer. As that Elastic Beanstalk application scales in response to demand, so would the number of deployed containers up to the available space of the hosts in the cluster as defined in ECS.

It turns out that this technique has a term - microscaling - and I’ve been researching it recently, including the offering from Force12.io - (http://force12.io/) - see http://microscaling.org for more details.

Once the number of containers is under dynamic control, the cluster itself could then expand or contract by responding to a separate Auto-Scaling Group rule.

Load-balancing

The other issue is load balancing across the cluster. In the scenario I want, the ports on the deployed containers would need to be routed to directly by the load balancer, so that means multiple different ports per host. This is something which would bend the ELB architecture out of shape, I think.

UPDATE: Now that Amazon have released their Application Load Balancer feature, then the load balancing problem is more or less solved on AWS. I must have been one of many people who were after this kind of feature as a senior ECS developer talked to me about it on Twitter over a year ago.

Use case and solution

Our use case is that I want to have an auto-scaling group of EC2 instances in an ECS cluster which host multiple similar containers that will listen on the same internal port, but are routed to via their external port that is assigned by Docker (e.g. while still appearing to be port 80 inside the container). They will be available to be load-balanced across all the containers that are running in the cluster. These large EC2 instances could host multiple different ‘services’ comprised of similar containers.

Essentially, we’d like to have a number of large container hosts and fill them on-demand with web servers or other cookie-cutter services, with load balancing going direct to the containers, across a cluster, across AZs.

I have created a system for performing both microscaling and load-balancing within ECS, and I will show the solution in the next blog post.

Twitter - We should talk.

Hi Twitter.

Please, come in and sit down. I think it's time we had a little chat.

I know things haven't been great lately, and I think you're coming undone at the seams a bit.

I'd like to lay a few things out for discussion.

Your position.

You are in a unique position of being a vital, real-time connection between content producers and interested parties, who may also be content producers. You enjoy a position which is (blessedly) outside Facebook and is not a walled garden. You have made amazing technical advances to be able to support this amazing real-time platform. You are, in general, viewed as a force for good. You have friends.

I have witnessed more news, events, conversations and opinions on Twitter than almost any other medium I've used in the past twenty years that I've been on the Internet. It's addictive to be part of this huge, global conversation and I love that.

Recent activity.

But you keep doing these things.

Layoffs.

Maybe these were justified because you had over-hired, but it looked reactionary which, if I was one of your investors, would make me quite nervous and/or irritated. Stop jerking your knees. Be more solid.

Moments.

Experimental. I think it's good to separate curated items like this, which leaves timelines alone. More on that later. Definitely interesting as a thing for surrounding current affairs and getting people interested in them. The "You're caught up" is very Circa-like (RIP Circa). The Trending items have been kind of filling that purpose already, though, but the presentation in Moments is very clean, so I hope that this can build on that kind of attention into something cool. But I'm really not sure that this is the new user grabber that you think it is.

Please don't retire trending or hashtags though. That would be uncool.

Polling.

The polling functionality is a really powerful thing. You can do a lot of good in this area. Keep up with the good work on this. Be a part of online democracy and live reaction. We like.

Periscope.

Still niche, but an excellent reporting tool. It would be nice to see better integration of that in things like the TweetDeck Chrome extension. Real problems here are bandwidth, video quality, people's data plans, vertical aspect video.

Vine.

I know Instagram do 15 seconds, but 6 seconds is still cool, right? The integration of this service makes sense because it lends itself to the concise nature of a Tweet. However, the only people I have really seen (in the UK) using Vine are comedians. Think about that for one of those 6 seconds.

Apparent lack of action regarding the tackling of online abuse.

You really, REALLY need to sort this shit out. Provide tools, streamline reporting processes, support victims, unleash rogue AI... Literally anything apart from sitting on your hands about bullying and threatening behaviour.

Favourites vs likes.

Stars vs hearts. Connotations and implications. Organisation vs emotion. I could go on. In fact ...

The whole point of Web 2.0 was the ability to customise your web experience. What happened? How can it not be clear that this change makes it means something totally different? It smacks of flag-waving. You must stop doing these kinds of changes. People (read: your user base) get very used to these semantics and symbols, and you'd be amazed what could drive a person away if they felt aggrieved enough about it. As it is, it's now fairly evident that the people who make these kinds of decisions don't appear to use Twitter or "get it" - the blog post from Akarshan Kumar made it clear that you are going for lowest common denominator, and that's extremely patronising.

The best possible thing you can do here is to bring back the Star icon, and call the collection of things "Starred", because that is completely agnostic to emotional connections. Then this feature can go back to its meaning being implied from context (which includes timing) and also have zero connotations when used as an organisational method. Please do this because what you've done makes no sense to me and many others.

Noise.

Most of Twitter is noisy. If I step outside of the accounts I follow then I find myself in a horrific sea of chatter. I'm actually cool with that. It supports the notion of 'tuning-in' to certain voices; apart from the Etsy bots - I'm f-ing sick of those. You should be able to spot them and many other types of spammer by now - so why aren't they automatically banned?

Concentration on in-house clients.

At the cost of pissing off every other development team who ever tried to make a 3rd party add on or client for Twitter. It's like you really don't want an ecosystem at all. You are a PLATFORM, a SERVICE, let people MAKE THINGS THAT USE YOUR PLATFORM. If this is about the whole advertising thing - i.e. that you can't control placement of adverts in 3rd party clients, then you're doing things backwards. FIND ANOTHER MODEL (see below). It's not like you're showing adverts in TweetDeck's chrome extension yet anyway (and please don't start now - I'll cry).

Perverse notion of messing with the chronology of timelines.

Twitter's utility is that it shows a chronological view of content. I have absolutely no idea why you think that modifying this is in any way a good thing to do. Even the the out-of-order replies reaction (the "blue line" fiasco) was a UX-disaster with people (like me) complaining about the cognitive dissonance it provoked and being the final nail in the coffin for staying the hell away from the web client. The considered plans for "Tweets-you-may-have-missed" is just barmy. If there are things that you think I've missed - PUT THEM INTO A SEPARATE LIST. Seriously, if you cock up the chronology of Tweets in a timeline then you might as well be a randomly ordered collection of things people have said at some point in the past X hours like Facebook. Just stop that line of thinking.

Consideration of Tweets longer than 140 characters.

No, no, no, no, NO. Twitter is concise. Keep it that way. How am I supposed to catch up on 500+ Tweets that happened overnight if they are a bunch of essays? That's what I have RSS for! If someone wants to post more than 140 characters then they should host content elsewhere which has meta-data on it that allows the previews in clients - just like they do now. CREATE AN ECOSYSTEM, NOT A WALLED GARDEN.

Also - please support RSS. RSS is your friend.

The web client.

I mean, seriously. I'm really glad that we got the Bootstrap project out of Twitter (I use it every day at work), but my God you guys really need to completely re-think your UX. Jesus F-ing Christ, what a mess. If I was approaching that project I would absolutely throw the entire template away and hope that no-one ever mentioned it at parties I went to.

Stock price.

By doubling down on bad ideas to please investors, all it goes to show is that you have collected the wrong investors, with the wrong promises whispered into the wrong ears.

The knock-on effects of promising swathes of new users is removing focus from what makes your platform great.

New users are only valuable if you can present advertising to them. And you've seen the current ad-block fuss.

What you need are customers.

Recommended actions.

Subscription model.

I would absolutely pay $5-10 dollars a month for a subscription to Twitter. That subscription would be ad-free, with a reasonable expectation of timely support for problems with my account, payment, online abuse, and the knowledge that my 3rd party Twitter client could be used without limitation.

I can envisage you running a subscription programme in parallel with your current operations, the only difference being the ad-free nature of the channel, a better SLA for support and the unlocked API access.

The more I think about it, (and I've been thinking about this for a few years), the more strongly I believe that this would be hugely successful for you.

But how much would subscription actually cost you? What change would that have on revenue when a (hopefully significant) percentage of your users are throwing you money each month? What would be the effect upon new user take-up, if they joined knowing that they could upgrade their experience? What would be the effect on API use if it was suddenly opened up to paying users?

I can't help but feel that these changes would be positive.

Content producer payment.

Given that many people join up to be able to follow people with something to say, celebrities, authors, musicians, news outlets, etc, then perhaps that should be rewarded somehow.

There are a few different services which allow users to make payment via Twitter. By taking that in-house this could be like your own version of Patreon, (minus the data leakage), offering content producers payback for being popular.

Again, this is about creating an ecosystem. Reward people for taking part.

Signing off.

Twitter, I don't know if you'll read this. I hope you do and I hope that some of it gets through to you. You are a vital, current thing, and it is in everyone's interest (apart from Facebook's) that you stop these self-harming actions and become something better.

Good day to you, whomever you are.

@fractos.