Sunday, 18 February 2018

3 - 10,000 days - Tool

progressive, psychedelic dark rock

10,000 days

Tool homepage

Quite late in the Tool canon, “10,000 days” sees them in full psychedelic rock flow, with quiet, noodly, introspective sections blossoming into complex, stoned soundscapes.

I’ve only had the stand-out track “Right In Two” on my phone for years, having given the full album a listen only a handful of times in the past. However, giving it another audition yields some different bits that I’m picking up on and I have restored the full track list to my working collection. I think maybe it was the rambling intro for “Lost Keys (blame Hoffman)” that originally did it, where a Doctor tries to get a patient to talk after presenting at an Emergency Room in a silent, troubled state. I guess it doesn’t feel quite so close to the bone now, so that’s progress, right?

Sunkist and Sudafed, gyroscopes and infrared

The sound they produce is immense; signature guitar setup pervades, the drums display various complex-timing trickery, and the chunky, intricate basslines, thick with harmonics, bounce around the low- and mid- range. Get a decent copy and bathe your ears in it.


Two of the tracks - “Wings for Marie (Pt 1)” and “10,000 days (Wings Pt 2)” can be layered together to form a single song (see which has an interesting call-response thing going on.


  • Right In Two
  • The Pot
  • Vicarious


It turns out that the packaging for the CD is EPIC - with a built-in stereogram viewer and about 15 illustrations.

Showing off the cover

Wednesday, 17 January 2018

2 - 1/f - immune

progressive hard rock?

Bandcamp link


Once upon a stumbling around Bandcamp, I found immune linked from somewhere, referenced as an interesting, Tool-esque, British band to explore. Was not disappointed!

I don’t really know that much about them aside from that they went through a name change at some point - they are now known as Master & The Mule - but they have a talent for melodic, dark, progressive rock. It’s quite thoughtful, introspective, and only shouty when it really needs to be.


  • Monkey
  • Selling Screen
  • Consume

1 - Brave enough - Lindsey Stirling

primarily instrumental violin

YouTube channel

Brave Enough

Note: This is first in my album list due to a naming quirk (the ID3 tags all have “Brave Enough” surrounded by quote marks).

I’ve followed Lindsey Stirling’s career for a few albums now, and hearing that she had a new one out last year was an automatic decision to buy and add to the collection. There are a few more vocal collaborations this time, but still the same rich, unpretentious melody lines and beats. Her music, like her, is perky, fun and can catch you emotionally with a set of heartfelt and brilliant tunes. And when she’s got an idea in her head she really lands it simply because she’s insanely good at expression.

The album is a tribute and celebration of a friend of hers who, tragically, died young, and the songs are weaved around memories of good and sad times, aspirations and regrets. The emotional tug of the songs is strong and conveys a great deal, though I think I preferred the big, instrumental tunes on her previous album “Shatter Me”.

Overall it’s bittersweet and that’s a wonderful thing. If you don’t know her work then please check it out. She’s a delight and a wonderful talent who may surprise you. Further, the relationship she has with her fans at live shows and via social media is infectious and commendable.


  • Brave Enough ft. Christina Perri
  • Something Wild ft. Andrew McMahon
  • Gavi’s Song

Tuesday, 16 January 2018

0 - Intro

And then I decided that I wanted to review every album that I have on my phone, in alphabetical order, as they play. “Good excuse to write!” says the aspirational bit of my brain which really does need a good kicking every so often. God help us all; here we go.

Album list

Thursday, 2 March 2017

Regarding Dependency Injection

A colleague of mine recently posted on an internal Slack channel about this blog article: How not to do dependency injection - the static or singleton container.
Reading the article, I became aware of something that a friend of mine had written about, namely the absolute vitriol that many coding blogs and generally good books seem to have against Service Location. For example, I’m reading Adaptive Code via C# at the moment and in that Gary McLean Hall has a small meltdown about it, which is a shame because it’s a pretty good book for people to learn about refactoring and use of basic patterns. However it also commits the cardinal sin of saying that the ‘D’ in SOLID is for Dependency Injection and not Dependency Inversion, and that betrays the architectural bias of the author.
The upshot being:
  • Anyone who tells you service location is an anti-pattern isn’t fully aware of the problems that it is supposed to be an answer for.
  • Dependency Injection moves away from the original point - separating the configuration of services from their use.
  • Dependency Injection vastly increases the surface area of an object via abuse of the constructor which isn’t bound by interface contract, thereby avoiding abstracting dependencies fully and leading to constructors being the medium by which relationships between objects are communicated - (this is not a virtue)
  • By making assumptions about state, Dependency Injection turns architectural uses-a relationships into has-a - which blocks the use of singletons and a bunch of other architectural patterns where they would be appropriate.
  • Further, it means that although relationships between data entities are modelled, behavioural relationships between objects are not because they become one great big dependency ball.
Another in-depth article Guy wrote about Inversion of Control can be found here:

Friday, 27 January 2017

Microscaling follow-up

Hello. Long time no follow-up blog about this.
My original post detailed the problem that I was seeing with using Amazon’s EC2 Container Service to manage a cluster of identical containers that required to have traffic routed to them individually from a load balancer. At the time, this couldn’t be performed without extra tooling around the system, and I hinted that I had come up with a solution that would allow the process of microscaling to be achieved on EC2.
I had a part two of that article sitting in draft for well over a year; various things happened - we bought our first house, my partner became pregnant and our daughter was born back in October, and part one even got cited by on their page in their collection of papers - so, you will perhaps forgive me for being a little distracted and not getting around to finishing that bit off.
Meantime, things moved along in the world of ECS.
The Application Load Balancer feature arrived and after a couple of months of me studiously ignoring its existence, I can now say that it does exactly what I needed Warden to do in the first place, namely that an ECS Service is now not limited to static port routing and a new Service instance with a dynamically allocated external port on the container host can be routed to automatically by the ALB. (Previously, it had meant that adding a new Service instance would require a new container host to be added to a cluster as the “Classic” Elastic Load Balancer could not route automatically to the dynamic ports.)
However, I will document what I’d come up with that filled the gap.


Warden ( was my hack for this, which enabled me to run multiple identical containers on an ECS cluster and have them load-balanced. It is the Go version of a set of Perl scripts that I had created which managed the ECS cluster of image servers.
It was impossible to manage the level (number of instances) of an ECS Service if its definition required a static container port and you wanted to re-use the same host for as many container instances as possible. So, instead of defining an ECS Service, Warden’s Manager would manage the number of Tasks that were programatically being run across the cluster, increasing or decreasing them according to a metric (e.g. number of Elastic Beanstalk instances in an upstream system that calls the image servers I’m working on). ECS would then schedule the new Tasks across the cluster and the Registrar process would detect a change in the running container instances on a host, updating the host’s local Nginx routing setup to match and enrolling or removing the host from an associated Elastic Load Balancer if required.
Each container host in the cluster would have two extra containers running:
  • Redx (
    Which is a modified version of Nginx that takes its routing configuration from a live Redis database.
  • Redis
    To serve as Redx’s configuration database. Lua code in the Nginx config allows the routing configuration to be read dynamically from Redis, so front-ends and back-ends can be added or removed without having to restart the Nginx process.
The cluster would need a central Redis instance that would serve as the service database and competition platform for Warden instances.
  1. Synchronised the list of currently active containers on a host with a local nginx configuration held in Redis - “Registrar”
  2. Managed the number of Tasks running across an ECS cluster for a particular Task Definition according to a connected metric - “Manager”

The Registrar

The Registrar periodically examines the list of Docker containers that are running on the host and, for those it recognises, it inspects the container, pulling out the local IP address that Docker has assigned. The IP addresses and the exposed port numbers are used to update the Nginx configuration held in a local Redis. Each Service has an Nginx front-end which is on a well-known port, unique for that Service. The separate container instances are added as Nginx back-ends associated with the Service front-end. Arriving traffic is therefore load-balanced internally across all the matching containers on the container host. If a Service has any container instances on the host then Warden ensures the EC2 instance is enrolled in the assigned Elastic Load Balancer, and removes it from the ELB if there are zero container instances.

The Manager

This process would also run on each container host, but the instances would hold a centralised competition for a leader. Each instance detects if the currently presiding leader’s Availability Zone is listed as currently active and checks for a recent heartbeat message from the leader. If there is a problem, then a competition is held and instances roll random numbers as their entry. The winner is picked from a Redis sorted set and they become the leader. A kill message is lodged for the previous leader to pick up if they return from whatever disaster befell their AZ.
Meanwhile, the new leader emits heartbeats and measures how many Tasks for a Service need to be running by using a specified metric. I’d hard-coded this in the Perl version to look at the number of Elastic Beanstalk instances were running for a particular application and then using a set of files in S3 to map between the metric and the number of Tasks that should run, like a very simple static database. Manager uses the AWS SDK to increase or decrease the running number of Tasks for a particular Task Definition across the ECS cluster before waiting and looping its lifecycle.


I never got around to writing a clean way to define a Service in such a way that meant Warden could pick up its configuration dynamically. Also, how to abstract away the metrics that Manager would use to decide on Task numbers. I’d like to look into how to produce a plug-in architecture for Go that is friendly to being completely agnostic for both of these factors.
Further, the network topology that the project moved towards made it unnecessary to run Warden’s Manager. Originally there was one ECS cluster that spread across the three Availability Zones in the eu-west-1 Region and there was no notion of an ECS Service to keep a desired number of container instances running across the cluster, so Manager filled that gap. Then there were changes that were made in response to realising that Elastic File System, once it eventually emerged from Preview, was more expensive to run than separate NFS volumes and servers per AZ. This meant splitting up the system into verticals - one per AZ - so that each layer of the system would only deal with one NFS volume and one ECS cluster per AZ. Warden’s Registrar still ran on each container host to synchronise the load balancing across the image server containers, but the number of containers running was managed by an ECS Service.
It was a good exploration of what was possible with a bit of scripting and glue. Experience of producing tooling in Go that was reactive to system conditions has also been invaluable.
For comparison, this is the original version of Warden which was written in Perl and effectively is just glue between the Redis, Docker and AWS CLIs:
Go was a natural choice for making a better, more solid version of Warden, mainly because of its Redis and AWS SDK access via libraries.

Wednesday, 7 December 2016

Configuring a containerised system


These are some basic methods of passing configuration into a container.

1. Building in configuration files

At build time, copy configuration files into the container image.
  • inflexible
  • insecure

2. Configuration fetched and applied during transient build step

Essentially this means pulling configuration data from somewhere and removing it before the end of the build step.
  • fragile
  • values sit in build cache

3. Environment variables

Discrete environment variable values injected at container instantiation time.
  • flexible - can re-configure per instance
  • good integration with Docker, ECS, shell

4. Holding configuration in S3

Pass S3 address to container by either an environment variable or a command parameter. Scripts or apps then retrieve dynamic configuration from the S3 bucket. Can be private, with ambient credentials assumed from an associated EC2 role on the container host, or by AWS credentials that are passed by environment variables.
  • allows dynamic configuration during container lifetime

5. Attach a configuration volume

A specific volume that holds configuration files can be attached to the container during instantiation.
  • allows dynamic configuration during container lifetime
  • have to start managing volumes across container hosts
  • Kubernetes uses this type of solution

6. Use a configuration server/service

Pass server details into the container at instantiation time. Scripts or apps will retrieve configuration from a service such as Vault, etcd, Redis, a database etc.

  • client dependencies
  • container contents may have to be adapted to integrate with a configuration server
  • allows dynamic configuration during container lifetime