Wednesday, 7 December 2016

Configuring a containerised system


These are some basic methods of passing configuration into a container.

1. Building in configuration files

At build time, copy configuration files into the container image.
  • inflexible
  • insecure

2. Configuration fetched and applied during transient build step

Essentially this means pulling configuration data from somewhere and removing it before the end of the build step.
  • fragile
  • values sit in build cache

3. Environment variables

Discrete environment variable values injected at container instantiation time.
  • flexible - can re-configure per instance
  • good integration with Docker, ECS, shell

4. Holding configuration in S3

Pass S3 address to container by either an environment variable or a command parameter. Scripts or apps then retrieve dynamic configuration from the S3 bucket. Can be private, with ambient credentials assumed from an associated EC2 role on the container host, or by AWS credentials that are passed by environment variables.
  • allows dynamic configuration during container lifetime

5. Attach a configuration volume

A specific volume that holds configuration files can be attached to the container during instantiation.
  • allows dynamic configuration during container lifetime
  • have to start managing volumes across container hosts
  • Kubernetes uses this type of solution

6. Use a configuration server/service

Pass server details into the container at instantiation time. Scripts or apps will retrieve configuration from a service such as Vault, etcd, Redis, a database etc.

  • client dependencies
  • container contents may have to be adapted to integrate with a configuration server
  • allows dynamic configuration during container lifetime

Friday, 9 September 2016

I can see the sea :)

I can see the sea from this train :)

It's quite far out, with bright orange beached buoys in the middle distance before a platoon of wind power generators that are standing in the eventual water toward the horizon.

Curlews are striding about, or maybe redshanks still in their summer plumage.

Always a thrill to catch sight of the shore and the different kind of life that teems there, all under a grey-blue coastal sky.

Tuesday, 23 August 2016

Canary farm

"Canary farm"

An IT system connected to various other brittle systems that invariably picks up the blame for them being unavailable.

Monday, 8 August 2016

Fractos-cumuli tipping point

Fractos-cumuli tipping point:

When the monthly cost of the Cloud project you are working on exceeds your monthly salary.

Sunday, 3 July 2016

Deploying a .Net website to Amazon Elastic Beanstalk with TeamCity

Note for posterity.

Steps I had to take to get a .Net website deployed to Elastic Beanstalk with TeamCity.

I was originally working from this tutorial, but soon realised that things weren't going to plan at all. This was a pain in the arse.


Install Visual Studio Community edition on the TeamCity server.
Install AWS Deployment Tool awsdeploy according to the instructions in the tutorial.

1. Web.config transform

The version of the website I wanted to deploy needed the transformation for Release build to be performed before it was packaged up.

I edited the .csproj of the website to include BeforeBuild and AfterBuild steps that would transform the Web.config file to its Release form:
<target condition=" '$(Configuration)' == 'Release' " name="BeforeBuild"> <copy destinationfiles="Web.temp.config" overwritereadonlyfiles="True" sourcefiles="Web.config"> <transformxml destination="Web.config" source="Web.temp.config" transform="Web.$(Configuration).config"> </transformxml></copy></target> <target name="AfterBuild"> <copy destinationfiles="Web.config" overwritereadonlyfiles="True" sourcefiles="Web.temp.config"> <delete files="Web.temp.config"> </delete></copy></target>

2.  Working around MSBuild being retarded about indirect references

MSBuild tries to be smart about including references in a deployment package. So smart that it doesn't actually include the dependencies of references that it includes - so you will probably end up with a website that doesn't work.

The indirect dependencies are included as references in the website project but MSBuild still filters them out.

I found some advice somewhere saying you should set the Copy Local property on the references to True. I dutifully did this, but it turns out that Visual Studio (including 2015 Update 2) has a huge bug in this area as it will not update the .csproj file if you set Copy Local to true. However, if you set it to False then click Save All, it will create the XML element in the .csproj file. Once you've set it to False and saved, ONLY THEN can you set it to True (and hit Save All again).  You can select all the references at once to perform this action in Visual Studio.

3. Release build project in TeamCity must export everything as an artefact

I was finding it impossible to run MSBuild in the deployment chain without having access to the compiled code. So, my Release build project in TeamCity has all the project and library folders marked as being artefacts, including an umbrella folder for any git submodules that are included.

Then, in the deployment build chain, I have both a snapshot dependency on the Release build chain (which means that it will only use successful build artefacts) and an artefact dependency which includes all the project and library folders that were exported as artefacts from the Release build chain above.

4. Deployment build chain copies transformed web.config back into website

The transformed version of the web.config ends up squirrelled away under the website's obj folder. I had to include an initial build step in the deployment chain to copy the transformed file back into the right folder so the packaging step can find it.

This looks something like this (obviously replace <website> with the name of the website's folder):

copy /Y <website>\obj\Release\TransformWebConfig\transformed\Web.config <website>\Web.config

5. Getting the command line parameters right for the packaging build step

This particular website is based at the root of the IIS folder on the remote server (i.e. c:\inetpub\wwwroot), so I had to include an instruction to deploy to the "Default Web Site" bare IIS path in the command line parameters.

Also this includes the Package instruction, the Configuration type, the SolutionDir definition (which is the TeamCity checkout directory - this is set as I use a particular scheme for cutting down on duplication of nested git submodules that involves telling included projects to use $(SolutionDir) as the base for their references), and the PackageTempRootDir property - an empty property which tells the packager not to make a deep hierarchy within the zipped output file.

/T:Package /P:Configuration=Release /property:SolutionDir=""\ /P:PackageTempRootDir= /P:DeployIISAppPath="Default Web Site"

6. Create a configuration file for awsdeploy for the specific Elastic Beanstalk environment

For reasons that escaped me, I found it impossible to include certain parameters on the command line for the awsdeploy command. I had to work around this by creating a static configuration file for a particular Elastic Beanstalk environment. Quite bare-bones data:

AWSProfileName = default
Region = <your aws region>
Template = ElasticBeanstalk
UploadBucket = <the s3 bucket that elastic beanstalk uses e.g. elasticbeanstalk-eu-west-1-accountnumber>
Application.Name = <elastic beanstalk application name>
Environment.Name = <elastic beanstalk environment name>

I put this file in a well known place on the build server that I knew TeamCity could get at.

7. Getting the command line parameters right for the awsdeploy build step

awsdeploy needs a few command line parameters and it was a bit hit and miss to sort it out, but here's what I ended up with:

/DAWSAccessKey=<api access key> /DAWSSecretKey=<api secret key> /\<website>\obj\Release\Package\<name of project website> /v /w /r <configuration file from step 6>

That should do the actual deployment to the Elastic Beanstalk environment. The deployment package is the pathname of the zip that the previous step creates which should be under the obj\Release\Package folder and have the same name as the .csproj that MSBuild was executed against.

Tuesday, 1 March 2016

Showing a stream in Immediate Window (for posterity)

Recording how to show a stream's contents in Immediate Window for posterity. In this example it's the response stream when a WebClient raises an exception:

System.Text.Encoding.UTF8.GetString((byte[])$exception.Response.GetResponseStream().GetType().GetMethod("InternalGetBuffer", System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic).Invoke($exception.Response.GetResponseStream(), null))