Saturday, 7 November 2015

Microscaling with Load-balancing in Amazon ECS (part 1)

Currently I am involved in the development of a digital imaging platform for libraries and other institutions (the Digital Library Cloud Service, see This platform uses a wide palette of Amazon Web Services for the orchestration of moving image assets between storage systems, retrieval of metadata and the presentation of deep-zoom image tiles. It’s sorta-kinda an “elastic image server” which supports the IIIF ( standard.
The system is split into two layers - Orchestration and Image-Serving. Both of these are intended to auto-scale independently, and this has been achieved with the Orchestration layer by deploying it as an Elastic Beanstalk application. Since this application is (currently) a .Net application that lives in IIS, then this environment suits it well.
The Image-Serving layer is Linux-based, comprising of a lightweight HTTP daemon (lighttpd) and Ruven Patel’s IIPSrv ( with optimised Jpeg-2000 support. This is quite easily deployed as a Docker container and that is what our current prototype uses.
Since we are using Docker anyway, I looked into how Amazon’s Elastic Container Service could be used to auto-scale the Image-Serving layer on demand.
What I found was that the concept of a Service in ECS did not cover how I thought it could be used.

Services under ECS

A Service in ECS represents parts of an application that are distributed across various Docker containers. A cluster of EC2 instances form a group of hosts that a Service may be deployed to. Each host may contain a maximum of one instance of a Service. When scaling occurs, the EC2 cluster can expand or contract with Auto-Scaling Group rules and the ECS Scheduler decides where new instances are placed.
The limitation of a maximum of a single Service instance per host is one that I was not expecting, but can understand given the way Elastic Load Balancers are configured. In ECS, the port numbers of containers in the Service are static because you’ll only get one instance of the Service per host, and therefore load balancing can occur across the cluster in a static way (from the Elastic Load Balancer’s point of view).
When I saw the setup dialog for a Task in ECS mention the internal and external ports for a Docker container, I had hoped that this implied that the external port on the container could be balanced across if it was left blank, but that turned out not to be the case.


What I wanted was the ability to work with multiple similar Docker containers populating EC2 cluster members dynamically in response to some metric - e.g. a CloudWatch alarm from our orchestration layer’s load balancer. As that Elastic Beanstalk application scales in response to demand, so would the number of deployed containers up to the available space of the hosts in the cluster as defined in ECS.
It turns out that this technique has a term - microscaling - and I’ve been researching it recently, including the offering from - ( - see for more details.
Once the number of containers is under dynamic control, the cluster itself could then expand or contract by responding to a separate Auto-Scaling Group rule.


The other issue is load balancing across the cluster. In the scenario I want, the ports on the deployed containers would need to be routed to directly by the load balancer, so that means multiple different ports per host. This is something which would bend the ELB architecture out of shape, I think.

UPDATE: Now that Amazon have released their Application Load Balancer feature, then the load balancing problem is more or less solved on AWS. I must have been one of many people who were after this kind of feature as a senior ECS developer talked to me about it on Twitter over a year ago.

Use case and solution

Our use case is that I want to have an auto-scaling group of EC2 instances in an ECS cluster which host multiple similar containers that will listen on the same internal port, but are routed to via their external port that is assigned by Docker (e.g. while still appearing to be port 80 inside the container). They will be available to be load-balanced across all the containers that are running in the cluster. These large EC2 instances could host multiple different ‘services’ comprised of similar containers.
Essentially, we’d like to have a number of large container hosts and fill them on-demand with web servers or other cookie-cutter services, with load balancing going direct to the containers, across a cluster, across AZs.
I have created a system for performing both microscaling and load-balancing within ECS, and I will show the solution in the next blog post.

Wednesday, 4 November 2015

Twitter - We should talk.

Hi Twitter.

Please, come in and sit down. I think it's time we had a little chat.

I know things haven't been great lately, and I think you're coming undone at the seams a bit.

I'd like to lay a few things out for discussion.

Your position.

You are in a unique position of being a vital, real-time connection between content producers and interested parties, who may also be content producers. You enjoy a position which is (blessedly) outside Facebook and is not a walled garden. You have made amazing technical advances to be able to support this amazing real-time platform. You are, in general, viewed as a force for good. You have friends.

I have witnessed more news, events, conversations and opinions on Twitter than almost any other medium I've used in the past twenty years that I've been on the Internet. It's addictive to be part of this huge, global conversation and I love that.

Recent activity.

But you keep doing these things.


Maybe these were justified because you had over-hired, but it looked reactionary which, if I was one of your investors, would make me quite nervous and/or irritated. Stop jerking your knees. Be more solid.


Experimental. I think it's good to separate curated items like this, which leaves timelines alone. More on that later. Definitely interesting as a thing for surrounding current affairs and getting people interested in them. The "You're caught up" is very Circa-like (RIP Circa). The Trending items have been kind of filling that purpose already, though, but the presentation in Moments is very clean, so I hope that this can build on that kind of attention into something cool. But I'm really not sure that this is the new user grabber that you think it is.

Please don't retire trending or hashtags though. That would be uncool.


The polling functionality is a really powerful thing. You can do a lot of good in this area. Keep up with the good work on this. Be a part of online democracy and live reaction. We like.


Still niche, but an excellent reporting tool. It would be nice to see better integration of that in things like the TweetDeck Chrome extension. Real problems here are bandwidth, video quality, people's data plans, vertical aspect video.


I know Instagram do 15 seconds, but 6 seconds is still cool, right? The integration of this service makes sense because it lends itself to the concise nature of a Tweet. However, the only people I have really seen (in the UK) using Vine are comedians. Think about that for one of those 6 seconds.

Apparent lack of action regarding the tackling of online abuse.

You really, REALLY need to sort this shit out. Provide tools, streamline reporting processes, support victims, unleash rogue AI... Literally anything apart from sitting on your hands about bullying and threatening behaviour.

Favourites vs likes.

Stars vs hearts. Connotations and implications. Organisation vs emotion. I could go on. In fact ...

The whole point of Web 2.0 was the ability to customise your web experience. What happened? How can it not be clear that this change makes it means something totally different? It smacks of flag-waving. You must stop doing these kinds of changes. People (read: your user base) get very used to these semantics and symbols, and you'd be amazed what could drive a person away if they felt aggrieved enough about it. As it is, it's now fairly evident that the people who make these kinds of decisions don't appear to use Twitter or "get it" - the blog post from Akarshan Kumar made it clear that you are going for lowest common denominator, and that's extremely patronising.

The best possible thing you can do here is to bring back the Star icon, and call the collection of things "Starred", because that is completely agnostic to emotional connections. Then this feature can go back to its meaning being implied from context (which includes timing) and also have zero connotations when used as an organisational method. Please do this because what you've done makes no sense to me and many others.


Most of Twitter is noisy. If I step outside of the accounts I follow then I find myself in a horrific sea of chatter. I'm actually cool with that. It supports the notion of 'tuning-in' to certain voices; apart from the Etsy bots - I'm f-ing sick of those. You should be able to spot them and many other types of spammer by now - so why aren't they automatically banned?

Concentration on in-house clients.

At the cost of pissing off every other development team who ever tried to make a 3rd party add on or client for Twitter. It's like you really don't want an ecosystem at all. You are a PLATFORM, a SERVICE, let people MAKE THINGS THAT USE YOUR PLATFORM. If this is about the whole advertising thing - i.e. that you can't control placement of adverts in 3rd party clients, then you're doing things backwards. FIND ANOTHER MODEL (see below). It's not like you're showing adverts in TweetDeck's chrome extension yet anyway (and please don't start now - I'll cry).

Perverse notion of messing with the chronology of timelines.

Twitter's utility is that it shows a chronological view of content. I have absolutely no idea why you think that modifying this is in any way a good thing to do. Even the the out-of-order replies reaction (the "blue line" fiasco) was a UX-disaster with people (like me) complaining about the cognitive dissonance it provoked and being the final nail in the coffin for staying the hell away from the web client. The considered plans for "Tweets-you-may-have-missed" is just barmy. If there are things that you think I've missed - PUT THEM INTO A SEPARATE LIST. Seriously, if you cock up the chronology of Tweets in a timeline then you might as well be a randomly ordered collection of things people have said at some point in the past X hours like Facebook. Just stop that line of thinking.

Consideration of Tweets longer than 140 characters.

No, no, no, no, NO. Twitter is concise. Keep it that way. How am I supposed to catch up on 500+ Tweets that happened overnight if they are a bunch of essays? That's what I have RSS for! If someone wants to post more than 140 characters then they should host content elsewhere which has meta-data on it that allows the previews in clients - just like they do now. CREATE AN ECOSYSTEM, NOT A WALLED GARDEN.

Also - please support RSS. RSS is your friend.

The web client.

I mean, seriously. I'm really glad that we got the Bootstrap project out of Twitter (I use it every day at work), but my God you guys really need to completely re-think your UX. Jesus F-ing Christ, what a mess. If I was approaching that project I would absolutely throw the entire template away and hope that no-one ever mentioned it at parties I went to.

Stock price.

By doubling down on bad ideas to please investors, all it goes to show is that you have collected the wrong investors, with the wrong promises whispered into the wrong ears.

The knock-on effects of promising swathes of new users is removing focus from what makes your platform great.

New users are only valuable if you can present advertising to them. And you've seen the current ad-block fuss.

What you need are customers.

Recommended actions.

Subscription model.

I would absolutely pay $5-10 dollars a month for a subscription to Twitter. That subscription would be ad-free, with a reasonable expectation of timely support for problems with my account, payment, online abuse, and the knowledge that my 3rd party Twitter client could be used without limitation.

I can envisage you running a subscription programme in parallel with your current operations, the only difference being the ad-free nature of the channel, a better SLA for support and the unlocked API access.

The more I think about it, (and I've been thinking about this for a few years), the more strongly I believe that this would be hugely successful for you.

But how much would subscription actually cost you? What change would that have on revenue when a (hopefully significant) percentage of your users are throwing you money each month? What would be the effect upon new user take-up, if they joined knowing that they could upgrade their experience? What would be the effect on API use if it was suddenly opened up to paying users?

I can't help but feel that these changes would be positive.

Content producer payment.

Given that many people join up to be able to follow people with something to say, celebrities, authors, musicians, news outlets, etc, then perhaps that should be rewarded somehow.

There are a few different services which allow users to make payment via Twitter. By taking that in-house this could be like your own version of Patreon, (minus the data leakage), offering content producers payback for being popular.

Again, this is about creating an ecosystem. Reward people for taking part.

Signing off.

Twitter, I don't know if you'll read this. I hope you do and I hope that some of it gets through to you. You are a vital, current thing, and it is in everyone's interest (apart from Facebook's) that you stop these self-harming actions and become something better.

Good day to you, whomever you are.


Monday, 21 September 2015

Fractos' axiom

Fractos' axiom:

There are only three types of software bug:

  1. Namespace issues
  2. Sequence mistakes
  3. Typos
  4. Out by one errors

Thursday, 30 July 2015

Books, mostly


My last set of posts about Using the Model-Store pattern in Inversion took a large heap of my blogging-concentration, but as they've been going since March it's good that they are finally out of the way.

Work's going well and I'm getting my teeth stuck into producing a cloud-based platform for digital library services for digitised collections, services like deep-zoom image serving, discovery, managing metadata and work with the emerging IIIF standard for viewers. Work days, I'm at Wellcome, which is a bit like being on a spaceship, although the food is probably better than NASA serve.

I'm cycling to work which is moderately terrifying. Drivers are terrifying. Cyclists are terrifying. Especially cyclists, frankly, considering some of the whacked-out shit I've seen them do in just a few weeks of riding.

At the moment I'm studying for the Microsoft Programming in C# Exam (70-483) which is on Monday afternoon. It helps out Digirati and definitely doesn't hurt me to go for the certification. However, the study has kinda precluded me making much progress with the stuff I'm reading.

Oh, I bought a Microsoft Band the other day, partly because geek and partly to mitigate an RSI I'm getting in my right index finger. Reaching for my phone in jeans pockets every 30 seconds is really causing a problem, it seems, so the Band picks up the notifications for me. The cycle / run tracking is really good too, although I was tipped over into buying one when I learned that they'd opened up the SDK to allow arbitrary RSS feeds to be added to the tiles.

Go get one! They're really cool.

Currently reading:

Assassin's Apprentice (Farseer Trilogy part 1)
Robin Hobb

Building Microservices
Sam Newmann

(Well, I lie - I just finished it the other day - really bloody good, though.)

And my exam book:
Programming in C# Exam Ref 70-483
Wouter de Kort

Which has got a whole bunch of text inconsistencies (calling things modules, for example), some really bad practice (if you generally mark your classes as sealed then you, whomever you are, are a sociopath), and some answers missing. So that's nice.

Tuesday, 21 July 2015

Using the Model-Store pattern in Inversion (part 5)

This post is the fifth in a series about using a Model-Store pattern in the Inversion behavioural micro-framework for .Net (see:
The first part can be found here.
The second part can be found here.
The third part can be found here.
The fourth part can be found here.


Looking back, I might as well have titled this essay - "I hate ORMs".

Firstly, because I really, really do - I think they are at the root of many evils in .Net software architecture. That's not to say that its the ORMs fault, because it really isn't. However, time after time we are told to "just use" Entity Framework or N-Hibernate because that's what Microsoft or the big boys have blogged about, and time after time I see people getting into the same mess with a half-cocked Repository pattern, absurdly heavyweight models and bloody DTOs springing up all over the place. What I want is a clean, quick, safe and lightweight pattern of data access, and an ORM has never convinced me that it is any of these things because people tend to abuse the functionality it provides without thinking about the consequences or that there might be other ways to approach data access.

Secondly, I didn't actually need to say that this was using Inversion, because Model-Store can be done in any .Net framework. However, it's been useful to write the foundations of the Inversion.Data namespace by considering the paths that writing this article has led me down.

The views in this set of articles almost feel heretical when looking at the current "best practice" of .Net application development. Having an immutable model POCO and a lightweight store (which eschews ORM-like functionality) shifts complexity away from the data access implementation, forcing any business logic back to the application where (in my opinion) it belongs. Data access should be dumb, MUST be dumb, if the application is to be agnostic about the storage technology to be used. This massively increases the platform and language options available for porting and the development of new features. I would counsel avoiding relying upon the features of a particular storage medium, because you never know when that medium could become suddenly unsuitable or unavailable.

Model objects are best designed very closely to the underlying table or document structures. This is for the simplicity of maintenance and the ease of introduction to new development staff. A Model should remain at a level of abstraction such that the properties would be the same across common types of database representation. Therefore, an application that uses this pattern can achieve granular control over the strategies used for reading, writing and updating individual data classes whilst also remaining agnostic to storage technology choices.

When developing the data analytics engine, we tended to write Stores for at least two storage mediums, e.g. MS Sql Server and Redis and/or MongoDB. This forced us into keeping data access straightforward while letting us concentrate on what the application actually needed to accomplish. These same patterns helped us again when writing a cloud-based quiz engine that would operate at scale and the new version of the website application. It forms part of a discipline that assists what I believe to be a fast and clean way of considering modern system architecture.

Immutable Models and CQRS/Event-Sourcing

There are other benefits that immutable Models can bring. If you take the basic tenet that a Model is immutable, then it follows that any change that is required to a Model can be represented as a separate thing. This is the basis of an architecture which gives CQRS-like benefits to an application that uses immutable Models in combination with read-only, optimised Model Stores, with changes persisted in the form of Events to write-capable Stores which could have a completely different storage medium that is optimised for write, thus separating the reads (a Query) from the writes (a Command).

This helps support eventual consistency, because any changes must be persisted explicitly and the source that the read-optimised Store fetches from can be lazily updated by a mechanism that is in the background. With this architecture, concurrent threads are guaranteed to read the same value when referencing an object, and modifications can be made without having to worry about locking or making a local copy.


It is not necessary to use the whole of Model-Store to get these advantages. However, the Store part of the pattern provides an efficient way of packaging data access which can use the immutable Model objects, and shows us that when implementing a DAL in the .Net ecosystem there are other options than Repository patterns and ORMs like Entity Framework or N-Hibernate.

If you haven't agreed with me so far, then perhaps the most compelling thing I can offer is that, most of the time, an application need only read what the values of a Model object are; updates and writes are scarce by comparison. Why weigh your application down with the ability to track changes in an object when you don't use it in most operations?

If you design your application to make changes explicitly then you can take advantage of the benefits of lightweight data access.

I've wanted to document some of my thoughts about this pattern for a while, and I do hope that some of you have stuck through to the end. There are many things that can be pointed to in this set of arguments which other developers may not agree with, and that's fine. As I said, this style of pattern may not be for you. However - I have seen it work in small, medium and production-scale systems, making them fly, whilst also providing a great deal of freedom as a developer.

At the very least - please check out the Inversion framework. Full disclosure being that I'm a major contributor for it and I am using it very happily in the production of a large, automatically scaling cloud-based system for a major institution at the moment, including the bits and pieces you will find in the Inversion.Data namespace.

Dream large.

Sunday, 12 July 2015

Using the Model-Store pattern in Inversion (part 4)

This post is the fourth in a series about using a Model-Store pattern in the Inversion behavioural micro-framework for .Net (see:

The code that I have been developing as part of this article series can be found at

The first part can be found here.
The second part can be found here.
The third part can be found here.

Using the Store

In this instalment, I'm going to show you what using the pattern looks like. I'll put together a few example functions that use the IUserStore interface to access and manipulate data stored in an underlying MongoDB instance.

Configuring and Instantiation

We are going to use the MongoDBUserStore class that I showed you in the previous article to perform operations in MongoDB. To instantiate it, we will need a few details that describe the MongoDB connection-string, the database name and the name of the collection used for users.

I am using a local MongoDB server, which has a database called 'test' and I want to access model objects in a collection called 'users':

Those details can then be passed to the MongoDBUserStore constructor:

But since this is an IDisposable and we only actually care about the interface methods, then more correctly we should be calling it like this:

At which point we are off to the races.

However, the Store could be gained by dependency injection behind an interface, or fetched from a service container (such as Inversion.Naiad, which implements Inversion.Process.IServiceContainer). I'll try and cover the use of IServiceContainer in a separate article, but the call might look something like this:

By using interfaces alone, you can keep your code free from references to the assemblies needed for accessing MongoDB, keeping the main application as agnostic as possible. All the code cares about is the IUserStore interface.

Asking a service container for an object of type IUserStore which has the label "store-user" will cause the container to perform any configuration required according to the instructions that are associated with the service label. The application doesn't have to care about the configuration of the Store because it is handled by the container. This is a form of loose coupling by service location.

The more agnostic your code is about its dependencies, the more flexible your application can be.

Creating a collection of Users in MongoDB

This method assumes that you have a list of User models which are ready to be written to the database. I've included the basic version of instantiating the Store just to keep it simple.

After this call, you should find that the User models have been written into MongoDB, complete with Object IDs.

Read a list of Users from MongoDB

This method will return all the users from MongoDB as a list of User models.

Note that the IEnumerable is converted to an IList with ToList() before the using statement has been terminated (and therefore the Store and its resources disposed) in order to avoid the enumeration code reading a closed connection.

Updating a list of Users

This function iterates through a passed list of User models, modifying their Password property and then writing them back into MongoDB. It makes use of the Mutate method on the User object in order to change an immutable object. The ID of the User model will remain unchanged, however, so the act of writing the data into MongoDB will perform an update rather than an insert.

Deleting a list of Users

This function will delete each of a list of User models from MongoDB.

Next time ...

In the next (and final, so far ...) instalment, I'll wrap up with a conclusion and some thoughts about how this process went.

Thursday, 14 May 2015

Reading progress ...

Finished William Gibson's "The Peripheral" earlier this week after reaching about 80% of the way through it, realising that I was totally absorbed in it and that I didn't want it to end. So, I restarted the book and I am so glad that I did. It's an amazing, lushly detailed work. Very happy that he has returned to sci-fi with what might possibly be his finest work.

There are so many books queued up on my Kindle. I've recently got back into going to the gym where I tend to get a good hour's worth of reading done if I find a free cross-trainer. Now I can start to chew through the pile of purchases I've made over the past six months since I moved into my fiancé's place from my old flat where the gym was just over the road.

First up, I'm hitting the start of the Farseer Trilogy by Robin Hobb. The whole series were on offer a while back and I was a bit click-happy.

Currently reading:

"Assassin's Apprentice (Book One of the Farseer Trilogy)" by Robin Hobb

Saturday, 9 May 2015

Using the Model-Store pattern in Inversion (part 3)

This post is the third in a series about using a Model-Store pattern in the Inversion behavioural micro-framework for .Net (see:

The first part can be found here.
The second part can be found here.


I finished the last post by describing the base Store for dealing with a MongoDB database. Now I will go through the specialised Store that uses it to perform operations with the User class I introduced in part 1.

Reminder - you can find the source code for this article at

Forced specialisation of Models

The first thing I need to cover is that I had to rewrite a whole bunch of stuff in this article after trying really hard to make the examples work with the original Model. This is because of the annoying behaviour in MongoDB of it requiring a property which it deserialises the database object ID into. It's not the only database that enforces this kind of thing (I'm looking at you, DynamoDB), and it's also an unwelcome reminder that sometimes we really are at the mercy of the storage gods.

I've also tried hard to make the examples work with simple serialisation and deserialisation to and from the Model's Builder object, but it just doesn't work. The automatic deserialisation forces you to change the original Model and none of my attempts at wrapping the Model in a way that serves as an adaptor seem to work. Maybe there is a way to wrap it, but the extra plumbing just doesn't feel worth the hassle.

As it is, the User class needs an ID, as the operations performed by the specialised Store become over-complicated without one. I've listed the new source of User.cs, below. Note that the ID property is set to a unique value in the User.Builder parameterless constructor, which means that any newly created User object will have a unique ID already assigned.

In order for the ID to be used correctly with the automatic deserialisation in MongoDB's driver, the type of the ID would have to be changed (to ObjectId) and the property itself marked with a Bson-specific attribute (BsonId). However, I'm not willing to do that as it invalidates the whole idea of having a Model class that is agnostic to the storage technology being used.

So I'm going to show you another way.

Another way around - BsonDocuments

As I mentioned in part two, BsonDocuments are objects that represent a piece of BSON data, which is a binary-serialised version of a JSON document peculiar to MongoDB. The driver is very happy dealing with these constructs as they provide a basic representation of the document. This representation is used internally in the database for binary serialisation. I use the BsonDocument method in all my projects that back onto MongoDB.

Properties are accessible by their name, and they can be any one of the primitive types that MongoDB supports which includes strings, dates, numbers, objects (i.e. another document) and arrays.

So, a way of getting around the forced specialisation of a Model type is to provide custom serialisation/deserialisation to and from a BsonDocument as presented and consumed by the MongoDB driver.

To do this, I use extension methods as a tidy solution for compartmentalisation of the conversion methods.

Using extensions to group conversion methods - MongoDBUserEx.cs

This is the static class that sits in the same namespace as the specialised store MongoDBUserStore. Inside are two extension methods - ConvertUserToBson and ConvertBsonToUser. They pretty much do what they say on the tin. Here's the code.

The ConvertUserToBson method extends an instance of the User class and returns a prepared BsonDocument containing a document representation of the User object's data. The document properties are accessed by name, hence the string indexer on doc. The reserved property name "_id" is used to set the document ID which is set from the User object's ID property.

The "username" and "password" document properties are simply set with the respective string values from the User object.

The "metadata" property, however, is set with a different BsonDocument which is made from the contents of the User object's Metadata property. The BsonDocument constructor can take a IDictionary of values and is quite happy to accept the raw Metadata dictionary which is exposed as IDictionary.

The ConvertBsonToUser method extends an instance of the BsonDocument class and returns a populated User object. Again, the document properties are accessed by name and are used to set the values of a User.Builder object. For the ID, the string value of the underlying ObjectId type is used.

The "metadata" property is retrieved as a BsonDocument, and then its Names property is used as the basis of a ToDictionary call which creates the keys and values of the Builder's Metadata object.

The act of returning the User.Builder object implicitly converts it into an immutable User object.

Interface for the User Store - IUserStore.cs

Here is the interface that I will be implementing for the specialised User Store.

The interface is defined as implementing the IStore interface, so certain methods will be expected to be implemented. However, these should be satisfied by the ancestor of our specialised Store - StoreBase.cs.

There is a Get method that accepts a username to search by, returning an immutable User object. Then there is a GetAll method that should return all the items in the database as an IEnumerable of User objects. I reckon the interface could probably do with a Get method that takes a user's ID value as a parameter, but feel free to do that yourselves (one of them will have to be renamed as they both accept a lone string parameter).

Then the Put method, taking an individual User object which will be entered into the database. I'm assuming that Put also serves as an Update operation in this implementation.

Finally there is a Delete operation that accepts a User object as its target to remove from the database.

Specialised Store for our User model - MongoDBUserStore.cs

Now we have the interface and those conversion tools available, we can proceed with the specialised Store for the User model.

I should point out that this Store flagrantly subverts the new, asynchronous capabilities of the MongoDB driver. This is because not all storage technologies will support async/await style calls and one of the main goals of this pattern is to allow any storage technology to be used behind the same interface. Having some implementations use async/await while others do not is therefore a really bad idea because it forms a leaky abstraction - you need to know from outside the component whether it can be called asynchronously. Further, if an async/await call to an IDisposable object is incorrectly executed, then resources could end up being disposed before an asynchronous call has completed. So, I believe that if you are dealing with async/await calls then you should define an explicit interface for them to make it clear to the developer calling the interface that they will be dealing with the side-effects themselves.

    public class MongoDBUserStore : MongoDBStore, IUserStore

The class signature shows that we are inheriting from the basic MongoDBStore class, and we will be exhibiting the behaviour specified in the IUserStore interface.

Constructor and properties


The constructor takes the information needed by the base Store - connStr and dbName, and passes them into the base class constructor. It also takes a collectionName argument which is stored in a private, read-only property. It is assumed that this specialised Store for the User object will only be dealing with a single MongoDB Collection.



Knowing the name of the collection up front allows us to perform this operation during the Start method, after having called the base implementation to change the Store state. This step retrieves an IMongoCollection object with which we can perform queries and commands upon the named collection.



Now we are starting to implement the IUserStore interface with the first method - Get. The first thing to do is to assert whether the Store is in the correct state to perform operations, i.e. whether Start has been called. The AssertIsStarted method is called which you may remember is implemented on the grandparent Store class.

The query itself is performed by calling the Find method of the IMongoCollection we initialised in the Start method. It is passed a BsonDocument that contains the information to evaluate appropriate documents in the collection. In this case, we set the "username" property in the document to the value that we are looking for. Only the first result is needed (or expected), and so the FirstAsync method is called on the result of the Find operation. Because we are wrapping asynchronous behaviour in this implementation, we will wait for the result of the FirstAsync method by accessing it's Result property. Finally, the Result property (which will be of type BsonDocument to match the generic type of the IMongoCollection) has the extension method ConvertBsonToUser called upon it, that will instantiate an immutable User object that contains the values from the document.

That's a lot of behaviour wrapped up in one line. Fluent interfaces like this which libraries like System.Linq allow are fantastic to use (when you know what you are doing), but unwinding the meaning of them by way of explanation can be a time-consuming business.



The next part of the IUserStore interface to be implemented is GetAll. This simply returns the entire contents of the collection. However, to do that we need to make a query that will return all items, then turn that list of results into an IEnumerable of the correct User objects. More fluency in the calling methods for this gives us those results.

Special note should be taken that the query used is represented by an empty BsonDocument passed to the FindAsync method - no matching criteria means that it matches against all the items in the collection. The result of the Find operation must be transformed into a list and that is performed by the ToListAsync method. Again, we must wait for the result and this gives us a List which then gives us access to the Select method that we can use to call the ConvertBsonToUser method upon each item finally resulting in an IEnumerable that is returned back to the caller.

Put (single User object)


We use the MongoDB driver's Replace method to effectively perform an Upsert operation on the database, creating or replacing an existing entry which has the same ID.



And finally the Delete method, giving us all the CRUD operations (provided you don't mind if the Create and Update are represented by the Put method). Again, we use the BsonDocument method of passing the matching criteria into the driver.

Next time ...

In the next part of the series, I'll show you how to use the Store in practice.

Thursday, 30 April 2015

Using the Model-Store pattern in Inversion (part 2)

This post is the second in a series about using a Model-Store pattern in the Inversion behavioural micro-framework for .Net (see:

The first part can be found here.


In the previous post in this series, I talked about the Model part of the Model-Store pattern; basically POCO (plain old C# objects) with a little extra infrastructure added to give them immutability (i.e. if a value changes then they become a different object with a different reference) and also decoration with Inversion's IData interface, which lets the object be more useful within the Inversion ecosystem.

In this post, I'm going to talk about the IStore interface and how it can be implemented to be a lightweight, efficient data store that uses immutable models.

The code for this post is not contained in the standard Inversion framework repository. Instead, I have created a new repository that contains the Inversion.Data namespace here:


A Store provides data access for a specific type of storage medium. This could be a database, a filesystem, or anything that you can perform basic CRUD operations upon.

It is generally tied to a particular type of Model, or a limited set of Models that could be described as being within a Bounded Context. This is in order to avoid duplication of access to the Model objects in storage. If a set of Models exist within a Bounded Context then they are not accessed by the rest of a system and so therefore access duplication should not occur. In other situations it is probably best to define a Store that handles as few Model types as possible so that the granularity of configuration could be on a per-Model basis.

In the example from part one of this series, a Store that is implemented for User objects would also cover UserMetadata objects since the UserMetadata class could be a private class within the User class's domain.

Tying a Store to a Model type has connotations. It is advised that the most suitable way of modelling with this pattern is to use simple objects that do not maintain a web of active references to other types of object and which do not have the expectation of these references to be kept up to date automatically by the actions of a DAL. This won't suit everyone's architectural styles.

The Model should only represent the data. The Store should only deal with the translation of that data in and out of a storage medium.

Now I think it's worth talking briefly about what a Store is not.

A Store is NOT a Repository

To reiterate, a Store uses immutable model objects. A Repository keeps track of changes, but a Store does not because immutable objects become a different object if their properties are changed. Changes in the underlying storage medium must be made explicitly by putting a data object back into the Store. Model objects do not have to track changes automatically so they can be primitive classes with no dependency upon libraries such as Entity Framework.

The IStore Interface

This is the basic interface that all Store classes must implement. It covers the following two areas of concern, those being that a Store is a resource and that a Store has state.

Stores are a resource

A Store is something that an application should be able to dip into; a lightweight component that is used only for data access. Once the data access is over, you should be able to throw away the store.

For this reason a Store implements the IDisposable interface. A Store may not have anything to dispose at the end of its use, but this enforces that there is the chance to perform actions at that point.

Stores have state

A Store should guard against misuse. Certain operations may only be valid when the Store is in a particular state. A primary use-case for this is being able to adjust the properties of a connection in a database store before the connection has been fully opened. The general solution for this is to implement a state property that defines whether the Store has been started or stopped explicitly.

For this reason, a Store implements methods that adjust its state and any data access methods should check the current state when they are called, raising an exception if the Store is in an unsuitable state.

The Store state is represented by a simple enumeration - StoreState:

A Store which has never been started will have a state equal to StoreState.Unstarted. When started explicitly, a Store will have a state equal to StoreState.Started. When stopped explicitly, a Store will have a state equal to StoreState.Stopped.

The IStore interface defines two methods for explicitly changing the state of the Store - Start and Stop:

    void Start();
    void Stop();

The IStore interface defines a public get property HasStarted, which should report whether the Store has had Start called upon it and has not yet had Stop called upon it:

    bool HasStarted { get; }

The IStore interface also defines another public get property HasStopped, which should report whether the Store has had Stop called upon it:

    bool HasStopped { get; }

The StoreBase abstract base class

This abstract class provides a base upon which to build other implementations of a Store, as we will see below.

The private property _state keeps track of whether the Store has ever been started, or is currently started or stopped. The initial value is StoreState.Unstarted. Only methods in the base class can adjust the _state property since it is private. Overriding methods for Start (and Stop) must call their base implementations as part of their instructions.

The public property HasStarted can report whether the Store is in a state where it can perform operations, i.e. whether the _state property is equal to StoreState.Started.

The public property HasStopped can report whether the Store is in a stopped state, i.e. whether the _state property is equal to StoreState.Stopped.

The public method Start is used to explicitly start any resources that the Store implementation needs. If the Store is already started, then an exception is raised.

The public method Stop is used to explicitly stop any resources that the Store implementation needs. This should be called by a descendant implementation of Dispose, and since both Stop or Dispose could be called explicitly (by a call from user code) or implicitly (when leaving a using statement block), then the Stop method does not try to raise an exception if the Store is in an incompatible state. Instead, it changes the state to Stopped if the Store is currently started. Generally, it is useful to keep a flag that records whether Dispose has been called already, and if the flag is set then no action is taken. This protects against the situation of a call to Stop being followed by an implicit call to Dispose. However, this may not be necessary if a Store has nothing to do during its disposal phase.

It is the descendant implementation's responsibility to have the correct behaviour within its overridden instructions for Stop and Dispose, and this is especially important for database connection types which must have their connection objects disposed (e.g. SqlClient). For this reason, Dispose is marked as abstract so that the descendant is forced to provide some form of implementation.

I have also added a simple method for asserting whether the Store has been started called AssertIsStarted. This will be put to a lot of use in our specialised Store.

StoreStartedException is a simple wrapper for ApplicationException. It is used to emit an exception from the Store when trying to start it if it is already in a started state:

Similarly there is a StoreStoppedException which should be emitted from a data access method when the Store is found to be in a stopped state.

Also, StoreProcessException - an instance or descendant of which should be emitted from a Store method if something goes wrong during processing a call, although that's not a hard and fast rule, just a useful convention.

I've omitted the source code for these as they are exactly the same as StoreStartedException apart from the name, but you can find it in the GitHub repository for Inversion.Data. The differentiation between them is just an exercise in taxonomy.

Now we are ready to see how a Store can be implemented for a particular storage technology, and then show how it can be used.

Implementing a Store for MongoDB

You probably already know this, but MongoDB by 10Gen is a cross-platform, "NoSQL" style document database which is used by many projects and applications across the globe. I use it extensively for most of my side-projects as a primary storage mechanism. My experience with it has led me to appreciate a horses-for-courses approach to databases - MongoDB can be a very quick leg-up to a working system and in production it is very stable and well-behaved, clustering notwithstanding. If you're looking to get into document databases then it's a really good starting place.

It's pretty straightforward to set up on Windows or Linux. Check out its home site and also the very good admin UI Windows application MongoVue at

A word about namespaces

The projects and assemblies that I am writing as part of this tutorial are arranged in a particular way. Essentially - Store, IStore, StoreState and the Exception classes are sitting in a core assembly called Inversion.Data. But my specialised classes, i.e. the Store and support classes that I am about to make live in a project called Inversion.Data.MongoDB. This has a default namespace of Inversion.Data and a Store folder within it, which means that by referencing the assembly Inversion.Data.MongoDB the Inversion.Data namespace is augmented with the contents of the root of the project, in this case that means the MongoDBStore class. The Store has the technology name used as a prefix to its class name to differentiate it within the namespace.

This convention is in place because we (Guy Murphy and I) have found that it aids a composition-by-augmentation approach; if you include an assembly with a particular technology stack, then it will appear within a well known namespace. No special knowledge of namespaces that are tied to a technology is required and, frankly, it just feels neater.

The base MongoDB implementation - MongoDBStore.cs

To use MongoDB, I start by adding the NuGet package "MongoDB.Driver". That package loads a couple of dependencies - MongoDB.Bson and MongoDB.Core.

The code for the MongoDBStore class is as follows:

This gives us some very basic functionality that can be used to build our specialised Store for a particular Model type.

The constructor for MongoDBStore requires the passing of both a connection string argument and a database name. I've made the base store centric around the named database in MongoDB because this allows basic functionality with collections that would be common across Stores that might want to access more than one collection within a database. Frankly, it's as good a place as any to put this level of organisation. If you feel that you want to change that around then go for it. The connection string and database name are stored in read-only properties which are private as they are unlikely to be of use to anything outside of the base implementation. Connection strings and database names are things that shouldn't leak outside of the Store's abstraction as they are a configuration concern.

Start has been overridden, taking the opportunity to create a MongoClient instance from the connection string and then use that client to gain an IMongoDatabase reference to the named database. The base Start method is called in order to correctly set the state of the Store. The result is that we now have an active MongoClient instance and a stored reference to the required database that is ready to go.

Stop has not been overridden as there is no action to take when stopping.

Similarly, Dispose has no actual work to do so it simply calls Stop on the base Store.

This basic Store has its index property overridden which allows access by name to a MongoDB collection within a database. The get-accessor performs a quick state check of the Store to determine if it has been started (and thus, whether it is safe to use the Database property), and then returns an IMongoCollection generic of type BsonDocument (objects that represent a piece of BSON data, which is a binary-serialised version of JSON peculiar to MongoDB).

Some OO-purists might want to mark this base Store as abstract, such that it cannot itself be instantiated and must be inherited from before it can perform any work. However, this is a fully working Store providing a basic mechanism for retrieving documents from a named collection, therefore I have not marked it as abstract as it could be put to immediate use as-is.

Next time ...

In the next part of this article, I will go through the specialised Store for our User object - MongoDBUserStore.cs.

Had to change name of Store to StoreBase to avoid a namespace conflict. Gists updated to reflect this.

Sunday, 19 April 2015

Shh... Write!

"Shh... Write!"

Thus speaks a voice the narrator of The Case Of Charles Dexter Ward does not recognise, in a room that he knows only one person resides within.

And so, I too write.

Working on the rest of the series of articles on using the Model-Store pattern in Inversion (link to part 1 here). It's taking some time to research and accurately voice the piece on Stores, hence the delay in posting. However, the second piece has now been split into a third as I really was trying to put across far too much in one go. Separating out some of the opinion has really helped.

I had an interesting week, pivoting around going to the AWS Summit at London ExCel on Wednesday. Pretty sure I spread the love of my Anker USB battery quite well that day too. I've got one of the 15,000 mAh versions and am totally happy with it. I use it as a base charger for my phones and it comes away on weekend trips and heavy-use days. Amazon currently sell the 16,000 mAh version which seems to come with a few extra features. Pretty good price too:

Currently reading:

Seven Databases in Seven Weeks
- A Guide to Modern Databases and the NoSQL Movement
Eric Redmond and Jim R. Wilson

Wednesday, 1 April 2015

Mirror people

When travelling around, something I have noticed more and more frequently is that people having mobile phone conversations do not behave like other people do. This appears to be especially true when the old-as-time situation of meeting someone walking in the other direction and having to decide which side to favour so you both can pass.

My experience is that people on phones will favour the same way as you, mirrored somehow, which causes a replay and reassessment as their attention is dragged back from the other side of the phone call.

If your body language is there you aren't here.

It's something about projection, I think. Perhaps this is something to do with how we relate to the person we are calling? Like a mirror has been placed in front of us - we relate our own self-image to the voice on the line.

Currently reading:

Monday, 23 March 2015

Using the Model-Store pattern in Inversion (part 1)

This post is the first in a series about using a Model-Store pattern in the Inversion behavioural micro-framework for .Net (see:

Model-Store Pattern

This is a general pattern than involves representing domain data as immutable models, and allowing the segregation of reading and writing of data in a manner that enables CQRS-like behaviour and optimisation in an application stack.

The key components of the pattern are domain model objects that exhibit an internal builder class that is used to mutate an existing object or create a new one (for this pattern's purposes, these are effectively the same thing), and a generalised store interface that is friendly towards service location, service injection and segregation. The store interface also enforces separation of concerns, enabling the storage methods and caching strategies to be transparent to the application.


I despair when joining a project and discovering that they are three foot deep in AutoMapper entrails and everything has a DTO equivalent for talking to a Web Service somewhere, or some kind of perverse partial class used as a View Model. Why create something that can't be used everywhere? Why can't we all just get along and respect each other as who we are?

The day I started to keep my data objects free of extraneous crap was a good day.

Just a POCO ...

Assuming that we have a User entity in our domain, our POCO for this might look like the following:

Taking control of changes

However, this can be changed by anybody, so in order to put that under control, we can make the properties read-only and only settable from the object's constructor.

This uses the ability to set a public property to only have a private setter, thereby cutting down on some code real-estate and making sure code external to the object instance cannot modify the value of a property.


However, the next step would be to enforce this policy for code internal to the object instance as well.

Extra constructor added that takes its values from an existing User object. This is mostly for completeness.

This is now more like a truly immutable data object. The instance is guaranteed by the CLR to have unmodified fields after initialisation.


When working with objects, at some point we will need to represent a change in value. This is known as mutation and it entails a little bit of extra internal structure to support it. The advantage of enforcing this really shows when dealing with multi-threaded applications, where a thread can guarantee that an object pointed to by specific reference will not change during the lifetime of that instance. Any change to the object creates a new instance. Using immutable objects in multi-threaded applications can lead to entire swathes of management code and concerns falling away.

Two components are needed to support mutation. First, an internal Builder class, and second, a Mutate method.

The internal Builder class

The Builder class is contained within the domain object class, as below. It contains publicly modifiable versions of the properties belonging to the immutable outer class.

It represents a mutable version of the data. This can be used and accessed by external code to prepare new or changed data. 

The ToModel method puts the Builder object's values into a new instance of a User object which has immutable properties.

The FromModel method puts a User object's values into a Builder object which has mutable properties.

The Mutate method

This method is responsible for facilitating a change to an immutable object by accepting a change that is represented by an anonymous method, and returning a new instance of the object that contains the changed data.

It creates a new instance of this model's Builder class, initialised with the current immutable instance's data. Then, it applies the passed anonymous method, or mutator, to the builder.

The mutator looks something like this:

The result of the mutation is returned as a User object, and normally this would cause a casting exception as Mutate is defined as returning a User object, but the addition of the following two methods allow us to override the implicit casting behaviour of the immutable User class and the mutable Builder class in a granular fashion:

Therefore, when the Mutate method returns a Builder object as a User, the implicit operator for User that takes a Builder as an input instead returns the result of the Builder's ToModel method - i.e. the immutable version of the Builder's data.

Similarly, when some code casts from a User object to a Builder object, then the implicit operator for Builder will be called, returning the result of new Builder(model), where model is the User object being cast.

Default values for fields

To provide fields with default values, they should be applied to the Builder object, as instantiating the immutable object requires either an existing User object or a full set of values for the properties.

One way of doing this is to change the parameterless constructor in the Builder class so that it contains the initialisation of properties with default values.

Another way would be to provide a private backing-field for a property that is being given a default value in the Builder object:

Dictionaries and Lists

There are existing implementations of immutable classes of Dictionary and List that may be used in the immutable outer class, but they require a little bit of initialisation and translation during construction and conversion. 

Suppose we want to add a Metadata collection to our User object which will be represented by a string,string dictionary. To do this, we specify an ImmutableDictionary in the outer class:

And the public, read-only property:

The two constructors of the outer class need some modification. The first constructor needs to be passed a dictionary of metadata to instantiate its immutable data from:

The second constructor is passed a complete User object, so the signature doesn't change but the body will have to take its metadata from there:

In the Builder class, the metadata is represented by just a normal dictionary:

And that will need initialising in the parameterless constructor:

The other constructors and methods adjust to accommodate:

For ImmutableList, the pattern is just the same.

Also, for the record, I hate using var, but the alternative didn't format very well. Weak excuse, I know.

Immutables, Immutables, Immutables, all the way down

Now that we have the User object with its immutability and Builder class, supposing we implemented its Metadata property as another immutable class instead of just an ImmutableDictionary.

Importantly, the Builder's Metadata property is of type UserMetadata.Builder. That type choice plus the implicit cast overrides (assuming there is one in the UserMetadata class as well) allows a chaining of the mutable versions of the objects, which means that within a Mutate method called on the User object, the mutable version of the Metadata is in scope and will be converted back into an immutable version when the result of Mutate is cast back to a User (internally it will chain the calls to ToModel).

Implementing the IData interface

Finally, in order to make this generally useful within the Inversion ecosystem, we make the object implement the Inversion.IData interface:

Implementing this interface allows Inversion to include the object in a ViewStep model which can be processed by the render pipeline. It's not a hard requirement for an object used in an Inversion application to implement IData, but it does enable the object to be represented easily by XML/Json in a way that easily interfaces with the framework.

The casting to IData in the Data method makes it explicitly clear to the compiler that we want the implementation of the IData's Data method within the Metadata object instance.

The full versions of the User and UserMetadata classes can be found here:

Next part ... Stores

Coming soon.
Update: Fixed the source code for the Mutate function.

Wednesday, 18 March 2015

Bending the world so log4net can work


Firstly, why the Hell would you change the public key of a strongly named assembly between versions?


Yes, I've read and that just reeks of the development team having some kind of argument with people outside of their anointed circle of coders.

Anyway, there's a requirement on some old CMS code in a project I'm working on, and the DLLs have got a reference to log4net 1.2.10 while the rest of the world is staring down the barrel of 1.2.13.

Incidentally, I think it's really bad form to put numbers into assembly names, and, further, to have version 1.2.13 to be a completely different NuGet version (2.0.3). Seriously, what the fuck?

I found this post which really helped - you can reference two different versions if they use a strong name as long as you stick them in different folders, whilst keeping a reference in the project to the later one but setting Copy Local to false. I also added a pre-build event on the website which would make the folders if they didn't exist and copy the appropriate DLLs into each.

However, the finishing touch was that since this was an endpoint - i.e. a website where lots of DLLs are bundled up together - then it needed to have the right path; in this case, it needed "bin\" prepended to it:

Currently reading:

Tuesday, 17 February 2015

Running unmanaged code in a managed C# .Net website

Encountered this problem in the Fusion Log of a .Net website that uses a wrapper for some unmanaged C++ code:

LOG: Assembly download was successful. Attempting setup of file: C:\dev\...\bin\kdu_a75R.dll
LOG: Entering download cache setup phase.
ERR: Error extracting manifest import from file (hr = 0x80131018).
ERR: Setup failed with hr = 0x80131018.
ERR: Failed to complete setup of assembly (hr = 0x80131018). Probing terminated.

After a great deal of looking at Dependency Walker traces, countless forum articles and advice, it turned out to be the simple fact that a .Net website will look at every .DLL file it can find in the Bin folder of the website and try to load a .Net assembly manifest from it, even if it isn't a .Net DLL. Because reasons.

The solution to this is as obvious as it is absurd:

  • Don't put unmanaged DLL files in a .Net website Bin folder.
  • Put them in Windows' System32 folder where IIS will be able to find them.
That's just terrifying, but it does work. It makes me taste a little bit of sick in my mouth, frankly.

Sunday, 1 February 2015

Promises, promises ... (Testing non-AngularJS websites with Selenium and Protractor)

A post about promises, web-testing and far too much indentation.

This post covers the tests defined in

Recently, I have been writing browser tests for a Javascript application. The tests were invoked via Protractor running under Node.js locally on a Windows laptop. The Automate service from BrowserStack was used for cross-browser capability checking, and to be honest I can't recommend that service enough. It's *very* clever and straightforward.

The tests use Cucumber to enable BDD-style spec statements for features and scenarios. This differs from the usual uses of Protractor tests which generally use Jasmine - a library that represents test scenarios programmatically. A lack of documentation surrounding the use of Cucumber in Protractor has been a major headache during the project.

The tests as received originally used Selenium's webdriverjs to drive the browser session. Before the previous developer left, we converted many of the test features over to using a Page Object which offers an abstracted set of functions that allow access to elements on the page whilst hiding the details of how the elements are found; the concerns of the test functionality are therefore separated from the concerns of how to locate an element.

Further, since we are using Protractor to test a website that does not use AngularJS, the synchronisation that Protractor will do under the hood is not available. This means that communication with Selenium webdriver must be done using a Promise - a callback-like way of scheduling functionality to occur after a call into a sub-system with handling for success and failure of operations. Although ostensibly a means to cut-down on code, using Promises has the same effect as callbacks in that *everything* becomes a callback and you can end up very far across your screen for even minor functionality.

A Page Object doesn't have to be very smart. All it has to do is hide the implementation of locating elements for the tests that call it.

I generalised three methods - find, findAll and sleep:
Almost all the abstracted functions for accessing elements in the Page Object use these methods. This was done primarily so that I could centralise the logging for what was going on. Here are some example element abstractions:
This makes the code that uses the Page Object a lot clearer, as it can refer to an abstraction of the element(s) it is considering instead of dealing with calls to the webdriver which are now hidden away.

So far, so good. The tests work really well in Chrome and Firefox. Problems occur, as ever, when Internet Explorer gets involved. The general advice you should heed when testing with Internet Explorer is to NOT TOUCH ANYTHING while the test is running. IE is effectively an embedded component of Windows and it really doesn't let you forget it. Changing focus to another window could be enough for it to lose its place and invalidate your test run. Further, when the thing being tested uses an HTML IFrame, IE will lose its place after EVERY SINGLE TIME you try to find anything in the DOM. This led to a complete re-write of the browser tests in order to maintain cross-browser compatibility whilst also trying to be robust enough for IE's behaviour.

The test rig HTML IFrame we use for the viewer is the only one on the page. Even with that fact, the following code is necessary to reset which frame Internet Explorer is considering before it attempts to locate an element:
This is utilised by the test code itself before it performs a locate operation:
As you can see from that gist, an element locator function from the Page Object returns something which offers a 'then' function - this is the implementation of the Promise functionality. The 'then' function can take two arguments; the first is a function that will be called if the operation was a success, and the (optional) other will be called in the event of a failure. In this case, there are two functions defined that handle failed operations - one for if the element cannot be located, and the other if the element's 'click' function fails in some way. The success route through the callbacks is indicated by the main flow from left to right - success in locating the element flows to a click which then flows to success in clicking followed by executing the Protractor/Cucumber callback that will announce a pass for this part of the testing scenario.

If I decide to do a refactoring job on this again, I would probably wrap the resetFrame call into the find/findAll functionality. This would mean passing the behaviour needed by the test into a callback function and passing it into the element locator function, frankly a bit like how the calls to resetFrame work now, but transparent to the feature files.

Okay, that's all for now. I wanted to record how to make these tests cross-browser compatible with the use of resetting the frame - this is just in case someone out there finds it useful in the future ... (HI, FUTURE ME!)