Using the Model-Store pattern in Inversion (part 2)

This post is the second in a series about using a Model-Store pattern in the Inversion behavioural micro-framework for .Net (see: https://github.com/guy-murphy/inversion-dev).

The first part can be found here.

Introduction

In the previous post in this series, I talked about the Model part of the Model-Store pattern; basically POCO (plain old C# objects) with a little extra infrastructure added to give them immutability (i.e. if a value changes then they become a different object with a different reference) and also decoration with Inversion's IData interface, which lets the object be more useful within the Inversion ecosystem.

In this post, I'm going to talk about the IStore interface and how it can be implemented to be a lightweight, efficient data store that uses immutable models.

The code for this post is not contained in the standard Inversion framework repository. Instead, I have created a new repository that contains the Inversion.Data namespace here: https://github.com/fractos/inversion-data

Stores

A Store provides data access for a specific type of storage medium. This could be a database, a filesystem, or anything that you can perform basic CRUD operations upon.

It is generally tied to a particular type of Model, or a limited set of Models that could be described as being within a Bounded Context. This is in order to avoid duplication of access to the Model objects in storage. If a set of Models exist within a Bounded Context then they are not accessed by the rest of a system and so therefore access duplication should not occur. In other situations it is probably best to define a Store that handles as few Model types as possible so that the granularity of configuration could be on a per-Model basis.

In the example from part one of this series, a Store that is implemented for User objects would also cover UserMetadata objects since the UserMetadata class could be a private class within the User class's domain.

Tying a Store to a Model type has connotations. It is advised that the most suitable way of modelling with this pattern is to use simple objects that do not maintain a web of active references to other types of object and which do not have the expectation of these references to be kept up to date automatically by the actions of a DAL. This won't suit everyone's architectural styles.

The Model should only represent the data. The Store should only deal with the translation of that data in and out of a storage medium.

Now I think it's worth talking briefly about what a Store is not.

A Store is NOT a Repository

To reiterate, a Store uses immutable model objects. A Repository keeps track of changes, but a Store does not because immutable objects become a different object if their properties are changed. Changes in the underlying storage medium must be made explicitly by putting a data object back into the Store. Model objects do not have to track changes automatically so they can be primitive classes with no dependency upon libraries such as Entity Framework.

The IStore Interface

This is the basic interface that all Store classes must implement. It covers the following two areas of concern, those being that a Store is a resource and that a Store has state.

Stores are a resource

A Store is something that an application should be able to dip into; a lightweight component that is used only for data access. Once the data access is over, you should be able to throw away the store.

For this reason a Store implements the IDisposable interface. A Store may not have anything to dispose at the end of its use, but this enforces that there is the chance to perform actions at that point.

Stores have state

A Store should guard against misuse. Certain operations may only be valid when the Store is in a particular state. A primary use-case for this is being able to adjust the properties of a connection in a database store before the connection has been fully opened. The general solution for this is to implement a state property that defines whether the Store has been started or stopped explicitly.

For this reason, a Store implements methods that adjust its state and any data access methods should check the current state when they are called, raising an exception if the Store is in an unsuitable state.

The Store state is represented by a simple enumeration - StoreState:




A Store which has never been started will have a state equal to StoreState.Unstarted. When started explicitly, a Store will have a state equal to StoreState.Started. When stopped explicitly, a Store will have a state equal to StoreState.Stopped.

The IStore interface defines two methods for explicitly changing the state of the Store - Start and Stop:

    void Start();
    void Stop();

The IStore interface defines a public get property HasStarted, which should report whether the Store has had Start called upon it and has not yet had Stop called upon it:

    bool HasStarted { get; }

The IStore interface also defines another public get property HasStopped, which should report whether the Store has had Stop called upon it:

    bool HasStopped { get; }

The StoreBase abstract base class

This abstract class provides a base upon which to build other implementations of a Store, as we will see below.

The private property _state keeps track of whether the Store has ever been started, or is currently started or stopped. The initial value is StoreState.Unstarted. Only methods in the base class can adjust the _state property since it is private. Overriding methods for Start (and Stop) must call their base implementations as part of their instructions.

The public property HasStarted can report whether the Store is in a state where it can perform operations, i.e. whether the _state property is equal to StoreState.Started.

The public property HasStopped can report whether the Store is in a stopped state, i.e. whether the _state property is equal to StoreState.Stopped.

The public method Start is used to explicitly start any resources that the Store implementation needs. If the Store is already started, then an exception is raised.

The public method Stop is used to explicitly stop any resources that the Store implementation needs. This should be called by a descendant implementation of Dispose, and since both Stop or Dispose could be called explicitly (by a call from user code) or implicitly (when leaving a using statement block), then the Stop method does not try to raise an exception if the Store is in an incompatible state. Instead, it changes the state to Stopped if the Store is currently started. Generally, it is useful to keep a flag that records whether Dispose has been called already, and if the flag is set then no action is taken. This protects against the situation of a call to Stop being followed by an implicit call to Dispose. However, this may not be necessary if a Store has nothing to do during its disposal phase.

It is the descendant implementation's responsibility to have the correct behaviour within its overridden instructions for Stop and Dispose, and this is especially important for database connection types which must have their connection objects disposed (e.g. SqlClient). For this reason, Dispose is marked as abstract so that the descendant is forced to provide some form of implementation.

I have also added a simple method for asserting whether the Store has been started called AssertIsStarted. This will be put to a lot of use in our specialised Store.

StoreStartedException is a simple wrapper for ApplicationException. It is used to emit an exception from the Store when trying to start it if it is already in a started state:



Similarly there is a StoreStoppedException which should be emitted from a data access method when the Store is found to be in a stopped state.

Also, StoreProcessException - an instance or descendant of which should be emitted from a Store method if something goes wrong during processing a call, although that's not a hard and fast rule, just a useful convention.

I've omitted the source code for these as they are exactly the same as StoreStartedException apart from the name, but you can find it in the GitHub repository for Inversion.Data. The differentiation between them is just an exercise in taxonomy.

Now we are ready to see how a Store can be implemented for a particular storage technology, and then show how it can be used.

Implementing a Store for MongoDB

You probably already know this, but MongoDB by 10Gen is a cross-platform, "NoSQL" style document database which is used by many projects and applications across the globe. I use it extensively for most of my side-projects as a primary storage mechanism. My experience with it has led me to appreciate a horses-for-courses approach to databases - MongoDB can be a very quick leg-up to a working system and in production it is very stable and well-behaved, clustering notwithstanding. If you're looking to get into document databases then it's a really good starting place.

It's pretty straightforward to set up on Windows or Linux. Check out its home site https://www.mongodb.org/ and also the very good admin UI Windows application MongoVue at http://www.mongovue.com/

A word about namespaces

The projects and assemblies that I am writing as part of this tutorial are arranged in a particular way. Essentially - Store, IStore, StoreState and the Exception classes are sitting in a core assembly called Inversion.Data. But my specialised classes, i.e. the Store and support classes that I am about to make live in a project called Inversion.Data.MongoDB. This has a default namespace of Inversion.Data and a Store folder within it, which means that by referencing the assembly Inversion.Data.MongoDB the Inversion.Data namespace is augmented with the contents of the root of the project, in this case that means the MongoDBStore class. The Store has the technology name used as a prefix to its class name to differentiate it within the namespace.

This convention is in place because we (Guy Murphy and I) have found that it aids a composition-by-augmentation approach; if you include an assembly with a particular technology stack, then it will appear within a well known namespace. No special knowledge of namespaces that are tied to a technology is required and, frankly, it just feels neater.

The base MongoDB implementation - MongoDBStore.cs

To use MongoDB, I start by adding the NuGet package "MongoDB.Driver". That package loads a couple of dependencies - MongoDB.Bson and MongoDB.Core.

The code for the MongoDBStore class is as follows:



This gives us some very basic functionality that can be used to build our specialised Store for a particular Model type.

The constructor for MongoDBStore requires the passing of both a connection string argument and a database name. I've made the base store centric around the named database in MongoDB because this allows basic functionality with collections that would be common across Stores that might want to access more than one collection within a database. Frankly, it's as good a place as any to put this level of organisation. If you feel that you want to change that around then go for it. The connection string and database name are stored in read-only properties which are private as they are unlikely to be of use to anything outside of the base implementation. Connection strings and database names are things that shouldn't leak outside of the Store's abstraction as they are a configuration concern.

Start has been overridden, taking the opportunity to create a MongoClient instance from the connection string and then use that client to gain an IMongoDatabase reference to the named database. The base Start method is called in order to correctly set the state of the Store. The result is that we now have an active MongoClient instance and a stored reference to the required database that is ready to go.

Stop has not been overridden as there is no action to take when stopping.

Similarly, Dispose has no actual work to do so it simply calls Stop on the base Store.

This basic Store has its index property overridden which allows access by name to a MongoDB collection within a database. The get-accessor performs a quick state check of the Store to determine if it has been started (and thus, whether it is safe to use the Database property), and then returns an IMongoCollection generic of type BsonDocument (objects that represent a piece of BSON data, which is a binary-serialised version of JSON peculiar to MongoDB).

Some OO-purists might want to mark this base Store as abstract, such that it cannot itself be instantiated and must be inherited from before it can perform any work. However, this is a fully working Store providing a basic mechanism for retrieving documents from a named collection, therefore I have not marked it as abstract as it could be put to immediate use as-is.

Next time ...

In the next part of this article, I will go through the specialised Store for our User object - MongoDBUserStore.cs.

Update:
Had to change name of Store to StoreBase to avoid a namespace conflict. Gists updated to reflect this.

Shh... Write!

"Shh... Write!"

Thus speaks a voice the narrator of The Case Of Charles Dexter Ward does not recognise, in a room that he knows only one person resides within.

And so, I too write.

Working on the rest of the series of articles on using the Model-Store pattern in Inversion (link to part 1 here). It's taking some time to research and accurately voice the piece on Stores, hence the delay in posting. However, the second piece has now been split into a third as I really was trying to put across far too much in one go. Separating out some of the opinion has really helped.

I had an interesting week, pivoting around going to the AWS Summit at London ExCel on Wednesday. Pretty sure I spread the love of my Anker USB battery quite well that day too. I've got one of the 15,000 mAh versions and am totally happy with it. I use it as a base charger for my phones and it comes away on weekend trips and heavy-use days. Amazon currently sell the 16,000 mAh version which seems to come with a few extra features. Pretty good price too:



Currently reading:

Seven Databases in Seven Weeks
- A Guide to Modern Databases and the NoSQL Movement
Eric Redmond and Jim R. Wilson

Mirror people

When travelling around, something I have noticed more and more frequently is that people having mobile phone conversations do not behave like other people do. This appears to be especially true when the old-as-time situation of meeting someone walking in the other direction and having to decide which side to favour so you both can pass.

My experience is that people on phones will favour the same way as you, mirrored somehow, which causes a replay and reassessment as their attention is dragged back from the other side of the phone call.

If your body language is there you aren't here.

It's something about projection, I think. Perhaps this is something to do with how we relate to the person we are calling? Like a mirror has been placed in front of us - we relate our own self-image to the voice on the line.

Currently reading:

Using the Model-Store pattern in Inversion (part 1)

This post is the first in a series about using a Model-Store pattern in the Inversion behavioural micro-framework for .Net (see: https://github.com/guy-murphy/inversion-dev).

Model-Store Pattern

This is a general pattern than involves representing domain data as immutable models, and allowing the segregation of reading and writing of data in a manner that enables CQRS-like behaviour and optimisation in an application stack.

The key components of the pattern are domain model objects that exhibit an internal builder class that is used to mutate an existing object or create a new one (for this pattern's purposes, these are effectively the same thing), and a generalised store interface that is friendly towards service location, service injection and segregation. The store interface also enforces separation of concerns, enabling the storage methods and caching strategies to be transparent to the application.

Models

I despair when joining a project and discovering that they are three foot deep in AutoMapper entrails and everything has a DTO equivalent for talking to a Web Service somewhere, or some kind of perverse partial class used as a View Model. Why create something that can't be used everywhere? Why can't we all just get along and respect each other as who we are?

The day I started to keep my data objects free of extraneous crap was a good day.

Just a POCO ...

Assuming that we have a User entity in our domain, our POCO for this might look like the following:

Taking control of changes

However, this can be changed by anybody, so in order to put that under control, we can make the properties read-only and only settable from the object's constructor.

This uses the ability to set a public property to only have a private setter, thereby cutting down on some code real-estate and making sure code external to the object instance cannot modify the value of a property.

Immutability

However, the next step would be to enforce this policy for code internal to the object instance as well.

Extra constructor added that takes its values from an existing User object. This is mostly for completeness.

This is now more like a truly immutable data object. The instance is guaranteed by the CLR to have unmodified fields after initialisation.

Mutation

When working with objects, at some point we will need to represent a change in value. This is known as mutation and it entails a little bit of extra internal structure to support it. The advantage of enforcing this really shows when dealing with multi-threaded applications, where a thread can guarantee that an object pointed to by specific reference will not change during the lifetime of that instance. Any change to the object creates a new instance. Using immutable objects in multi-threaded applications can lead to entire swathes of management code and concerns falling away.

Two components are needed to support mutation. First, an internal Builder class, and second, a Mutate method.

The internal Builder class

The Builder class is contained within the domain object class, as below. It contains publicly modifiable versions of the properties belonging to the immutable outer class.

It represents a mutable version of the data. This can be used and accessed by external code to prepare new or changed data. 

The ToModel method puts the Builder object's values into a new instance of a User object which has immutable properties.

The FromModel method puts a User object's values into a Builder object which has mutable properties.

The Mutate method

This method is responsible for facilitating a change to an immutable object by accepting a change that is represented by an anonymous method, and returning a new instance of the object that contains the changed data.

It creates a new instance of this model's Builder class, initialised with the current immutable instance's data. Then, it applies the passed anonymous method, or mutator, to the builder.

The mutator looks something like this:

The result of the mutation is returned as a User object, and normally this would cause a casting exception as Mutate is defined as returning a User object, but the addition of the following two methods allow us to override the implicit casting behaviour of the immutable User class and the mutable Builder class in a granular fashion:

Therefore, when the Mutate method returns a Builder object as a User, the implicit operator for User that takes a Builder as an input instead returns the result of the Builder's ToModel method - i.e. the immutable version of the Builder's data.

Similarly, when some code casts from a User object to a Builder object, then the implicit operator for Builder will be called, returning the result of new Builder(model), where model is the User object being cast.

Default values for fields

To provide fields with default values, they should be applied to the Builder object, as instantiating the immutable object requires either an existing User object or a full set of values for the properties.

One way of doing this is to change the parameterless constructor in the Builder class so that it contains the initialisation of properties with default values.

Another way would be to provide a private backing-field for a property that is being given a default value in the Builder object:

Dictionaries and Lists

There are existing implementations of immutable classes of Dictionary and List that may be used in the immutable outer class, but they require a little bit of initialisation and translation during construction and conversion. 

Suppose we want to add a Metadata collection to our User object which will be represented by a string,string dictionary. To do this, we specify an ImmutableDictionary in the outer class:

And the public, read-only property:

The two constructors of the outer class need some modification. The first constructor needs to be passed a dictionary of metadata to instantiate its immutable data from:

The second constructor is passed a complete User object, so the signature doesn't change but the body will have to take its metadata from there:

In the Builder class, the metadata is represented by just a normal dictionary:

And that will need initialising in the parameterless constructor:

The other constructors and methods adjust to accommodate:

For ImmutableList, the pattern is just the same.

Also, for the record, I hate using var, but the alternative didn't format very well. Weak excuse, I know.

Immutables, Immutables, Immutables, all the way down

Now that we have the User object with its immutability and Builder class, supposing we implemented its Metadata property as another immutable class instead of just an ImmutableDictionary.

Importantly, the Builder's Metadata property is of type UserMetadata.Builder. That type choice plus the implicit cast overrides (assuming there is one in the UserMetadata class as well) allows a chaining of the mutable versions of the objects, which means that within a Mutate method called on the User object, the mutable version of the Metadata is in scope and will be converted back into an immutable version when the result of Mutate is cast back to a User (internally it will chain the calls to ToModel).

Implementing the IData interface

Finally, in order to make this generally useful within the Inversion ecosystem, we make the object implement the Inversion.IData interface:

Implementing this interface allows Inversion to include the object in a ViewStep model which can be processed by the render pipeline. It's not a hard requirement for an object used in an Inversion application to implement IData, but it does enable the object to be represented easily by XML/Json in a way that easily interfaces with the framework.

The casting to IData in the Data method makes it explicitly clear to the compiler that we want the implementation of the IData's Data method within the Metadata object instance.

The full versions of the User and UserMetadata classes can be found here:

https://gist.github.com/fractos/6c7f53dd88aeb4c00b5f

Next part ... Stores

Coming soon.

Update: Fixed the source code for the Mutate function.

Bending the world so log4net can work

log4net.

Firstly, why the Hell would you change the public key of a strongly named assembly between versions?

Secondly, WHY THE HELL WOULD YOU CHANGE THE PUBLIC KEY? Jesus.

Yes, I've read http://logging.apache.org/log4net/release/faq.html#two-snks and that just reeks of the development team having some kind of argument with people outside of their anointed circle of coders.

Anyway, there's a requirement on some old CMS code in a project I'm working on, and the DLLs have got a reference to log4net 1.2.10 while the rest of the world is staring down the barrel of 1.2.13.

Incidentally, I think it's really bad form to put numbers into assembly names, and, further, to have version 1.2.13 to be a completely different NuGet version (2.0.3). Seriously, what the fuck?

I found this post which really helped - you can reference two different versions if they use a strong name as long as you stick them in different folders, whilst keeping a reference in the project to the later one but setting Copy Local to false. I also added a pre-build event on the website which would make the folders if they didn't exist and copy the appropriate DLLs into each.

http://stackoverflow.com/questions/3158928/referencing-2-differents-versions-of-log4net-in-the-same-solution

However, the finishing touch was that since this was an endpoint - i.e. a website where lots of DLLs are bundled up together - then it needed to have the right path; in this case, it needed "bin\" prepended to it:


Currently reading:

Running unmanaged code in a managed C# .Net website

Encountered this problem in the Fusion Log of a .Net website that uses a wrapper for some unmanaged C++ code:

LOG: Assembly download was successful. Attempting setup of file: C:\dev\...\bin\kdu_a75R.dll
LOG: Entering download cache setup phase.
ERR: Error extracting manifest import from file (hr = 0x80131018).
ERR: Setup failed with hr = 0x80131018.
ERR: Failed to complete setup of assembly (hr = 0x80131018). Probing terminated.

After a great deal of looking at Dependency Walker traces, countless forum articles and advice, it turned out to be the simple fact that a .Net website will look at every .DLL file it can find in the Bin folder of the website and try to load a .Net assembly manifest from it, even if it isn't a .Net DLL. Because reasons.

The solution to this is as obvious as it is absurd:

• Don't put unmanaged DLL files in a .Net website Bin folder.
• Put them in Windows' System32 folder where IIS will be able to find them.

That's just terrifying, but it does work. It makes me taste a little bit of sick in my mouth, frankly.

Promises, promises ... (Testing non-AngularJS websites with Selenium and Protractor)

A post about promises, web-testing and far too much indentation.

This post covers the tests defined in https://github.com/britishlibrary/UniversalViewerTests

Recently, I have been writing browser tests for a Javascript application. The tests were invoked via Protractor running under Node.js locally on a Windows laptop. The Automate service from BrowserStack was used for cross-browser capability checking, and to be honest I can't recommend that service enough. It's *very* clever and straightforward.

The tests use Cucumber to enable BDD-style spec statements for features and scenarios. This differs from the usual uses of Protractor tests which generally use Jasmine - a library that represents test scenarios programmatically. A lack of documentation surrounding the use of Cucumber in Protractor has been a major headache during the project.

The tests as received originally used Selenium's webdriverjs to drive the browser session. Before the previous developer left, we converted many of the test features over to using a Page Object which offers an abstracted set of functions that allow access to elements on the page whilst hiding the details of how the elements are found; the concerns of the test functionality are therefore separated from the concerns of how to locate an element.

Further, since we are using Protractor to test a website that does not use AngularJS, the synchronisation that Protractor will do under the hood is not available. This means that communication with Selenium webdriver must be done using a Promise - a callback-like way of scheduling functionality to occur after a call into a sub-system with handling for success and failure of operations. Although ostensibly a means to cut-down on code, using Promises has the same effect as callbacks in that *everything* becomes a callback and you can end up very far across your screen for even minor functionality.

A Page Object doesn't have to be very smart. All it has to do is hide the implementation of locating elements for the tests that call it.

I generalised three methods - find, findAll and sleep:
Almost all the abstracted functions for accessing elements in the Page Object use these methods. This was done primarily so that I could centralise the logging for what was going on. Here are some example element abstractions:
This makes the code that uses the Page Object a lot clearer, as it can refer to an abstraction of the element(s) it is considering instead of dealing with calls to the webdriver which are now hidden away.

So far, so good. The tests work really well in Chrome and Firefox. Problems occur, as ever, when Internet Explorer gets involved. The general advice you should heed when testing with Internet Explorer is to NOT TOUCH ANYTHING while the test is running. IE is effectively an embedded component of Windows and it really doesn't let you forget it. Changing focus to another window could be enough for it to lose its place and invalidate your test run. Further, when the thing being tested uses an HTML IFrame, IE will lose its place after EVERY SINGLE TIME you try to find anything in the DOM. This led to a complete re-write of the browser tests in order to maintain cross-browser compatibility whilst also trying to be robust enough for IE's behaviour.

The test rig HTML IFrame we use for the viewer is the only one on the page. Even with that fact, the following code is necessary to reset which frame Internet Explorer is considering before it attempts to locate an element:
This is utilised by the test code itself before it performs a locate operation:
As you can see from that gist, an element locator function from the Page Object returns something which offers a 'then' function - this is the implementation of the Promise functionality. The 'then' function can take two arguments; the first is a function that will be called if the operation was a success, and the (optional) other will be called in the event of a failure. In this case, there are two functions defined that handle failed operations - one for if the element cannot be located, and the other if the element's 'click' function fails in some way. The success route through the callbacks is indicated by the main flow from left to right - success in locating the element flows to a click which then flows to success in clicking followed by executing the Protractor/Cucumber callback that will announce a pass for this part of the testing scenario.

If I decide to do a refactoring job on this again, I would probably wrap the resetFrame call into the find/findAll functionality. This would mean passing the behaviour needed by the test into a callback function and passing it into the element locator function, frankly a bit like how the calls to resetFrame work now, but transparent to the feature files.

Okay, that's all for now. I wanted to record how to make these tests cross-browser compatible with the use of resetting the frame - this is just in case someone out there finds it useful in the future ... (HI, FUTURE ME!)