Porting Inversion part 2

Things have progressed and I think it’s time to write up and highlight a few items.

Choices of what to port

There are some things which I have chosen to abandon for now. These include my Inversion.Web.Razor extensions in inversion-razor, and the ConfigurationHelper and Pipeline models for service containers in Inversion.Extensibility.

In the case of the Razor extensions, I think I will need to rewrite this entirely in light of the AspNetCore support for Razor. I’d much rather be integrated with that than remain off-piste with Antaris RazorEngine.

Regarding ConfigurationHelper and Pipeline interfaces, the configuration options are so starkly different in .Net Standard, and so much easier to deal with, that there’s no point having a shim to pull configuration from a database etc when the new methods are so much more accessible.

Publishing to NuGet

I’m not sure which sociopath designed the web UI for NuGet, but they need their head examining; they’re clearly insane. It’s not quite as bad as the Visual Studio interface though, at least.

Having fought through the various stuff it puts in your way to prevent you getting a clean package uploaded, the various repos - inversion-dev, inversion-data, inversion-data and inversion-messaging - have been made available. The assemblies that make up those repos are available separately in order to keep project tech hierarchies segregated, e.g. Inversion.Data base library is separate to Inversion.Data.AmazonSQS which is separate to Inversion.Data.Redis etc. Don’t cross the streams unless you have to.

The versions are in the 1.0.x range and the references of child packages to their parents are currently set to 1.0.* as I was pretty shocked how stupid the management of minor versions was in the NuGet CLI. Anyway, working now.

GitHub organisation

The main libraries are now under the newly formed inversion-org GitHub organisation - https://github.com/inversion-org

Guy, Rob and myself are owners.

Here you will find:

• Inversion.Data
• Inversion.Extensibity
• Inversion.Messaging

The main Inversion library hasn’t moved yet but this will be its home in the future. You can still find it here:

• Inversion

Other libraries, such as Inversion.Ultrastructure will also be moved into the organisation shortly.

Adding Travis automation

Some Travis automation has been added to build and publish the NuGet packages, which it does sort of blindly as it doesn’t check first if the version already exists and so seeks forgiveness rather than permission when it fails to upload. I suppose there might be ways to automate incrementing the patch number on the .csproj file, but at the moment it is manual.

Things to do next:

• add unit tests (with automation for commit status updates)
• test the libraries work!
• create basic application that uses the libraries via NuGet and can be deployed as a container

Regarding Dependency Injection

A colleague of mine recently posted on an internal Slack channel about this blog article: How not to do dependency injection - the static or singleton container.

Reading the article, I became aware of something that a friend of mine had written about, namely the absolute vitriol that many coding blogs and generally good books seem to have against Service Location. For example, I’m reading Adaptive Code via C# at the moment and in that Gary McLean Hall has a small meltdown about it, which is a shame because it’s a pretty good book for people to learn about refactoring and use of basic patterns. However it also commits the cardinal sin of saying that the ‘D’ in SOLID is for Dependency Injection and not Dependency Inversion, and that betrays the architectural bias of the author.

The upshot being:

• Anyone who tells you service location is an anti-pattern isn’t fully aware of the problems that it is supposed to be an answer for.
• Dependency Injection moves away from the original point - separating the configuration of services from their use.
• Dependency Injection vastly increases the surface area of an object via abuse of the constructor which isn’t bound by interface contract, thereby avoiding abstracting dependencies fully and leading to constructors being the medium by which relationships between objects are communicated - (this is not a virtue)
• By making assumptions about state, Dependency Injection turns architectural uses-a relationships into has-a - which blocks the use of singletons and a bunch of other architectural patterns where they would be appropriate.
• Further, it means that although relationships between data entities are modelled, behavioural relationships between objects are not because they become one great big dependency ball.

Guy Murphy - Service Locator vs Dependency Injection

Another in-depth article Guy wrote about Inversion of Control can be found here:

Guy Murphy - I come not to bury IoC, but to praise it

Canary farm

"Canary farm"

An IT system connected to various other brittle systems that invariably picks up the blame for them being unavailable.

Fractos' axiom

Fractos' axiom:

There are only three types of software bug:

1. Namespace issues
2. Sequence mistakes
3. Typos
4. Out by one errors

Bookmarks for 20140912

Some bookmarks as I haven't done this for ages and I need to test some automation.

My friend @PullMyDaisy 's new KickStarter, "Glass Senses"
https://www.kickstarter.com/projects/1337820983/glass-senses

Great Sci-Fi short story by Tobias Buckell - "System Reset"
http://io9.com/read-this-intense-pre-apocalyptic-hacker-story-by-tobi-1542763261

The art of Nick Gentry, which I am a particular fan of, especially the more abstract floppy disk pieces
http://www.nickgentry.com

A piece I found on using database tables as queues which is just useful to read
http://rusanu.com/2010/03/26/using-tables-as-queues/

That Manchester Evening News donation drive for Manchester Dogs Home just hit £857,000. Stunning.

Don't fear the Bulk Copy

While searching for a way to optimise parts of a particular system today, I had managed to get a write to SQL Server (2005, non-local) down to about 0.5 seconds for 121 rows. Not great, but I was prepared to believe that it was working hard as it is a fat table with a dodgy looking schema, with the less said the better about the network in-between.

My boss pointed me at this page: http://msdn.microsoft.com/en-us/library/1y8tb169.aspx which has details on the SqlBulkCopy object in the System.Data library.

To be honest, I was a bit cagey about this. My belief has always been that there is a lower limit of rows before the pros of using a bulk copy operation outweigh the cons. Perhaps that is true, but certainly for a mere 121 rows, it is not an issue at all.

Previously, I had implemented this insert operation using a variety of methods:

• Injection via an XML parameter, processed by use of @xml.nodes style queries. 
• Dynamically generated SQL, creating a large INSERT statement with multiple rows using SELECT and UNION ALL. 
• Individual parameterised INSERT statements. 

Of these, the XML method had the worst performance; SQL Server may have tools to navigate and process XML, but they are definitely not quick and this was to be expected. This process is *marginally* quicker than doing individual insert statements, but only after the number of rows has increased past, say, 50.

The dynamically generated query looked a bit like this:

INSERT INTO [dbo].[tblData] ( [UserId], [CreateDate], [Value] )
SELECT 1001, '2013-03-04 00:14:30.00', 25
UNION ALL SELECT 1023, '2013-03-04 00:14:30.15', 67
UNION ALL SELECT 1038, '2013-03-04 00:14:30.32', 21

Examining the execution plan for this yielded that it spent most of its time doing a clustered index insert; about 98% of its time to be exact. Although performance for this increased between runs (eventually down to 21 milliseconds for the writing of 121 rows), there was still room for improvement. Time for 1000 users in a Parallel.ForEach loop of the entire operation (which included this write) was 00:02:30

It was hoped that the use of parameterised INSERT statements would allow SQL Server to cache an execution plan and use it. In tests, it does perform faster. Time for 1000 users, as above, was 00:01:45

So... on to an implementation using SqlBulkCopy.

If you are throwing arbitrary data at it then this entails a little bit of set up. The WriteToServer method accepts a DataTable object and this must be tailored exactly to the schema for the bulk copy to work.

(Please note that this is an extremely cut down / Noddy version of the table for brevity.)

public override void Put(int userID, List; myData)
{
    DataTable dt = new DataTable();
    dt.Columns.Add(new DataColumn("DataId", typeof (System.Int32)));
    dt.Columns.Add(new DataColumn("UserId", typeof(System.Int32)));
    dt.Columns.Add(new DataColumn("CreateDate",typeof(System.DateTime)));
    dt.Columns.Add(new DataColumn("Value",typeof(System.Int32)));

After this, you add the data, row by row into the table. If you have a nullable field, then test for HasValue and use either the Value or enter DBNull.Value for the assignment.

foreach (MyData row in myData) {
    DataRow dr = dt.NewRow();
    dr["DataId"] = DBNull.Value;
    dr["UserId"] = row.UserId;
    dr["CreateDate"] = row.CreateDate;
    dr["Value"] = row.Value;
    dt.Rows.Add(dr);
}

Now we set up the SqlBulkCopy object. I've added two option flags for it - KeepNulls and KeepIdentity. KeepNulls so it will honour the DBNull.Value encountered on some fields, and KeepIdentity so that it leaves the destination table in control of the assignment of row identity. I have included the row ID in the DataTable's columns, set it to DBNull.Value in the rows themselves, but I shall now make sure that it is removed from the column mappings by clearing them and re-adding the columns I require.


CORRECTION:
The KeepIdentity flag does NOT do that. This code only works because I do not include the identity column in the ColumnMappings collection.

using (
    SqlBulkCopy bulkCopy = new SqlBulkCopy(
        _connectionString,
        SqlBulkCopyOptions.KeepNulls | SqlBulkCopyOptions.KeepIdentity))
    {

        bulkCopy.DestinationTableName = "dbo.tblData";
        bulkCopy.ColumnMappings.Clear();
        bulkCopy.ColumnMappings.Add("UserId", "UserId");
        bulkCopy.ColumnMappings.Add("CreateDate", "CreateDate");
        bulkCopy.ColumnMappings.Add("Value", "Value");

Then I can perform the write.

        try
        {
            bulkCopy.WriteToServer(dt);
        }
        catch (Exception)
        {
            throw;
        }
    }
}

The performance of the write, at 121 rows, was 0.025 seconds, instead of 0.5 seconds. Time for 1000 users, in the parallel test mentioned above, was 00:00:45.

This technique is *lightning* fast and totally worth doing for much smaller numbers of rows than I originally thought.

Don't fear the Bulk Copy.


EDIT:
It should be noted that the default behaviour of the WriteToServer command is that the following apply:

• Table constraints will not be enforced
• Insert triggers will not fire
• The operation will use Row locks

This behaviour can be tailored using the SqlBulkCopyOptions enumeration, as detailed here: http://msdn.microsoft.com/en-gb/library/system.data.sqlclient.sqlbulkcopyoptions.aspx

GiveCamp project continuing...

So I'm having a cheap Friday night by staying in and doing some more coding on the project that was started at #GiveCampUK back in October. We've pretty much decided on the database schema now which has meant that I can spend time setting up the entities and support libraries for the application.

Originally, this was going to be an application written using Microsoft Lightswitch, a simple query screen development platform sitting on top of Entity Framework 4. However this was canned due to limitations in Lightswitch preventing us writing queries that would perform well. It's a nice idea and environment (once you've got your head around it), but it's profoundly limiting with respect to what it allows you to do with the queries it generates. So, as I said this was canned after it became clear that it wasn't going to play nicely.

Therefore, I find myself writing a .Net WinForms application. But I've hit a problem which may see this just go back to being a website.

I've put a lot of effort into the model and store patterns used, into the caching that will be available to the application, into the Spring.Net implementation which keeps the whole thing easy to configure, into the stored procedures performing basic data access to the entity tables; but it's no good:

The problem is that this will be used by more than one person, at the same time.

Therefore, if something changes in the database, every connected application must be kept up to date.

This causes me a massive, massive problem for consistency. But I don't know if I am worrying about this unduly? After all, the people using this aren't going to be doing break-neck speed data entry, they aren't going to be removing large swathes of the database, they're probably going to be doing one thing each, and they're probably going to be in the same room. So, how much do I worry about it?

What I have currently is a watchdog system. A scheduled background task in the application calls 'home' to the database to determine if there has been a change since the last time it checked / a change occurred. This notifies the master cache that it should reload its data again. The application needs to make sure that this behaviour does not interfere with what the user is doing too much, and I don't know at this point whether it will do that.

I'm going to try and press in with this as a simple WinForms application, but it is very tempting to just make it an ASP.Net website which means that all the locking can be performed locally and safely.

Maybe I should've gone to the pub after all...

Deadlines schmedlines

Ok, so we missed a deployment window today. It was for a Spring-based job processor that will support the website by handling fire and forget jobs. Luckily the only fallout will be the twiddling of thumbs tomorrow and a few people inconvenienced. It was close, but the prospect of releasing such a thing without it being fully tested and proven on staging was too much for both our team lead and myself to bear. However, through the course of the afternoon we put the system through a whole bunch of failure scenarios which should mean that Monday's deployment will go better in the light of the knowledge gained and the configs puzzled over.