Migration to Blogger

The process of moving my Dirty Fire Project blog to Blogger has been, frankly, hair-pullingly annoying.

However, I would like to save for posterity how I managed to get it working.

Importing the articles proved impossible as I couldn't give it an RSS feed and let it do the rest, which was a shame. The utility of that would be immense. So, I ended up copy pasting work in and then stripping the formatting (after realising that it had kept the background colour of the text the first time I did it). It did give me a chance to go through every article and add edits, fix links and so on (desperately trying to find silver cloud here).

The main problem with migration was the use of a custom domain, or rather, persuading Blogger to accept that I had authority for my existing domain.

What it boils down to is that to make it work I have the following zone entries:

(my verify key) CNAME (my verify token).googlehosted.com.
www CNAME ghs.google.com.

Where (my verify key) is the CNAME key and (my verify token) is the CNAME domain they want you to add in the domain authentication settings.

The most critical part of this that mattered was the inclusion of the full stops after the two CNAME values.

Basically, the first page of instructions, the one on the Settings->Basic->Publishing screen, and critically the one you see when you enter information which hasn't worked (giving the infamous Error 12 / Error 13 messages) - has the wrong information. It advises putting the address in without the full stop.

If I was more clear about how to put DNS records together then I probably would have spotted this sooner. As it is, it took until the next morning when I stumbled across a different page - the one that is linked as "settings instructions" and tailors its information based on what registrar your domain uses - before I noticed that it included a trailing full stop in the domain verification value.

Once that was set up, it all went quite well. The "missing files host" facility works really well, so I can point that back at my old server and it picks up any files that are not present on the new www host.

The site looks a lot better. And I don't have to run a really contrived set of operations to add an article any more. Though, in all fairness, I did *write* that contrived set of operations. My bad :)

EDIT:
I've isolated what the confusion is. The set of instructions that appear when you first type in your custom domain, where the operation fails with an Error 12/13 - THOSE INSTRUCTIONS ARE WRONG. Then, the linked page called "settings instructions" claims that you need a full stop after the name of the verify token (which is correct), but doesn't include one after the "ghs.google.com" address - THOSE INSTRUCTIONS ARE THEREFORE WRONG.
The only page with the correct information on is the "webmaster's verification tool" page which you have to click around to find. The CNAME method verification page in that area has the full stops added to the back of both the www / ghs.google.com and verify token addresses.

Don't fear the Bulk Copy

While searching for a way to optimise parts of a particular system today, I had managed to get a write to SQL Server (2005, non-local) down to about 0.5 seconds for 121 rows. Not great, but I was prepared to believe that it was working hard as it is a fat table with a dodgy looking schema, with the less said the better about the network in-between.

My boss pointed me at this page: http://msdn.microsoft.com/en-us/library/1y8tb169.aspx which has details on the SqlBulkCopy object in the System.Data library.

To be honest, I was a bit cagey about this. My belief has always been that there is a lower limit of rows before the pros of using a bulk copy operation outweigh the cons. Perhaps that is true, but certainly for a mere 121 rows, it is not an issue at all.

Previously, I had implemented this insert operation using a variety of methods:

• Injection via an XML parameter, processed by use of @xml.nodes style queries. 
• Dynamically generated SQL, creating a large INSERT statement with multiple rows using SELECT and UNION ALL. 
• Individual parameterised INSERT statements. 

Of these, the XML method had the worst performance; SQL Server may have tools to navigate and process XML, but they are definitely not quick and this was to be expected. This process is *marginally* quicker than doing individual insert statements, but only after the number of rows has increased past, say, 50.

The dynamically generated query looked a bit like this:

INSERT INTO [dbo].[tblData] ( [UserId], [CreateDate], [Value] )
SELECT 1001, '2013-03-04 00:14:30.00', 25
UNION ALL SELECT 1023, '2013-03-04 00:14:30.15', 67
UNION ALL SELECT 1038, '2013-03-04 00:14:30.32', 21

Examining the execution plan for this yielded that it spent most of its time doing a clustered index insert; about 98% of its time to be exact. Although performance for this increased between runs (eventually down to 21 milliseconds for the writing of 121 rows), there was still room for improvement. Time for 1000 users in a Parallel.ForEach loop of the entire operation (which included this write) was 00:02:30

It was hoped that the use of parameterised INSERT statements would allow SQL Server to cache an execution plan and use it. In tests, it does perform faster. Time for 1000 users, as above, was 00:01:45

So... on to an implementation using SqlBulkCopy.

If you are throwing arbitrary data at it then this entails a little bit of set up. The WriteToServer method accepts a DataTable object and this must be tailored exactly to the schema for the bulk copy to work.

(Please note that this is an extremely cut down / Noddy version of the table for brevity.)

public override void Put(int userID, List; myData)
{
    DataTable dt = new DataTable();
    dt.Columns.Add(new DataColumn("DataId", typeof (System.Int32)));
    dt.Columns.Add(new DataColumn("UserId", typeof(System.Int32)));
    dt.Columns.Add(new DataColumn("CreateDate",typeof(System.DateTime)));
    dt.Columns.Add(new DataColumn("Value",typeof(System.Int32)));

After this, you add the data, row by row into the table. If you have a nullable field, then test for HasValue and use either the Value or enter DBNull.Value for the assignment.

foreach (MyData row in myData) {
    DataRow dr = dt.NewRow();
    dr["DataId"] = DBNull.Value;
    dr["UserId"] = row.UserId;
    dr["CreateDate"] = row.CreateDate;
    dr["Value"] = row.Value;
    dt.Rows.Add(dr);
}

Now we set up the SqlBulkCopy object. I've added two option flags for it - KeepNulls and KeepIdentity. KeepNulls so it will honour the DBNull.Value encountered on some fields, and KeepIdentity so that it leaves the destination table in control of the assignment of row identity. I have included the row ID in the DataTable's columns, set it to DBNull.Value in the rows themselves, but I shall now make sure that it is removed from the column mappings by clearing them and re-adding the columns I require.


CORRECTION:
The KeepIdentity flag does NOT do that. This code only works because I do not include the identity column in the ColumnMappings collection.

using (
    SqlBulkCopy bulkCopy = new SqlBulkCopy(
        _connectionString,
        SqlBulkCopyOptions.KeepNulls | SqlBulkCopyOptions.KeepIdentity))
    {

        bulkCopy.DestinationTableName = "dbo.tblData";
        bulkCopy.ColumnMappings.Clear();
        bulkCopy.ColumnMappings.Add("UserId", "UserId");
        bulkCopy.ColumnMappings.Add("CreateDate", "CreateDate");
        bulkCopy.ColumnMappings.Add("Value", "Value");

Then I can perform the write.

        try
        {
            bulkCopy.WriteToServer(dt);
        }
        catch (Exception)
        {
            throw;
        }
    }
}

The performance of the write, at 121 rows, was 0.025 seconds, instead of 0.5 seconds. Time for 1000 users, in the parallel test mentioned above, was 00:00:45.

This technique is *lightning* fast and totally worth doing for much smaller numbers of rows than I originally thought.

Don't fear the Bulk Copy.


EDIT:
It should be noted that the default behaviour of the WriteToServer command is that the following apply:

• Table constraints will not be enforced
• Insert triggers will not fire
• The operation will use Row locks

This behaviour can be tailored using the SqlBulkCopyOptions enumeration, as detailed here: http://msdn.microsoft.com/en-gb/library/system.data.sqlclient.sqlbulkcopyoptions.aspx

Bookmarks for 20130305

DesignModo's Flat-UI, a free UI kit based on Bootstrap, JQuery:

EDIT: This is now defunct. GitHub got hit with a DMCA takedown on the software as the guy's previous employer deigned it to be their property.

http://designmodo.github.com/Flat-UI/

TIL that SqlBulkCopy is really awesome:

http://msdn.microsoft.com/en-us/library/1y8tb169.aspx

Bookmarks for 20130304

.Net context shell items

http://www.codeproject.com/Articles/512956/NET-Shell-Extensions-Shell-Context-Menus

Ils has a new album out

http://www.distinctiverecords.com/disn216.html

Comet PANSTARRS visible to Northern Hemisphere from March 7th with naked eye

http://www.wired.com/wiredscience/2013/03/comet-panstarrs/?cid=co6167324

Wine and DNS

Wake up. Blink. Rehydrate. Discover emailed receipt for two domains purchased sometime on Friday night.

Gulp.

I'd better finish this project then!

Thai Fish Curry

Thai Fish Curry with Noodles

serves 4

2 tbsp vegetable oil

1 large onion, chopped

2-3 garlic cloves, chopped

handful of white mushrooms

350g white/yellow fish fillet (cod or haddock), cut into cubes

350g salmon fillet, cut into cubes

2 tbsp Thai red curry paste

400g / 1 can coconut milk

handful of fresh coriander, chopped

1 tsp brown sugar

1 tsp fish sauce

2-3 spring onions, chopped

2oz beansprouts

3-4 basil leaves (pref Thai basil), finely chopped

method

1. Heat the oil in a wok and gently sauté the onion, garlic and mushrooms until softened but not browned.
   
2. Add the fish, curry paste and coconut milk and bring gently to the boil. Simmer for 2-3 minutes before adding half the coriander, the sugar and the fish sauce. Keep warm.
   
3. Meanwhile, soak the noodles for 3-4 minutes or until tender, and drain well through a colander. Put the colander and noodles over a pan of simmering water. Add the spring onions, beansprouts and most of the basil and steam on top of the noodles for 1-2 minutes or until just wilted.
   
4. Pile the noodles into a serving bowl and to pwith the fish curry. Sprinkle the remaining coriander and basil over the top and serve immediately.

Taken from "Wok & Stir-Fry" published by Parragon Books Ltd and with a few alterations by me.

Stone Code Sober (also, don't use ConcurrentBag in Mono)

Ugh. Saturday night, sans alcohol, post-Prometheus. Sobriety on a weekend evening can only mean one thing... staying up till fucking 4am and writing code.

As it happens, I did have something to try and achieve. Namely, getting some prototype Spring.Net-based code working in Mono, running under XSP, running on an Ubuntu box.

The code itself is based on some of the same ideas as a current piece of work - a custom http handler and framework that is built for speed and simplicity. At the heart of it is a message bus with attached listeners waiting for commands and an application context to be delivered. The listeners act on the context and it ends up holding the output state ready for the response to be marshalled. It is quite straightforward and /very/ fast, utilising singleton objects from Spring.Net.

Now, I have an Ubuntu box (shard, a little ultra-small-form-factor Dell, bless it) and I've already been through the pain in the arse of getting Spring.Net working under Mono. Well, just Spring.Core really, but that's the only bit I'm interested in so fuck it, whatever. I'm using Mono 2.8 which gives me .Net 4 functionality. If you're interested in getting Mono/XSP working on Ubuntu or Fedora Linux then I highly recommend this page which has a comprehensive install script. I did have to do a little bit of make check; make install afterwards to bring my MonoDevelop stuff up to date, but aside from that it went smoothly (although it took about 45 minutes on shard!).

So, I have .Net 4 capability, a working Spring.Net/Common.Logging build, and that script gave me XSP version 4. I've never used XSP before but there are quite a few good pages on it, even if they do appear to be from 2006. Hell, I can't even remember if we had Google back in 2006. There followed a couple of bumps of a) remembering that I had to set up the custom http handler part of it in the web.config, and b) my .Net solution in MonoDevelop won't make an ASP.Net web application project that uses anything above version 3.5 for reasons best known to itself, which stops you referencing the projects directly from the web host. However, XSP and Mono are more than happy with it, so I made an empty library project that referenced the projects and now have a pre-XSP step that copies the latest DLLs into the web host application.

Alright. Now I have a working application. Almost. This happened:



This kept me going for over an hour, having to break bits of code by design in order to get a sensible answer out of Mono since I couldn't attach a debugger to it.

The problem occurred whenever the Store behaviour was called. Now it turns out that this behaviour happens to be the only one which refuses to do anything if the user isn't authenticated. Like so:


When the behaviour returns false, that signals that the command that is being delivered on the message bus has not been fully satisfied. So therefore it will move on to the next listener it has. Here is the delivery code:


Now what I couldn't fathom, is why this somehow breaks in Mono but runs fine under Windows. The bug is essentially implying that there is a listener in the collection which is null. I was feeling paranoid about this and so I made sure that there was a null check on the Add routine in the message bus.


Ok, so. All good, yeah?

No. Still fucked. In desperation, I added some code to show the listeners collection every time a message was being delivered. The result?


There were 2K of listeners in each of my two message buses listener collections. WTF?

The listener collection is pretty straightforward:


So just a ConcurrentBag of my interface type. That's absolutely fine and to my knowledge I had no reason to believe that it would spawn extra capacity.

I ran that particular test with an unmodified listeners collection on my Windows build and received the following result:


Which is correct behaviour. WTF?!? It turns out that there appears to be different behaviour for the System.Collections.Concurrent.ConcurrentBag in Microsoft.Net 4.0 and in Mono 2.8, where the Microsoft version enumerates correctly based on number of added items, and the Mono version seems to enumerate over an extended capacity.

Given that there is little practical difference in contention handling between locking on a System.Collections.Generic.List and using a ConcurrentBag, I started to use a List as it doesn't have the same retarded behaviour.


Yay! Working application... And now when running the upload tester...


Which gives me one of these...


Phew.

At which point it was time for sleepz.