Wednesday, 1 April 2015

Mirror people

When travelling around, something I have noticed more and more frequently is that people having mobile phone conversations do not behave like other people do. This appears to be especially true when the old-as-time situation of meeting someone walking in the other direction and having to decide which side to favour so you both can pass.

My experience is that people on phones will favour the same way as you, mirrored somehow, which causes a replay and reassessment as their attention is dragged back from the other side of the phone call.

If your body language is there you aren't here.

It's something about projection, I think. Perhaps this is something to do with how we relate to the person we are calling? Like a mirror has been placed in front of us - we relate our own self-image to the voice on the line.

Currently reading:


Monday, 23 March 2015

Using the Model-Store pattern in Inversion (part 1)

This post is the first in a series about using a Model-Store pattern in the Inversion behavioural micro-framework for .Net (see: https://github.com/guy-murphy/inversion-dev).

Model-Store Pattern

This is a general pattern than involves representing domain data as immutable models, and allowing the segregation of reading and writing of data in a manner that enables CQRS-like behaviour and optimisation in an application stack.

The key components of the pattern are domain model objects that exhibit an internal builder class that is used to mutate an existing object or create a new one (for this pattern's purposes, these are effectively the same thing), and a generalised store interface that is friendly towards service location, service injection and segregation. The store interface also enforces separation of concerns, enabling the storage methods and caching strategies to be transparent to the application.

Models

I despair when joining a project and discovering that they are three foot deep in AutoMapper entrails and everything has a DTO equivalent for talking to a Web Service somewhere, or some kind of perverse partial class used as a View Model. Why create something that can't be used everywhere? Why can't we all just get along and respect each other as who we are?

The day I started to keep my data objects free of extraneous crap was a good day.

Just a POCO ...

Assuming that we have a User entity in our domain, our POCO for this might look like the following:


Taking control of changes

However, this can be changed by anybody, so in order to put that under control, we can make the properties read-only and only settable from the object's constructor.


This uses the ability to set a public property to only have a private setter, thereby cutting down on some code real-estate and making sure code external to the object instance cannot modify the value of a property.

Immutability

However, the next step would be to enforce this policy for code internal to the object instance as well.


Extra constructor added that takes its values from an existing User object. This is mostly for completeness.

This is now more like a truly immutable data object. The instance is guaranteed by the CLR to have unmodified fields after initialisation.

Mutation

When working with objects, at some point we will need to represent a change in value. This is known as mutation and it entails a little bit of extra internal structure to support it. The advantage of enforcing this really shows when dealing with multi-threaded applications, where a thread can guarantee that an object pointed to by specific reference will not change during the lifetime of that instance. Any change to the object creates a new instance. Using immutable objects in multi-threaded applications can lead to entire swathes of management code and concerns falling away.

Two components are needed to support mutation. First, an internal Builder class, and second, a Mutate method.

The internal Builder class

The Builder class is contained within the domain object class, as below. It contains publicly modifiable versions of the properties belonging to the immutable outer class.

It represents a mutable version of the data. This can be used and accessed by external code to prepare new or changed data. 


The ToModel method puts the Builder object's values into a new instance of a User object which has immutable properties.

The FromModel method puts a User object's values into a Builder object which has mutable properties.

The Mutate method

This method is responsible for facilitating a change to an immutable object by accepting a change that is represented by an anonymous method, and returning a new instance of the object that contains the changed data.


It creates a new instance of this model's Builder class, initialised with the current immutable instance's data. Then, it applies the passed anonymous method, or mutator, to the builder.

The mutator looks something like this:


The result of the mutation is returned as a User object, and normally this would cause a casting exception as Mutate is defined as returning a User object, but the addition of the following two methods allow us to override the implicit casting behaviour of the immutable User class and the mutable Builder class in a granular fashion:


Therefore, when the Mutate method returns a Builder object as a User, the implicit operator for User that takes a Builder as an input instead returns the result of the Builder's ToModel method - i.e. the immutable version of the Builder's data.

Similarly, when some code casts from a User object to a Builder object, then the implicit operator for Builder will be called, returning the result of new Builder(model), where model is the User object being cast.

Default values for fields

To provide fields with default values, they should be applied to the Builder object, as instantiating the immutable object requires either an existing User object or a full set of values for the properties.

One way of doing this is to change the parameterless constructor in the Builder class so that it contains the initialisation of properties with default values.



Another way would be to provide a private backing-field for a property that is being given a default value in the Builder object:


Dictionaries and Lists

There are existing implementations of immutable classes of Dictionary and List that may be used in the immutable outer class, but they require a little bit of initialisation and translation during construction and conversion. 

Suppose we want to add a Metadata collection to our User object which will be represented by a string,string dictionary. To do this, we specify an ImmutableDictionary in the outer class:


And the public, read-only property:


The two constructors of the outer class need some modification. The first constructor needs to be passed a dictionary of metadata to instantiate its immutable data from:


The second constructor is passed a complete User object, so the signature doesn't change but the body will have to take its metadata from there:


In the Builder class, the metadata is represented by just a normal dictionary:


And that will need initialising in the parameterless constructor:


The other constructors and methods adjust to accommodate:


For ImmutableList, the pattern is just the same.

Also, for the record, I hate using var, but the alternative didn't format very well. Weak excuse, I know.

Immutables, Immutables, Immutables, all the way down

Now that we have the User object with its immutability and Builder class, supposing we implemented its Metadata property as another immutable class instead of just an ImmutableDictionary.


Importantly, the Builder's Metadata property is of type UserMetadata.Builder. That type choice plus the implicit cast overrides (assuming there is one in the UserMetadata class as well) allows a chaining of the mutable versions of the objects, which means that within a Mutate method called on the User object, the mutable version of the Metadata is in scope and will be converted back into an immutable version when the result of Mutate is cast back to a User (internally it will chain the calls to ToModel).


Implementing the IData interface

Finally, in order to make this generally useful within the Inversion ecosystem, we make the object implement the Inversion.IData interface:


Implementing this interface allows Inversion to include the object in a ViewStep model which can be processed by the render pipeline. It's not a hard requirement for an object used in an Inversion application to implement IData, but it does enable the object to be represented easily by XML/Json in a way that easily interfaces with the framework.


The casting to IData in the Data method makes it explicitly clear to the compiler that we want the implementation of the IData's Data method within the Metadata object instance.

The full versions of the User and UserMetadata classes can be found here:

https://gist.github.com/fractos/6c7f53dd88aeb4c00b5f

Next part ... Stores

Coming soon.
Update: Fixed the source code for the Mutate function.

Wednesday, 18 March 2015

Bending the world so log4net can work

log4net.

Firstly, why the Hell would you change the public key of a strongly named assembly between versions?

Secondly, WHY THE HELL WOULD YOU CHANGE THE PUBLIC KEY? Jesus.

Yes, I've read http://logging.apache.org/log4net/release/faq.html#two-snks and that just reeks of the development team having some kind of argument with people outside of their anointed circle of coders.

Anyway, there's a requirement on some old CMS code in a project I'm working on, and the DLLs have got a reference to log4net 1.2.10 while the rest of the world is staring down the barrel of 1.2.13.

Incidentally, I think it's really bad form to put numbers into assembly names, and, further, to have version 1.2.13 to be a completely different NuGet version (2.0.3). Seriously, what the fuck?

I found this post which really helped - you can reference two different versions if they use a strong name as long as you stick them in different folders, whilst keeping a reference in the project to the later one but setting Copy Local to false. I also added a pre-build event on the website which would make the folders if they didn't exist and copy the appropriate DLLs into each.

http://stackoverflow.com/questions/3158928/referencing-2-differents-versions-of-log4net-in-the-same-solution

However, the finishing touch was that since this was an endpoint - i.e. a website where lots of DLLs are bundled up together - then it needed to have the right path; in this case, it needed "bin\" prepended to it:


Currently reading:

Tuesday, 17 February 2015

Running unmanaged code in a managed C# .Net website


Encountered this problem in the Fusion Log of a .Net website that uses a wrapper for some unmanaged C++ code:

LOG: Assembly download was successful. Attempting setup of file: C:\dev\...\bin\kdu_a75R.dll
LOG: Entering download cache setup phase.
ERR: Error extracting manifest import from file (hr = 0x80131018).
ERR: Setup failed with hr = 0x80131018.
ERR: Failed to complete setup of assembly (hr = 0x80131018). Probing terminated.


After a great deal of looking at Dependency Walker traces, countless forum articles and advice, it turned out to be the simple fact that a .Net website will look at every .DLL file it can find in the Bin folder of the website and try to load a .Net assembly manifest from it, even if it isn't a .Net DLL. Because reasons.

The solution to this is as obvious as it is absurd:

  • Don't put unmanaged DLL files in a .Net website Bin folder.
  • Put them in Windows' System32 folder where IIS will be able to find them.
That's just terrifying, but it does work. It makes me taste a little bit of sick in my mouth, frankly.

Sunday, 1 February 2015

Promises, promises ... (Testing non-AngularJS websites with Selenium and Protractor)

A post about promises, web-testing and far too much indentation.

This post covers the tests defined in https://github.com/britishlibrary/UniversalViewerTests

Recently, I have been writing browser tests for a Javascript application. The tests were invoked via Protractor running under Node.js locally on a Windows laptop. The Automate service from BrowserStack was used for cross-browser capability checking, and to be honest I can't recommend that service enough. It's *very* clever and straightforward.

The tests use Cucumber to enable BDD-style spec statements for features and scenarios. This differs from the usual uses of Protractor tests which generally use Jasmine - a library that represents test scenarios programmatically. A lack of documentation surrounding the use of Cucumber in Protractor has been a major headache during the project.

The tests as received originally used Selenium's webdriverjs to drive the browser session. Before the previous developer left, we converted many of the test features over to using a Page Object which offers an abstracted set of functions that allow access to elements on the page whilst hiding the details of how the elements are found; the concerns of the test functionality are therefore separated from the concerns of how to locate an element.

Further, since we are using Protractor to test a website that does not use AngularJS, the synchronisation that Protractor will do under the hood is not available. This means that communication with Selenium webdriver must be done using a Promise - a callback-like way of scheduling functionality to occur after a call into a sub-system with handling for success and failure of operations. Although ostensibly a means to cut-down on code, using Promises has the same effect as callbacks in that *everything* becomes a callback and you can end up very far across your screen for even minor functionality.

A Page Object doesn't have to be very smart. All it has to do is hide the implementation of locating elements for the tests that call it.

I generalised three methods - find, findAll and sleep:
Almost all the abstracted functions for accessing elements in the Page Object use these methods. This was done primarily so that I could centralise the logging for what was going on. Here are some example element abstractions:
This makes the code that uses the Page Object a lot clearer, as it can refer to an abstraction of the element(s) it is considering instead of dealing with calls to the webdriver which are now hidden away.

So far, so good. The tests work really well in Chrome and Firefox. Problems occur, as ever, when Internet Explorer gets involved. The general advice you should heed when testing with Internet Explorer is to NOT TOUCH ANYTHING while the test is running. IE is effectively an embedded component of Windows and it really doesn't let you forget it. Changing focus to another window could be enough for it to lose its place and invalidate your test run. Further, when the thing being tested uses an HTML IFrame, IE will lose its place after EVERY SINGLE TIME you try to find anything in the DOM. This led to a complete re-write of the browser tests in order to maintain cross-browser compatibility whilst also trying to be robust enough for IE's behaviour.

The test rig HTML IFrame we use for the viewer is the only one on the page. Even with that fact, the following code is necessary to reset which frame Internet Explorer is considering before it attempts to locate an element:
This is utilised by the test code itself before it performs a locate operation:
As you can see from that gist, an element locator function from the Page Object returns something which offers a 'then' function - this is the implementation of the Promise functionality. The 'then' function can take two arguments; the first is a function that will be called if the operation was a success, and the (optional) other will be called in the event of a failure. In this case, there are two functions defined that handle failed operations - one for if the element cannot be located, and the other if the element's 'click' function fails in some way. The success route through the callbacks is indicated by the main flow from left to right - success in locating the element flows to a click which then flows to success in clicking followed by executing the Protractor/Cucumber callback that will announce a pass for this part of the testing scenario.

If I decide to do a refactoring job on this again, I would probably wrap the resetFrame call into the find/findAll functionality. This would mean passing the behaviour needed by the test into a callback function and passing it into the element locator function, frankly a bit like how the calls to resetFrame work now, but transparent to the feature files.

Okay, that's all for now. I wanted to record how to make these tests cross-browser compatible with the use of resetting the frame - this is just in case someone out there finds it useful in the future ... (HI, FUTURE ME!)

Tuesday, 27 January 2015

Spite Coding (Asura.Schema)

It's only recently I've realised how much programming I do that is purely done out of spite.

Latest example - Newtonsoft.Json.Schema library being released and it coming with licence restrictions.

This really annoyed me*, and over last weekend, whilst sick and feverish from a chest and sinus infection, I wrote Asura.Schema which covers about 75% of the draft 4 standard for JSON schema. It's still a work in progress and has a few bits to add, (like, you know, primitive constraint checking), but it (almost) does all that I need it to do. I should have the bits of it that I need done in the rest of this week.

Like some kind of delirious grudge fuck, it's afforded me that rare combination of effort spent and a sick sense of enjoyment. I am okay with this; it's like I've exorcised a particular demon by thrashing it out in code.

I think this is my first piece of code I've made public like this.

*: the why is another story - it could well have just caught an edge because I was ill.

Wednesday, 21 January 2015

JPEG2000 Support with IIPImage / IIPSrv and Kakadu 7.5 on Ubuntu 14.04 LTS via Lighttpd 1.4

Been having fun getting IIPSrv working with JPEG2000 support via Kakadu. After a lot of head scratching yesterday, I got a working config together using Lighttpd's FastCGI support.

Figured I'd best note down what steps I took so that others may benefit ... or indeed future me. (HI, FUTURE ME!)

Be warned, these are notes scavenged from my bash history, so some steps may have been more efficiently done. I'm a little rusty on my Linux and slightly clueless about some package management.

Setup

1. Install Ubuntu 14.04 Server LTS  (tested under 12.04 LTS as well)
2. Select LAMP and OpenSSH server roles from installation menu (or use Tasksel on an existing server)
3. Once installed, log in and update packages using: sudo apt-get update
4. Then: sudo apt-get upgrade
5. Reboot, etc.

6. Add some packages for compilation and other things:
6.1 sudo apt-get install make
6.2 sudo apt-get install g++
6.3 sudo apt-get install openjdk-7-jdk
6.4 sudo apt-get install vsftpd
6.5 sudo apt-get install git
6.6 sudo apt-get install autoconf
6.7 sudo apt-get install libtool
6.8 sudo apt-get install libfcgi-dev libjpeg-dev libtiff-dev

Transferring files

7. I installed an FTP server (vsftpd) to get the package of Kakadu's SDK and sample images on to the server, but it needed a little configuration first:
7.1 Edited /etc/vsftpd.conf: 
Uncommented the following lines:
local_enable=YES
write_enable=YES

Comment the following line:
#anonymous_enable=YES
7.2 sudo service vsftpd restart
Compiling Kakadu

I transferred the Kakadu SDK onto the box using FTP. The location that Kakadu is placed is important. I ended up putting it into a "kakadu" folder in my user's home folder, then after compiling it I placed it under /opt/kakadu and renamed the SDK folder itself to /opt/kakadu/kakadu-7.5 (previously it was v7_5-01573L).

The reason it is important is that IIPSrv is compiled against the Kakadu binaries, so they must be in a place that is accessible to the Lighttpd process.

The first time I did this I hit a brick wall as Lighttpd's mod_fastcgi module would say it had failed to start and that the child process exited with a status of 127, and it was not until I had taken the Lighttpd 1.4 sources from GitHub and ran the version I compiled myself that it told me exactly why - because it couldn't access the Kakadu library.

The problem was that I had compiled both Kakadu and IIPSrv in my user's home directory, so when I installed the version of IIPSrv I had compiled, it was still referencing my home directory which was inaccessible from the Lighttpd's www-data user. By placing the Kakadu SDK under the /opt/kakadu folder before I compiled IIPSrv, the binary would reference the Kakadu libraries in a place it can have permission to read them.

The Kakadu makefiles in three directories that I compiled separately as I couldn't get the makefile in the root 'make' folder of the SDK working.

I had to edit the makefiles as there is currently an incompatibility between the latest version of IIPSrv (0.99) and Kakadu 7.5, explained here: https://github.com/ruven/iipsrv/issues/22 - the solution for which is to fiddle with the compilation flags for Kakadu with regards to SSSE3 support. However, I found that I had to make a further modification involving the AVX2 support to get a clean compile. So:

10. Edit makefiles for optimisation support:

(from the Kakadu SDK root folder)
10.1 cd coresys/make
10.2 Edit Makefile-Linux-x86-64-gcc:
Uncomment the following lines:
C_OPT += -DKDU_NO_SSSE3
C_OPT += -DKDU_NO_AVX2
Comment the following lines:
#SSSE3FLAGS = -mssse3
#AVX2FLAGS = -mavx2 -mfma 
10.3 make -f ./Makefile-Linux-x86-64-gcc
10.4 cd ../../managed/make
10.5 Edit Makefile-Linux-x86-64-gcc:
Uncomment the following lines:
C_OPT += -DKDU_NO_SSSE3
C_OPT += -DKDU_NO_AVX2
Comment the following lines:
#SSSE3FLAGS = -mssse3
#AVX2FLAGS = -mavx2 -mfma 
10.6 export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
10.7 make -f ./Makefile-Linux-x86-64-gcc
This makefile tries to create a folder called "java" above the kakadu source folder, which is a bit outrageous. Hence, the reason why I compile the Kakadu source in my user's folder then move it to /opt/kakadu/kakadu-7.5 so it is more accessible.
10.8 cd ../../apps/make
10.9 Edit Makefile-Linux-x86-64-gcc:
Uncomment the following lines:
C_OPT += -DKDU_NO_SSSE3
C_OPT += -DKDU_NO_AVX2
Comment the following lines:
#SSSE3FLAGS = -mssse3
#AVX2FLAGS = -mavx2 -mfma 
 10.10 make -f ./Makefile-Linux-x86-64-gcc
Now, hopefully, you should have the object files for Kakadu compiled.

11. Move Kakadu SDK to a better place and fix permissions:
11.1 cd ~/kakadu   (or whichever folder the SDK is sitting beneath in your home folder)
11.2 sudo mkdir /opt/kakadu
11.3 sudo mv v7_5-01573L /opt/kakadu/kakadu-7.5
11.4 sudo find /opt/kakadu -type d -exec chmod 755 {} +
This makes all the directories under /opt/kakadu accessible.
11.5 sudo find /opt/kakadu -type f -exec chmod +r {} +
This makes all the files under /opt/kakadu world-readable. Please note that I would bet a whole Mars bar that there are more graceful ways of doing this.
Compiling IIPSrv

The version of IIPSrv I used was from GitHub, here's a link to the exact commit - https://github.com/ruven/iipsrv/commit/5edcdb7bb8b22538f467da6bbefbbae2649b59c6

12. Get IIPSrv source from GitHub:
12.1 cd ~
12.2 git clone https://github.com/ruven/iipsrv.git
13. Configure IIPSrv:
13.1 cd iipsrv
13.2 ./autogen.sh
13.3 ./configure --with-kakadu=/opt/kakadu/kakadu-7.5
At the end of configuration, you should hopefully see a message saying that JPEG2000 support is enabled. If not, then check through the configure output in scroll-back for clues.
14. Compile IIPSrv:
14.1 make
The output from a successful make should be a file called src/iipsrv.fcgi. This will be utilised in the next steps.

Preparing FastCGI for Lighttpd

A few things to do here, so:

15. Install Lighttpd:
15.1 sudo apt-get install lighttpd
16. Prepare a FastCGI area for Lighttpd:
16.1 cd /var/www
16.2 sudo mkdir localhost
16.3 sudo chmod 755 localhost
16.4 cd localhost
16.5 sudo mkdir fcgi-bin
16.6 sudo chmod 755 fcgi-bin
16.7 sudo mkdir data
16.8 sudo chmod 755 data
17. Copy IIPSrv binary to FastCGI area:
17.1 sudo cp ~/iipsrv/src/iipsrv.fcgi /var/www/localhost/fcgi-bin/.
17.2 sudo chmod 755 /var/www/localhost/fcgi-bin/iipsrv.fcgi
18. Edit /etc/lighttpd/lighttpd.conf:

Add the following section, as per http://iipimage.sourceforge.net/documentation/server/

fastcgi.server = ( "/fcgi-bin/iipsrv.fcgi" =>
  (( "host" => "127.0.0.1",
     "port" => 9000,
     "check-local" => "disable",
     "min-procs" => 1,
     "max-procs" => 1,
     "bin-path" => "/var/www/localhost/fcgi-bin/iipsrv.fcgi",
     "bin-environment" => (
        "LOGFILE" => "/tmp/iipsrv.log",
        "VERBOSITY" => "5",
        "MAX_IMAGE_CACHE_SIZE" => "10",
        "FILENAME_PATTERN" => "_pyr_",
        "JPEG_QUALITY" => "50",
        "MAX_CVT" => "3000"
      )
  ))
)

And I changed the HTTP port to 8080 as I couldn't be bothered uninstalling Apache2:
server.port                 = 8080
19. Enable mod_fastcgi:
19.1 sudo lighttpd-enable-mod fastcgi
20. Restart Lighttpd:
20.1 sudo /etc/init.d/lighttpd stop
20.2 sudo /etc/init.d/lighttpd start
"force-reload" as a command to that script probably works too, but it didn't for me.

Sample images

I found sample TIF and JPEG2000 images at:

from http://iipimage.sourceforge.net/documentation/images/
http://merovingio.c2rmf.cnrs.fr/iipimage/PalaisDuLouvre.tif
(1.87mb TIF file 4000x828)

from http://www.microimages.com/gallery/jp2/
http://www.microimages.com/gallery/jp2/CB_TM_QQ432.jp2
(17.8mb JPEG2000 file 3164x2982)

NOTE: This was unavailable last time I checked, try this one instead:
from http://www.loc.gov/item/cmns000145/
http://cdn.loc.gov/service/afc/afc1999008/afc1999008_crf_lec05011.jp2
(1.8mb JPEG2000 file 2048x1402)

21. I transferred these to the box and then moved them into the data folder I made in stage 16:
21.1 sudo cp ~/PalaisDuLouvre.tif /var/www/localhost/data/sample.tif
21.2 sudo cp ~/CB_TM_QQ432.jp2 /var/www/localhost/data/sample.jp2
21.3 sudo chmod 644 /var/www/localhost/data/*
Time to test

Okay, theoretically now you should have a functional Lighttpd instance. This can be tested by pointing a web browser at your VM / box on whatever HTTP port that your /etc/lighttpd/lighttpd.conf is set to (bearing in mind that Apache2 will be on port 80 already.)

So in my case I did:

http://192.168.137.80:8080/

And got the Lighttpd placeholder page. If you see that then you know that the FastCGI config has worked as otherwise Lighttpd will bomb out on startup. If you don't see the placeholder page then you need to look at the entries in the /var/log/lighttpd/error.log file for clues.

Next, try the IIPSrv module with a TIF:

http://192.168.137.80:8080/fcgi-bin/iipsrv.fcgi?FIF=/var/www/localhost/data/sample.tif&HEI=2000&CVT=jpeg

Hopefully you should see a panoramic picture of the Lourve ...

If that works, then try with the JPEG2000 file:

http://192.168.137.80:8080/fcgi-bin/iipsrv.fcgi?FIF=/var/www/localhost/data/sample.jp2&HEI=2000&CVT=jpeg

If that works - you've done it!