TeamCity and Mercurial Feature Branches

Rationale

I wanted to write this blog as I have used up a small quotient of my life attempting to get TeamCity and Mercurial to work together, and somebody somewhere should benefit from my findings.

The company I work for use Microsoft .Net with Mercurial as their de facto version control system, which I am completely fine with. Mercurial is a wonderful VCS and especially so if you've lived your arcana of SourceSafe, TFS and Subversion. I'm not here to defend our choice in not moving to Git because frankly I can't be bothered. Basically, if you use Mercurial and you want to use TeamCity as a build and deployment server with feature branches and something which is a little bit like continuous integration if you squint your eyes, here's a way I found of making them work together in something approaching harmony.

A Tale Of Two Build Chains

So, initially we had this idea of having a build chain permanently churning "default" in Debug mode, and another which would build anything with the prefix "Release-" on its branch name in Release mode. This kinda worked for a month or so but then it was starting to make it difficult to logically separate features since people started forgetting to use the prefix, then pointing out that their feature on the UAT server clearly wasn't ready to be released so there was a nomenclature problem there. Further, when we started trying to coerce the VCS roots to only consider certain feature branches using the Default Branch and Branch Specifications, things started going really awry and I realised that we were losing control of how TeamCity was responding, including not being able to tell what had just been deployed and why.

This week I've been anxious to get anything working, let alone things like this that have been driving me mad, so when I came in on Monday I ditched what I was currently doing and rolled up sleeves to work on this problem.

It turns out that our thinking on this was pretty much backwards and then things were getting more out of whack with the various hacks we were trying to try and make it work. Limiting the build chains with a Branch Specification seems to be actively fighting against how the TeamCity feature was designed, so I needed to re-think how to achieve what we needed.

After re-reading the TeamCity 8.x documentation around Mercurial, which swear blind that this should all just work, I decided to try a new topology with respect to the original two build chains we had set up. I got the feeling that the builds produced by a build chain could be used as a palette by the deployment chains, and to that end I re-tasked Trunk and Release into chains that would consider *every* branch, but would compile them with Debug or Release mode, respectively.

Setup for "All branches (debug mode)":
VCS Root:
  Name: "Trunk" (This isn't the right name for this now, I realise)
  Mercurial, etc
  Default branch: [blank]
  Branch specification: +:*
Build steps:
  1) Build Solution (Debug mode)
  2) Run Unit Tests (Debug mode)
Build triggers:
  Branch filter: +:*  (trigger one build per each VCS check-in, include several check-ins if from same committer)
Build parameters: (various things but importantly:)
  system.Configuration = Debug

Setup for "All branches (release mode)":
VCS Root:
  Name: "Release"
  Mercurial, etc
  Default branch: [blank]
  Branch specification: +:*
Build steps:
  1) Build Solution (Release mode)
  2) Run Unit Tests (Release mode)
Build triggers:
  Branch filter: +:*  (trigger one build per each VCS check-in, include several check-ins if from same committer)
Build parameters: (various things but importantly:)
  system.Configuration = Release

In all fairness, I could attach the same VCS Root to both of these build chains and I think it would work fine. I kept these as separate Roots since it was how it was originally and that gives the option of pointing it at a different source repository, although Christ knows why we'd do that.

Deployment Chains - 1 - Sorta-Kinda Continuous Integration

In a vague nod towards doing some kind of Continuous Integration, we want to have our two testing servers (Testing and UAT) continuously updated with the latest build from certain feature branches. I found I could do this by using the Build Triggers functionality. This appears to be because the Branch Specification parameter in the build chains uses a wildcard - that wildcard becomes a logical branch name which the Build Trigger's Branch filter can then pick up.

Sub-project: Continuous Integration - Automatic deployments to Testing / UAT

Setup for Deploy Testing - "Feature Branch 1"

(I started renaming the chains to what branch they were tuned to)

VCS Root: "Trunk"  (as above)

Build steps:

  1) Compile and deploy

  2) Remove output directory

Build Triggers:

  Branch filter: +:Feature Branch 1

  Trigger a build on each check-in

  Include several check-ins in a build if they are from the same committer

Dependencies:

  Depend on: "All branches (debug mode)" (the Debug mode build chain)

  Do not run new build if there is a suitable one

  Only use successful builds from suitable ones

Build parameters: (whatever you need for your deploy job, plus:)

  system.Configuration = Testing

Setup for Deploy UAT - "Feature Branch 2"

VCS Root: "Trunk"  (as above)

Build steps:

  1) Compile and deploy

  2) Remove output directory

Build Triggers:

  Branch filter: +:Feature Branch 2

  Trigger a build on each check-in

  Include several check-ins in a build if they are from the same committer

Dependencies:

  Depend on: "All branches (debug mode)" (the Debug mode build chain)

  Do not run new build if there is a suitable one

  Only use successful builds from suitable ones

Build parameters: (whatever you need for your deploy job, plus:)

  system.Configuration = UAT

With these in place, a commit to "Feature Branch 1" will spawn three builds:

1. "Feature Branch 1" in Debug mode on "All branches (debug mode)" build chain
2. "Feature Branch 1" in Release mode on "All branches (release mode)" build chain
3. "Feature Branch 1" deployed in Debug mode from Deploy Testing - "Feature Branch 1" build chain

The deployment job will queue until the "All branches (debug mode)" build chain has completed successfully.

Similarly, a commit to "Feature Branch 2" will spawn the Debug, Release and deployment builds, but Deploy UAT - "Feature Branch 2" will run instead.

Deployment Chains - 2 - Manual Deployments

Along with the Testing and UAT deployment chains that do this pretence of Continuous Integration, we also wanted to ability to manually deploy particular feature branches across our Staging and dual-head Live environments.

For this, I have attached the Release mode VCS Root, so they will receive the correctly compiled build, but I have not added any Build Triggers, relying instead upon the GUI in TeamCity to request the appropriate feature branch from the provided drop-down menus.

The dependency means that it is still requires a successful build plus unit test run to deploy.

Setup for Deploy Staging

VCS Root: "Release"  (as above)

Build steps:

  1) Compile and deploy

  2) Remove output directory

Dependencies:

  Depend on: "All branches (release mode)" (the Release mode build chain)

  Do not run new build if there is a suitable one

  Only use successful builds from suitable ones

Build parameters: (whatever you need for your deploy job, plus:)

  system.Configuration = Staging

Conclusion

I'm utterly relieved that I'm getting some sense out of TeamCity now. I've really fought for keeping the use of feature branches and to make sure that we continue with Mercurial, so this has felt like some kind of small victory. The automatic "continuous integration" deployments to Testing and UAT work well (when I remember to tick all the right boxes in the Build Triggers), and they have meant that I haven't had to escalate privileges for a remote-working developer. The only drawback is that you have to fiddle with the Build Triggers to be able to "tune" those deployment chains to a particular feature branch, but I think that's a decent trade-off.

Although the Staging and Live deployment chains involve a manual selection step, this is basically acceptable considering that the rest of the build server appears to be running in a sane fashion now.

Well, apart from one thing. When feature branches are added and their first commits are pushed, Team City will spawn a build per group of commits based on user, and that has meant that I've occasionally spotted 20+ builds being added to the queue including a bunch of re-deployments. In all fairness, this isn't the end of everything, but it can be a little bit irritating.

So, yeah. TeamCity and Mercurial feature branches. It does actually work as long as you think about it right.

Carbon for Android v2 - some niggles

The new version of Carbon for Android is out which features a number of changes including some quite striking visual differences to the original. My install was automatically upgraded to the new version yesterday and I must say that the interface is even more slick than the old one, but I have a number of issues with the new design.

1. The refresh icon rotation is off-centre.
I'm not quite sure why this has been done as it frankly looks a bit scrappy. Centering this would make the feature complete and professional.

2. A lot of space taken up by the 3 dots at the top.
Seriously, you could fit an entire line of text in the space occupied by those 3 little dots. Better use of that space could be made.

3. The font has a weird aspect ratio.
It is clear that you love this font but I find it much harder to read with its oddly vertically stretched aspect. It would be so welcome to have the option of a shorter font. This is practically a deal-breaker for me and the thing I can least understand as to why it has been forced.

4. No Tweet size of "small" ?
The menu option only starts at a size of "medium"; that's pretty weird.

5. Retweet mesages.
It's hard to tell immediately whether the message saying that something has been retweeted belongs to the tweet above or below the message. A hint of a border here would really help.

6. Bug - faded tweets.
Some tweets appear faded suddenly for no reason but then are fine when restarting Carbon.

I am so glad that I can now get to my own favourites at last, and the work on the profile view is pretty epic. Carbon itself is an amazing piece of software. But there are these few things that need to be addressed before I will be happy with the update, sorry guys.

Please note that if the font thing is addressed then I will definitely donate. I don't mean to hold to ransom but I feel pretty strongly about this issue.

Update: They fixed the font issue!

Static Content in IIS 7.5

It's amazing how almost every piece of advice you read about something can be completely, utterly, totally useless. I think it's an endemic problem of developers who only spend time clicking the shiny buttons in the GUI of products without ever actually getting into the guts of a thing to see what is happening.

Anyway, a designer was trying to add .SVG, .ICO and .WOFF files to our website which worked fine locally but when viewed on our UAT server it was coming up with 404 errors for those files.

After a little head scratching, I found out that IIS disables transport for those extensions by default. When looking around for advice as to how to switch transport back on for these, I read so many articles where clearly the author never bothered to go near the web.config file.

And of the advice that did cover the web.config file, most of the received wisdom goes along the lines of:

Add these lines to the web.config:

    <system.webServer>
    ...
    <staticContent>
<mimeMap fileExtension=".woff" mimeType="application/x-font-woff" />
<mimeMap fileExtension=".svg" mimeType="image/svg+xml" />
<mimeMap fileExtension=".ico" mimeType="image/x-icon" />
        </staticContent>
    </system.webServer>

Which is great in theory, but this actually results massive amounts of errors on the site as it suddenly fails to load ALL STATIC CONTENT. So no CSS, JS, images, anything, plus a load of odd redirections for other types.

I figured that having these rules in a separate web.config file in our static resources folder might help, so I tried that and although the site was now generally working again, there were mysterious 500 errors for the .SVG etc requests.

After a little digging, I found this post on Stack Overflow:

http://stackoverflow.com/questions/13677458/asp-net-mvc-iis-7-5-500-internal-server-error-for-static-content-only/13677506#13677506

This saved me a complete metric fuck-tonne of time.

What all the advice I had read was missing is that you should remove the mappings in the static content before you add them, or you will get a 500 error that doesn't appear in any log unless you enable the really paranoid stuff in IIS.

So, the proper version of what the file looks like is this, remembering that this is a partial web.config sitting in our static resources folder:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
    <staticContent>
<remove fileExtension=".woff"/>
<mimeMap fileExtension=".woff" mimeType="application/x-font-woff" />
<remove fileExtension=".svg"/>
<mimeMap fileExtension=".svg" mimeType="image/svg+xml" />
<remove fileExtension=".ico"/>
<mimeMap fileExtension=".ico" mimeType="image/x-icon" />
        </staticContent>
    </system.webServer>
</configuration>

Antaris RazorEngine and Mono-3.2.3

After some machinations, I managed to get Antaris RazorEngine working with Mono-3.2.3.

After making and installing mono from a tarball (http://download.mono-project.com/sources/mono/), I took the latest version of RazorEngine down from github (https://github.com/Antaris/RazorEngine), compiled using mono's xbuild tool, and then took the version of RazorEngine.dll and System.Web.Razor.dll from the RazorEngine.Core folder and used that within my applications. I removed all instances of System.Web.Helpers, System.Web.Mvc, System.Web.WebPages, WebMatrix, etc as those would have conflicted references.

And RazorEngine is now really horribly strict :( It throws compile exceptions on warnings which is wholly irritating. I'll have to revisit and police every single template and make sure they are compatible.

I also noticed that the new version of xsp4 required me to be in the root folder of the website even though I was setting the root on the command line.

Anyway, glad to see it working again, even if it will be a chore to get it fully compatible. I guess it is balanced out by now being on .Net 4.5 compatible mono.

Update: I've made an adjustment to the RazorEngine code which stops it treating warnings as errors during template compilation.

Modify RazorEngine.Core/Compilation/DirectCompilerServiceBase.cs, Compile method thusly:

            var @params = new CompilerParameters
            {
                GenerateInMemory = true,
                GenerateExecutable = false,
                IncludeDebugInformation = false,
                CompilerOptions = "/target:library /optimize",
                TreatWarningsAsErrors = false,
                WarningLevel = 0
            };

Although the default for TreatWarningsAsErrors is false in Mono, I think the default for WarningLevel is -1, which doesn't really mean anything. So this might actually be a Mono porting issue.

Bookmarks for 20130922

Ello. Not done one of these in a while. Here's the shit.

Incredible illustrations, in a very old style. http://danielmartindiaz.com

The one that caught my eye was this one, "Binary Predator" -

which can be found in an animated form in Warren Ellis's introduction to his short story "Lich House" - http://www.youtube.com/watch?v=RYScwq3EOD0



You should check out TRSST - a secure micro blogging system that recently hit its funding target on Kickstarter.

http://www.trsst.com

This is both terrifying and gorgeous. Poet CJ Allen - "Explaining the Plot of Blade Runner to my Mother who has Alzheimer's".

http://www.forwardartsfoundation.org/poetry/explaining-the-plot-of-blade-runner-to-my-mother-who-has-alzheimers/

Antaris RazorEngine, Site Layouts

I have been using Antaris RazorEngine (v3) both at work and in the Asura framework used by Xizi. It is a light implementation of Razor that isn't strongly tied to MVC.

That said, documentation has been pretty thin on the ground. Especially around what I've been trying to do today which is to use the Layout functionality from within a file that is effectively in a .cshtml format.

So, the problem I am trying to address is doing ASP.Net MasterPage-like things but with isolated Razor templates using RazorEngine.

Certainly there are bits of documentation about how to use Layouts when creating and treating templates from code, but I couldn't find any complete examples on how to do it with template files.

The Layout file - mylayout.cshtml

<!DOCTYPE html>

<html>

@RenderSection("Head")

@RenderSection("Body")

</html>

This is a very basic layout comprised of the beginning and end of the markup and two section placeholders for "Head" and "Body".

That's all quite simple, but the problem is that if you are using RazorEngine in the raw way that I am, then when a template tries to resolve that layout it will fail as it is not in the template cache and I do not specify a resolver delegate to be able to do it.

Instead, the layout file must be compiled ahead of time. This can be performed by the following which, given an existing pathname for a .cshtml, will compile it into the template cache:

string viewPath = @"C:\code\razor\mylayout.cshtml";
string layoutName = @"mylayout";
if (File.Exists(viewPath))
{
ITemplate template = Razor.Resolve(viewPath);
if (template == null)
{
string templateContents = File.ReadAllText(viewPath);
Razor.Compile(
templateContents,
typeof(IDictionary<string, object>),
layoutName);
}
else
{
// already in cache
}
}

Assuming viewPath is the physical path of the mylayout.cshtml file and layoutName is the simple name that this layout will be known as in the template cache. Also, there should be a critical section between the Razor.Resolve and the Razor.Compile operations.

The Razor.Compile operation takes the contents of the mylayout.cshtml file and considers it with having a Model object of type IDictionary passed to it. This type should be repeated in the template file, as shown below.

Once the layout has been cached, any template file can use the layout.

The Template file - layout_test.cshtml

Here we go, then. This is the contents of my template, which is the one I want to use to fill the layout's section placeholders:

@inherits RazorEngine.Templating.TemplateBase<IDictionary<string, object>>
@{   
    this.Layout = @"mylayout";
}

@section Head
{
<head>
    <title>@Model["facebookAppID"].ToString()</title>
</head>
}

@section Body
{
<body>
my body
</body>
}

Firstly, the @inherits statement uses the generic version of RazorEngine's TemplateBase class to define the type that the @Model object will be interpreted as. This type should match whatever you are passing into the RazorEngine.Razor.Compile method when you compile the template. This should also match the type passed into the layout's compile operation.

Next, the setting of this.Layout. This is the name by which the layout's template is known within the RazorEngine cache, so it must match what was used to compile that template. Setting this string activates RazorEngine's ability to process this template while considering the contents of the layout.

After that are two @section definitions. One of which uses the @Model object which is treated as an IDictionary just as an example proving that it has access to @Model.

Note: You can also access @Model from the Layout template, without having to use the @inherit statement.

The template is compiled in much the same way, although in the Asura framework, templates are compiled on application startup and also when the last file write time is detected to be greater than that which we have recorded most recently.

Conclusion

So, long story, short, in order to get the Layout functionality working from within a .cshtml style use of RazorEngine, you have to pre-compile the templates that you will be using as a Layout before they are accessed, because RazorEngine will not do that work for you.

Hope that helps some people.

Bookmarks for 20130729

Well this just pulled me out of a terrible hole ... the dreaded "case folding error" in Mercurial. The following advice worked:
http://mercurial.selenic.com/wiki/FixingCaseCollisions

And this seemed useful...

Failing over from CDN to local storage:
http://www.hanselman.com/blog/CDNsFailButYourScriptsDontHaveToFallbackFromCDNToLocalJQuery.aspx

Twitter Bootstrap has a release candidate for version 3. Not sure how I feel about this yet. On the one hand the new stuff looks amazing, on the other, I'm not sure I like the new grid classes. But, worth a look and it is admirable that they are going for "mobile-first", so rolling the responsive stuff into the main build as default:
http://twitter.github.io/bootstrap