Wednesday, 19 February 2014

TeamCity and Mercurial Feature Branches

Rationale


I wanted to write this blog as I have used up a small quotient of my life attempting to get TeamCity and Mercurial to work together, and somebody somewhere should benefit from my findings.

The company I work for use Microsoft .Net with Mercurial as their de facto version control system, which I am completely fine with. Mercurial is a wonderful VCS and especially so if you've lived your arcana of SourceSafe, TFS and Subversion. I'm not here to defend our choice in not moving to Git because frankly I can't be bothered. Basically, if you use Mercurial and you want to use TeamCity as a build and deployment server with feature branches and something which is a little bit like continuous integration if you squint your eyes, here's a way I found of making them work together in something approaching harmony.

A Tale Of Two Build Chains

So, initially we had this idea of having a build chain permanently churning "default" in Debug mode, and another which would build anything with the prefix "Release-" on its branch name in Release mode. This kinda worked for a month or so but then it was starting to make it difficult to logically separate features since people started forgetting to use the prefix, then pointing out that their feature on the UAT server clearly wasn't ready to be released so there was a nomenclature problem there. Further, when we started trying to coerce the VCS roots to only consider certain feature branches using the Default Branch and Branch Specifications, things started going really awry and I realised that we were losing control of how TeamCity was responding, including not being able to tell what had just been deployed and why.

This week I've been anxious to get anything working, let alone things like this that have been driving me mad, so when I came in on Monday I ditched what I was currently doing and rolled up sleeves to work on this problem.

It turns out that our thinking on this was pretty much backwards and then things were getting more out of whack with the various hacks we were trying to try and make it work. Limiting the build chains with a Branch Specification seems to be actively fighting against how the TeamCity feature was designed, so I needed to re-think how to achieve what we needed.

After re-reading the TeamCity 8.x documentation around Mercurial, which swear blind that this should all just work, I decided to try a new topology with respect to the original two build chains we had set up. I got the feeling that the builds produced by a build chain could be used as a palette by the deployment chains, and to that end I re-tasked Trunk and Release into chains that would consider *every* branch, but would compile them with Debug or Release mode, respectively.

Setup for "All branches (debug mode)":
VCS Root:
  Name: "Trunk" (This isn't the right name for this now, I realise)
  Mercurial, etc
  Default branch: [blank]
  Branch specification: +:*
Build steps:
  1) Build Solution (Debug mode)
  2) Run Unit Tests (Debug mode)
Build triggers:
  Branch filter: +:*  (trigger one build per each VCS check-in, include several check-ins if from same committer)
Build parameters: (various things but importantly:)
  system.Configuration = Debug

Setup for "All branches (release mode)":
VCS Root:
  Name: "Release"
  Mercurial, etc
  Default branch: [blank]
  Branch specification: +:*
Build steps:
  1) Build Solution (Release mode)
  2) Run Unit Tests (Release mode)
Build triggers:
  Branch filter: +:*  (trigger one build per each VCS check-in, include several check-ins if from same committer)
Build parameters: (various things but importantly:)
  system.Configuration = Release

In all fairness, I could attach the same VCS Root to both of these build chains and I think it would work fine. I kept these as separate Roots since it was how it was originally and that gives the option of pointing it at a different source repository, although Christ knows why we'd do that.

Deployment Chains - 1 - Sorta-Kinda Continuous Integration

In a vague nod towards doing some kind of Continuous Integration, we want to have our two testing servers (Testing and UAT) continuously updated with the latest build from certain feature branches. I found I could do this by using the Build Triggers functionality. This appears to be because the Branch Specification parameter in the build chains uses a wildcard - that wildcard becomes a logical branch name which the Build Trigger's Branch filter can then pick up.

Sub-project: Continuous Integration - Automatic deployments to Testing / UAT
Setup for Deploy Testing - "Feature Branch 1"
(I started renaming the chains to what branch they were tuned to)
VCS Root: "Trunk"  (as above)
Build steps:
  1) Compile and deploy
  2) Remove output directory
Build Triggers:
  Branch filter: +:Feature Branch 1
  Trigger a build on each check-in
  Include several check-ins in a build if they are from the same committer
Dependencies:
  Depend on: "All branches (debug mode)" (the Debug mode build chain)
  Do not run new build if there is a suitable one
  Only use successful builds from suitable ones
Build parameters: (whatever you need for your deploy job, plus:)
  system.Configuration = Testing

Setup for Deploy UAT - "Feature Branch 2"
VCS Root: "Trunk"  (as above)
Build steps:
  1) Compile and deploy
  2) Remove output directory
Build Triggers:
  Branch filter: +:Feature Branch 2
  Trigger a build on each check-in
  Include several check-ins in a build if they are from the same committer
Dependencies:
  Depend on: "All branches (debug mode)" (the Debug mode build chain)
  Do not run new build if there is a suitable one
  Only use successful builds from suitable ones
Build parameters: (whatever you need for your deploy job, plus:)
  system.Configuration = UAT

With these in place, a commit to "Feature Branch 1" will spawn three builds:
  1. "Feature Branch 1" in Debug mode on "All branches (debug mode)" build chain
  2. "Feature Branch 1" in Release mode on "All branches (release mode)" build chain
  3. "Feature Branch 1" deployed in Debug mode from Deploy Testing - "Feature Branch 1" build chain
The deployment job will queue until the "All branches (debug mode)" build chain has completed successfully.

Similarly, a commit to "Feature Branch 2" will spawn the Debug, Release and deployment builds, but Deploy UAT - "Feature Branch 2" will run instead.

Deployment Chains - 2 - Manual Deployments

Along with the Testing and UAT deployment chains that do this pretence of Continuous Integration, we also wanted to ability to manually deploy particular feature branches across our Staging and dual-head Live environments.

For this, I have attached the Release mode VCS Root, so they will receive the correctly compiled build, but I have not added any Build Triggers, relying instead upon the GUI in TeamCity to request the appropriate feature branch from the provided drop-down menus.

The dependency means that it is still requires a successful build plus unit test run to deploy.

Setup for Deploy Staging
VCS Root: "Release"  (as above)
Build steps:
  1) Compile and deploy
  2) Remove output directory
Dependencies:
  Depend on: "All branches (release mode)" (the Release mode build chain)
  Do not run new build if there is a suitable one
  Only use successful builds from suitable ones
Build parameters: (whatever you need for your deploy job, plus:)
  system.Configuration = Staging

Conclusion

I'm utterly relieved that I'm getting some sense out of TeamCity now. I've really fought for keeping the use of feature branches and to make sure that we continue with Mercurial, so this has felt like some kind of small victory. The automatic "continuous integration" deployments to Testing and UAT work well (when I remember to tick all the right boxes in the Build Triggers), and they have meant that I haven't had to escalate privileges for a remote-working developer. The only drawback is that you have to fiddle with the Build Triggers to be able to "tune" those deployment chains to a particular feature branch, but I think that's a decent trade-off.

Although the Staging and Live deployment chains involve a manual selection step, this is basically acceptable considering that the rest of the build server appears to be running in a sane fashion now.

Well, apart from one thing. When feature branches are added and their first commits are pushed, Team City will spawn a build per group of commits based on user, and that has meant that I've occasionally spotted 20+ builds being added to the queue including a bunch of re-deployments. In all fairness, this isn't the end of everything, but it can be a little bit irritating.

So, yeah. TeamCity and Mercurial feature branches. It does actually work as long as you think about it right.