JPEG2000 Support with IIPImage / IIPSrv and Kakadu 7.5 on Ubuntu 14.04 LTS via Lighttpd 1.4

Been having fun getting IIPSrv working with JPEG2000 support via Kakadu. After a lot of head scratching yesterday, I got a working config together using Lighttpd's FastCGI support.

Figured I'd best note down what steps I took so that others may benefit ... or indeed future me. (HI, FUTURE ME!)

Be warned, these are notes scavenged from my bash history, so some steps may have been more efficiently done. I'm a little rusty on my Linux and slightly clueless about some package management.

Setup

1. Install Ubuntu 14.04 Server LTS  (tested under 12.04 LTS as well)
2. Select LAMP and OpenSSH server roles from installation menu (or use Tasksel on an existing server)
3. Once installed, log in and update packages using: sudo apt-get update
4. Then: sudo apt-get upgrade
5. Reboot, etc.

6. Add some packages for compilation and other things:

6.1 sudo apt-get install make
6.2 sudo apt-get install g++
6.3 sudo apt-get install openjdk-7-jdk
6.4 sudo apt-get install vsftpd
6.5 sudo apt-get install git
6.6 sudo apt-get install autoconf
6.7 sudo apt-get install libtool
6.8 sudo apt-get install libfcgi-dev libjpeg-dev libtiff-dev

Transferring files

7. I installed an FTP server (vsftpd) to get the package of Kakadu's SDK and sample images on to the server, but it needed a little configuration first:

7.1 Edited /etc/vsftpd.conf: 

Uncommented the following lines:

local_enable=YES
write_enable=YES

Comment the following line:

#anonymous_enable=YES

7.2 sudo service vsftpd restart

Compiling Kakadu

I transferred the Kakadu SDK onto the box using FTP. The location that Kakadu is placed is important. I ended up putting it into a "kakadu" folder in my user's home folder, then after compiling it I placed it under /opt/kakadu and renamed the SDK folder itself to /opt/kakadu/kakadu-7.5 (previously it was v7_5-01573L).

The reason it is important is that IIPSrv is compiled against the Kakadu binaries, so they must be in a place that is accessible to the Lighttpd process.

The first time I did this I hit a brick wall as Lighttpd's mod_fastcgi module would say it had failed to start and that the child process exited with a status of 127, and it was not until I had taken the Lighttpd 1.4 sources from GitHub and ran the version I compiled myself that it told me exactly why - because it couldn't access the Kakadu library.

The problem was that I had compiled both Kakadu and IIPSrv in my user's home directory, so when I installed the version of IIPSrv I had compiled, it was still referencing my home directory which was inaccessible from the Lighttpd's www-data user. By placing the Kakadu SDK under the /opt/kakadu folder before I compiled IIPSrv, the binary would reference the Kakadu libraries in a place it can have permission to read them.

The Kakadu makefiles in three directories that I compiled separately as I couldn't get the makefile in the root 'make' folder of the SDK working.

I had to edit the makefiles as there is currently an incompatibility between the latest version of IIPSrv (0.99) and Kakadu 7.5, explained here: https://github.com/ruven/iipsrv/issues/22 - the solution for which is to fiddle with the compilation flags for Kakadu with regards to SSSE3 support. However, I found that I had to make a further modification involving the AVX2 support to get a clean compile. So:

10. Edit makefiles for optimisation support:

(from the Kakadu SDK root folder)

10.1 cd coresys/make
10.2 Edit Makefile-Linux-x86-64-gcc:

Uncomment the following lines:

C_OPT += -DKDU_NO_SSSE3
C_OPT += -DKDU_NO_AVX2

Comment the following lines:

#SSSE3FLAGS = -mssse3
#AVX2FLAGS = -mavx2 -mfma 

10.3 make -f ./Makefile-Linux-x86-64-gcc
10.4 cd ../../managed/make
10.5 Edit Makefile-Linux-x86-64-gcc:

Uncomment the following lines:

C_OPT += -DKDU_NO_SSSE3
C_OPT += -DKDU_NO_AVX2

Comment the following lines:

#SSSE3FLAGS = -mssse3
#AVX2FLAGS = -mavx2 -mfma 

10.6 export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
10.7 make -f ./Makefile-Linux-x86-64-gcc

This makefile tries to create a folder called "java" above the kakadu source folder, which is a bit outrageous. Hence, the reason why I compile the Kakadu source in my user's folder then move it to /opt/kakadu/kakadu-7.5 so it is more accessible.

10.8 cd ../../apps/make
10.9 Edit Makefile-Linux-x86-64-gcc:

Uncomment the following lines:

C_OPT += -DKDU_NO_SSSE3
C_OPT += -DKDU_NO_AVX2

Comment the following lines:

#SSSE3FLAGS = -mssse3
#AVX2FLAGS = -mavx2 -mfma 

 10.10 make -f ./Makefile-Linux-x86-64-gcc

Now, hopefully, you should have the object files for Kakadu compiled.

11. Move Kakadu SDK to a better place and fix permissions:

11.1 cd ~/kakadu   (or whichever folder the SDK is sitting beneath in your home folder)
11.2 sudo mkdir /opt/kakadu
11.3 sudo mv v7_5-01573L /opt/kakadu/kakadu-7.5
11.4 sudo find /opt/kakadu -type d -exec chmod 755 {} +

This makes all the directories under /opt/kakadu accessible.

11.5 sudo find /opt/kakadu -type f -exec chmod +r {} +

This makes all the files under /opt/kakadu world-readable. Please note that I would bet a whole Mars bar that there are more graceful ways of doing this.

Compiling IIPSrv

The version of IIPSrv I used was from GitHub, here's a link to the exact commit - https://github.com/ruven/iipsrv/commit/5edcdb7bb8b22538f467da6bbefbbae2649b59c6

12. Get IIPSrv source from GitHub:

12.1 cd ~
12.2 git clone https://github.com/ruven/iipsrv.git

13. Configure IIPSrv:

13.1 cd iipsrv
13.2 ./autogen.sh
13.3 ./configure --with-kakadu=/opt/kakadu/kakadu-7.5

At the end of configuration, you should hopefully see a message saying that JPEG2000 support is enabled. If not, then check through the configure output in scroll-back for clues.

14. Compile IIPSrv:

14.1 make

The output from a successful make should be a file called src/iipsrv.fcgi. This will be utilised in the next steps.

Preparing FastCGI for Lighttpd

A few things to do here, so:

15. Install Lighttpd:

15.1 sudo apt-get install lighttpd

16. Prepare a FastCGI area for Lighttpd:

16.1 cd /var/www
16.2 sudo mkdir localhost
16.3 sudo chmod 755 localhost
16.4 cd localhost
16.5 sudo mkdir fcgi-bin
16.6 sudo chmod 755 fcgi-bin
16.7 sudo mkdir data
16.8 sudo chmod 755 data

17. Copy IIPSrv binary to FastCGI area:

17.1 sudo cp ~/iipsrv/src/iipsrv.fcgi /var/www/localhost/fcgi-bin/.
17.2 sudo chmod 755 /var/www/localhost/fcgi-bin/iipsrv.fcgi

18. Edit /etc/lighttpd/lighttpd.conf:

Add the following section, as per http://iipimage.sourceforge.net/documentation/server/

fastcgi.server = ( "/fcgi-bin/iipsrv.fcgi" =>
  (( "host" => "127.0.0.1",
     "port" => 9000,
     "check-local" => "disable",
     "min-procs" => 1,
     "max-procs" => 1,
     "bin-path" => "/var/www/localhost/fcgi-bin/iipsrv.fcgi",
     "bin-environment" => (
        "LOGFILE" => "/tmp/iipsrv.log",
        "VERBOSITY" => "5",
        "MAX_IMAGE_CACHE_SIZE" => "10",
        "FILENAME_PATTERN" => "_pyr_",
        "JPEG_QUALITY" => "50",
        "MAX_CVT" => "3000"
      )
  ))
)

And I changed the HTTP port to 8080 as I couldn't be bothered uninstalling Apache2:

server.port                 = 8080

19. Enable mod_fastcgi:

19.1 sudo lighttpd-enable-mod fastcgi

20. Restart Lighttpd:

20.1 sudo /etc/init.d/lighttpd stop
20.2 sudo /etc/init.d/lighttpd start

"force-reload" as a command to that script probably works too, but it didn't for me.

Sample images

I found sample TIF and JPEG2000 images at:

from http://iipimage.sourceforge.net/documentation/images/
http://merovingio.c2rmf.cnrs.fr/iipimage/PalaisDuLouvre.tif
(1.87mb TIF file 4000x828)

from http://www.microimages.com/gallery/jp2/
http://www.microimages.com/gallery/jp2/CB_TM_QQ432.jp2
(17.8mb JPEG2000 file 3164x2982)

NOTE: This was unavailable last time I checked, try this one instead:
from http://www.loc.gov/item/cmns000145/
http://cdn.loc.gov/service/afc/afc1999008/afc1999008_crf_lec05011.jp2
(1.8mb JPEG2000 file 2048x1402)

21. I transferred these to the box and then moved them into the data folder I made in stage 16:

21.1 sudo cp ~/PalaisDuLouvre.tif /var/www/localhost/data/sample.tif
21.2 sudo cp ~/CB_TM_QQ432.jp2 /var/www/localhost/data/sample.jp2
21.3 sudo chmod 644 /var/www/localhost/data/*

Time to test

Okay, theoretically now you should have a functional Lighttpd instance. This can be tested by pointing a web browser at your VM / box on whatever HTTP port that your /etc/lighttpd/lighttpd.conf is set to (bearing in mind that Apache2 will be on port 80 already.)

So in my case I did:

http://192.168.137.80:8080/

And got the Lighttpd placeholder page. If you see that then you know that the FastCGI config has worked as otherwise Lighttpd will bomb out on startup. If you don't see the placeholder page then you need to look at the entries in the /var/log/lighttpd/error.log file for clues.

Next, try the IIPSrv module with a TIF:

http://192.168.137.80:8080/fcgi-bin/iipsrv.fcgi?FIF=/var/www/localhost/data/sample.tif&HEI=2000&CVT=jpeg

Hopefully you should see a panoramic picture of the Lourve ...

If that works, then try with the JPEG2000 file:

http://192.168.137.80:8080/fcgi-bin/iipsrv.fcgi?FIF=/var/www/localhost/data/sample.jp2&HEI=2000&CVT=jpeg

If that works - you've done it!