Docking to Docker

Another month has passed and we have been very busy in ARCCA at Cardiff University with installation of the new hardware and installing new services. These are a few of the reasons why updates in the past few weeks have been infrequent but as I keep saying to myself:

Must do better…

However recently we have been thinking about how we could use one bit of technology to deliver services to our community – this technology is called Docker.

What is Docker?
If you have been reading IT related news sites for the past few years then “Docker” should be a fairly familiar technology. Docker could be described as a way to revolutionise how we can virtualise services. The fairly traditional method of virtualisation is to create a virtual machine (a VM) and to setup a service within a whole operating system running in its own (virtual) machine in isolation from anything else. With Docker we can now create a container within an operating system for our service, including the dependencies and run this on any other system which supports Docker – thus avoiding having to spend time and effort managing and resourcing another machine (albeit virtual).

Using this to deliver a better service
In high-performance computing (HPC) one of the most time-consuming aspects is to compile fairly technical software or to optimise the software for your particular platform. RPM and DEBs do not help as they tend to only be well supported for general-purpose computing.

Docker would allow software (and its dependencies) to be installed in isolation and run on systems across a HPC cluster. When a software package needs updating (possibly with dependencies) this can be done, knowing that it will not interact with other installed software. Technologies such as the Galaxy project (mentioned in a previous post) has already started delivering methods to allow Docker to be used to manage the extra tools that can be provided by the community. Integrating Docker with a provisioning service such as Ansible will provide greater confidence that we can launch and deliver any service, anywhere.

How does Docker perform – does it perform “highly” for HPC?
This is something which we would like to get to grips with. Docker is reported to have minimal system overhead (unlike virtualisation). How Docker interacts with networks is another issue, where software using MPI over Infiniband is common. There is much to learn and test, especially how best to integrate it with a job scheduler. PBS Pro has something called provisioning where the system can be setup before a job is run, maybe docker could be treated automatically like that rather than the user having to know how to run a docker application.

We feel that Docker is worth investigating and to benchmark against other methods to manage software (maybe that is for another post in the future). With Docker being fully supported by big names such as Redhat and Microsoft it will definitely be around for the longer term than other technologies that can come and go quickly.