Singularity on the HPC

Maintainer: Dianne Patterson Ph.D. dkp @ email.arizona.edu
Date Created: 2019_07_23
Date Updated: 2020_04_06
Tags: UNIX, containers, bids
OS: UNIX

Introduction

The goal of this page is to describe running and building containerized BIDS workflows on the HPC. Containerized technologies are changing fast, so check the Date Updated above to ensure you are looking at reasonably current documentation.

For security reasons:

  • Docker cannot be used on the HPC, but Docker containers can be converted to Singularity containers which can be run on the HPC.
  • It is also not possible to build a Singularity container from a Singularity recipe file on the HPC, but we can build a Singularity container by pulling an existing Docker container such as a BIDS container.

For details of working with BIDS containers available to you on our HPC, see the page Bids Containers.

Build Singularity Containers from Dockerhub

To get a basic idea of how singularity containers work, you should try this. You do not even have to be in interactive mode. You can build a Singularity *.sif container on the HPC from an existing Docker container posted on dockerhub:

module load singularity
module list
module help
singularity build lolcow.sif docker://godlovedc/lolcow

Above, we run four commands:

  • The first command, module load singularity, makes singularity tools available to us on the HPC by loading the associated module.

  • The second command module list lists what modules we have loaded. This is not strictly necessary, but it will tell you which version of singularity has been loaded.

  • The third command module help is also not necessary, but tells you more about the module command and how to use it.

  • The fourth command singularity build lolcow.sif docker://godlovedc/lolcow builds a singularity file lolcow.sif (call it whatever you want), by pulling the associated Docker container from dockerhub.

  • Once the container has built, you can run the Singularity container thus:

    singularity run lolcow.sif

  • You should see a clever saying produced by an ascii cow, something different every time you run it.

Build Singularity Containers from Recipes

  • If you have a Singularity recipe file, you cannot build it on the HPC.
  • You can build a Singularity container from a recipe on Sylabs Cloud. I believe there are some size and time limits, but I have successfully built the 1.6 GB BIP2 container in 7 minutes on Sylabs Cloud.
  • If your recipe is too resource-intensive to build on Sylabs cloud, you can use Cyverse Atmosphere if you sign up for an account. See Singularity on Cyverse.
  • You could build a linux machine and install Singularity (not for the faint of heart).

Running a BIDS Singularity container

  • Running a job like fMRIPrep or mrtrix3_connectome will likely take overnight (depending on your options). It is better to write a batch job specification that overestimates what you need to finish running a subject than to underestimate.
  • If you underestimate, the job will be killed before it is finished (so, 3 hours is not enough, 2 CPUs might not be enough).
  • If you overestimate, you’ll be charged only for the time you actually use (so, maybe try 48 hours?).
  • If you vastly overestimate (e.g 200 hours) then your job may not start until the HPC has that much time available to allocate to you. So, it may take some experimentation to find the right balance.
  • If you have run the Docker container on your local machine, that will give you a rough estimate. You could double that number of hours and see how it goes.
  • See BIDS containers for details about individual BIDS containers available on the U of A HPC system and the corresponding run scripts and resource estimates.

Host OS and Singularity Interactions

For the most part, what is inside your container is separate from what is outside your container. That’s the point of containers. However, there can be interactions. Consider the following case:

MRtrix3_connectome is built on Ubuntu 18.04. You can see this by looking at the first line in the dockerfile.

Ocelote is (at the time of this writing) on CentOS release 6.10:

[dkp@login2 dkp]$ cat /etc/issue
CentOS release 6.10 (Final)

[dkp@login2 dkp]$ uname -a
Linux login2 2.6.32-754.12.1.el6.x86_64  #1  SMP Tue Apr 9 14:52:26 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[dkp@login2 dkp]$ cat /etc/*elease
CentOS release 6.10 (Final)
Cluster Manager v8.1
slave
LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
CentOS release 6.10 (Final)
CentOS release 6.10 (Final)

If I try to run even the simplest command calling the mrtrix3_connectome.sif container from the ocelote shell prompt, I get this message:

[dkp@login2 singularity-images]$ singularity run mrtrix3_connectome.sif -v
FATAL: kernel too old

If I log in to el gato, several things are different. First cat /etc/issue isn’t helpful on el gato:

[dkp@login dkp]$ cat /etc/issue
\S
Kernel \r on an \m

The other two commands work fine on el gato:

[dkp@login dkp]$ uname -a
Linux login 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[dkp@login dkp]$ cat /etc/*elease
CentOS Linux release 7.6.1810 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.6.1810 (Core)
CentOS Linux release 7.6.1810 (Core)

As you can see, the operating system is much newer on el gato (CentOS Linux release 7.6.1810 (Core)) than on Ocelote (CentOS release 6.10 (Final)). It turns out that el gato is new enough to handle a container built on Ubuntu 18.04. When I ask for the version of the mrtrix3_connectome.sif from el gato, I get a reasonable answer:

[dkp@login singularity-images]$ singularity run mrtrix3_connectome.sif -v
BIDS-App 'MRtrix3_connectome' version 0.4.2

Note

As of 2/14/2020: Some nodes on Ocelote are available with CentOS7. See the Including CentOS7 documentaion. To use these nodes, you must add os7=True to your qsub select statement, e.g., select=1:ncpus=28:mem=12gb:np100s=1:os7=True

Building Large Containers

The process for building lolcow.sif works fine for a small container, but for large containers, you run into two problems: Running out of Space and Running out of Time.

Note

You do not have to build containers yourself if you are happy with the ones I have built. Look in /extra/dkp/singularity-images. The permissions should be such that you can run any container I have in here.

Running Out of Space

On our University of Arizona HPC and on Cyverse virtual machines (Atmosphere), even if you have access to many GB of space, the space IN YOUR HOME DIRECTORY is likely to be very small. When Singularity builds a *.sif file, as illustrated for lolcow.sif, it pulls tarballs from Dockerhub into the hidden .singularity/docker and .singularity/cache subdirectories in your home directory. Singularity quickly runs out of available space in your home directory and then complains and dies. Below I describe how to circumvent this problem:

  • Move .singularity/docker and .singularity/cache FROM your home directory TO a directory with more space (e.g., /extra/dkp in my case)

  • Add information to your shell configuration file to tell Singularity where to find these important directories. I added the lines below to my .bash_profile:

    export SINGULARITY_PULLFOLDER=/extra/dkp/build_sif/docker
export SINGULARITY_CACHEDIR=/extra/dkp/build_sif/cache

You should modify these lines for your own directory (not dkp please). Now you can submit a batch job to build the Singularity container, for example you could modify buildSIF.sh for your own group and chosen directory.

Note

As of 2/14/2020: We are expecting to have 50 GB of space in home sometime during the Spring of 2020 as our new cluster and enhanced storage comes online. You ought to be able to pull large Docker containers and build them into Singularity containers with this much space (unless you fill up the space). Other Singularity-related tools for building and storage may be coming our way, so stay vigilant.

Running Out of Time

As described in Batch Jobs and Interactive Mode when you log on to the HPC, you are in login mode. In login mode, you are not using any of your allocated monthly time allowance but you can only do small tasks (like building lolcow.sif). The HPC will kill your job if you try to build a large Singularity container (like any of the BIDS containers). One solution is to submit a batch job with a script like the buildSIF.sh (this builds the fMRIPrep container). Batch jobs are special scripts submitted with qsub. These scripts combine instructions to the PBS batch system and bash commands. Submit the script with qsub, like this:

qsub buildSIF.sh

If you want to actually work at the commandline (like you did with the lolcow build), then you have to submit a batch job that puts you into interactive mode at the command line.