Singularity on the HPC

Maintainer: Dianne Patterson Ph.D. dkp @ email.arizona.edu
Date Created: 2019_07_23
Date Updated: 2022_03_10
Tags: UNIX, containers, bids
OS: UNIX

Introduction

The goal of this page is to describe running and building containerized BIDS workflows on the HPC. Containerized technologies are changing fast, so check the Date Updated above to ensure you are looking at reasonably current documentation.

For security reasons:

  • Docker cannot be used on the HPC, but Docker containers can be converted to Singularity containers which can be run on the HPC.

  • It is also not possible to build a Singularity container from a Singularity recipe file on the HPC, but we can build a Singularity container by pulling an existing Docker container such as a BIDS container.

For details of working with BIDS containers available to you on our HPC, see the page Bids Containers.

Build Singularity Containers from Dockerhub

Typically, you will want to convert a BIDS Docker container to Singularity for use on the HPC. Build a Singularity *.sif container on the HPC from an existing Docker container posted on dockerhub:

interact
singularity build lolcow.sif docker://godlovedc/lolcow

Above, we run two commands:

  • The first command starts an interactive session. It assumes you have an alias in your .bashrc that looks like this, but references your account:

    alias interact='srun --nodes=1 --ntasks=1 --mem-per-cpu=4GB --time=01:00:00 --job-name=interact --account=dkp --partition=standard --pty bash -i'
    
  • The second command singularity build lolcow.sif docker://godlovedc/lolcow builds a singularity file lolcow.sif (call it whatever you want), by pulling the associated Docker container from Dockerhub.

  • Once the container has built, you can run the Singularity container thus:

    singularity run lolcow.sif
    
  • Every time you run lolcow.sif, you see a different clever saying produced by an ascii cow.

  • Terminate your interactive session when you have seen enough cute cow sayings:

    exit
    

Build Singularity Containers from Recipes

  • Because you do not have administrative privileges, you cannot build a Singularity container from a Singularity recipe file on the HPC.

  • You can build a Singularity container from a recipe on Sylabs Cloud. There are some size and time limits, but I have successfully built the 1.6 GB BIP2 container in 7 minutes on Sylabs Cloud.

  • If your recipe is too resource-intensive to build on Sylabs cloud, you can use Cyverse Atmosphere if you sign up for an account. See Singularity on Cyverse.

  • You could build a Linux machine and install Singularity (not for the faint of heart).

Running a BIDS Singularity container

  • Running a job like fMRIPrep or mrtrix3_connectome will likely take overnight (depending on your options).

  • It is better to write a batch job specification that overestimates what you need to finish running a subject than to underestimate.

  • If you underestimate, the job will be killed before it is finished (so, 3 hours is not enough, 2 CPUs might not be enough).

  • If you overestimate, you’ll be charged only for the time you actually use (so, maybe try 48 hours?).

  • If you vastly overestimate (e.g 200 hours) then your job may not start until the HPC has that much time available to allocate to you. So, it may take some experimentation to find the right balance.

  • If you have run the Docker container on your local machine, that will give you a rough estimate. You could double that number of hours and see how it goes.

  • See BIDS containers for details about individual BIDS containers available on the U of A HPC system and the corresponding run scripts and resource estimates.

Host OS and Singularity container OS Interactions

For the most part, what is inside your container is separate from what is outside your container. That’s the point of containers. However, there can be interactions. Consider the following case:

MRtrix3_connectome is built on Ubuntu 18.04. You can see this by looking at the first line in the dockerfile. The MRtrix3_connectome container will not run on a host Linux system running an older version of CentOS. It is possible to have a container built on a newer version of linux that is not backward-compatible with the host operating system! So keep that in mind if a container will not run on a particular machine.

To determine the version of Linux that is running, type the following:

cat /etc/redhat-release

As of Fall 2021, all UofA hosts are running CentOS7 (and MRtrix3_connectome is compatible with CentOS7).

Building Large Containers

The process for building lolcow.sif works fine for a small container, but for large containers, you run into two problems: Running out of Space

Note

You do not have to build containers yourself if you are happy with the ones provided in /contrib/singularity/shared/neuroimaging. You must be in interactive mode to access this directory!

Running Out of Space to Build Containers

On our University of Arizona HPC and on Cyverse virtual machines (Atmosphere), even if you have access to many GB of space, the space IN YOUR HOME DIRECTORY is likely to be very small. When Singularity builds a *.sif file, as illustrated for lolcow.sif, it pulls tarballs from Dockerhub into the hidden .singularity/docker and .singularity/cache subdirectories in your home directory. Singularity quickly runs out of available space in your home directory and then complains and dies. Below I describe how to circumvent this problem:

  • Move .singularity/docker and .singularity/cache FROM your home directory TO a directory with more space (e.g., /groups/dkp/ in my case)

  • Add information to your shell configuration file to tell Singularity where to find these important directories. I added the lines below to my .bash_profile:

    export SINGULARITY_PULLFOLDER=/groups/dkp/build_sif/docker
    export SINGULARITY_CACHEDIR=/groups/dkp/build_sif/cache
    

You should modify these lines for your own directory (not dkp please). Now you can build the Singularity container for your own group and chosen directory.

Warning

Your HPC home directory has 50 GB of space. You ought to be able to build a Singularity container by pulling from Dockerhub UNLESS you fill up your SINGULARITY_PULLFOLDER with tarballs. You accumulate these tarballs every time you build a Singularity container from Dockerhub. These tarballs are useful if you have to rebuild a container frequently (the build will not have to download as much) BUT they could easily fill up your home directory with hidden files.

Running Out of Time to Build Containers

As described in Running your first SLURM Batch Job when you log on to the HPC, you are in login mode. In login mode, you are not using any of your allocated monthly time allowance, but you can only do small tasks. The HPC will kill your job if you try to build a large Singularity container (like any of the BIDS containers). You should start an interactive session to build your container. I’ve occasionally seen it take more than an hour to build a Singularity container from Dockerhub on the HPC.