Singularity (Apptainer) on the HPC

Maintainer: Dianne Patterson Ph.D. dkp @ arizona.edu

Date Created: 2019_07_23

Date Updated: 2023_03_07

Tags: UNIX, containers, bids

OS: UNIX

Introduction

The goal of this page is to describe running and building containerized BIDS workflows on the HPC. Containerized technologies are changing fast, so check the Date Updated above to ensure you are looking at reasonably current documentation.

For security reasons:

• Docker cannot be used on the HPC, but Docker containers can be converted to Singularity containers which can be run on the HPC.

• Now that the University uses Apptainer, it may be possible to build a Singularity container from a Singularity recipe file on the HPC, but we can still build a Singularity container by pulling an existing Docker container such as a BIDS container.

For details of working with BIDS containers available to you on our HPC, see the page Bids Containers.

Build Singularity (or Apptainer) Containers from Dockerhub

Typically, you will want to convert a BIDS Docker container to Singularity for use on the HPC. Build a Singularity *.sif container on the HPC from an existing Docker container posted on dockerhub:

interact
singularity build lolcow.sif docker://godlovedc/lolcow

Above, we run two commands:

• The first command starts an interactive session. It assumes you have an alias in your .bashrc that looks like this, but references your account:

  alias interact='srun --nodes=1 --ntasks=1 --mem-per-cpu=4GB --time=01:00:00 --job-name=interact --account=dkp --partition=standard --pty bash -i'
  

• The second command singularity build lolcow.sif docker://godlovedc/lolcow builds a singularity file lolcow.sif (call it whatever you want), by pulling the associated Docker container from Dockerhub.

• Once the container has built, you can run the Singularity container thus:

  singularity run lolcow.sif
  

• Every time you run lolcow.sif, you see a different clever saying produced by an ascii cow.

• Terminate your interactive session when you have seen enough cute cow sayings:

  exit
  

Build Singularity Containers from Recipes

• Singularity has been replaced by apptainer on the HPC. They are very similar, and you can use singularity in your calls (but you’ll actually be getting apptainer).

• Remote builds via SyLabs are no longer supported. Instead, in many cases you may build your image directly on a compute node using

  apptainer build local_image.sif container.recipe

• For example, there is a Singularity recipe called cow.def in /groups/dkp/neuroimaging/scripts

• Start an interactive terminal session (a few minutes should be fine) and build the cow container from the recipe like this:

  interact
  apptainer build cow.sif /groups/dkp/neuroimaging/scripts/cow.def
  

• Try it like this:

  apptainer run cow.sif
  

Running a BIDS Singularity container

• Running a job like fMRIPrep or mrtrix3_connectome will likely take overnight (depending on your options).

• It is better to write a batch job specification that overestimates what you need to finish running a subject than to underestimate.

• If you underestimate, the job will be killed before it is finished (so, 3 hours is not enough, 2 CPUs might not be enough).

• If you overestimate, you’ll be charged only for the time you actually use (so, maybe try 48 hours?).

• If you vastly overestimate (e.g 200 hours) then your job may not start until the HPC has that much time available to allocate to you. So, it may take some experimentation to find the right balance.

• If you have run the Docker container on your local machine, that will give you a rough estimate. You could double that number of hours and see how it goes.

• See BIDS containers for details about individual BIDS containers available on the U of A HPC system and the corresponding run scripts and resource estimates.

Host OS and Apptainer container OS Interactions

For the most part, what is inside your container is separate from what is outside your container. That’s the point of containers. However, there can be interactions. Consider the following case:

MRtrix3_connectome is built on Ubuntu 18.04. You can see this by looking at the first line in the dockerfile. The MRtrix3_connectome container will not run on a host Linux system running an older version of CentOS. It is possible to have a container built on a newer version of linux that is not backward-compatible with the host operating system! So keep that in mind if a container will not run on a particular machine.

To determine the version of Linux that is running, type the following:

cat /etc/redhat-release

As of Fall 2021, all UofA hosts are running CentOS7 (and MRtrix3_connectome is compatible with CentOS7).

Building Large Containers

The process for building lolcow.sif works fine for a small container, but for large containers, you run into two problems: Running out of Space

Note

You do not have to build containers yourself if you are happy with the ones provided in /contrib/singularity/shared/neuroimaging. You must be in interactive mode to access this directory!

Running Out of Space to Build Containers

On our University of Arizona HPC, even if you have access to many GB of space, the space IN YOUR HOME DIRECTORY is likely to be very small (~50 GB). When apptainer builds a *.sif file, as illustrated for lolcow.sif, it defaults to pulling tarballs from Dockerhub into the hidden .apptainer subdirectory in your home directory. Apptainer then quickly runs out of available space in your home directory and then complains and dies. Below I describe how to circumvent this problem:

• Set the apptainer cachedir TO a directory with more space than your home dir (e.g., /groups/dkp/ in my case) by adding information to your shell configuration file. I added the lines below to my .bashrc:

  export APPTAINER_CACHEDIR=/groups/dkp/build_sif/cache
  

If you want to pull large containers, you should modify this lines for your own directory (not dkp please). Now you can build the apptainer container for your own group and chosen directory.

Running Out of Time to Build Containers

As described in Running your first SLURM Batch Job when you log on to the HPC, you are in login mode. In login mode, you are not using any of your allocated monthly time allowance, but you can only do small tasks. The HPC will kill your job if you try to build a large Apptainer container (like any of the BIDS containers). You should start an interactive session to build your container. I’ve occasionally seen it take more than an hour to build an apptainer container from Dockerhub on the HPC.

BIDS Containers