ROAR User Guide » Roar User Guide » Ways of Using Singularity Containers

Ways of Using Singularity Containers

Once a container has been built and is placed on Roar, you can use it in a variety of ways. Here, we explain a few ways, including:

Roar interactive sessions
Roar batch sessions
Interactive shells
Executing commands
Running a container
Working with files

Roar Interactive Sessions

You can start an interactive sessions with X-forwarding using the following command:

(script: qsub -l walltime=04:00:00 -l nodes=1:ppn=4 -A open -I -X)O

Warning: Do not run interactive sessions on the log-in (head) nodes. These nodes are shared among all users and can quickly be rendered unusable by computationally intensive jobs on the sessions. Use of the head nodes to perform computationally demanding tasks can lead to the programs being terminated, or deactivation of the user’s access to Roar.

Roar Batch Sessions

Here are a few examples that will run an application or command non-interactively in a PBS job file.

#!/bin/bash

#PBS -N Lammps-singularity

#PBS -A open

#PBS -l walltime=04:00:00

#PBS -l nodes=2:ppn=20

#PBS -j oe

module load gcc/5.3.1 mpich/3.2

cd $PBS_O_WORKDIR

mpirun --hostfile $PBS_NODEFILE --np 40 singularity run /path/to/lammps_mpi.simg   in.

˓→friction \

> out.friction

Interactive Shells

If you need to use many interactive tools or applications, you may want to start an interactive shell. Use the following command to do this:

[cjb47@comp-bc-0226 images]$ cat /etc/issue

Red Hat Enterprise Linux Server release 6.10 (Santiago)

Kernel \r on an \m

[cjb47@comp-bc-0226 images]$ singularity shell ubuntu_aci.simg

** NOTE: Singularity is in a testing phase on ACI-b and is currently unsupported.

Singularity: Invoking an interactive shell within container…

Singularity ubuntu_aci.simg:/gpfs/group/dml129/default/cjb47/singularity/images> cat /

˓→etc/issue

Ubuntu 16.04.5 LTS \n \l

You will be able to interact with directories that have been bound.

Executing Commands

In some cases, you may only want to run a single command from within the container. The following is an example of this:

[cjb47@comp-bc-0226 images]$ R

R version 3.5.0 (2018-04-23) — “Joy in Playing”

Platform: x86_64-redhat-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.

You are welcome to redistribute it under certain conditions.

Type ‘license()’ or ‘licence()’ for distribution details.

R is a collaborative project with many contributors.

Type ‘contributors()’ for more information and

‘citation()’ on how to cite R or R packages in publications.

Type ‘demo()’ for some demos, ‘help()’ for on-line help, or

‘help.start()’ for an HTML browser interface to help.

Type ‘q()’ to quit R.

> q()

Save workspace image? [y/n/c]: n

[cjb47@comp-bc-0226 images]$ singularity run shub://jekriske/r-base:3.4.4 R

** NOTE: Singularity is in a testing phase on ACI-b and is currently unsupported.

Progress |===================================| 100.0%

WARNING: Non existent mountpoint (directory) in container: ‘/var/singularity/mnt/

˓→final/storage/home’

WARNING: Non existent mountpoint (directory) in container: ‘/var/singularity/mnt/

˓→final/storage/work’

WARNING: Non existent mountpoint (directory) in container: ‘/var/singularity/mnt/

˓→final/gpfs/scratch’

WARNING: Non existent mountpoint (directory) in container: ‘/var/singularity/mnt/

˓→final/gpfs/group’

WARNING: Non existent mountpoint (directory) in container: ‘/var/singularity/mnt/

˓→final/var/spool/torque’

WARNING: Could not chdir to home: /storage/home/cjb47

ARGUMENT ‘R’ __ignored__

R version 3.4.4 (2018-03-15) — “Someone to Lean On”

Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.

You are welcome to redistribute it under certain conditions.

Type ‘license()’ or ‘licence()’ for distribution details.

Natural language support but running in an English locale

R is a collaborative project with many contributors.

Type ‘contributors()’ for more information and

‘citation()’ on how to cite R or R packages in publications.

Type ‘demo()’ for some demos, ‘help()’ for on-line help, or

‘help.start()’ for an HTML browser interface to help.

Type ‘q()’ to quit R.

> q()

Save workspace image? [y/n/c]: n

Running a Container

Some containers may have one or more runscripts, which allow a user to define a set of actions a container will run when it is called. For example:

[cjb47@comp-bc-0226 images]$ singularity run ./hello-world-aci.simg

** NOTE: Singularity is in a testing phase on ACI-b and is currently unsupported.

Hello there cjb47, from ICDS

A container may have multiple runscripts; in Singularity terminology, this is known as an “application.” Here is an example, including instructions for interacting with applications:

[cjb47@comp-sc-0120 images]$ singularity apps ./multiapps-aci.simg

cowsay

fortune

lolcat

[cjb47@comp-sc-0120 images]$ singularity run –app fortune ./multiapps-aci.simg -a

** NOTE: Singularity is in a testing phase on ACI-b and is currently unsupported.

FORTUNE PROVIDES QUESTIONS FOR THE GREAT ANSWERS: #19

A: To be or not to be.

Q: What is the square root of 4b^2?

[cjb47@comp-sc-0120 images]$ echo “Hello from lolcat” > file

[cjb47@comp-sc-0120 images]$ singularity run –app lolcat ./multiapps-aci.simg file

** NOTE: Singularity is in a testing phase on ACI-b and is currently unsupported.

Hello from lolcat

Working with Files

As long as there are corresponding binding points, you will be able to reach files on the host from within the container. Since Roar has user storage in non-standard locations (compared to distribution default), you will need to add the appropriate locations to a recipe.

In the following example, we need binding for ACI-b:

%post

#ACI mappings so you can access your files.

mkdir -p /storage/home

mkdir -p /storage/work

mkdir -p /gpfs/group

mkdir -p /gpfs/scratch

mkdir -p /var/spool/torque

More information on building Singularity containers

The official Singularity user guide has additional information and examples related to building and using Singularity containers.

Following are several other use cases:

Building containers with GUI support
Building an image with MPI support
Images with GPU support
Running Services
Using sandbox and writable images
Building Containers with GUI Support

A Singularity container can contain a GUI system so that the user can run a GUI application that may not easily be installed on Roar. Here is a simple example of a recipe that has a GUI available:

Bootstrap: docker

From: centos:centos7

%post

yum -y upgrade

yum -y groupinstall “X Window System”

rpm –import https://packages.microsoft.com/keys/microsoft.asc

echo -e “[code]\nname=Visual Studio Code\nbaseurl=https://packages.microsoft.com/

˓→yumrepos/vscode\nenabled=1\ngpgcheck=1\ngpgkey=https://packages.microsoft.com/keys/

˓→microsoft.asc” > /etc/yum.repos.d/vscode.repo

yum install -y nano emacs vim gedit kate nedit kwrite jed code

yum -y install which xorg-x11-fonts-Type1 liberation-sans-fonts

yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

yum -y install geany geany-plugins*

mkdir -p /storage/home

mkdir -p /storage/work

mkdir -p /gpfs/group

mkdir -p /gpfs/scratch

mkdir -p /var/spool/torque

mkdir -p /run/user/1000/dconf

touch /run/user/1000/dconf/user

%runscript

Here is a screenshot of an editor running within that container on an ACI-b compute node:

Building an Image with MPI Support

Many HPC applications require the use of MPI. Singularity supports this; however, there are requirements for this:

Install InfiniBand libraries in the container
Make the MPI version available to the container, which may be accomplished by setting a bind path to the MPI location
The application must be linked the proper version of MPI

Here is an example of the MPI build file:

BootStrap: yum

OSVersion: 7

MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/

Include: yum

%setup

cd $SINGULARITY_ROOTFS/opt

wget http://www.mpich.org/static/downloads/3.2/mpich-3.2.tar.gz

wget http://lammps.sandia.gov/tars/lammps-stable.tar.gz

%post

yum -y groupinstall “Development Tools”

mkdir -p /opt/mpich

cd /opt/mpich

tar xf ../mpich-3.2.tar.gz –strip-components 1

./configure –prefix=/usr/local |& tee log.configure

make -j |& tee log.make

make install |& tee log.make_install

mkdir -p /opt/lammps

cd /opt/lammps

tar xf ../lammps-stable.tar.gz –strip-components 1

cd src

make yes-granular |& tee log.make_yes_granular

make -j mpi |& tee log.make_mpi

#ACI mappings so you can access your files.

mkdir -p /storage/home

mkdir -p /storage/work

mkdir -p /gpfs/group

mkdir -p /gpfs/scratch

mkdir -p /var/spool/torque

%runscript

/opt/lammps/src/lmp_mpi “$@”

Here is an example of an MPI run:

[cjb47@comp-sc-0174 images]$ module load gcc/5.3.1 mpich/3.2

[cjb47@comp-sc-0174 images]$ mpirun –hostfile $PBS_NODEFILE -np 8 singularity run ./

˓→lammps_mpi.simg in.friction

** NOTE: Singularity is in a testing phase on ACI-b and is currently unsupported.

LAMMPS (22 Aug 2018)

Lattice spacing in x,y,z = 1.1327 1.96189 1.1327

Created orthogonal box = (0 0 -0.283174) to (56.6348 43.1615 0.283174)

4 by 2 by 1 MPI processor grid

Created 750 atoms

Time spent = 0.00195265 secs

Created 750 atoms

Time spent = 0.00053978 secs

Created 112 atoms

Time spent = 0.000246286 secs

Created 112 atoms

Time spent = 0.0002985 secs

750 atoms in group lo

862 atoms in group lo

750 atoms in group hi

862 atoms in group hi

150 atoms in group lo-fixed

150 atoms in group hi-fixed

300 atoms in group boundary

1424 atoms in group mobile

Setting atom values …

150 settings made for type

Setting atom values …

150 settings made for type

WARNING: Temperature for thermo pressure is not for group all (../thermo.cpp:488)

Neighbor list info …

update every 1 steps, delay 5 steps, check yes

max neighbors/atom: 2000, page size: 100000

master list distance cutoff = 2.8

ghost atom cutoff = 2.8

binsize = 1.4, bins = 41 31 1

1 neighbor lists, perpetual/occasional/extra = 1 0 0

(1) pair lj/cut, perpetual

attributes: half, newton on

pair build: half/bin/atomonly/newton

stencil: half/bin/2d/newton

bin: standard

Setting up Verlet run …

Unit style : lj

Current step : 0

Time step : 0.0025

Per MPI rank memory allocation (min/avg/max) = 3.048 | 3.049 | 3.049 Mbytes

Step Temp E_pair E_mol TotEng Press Volume

0 0.1 -3.1333672 0 -3.0920969 -1.1437663 2444.9333

1000 0.1 -3.0917089 0 -3.0504386 -0.023690937 2444.9333

2000 0.082122114 -3.0852042 0 -3.0513121 -0.43261548 2444.9333

3000 0.081076017 -3.0813279 0 -3.0478675 -0.34285337 2444.9333

4000 0.094734274 -3.0722764 0 -3.0331792 -0.31676394 2444.9333

5000 0.11433917 -3.0594274 0 -3.0122393 -0.14791034 2444.9333

6000 0.11055427 -3.046338 0 -3.0007119 -0.22376263 2444.9333

7000 0.1 -3.045677 0 -3.0044067 -0.42807494 2444.9333

8000 0.11471279 -3.0383911 0 -2.9910488 -0.30901046 2444.9333

9000 0.11181441 -3.037818 0 -2.9916719 -0.41346773 2444.9333

10000 0.10709722 -3.0390765 0 -2.9948772 -0.27785942 2444.9333

11000 0.1 -3.0404147 0 -2.9991444 -0.50482354 2444.9333

12000 0.11767118 -3.0483134 0 -2.9997502 -0.12862642 2444.9333

13000 0.11773859 -3.0569926 0 -3.0084016 -0.36892682 2444.9333

14000 0.11272521 -3.0514207 0 -3.0048987 -0.36445405 2444.9333

15000 0.10522749 -3.0506428 0 -3.0072151 -0.35624388 2444.9333

16000 0.11015277 -3.0509982 0 -3.0055378 -0.19177436 2444.9333

17000 0.1081148 -3.0478773 0 -3.003258 -0.3475267 2444.9333

18000 0.11109139 -3.0476586 0 -3.0018109 -0.33148148 2444.9333

19000 0.10911522 -3.0523013 0 -3.0072692 -0.25645655 2444.9333

20000 0.11656944 -3.0534574 0 -3.0053488 -0.33684091 2444.9333

Loop time of 6.56253 on 8 procs for 20000 steps with 1724 atoms

Performance: 658283.185 tau/day, 3047.607 timesteps/s

74.8% CPU use with 8 MPI tasks x no OpenMP threads

MPI task timing breakdown:

—————————————————————

Pair | 0.4896 | 0.57526 | 0.75825 | 10.7 | 8.77

Neigh | 0.029521 | 0.03774 | 0.052896 | 4.3 | 0.58

Comm | 3.6847 | 4.1707 | 5.064 | 25.3 | 63.55

Output | 0.0028827 | 0.0031309 | 0.0033143 | 0.3 | 0.05

Modify | 0.076676 | 0.083922 | 0.09419 | 1.8 | 1.28

Other | | 1.692 | | | 25.78

Nlocal: 215.5 ave 286 max 189 min

Histogram: 5 1 0 0 0 0 0 0 1 1

Nghost: 97.875 ave 131 max 77 min

Histogram: 2 1 1 1 1 0 0 0 1 1

Neighs: 1802 ave 2442 max 1569 min

Histogram: 5 1 0 0 0 0 0 0 1 1

Total # of neighbors = 14416

Ave neighs/atom = 8.36195

Neighbor list builds = 720

Dangerous builds = 0

Total wall time: 0:00:06

Images with GPU Support

You can build a Singularity image with GPU support or find a pre-built one. You will need to make sure the version of CUDA that is to be used (i.e., the version of the CUDA that will be in the environment, check GPU documentation for more information) matches that of the image you plan to use. The Official Singularity User Guide has more details.

Note that many services require sudo access and thus cannot be run on Roar.

Running Services

In some cases, a piece of software, such as database or web server, is meant to be run in the background and accessed as a server. In some cases, it is possible to run these as an instance.

Using Sandbox and Writable Images

Recipes are the most reproducible method of preparing images, but you can use writable and sandbox images to build the image interactively. By default, all Singularity images are temporary, which means changes are not retained when the image is stopped (i.e., when

the shell is exited, the command has completed running, or the instance is stopped by the instance.stop command).