diff --git a/_episodes/12.docker.md b/_episodes/12.docker.md index d0aa22c..b815216 100644 --- a/_episodes/12.docker.md +++ b/_episodes/12.docker.md @@ -133,7 +133,7 @@ int main(int argc, char** argv) { > ## Hints: > -> * You can start with the image "ecpe4s/ubuntu20.04:latest". It already has MPI installed with Spack and E4S. +> * You can start with the image "ecpe4s/ubuntu22.04:latest". It already has MPI installed with Spack and E4S. > * You compile with "mpicc helloworld.c -o /app/hello" {: .callout} @@ -142,7 +142,7 @@ int main(int argc, char** argv) { > > ```bash > # MPI Dockerfile ->FROM ecpe4s/ubuntu20.04:latest +>FROM ecpe4s/ubuntu22.04:latest > > ADD helloworld.c /app/ > diff --git a/_episodes/23.run-hpc.md b/_episodes/23.run-hpc.md index 0af5f98..ab43562 100644 --- a/_episodes/23.run-hpc.md +++ b/_episodes/23.run-hpc.md @@ -32,31 +32,28 @@ cd ~/sc-tutorials/exercises #### Singularity -Download ECP's Ubuntu 20.04 image using: +Download ECP's Ubuntu 22.04 image using: ```bash -singularity pull docker://ecpe4s/ubuntu20.04 +singularity pull docker://ecpe4s/ubuntu22.04 ``` ```output INFO: Converting OCI blobs to SIF format INFO: Starting build... Getting image source signatures -Copying blob 675920708c8b done -Copying blob 8ba2454310fe done -Copying blob 14833710fc9f done +Copying blob d19f32bd9e41 done +Copying blob 63f70a975ff1 done +Copying blob f7c04a75ff42 done ... -Copying blob ab9bb56692d1 done -Copying blob 6d1c299f692a done -Copying config 713eae3dbe done +Copying config 0cad836100 done Writing manifest to image destination Storing signatures -2022/10/11 05:10:39 info unpack layer: sha256:675920708c8bf10fbd02693dc8f43ee7dbe0a99cdfd55e06e6f1a8b43fd08e3f -2022/10/11 05:10:40 info unpack layer: sha256:8ba2454310fee5d84714acde4437d48a888844a0e569b9df36b5637073fad9fb -2022/10/11 05:10:41 info unpack layer: sha256:14833710fc9f06524846486bbb96c1a1043689386133e714c24ad3ebfad720db +2023/10/27 06:50:17 info unpack layer: sha256:d19f32bd9e4106d487f1a703fc2f09c8edadd92db4405d477978e8e466ab290d +2023/10/27 06:50:18 info unpack layer: sha256:63f70a975ff19fb1c92fc6a8ef75b85644211c8ca37f56aecc99629b9444be3c +2023/10/27 06:50:19 info unpack layer: sha256:f7c04a75ff427331d2133d2bfebd3358a7e1a1d6448de27c0920f13f64c95af5 +2023/10/27 06:50:19 warn rootless{usr/bin/ping} ignoring (usually) harmless EPERM on setxattr "security.capability" ... -2022/10/11 05:11:09 info unpack layer: sha256:b0bd5c6e49cc76e93dab9251446c4294eac10028483fef8c5209d0f9af1bfc60 -2022/10/11 05:11:09 info unpack layer: sha256:ab9bb56692d116c3c07c5a2dd427186196e2e0a3417c791404ba09bd69e80e91 -2022/10/11 05:11:09 info unpack layer: sha256:6d1c299f692aab503b4041bdf7896c185248d1acc4f8b7d73740452f0cfbe4d3 +2023/10/27 06:50:50 info unpack layer: sha256:d91732cb657c274bfef42cb46ddad946c6670575e33b37434742c5c32c00f4a6 INFO: Creating SIF file... ``` @@ -68,34 +65,34 @@ The image file is just in your current directory: ls ``` ```output -ubuntu20.04_latest.sif +ubuntu22.04_latest.sif ``` Now let's execute some Linux commands from within the container, `whoami` and `cat /etc/os-release`: ```bash -singularity exec ubuntu20.04_latest.sif whoami +singularity exec ubuntu22.04_latest.sif whoami ``` ```output tutorial ``` ```bash -singularity exec ubuntu20.04_latest.sif cat /etc/os-release +singularity exec ubuntu22.04_latest.sif cat /etc/os-release ``` ```output +PRETTY_NAME="Ubuntu 22.04.1 LTS" NAME="Ubuntu" -VERSION="20.04.5 LTS (Focal Fossa)" +VERSION_ID="22.04" +VERSION="22.04.1 LTS (Jammy Jellyfish)" +VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian -PRETTY_NAME="Ubuntu 20.04.5 LTS" -VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" -VERSION_CODENAME=focal -UBUNTU_CODENAME=focal +UBUNTU_CODENAME=jammy ``` Note how with Singularity the user in the container is the same as in the host machine. @@ -103,7 +100,7 @@ Note how with Singularity the user in the container is the same as in the host m Singularity has a dedicated syntax to open an interactive shell prompt in a container: ```bash -singularity shell ubuntu20.04_latest.sif +singularity shell ubuntu22.04_latest.sif ``` ```output Singularity> @@ -145,8 +142,7 @@ VERSION_CODENAME=xenial UBUNTU_CODENAME=xenial ``` -By default, the cache is stored in `~/.singularity`; this location can be customised using the environment variable `SINGULARITY_CACHEDIR`. -A subcommand, `singularity cache`, can be used to manage the cache. +By default, the cache is stored in `~/.singularity`; this location can be customised using the environment variable `SINGULARITY_CACHEDIR`. This cache directory can use a non-negligible amount of space. A subcommand, `singularity cache`, can be used to manage the cache or clear it completely. #### Shifter @@ -154,10 +150,10 @@ A subcommand, `singularity cache`, can be used to manage the cache. Let's download the same Ubuntu image as above, using `shifterimg`: ```bash -shifterimg pull ecpe4s/ubuntu20.04 +shifterimg pull ecpe4s/ubuntu22.04 ``` ```output -2022-10-11T05:25:11 Pulling Image: docker:ecpe4s/ubuntu20.04, status: READY +2022-10-11T05:25:11 Pulling Image: docker:ecpe4s/ubuntu22.04, status: READY ``` Locally stored images are managed by Shifter itself: @@ -166,20 +162,20 @@ Locally stored images are managed by Shifter itself: shifterimg images ``` ```output -mycluster docker READY 369c047181 2022-10-11T05:25:11 ecpe4s/ubuntu20.04:latest +mycluster docker READY 369c047181 2022-10-11T05:25:11 ecpe4s/ubuntu22.04:latest ``` What's the container user with Shifter? Let's use both `id -u` and `whoami`: ```bash -shifter --image=ecpe4s/ubuntu20.04 whoami +shifter --image=ecpe4s/ubuntu22.04 whoami ``` ```output tutorial ``` ```bash -shifter --image=ecpe4s/ubuntu20.04 id -u +shifter --image=ecpe4s/ubuntu22.04 id -u ``` ```output 1001 @@ -190,26 +186,28 @@ Again, these come from the host. You can try more Linux commands: ```bash -shifter --image=ecpe4s/ubuntu20.04 cat /etc/os-release +shifter --image=ecpe4s/ubuntu22.04 cat /etc/os-release ``` ```output +PRETTY_NAME="Ubuntu 22.04.1 LTS" NAME="Ubuntu" -VERSION="20.04.5 LTS (Focal Fossa)" +VERSION_ID="22.04" +VERSION="22.04.1 LTS (Jammy Jellyfish)" +VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian -PRETTY_NAME="Ubuntu 20.04.5 LTS" -VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" -VERSION_CODENAME=focal -UBUNTU_CODENAME=focal +UBUNTU_CODENAME=jammy ``` NOTE: If you need to open an interactive shell in the container with Shifter, just execute `bash` in the container. +#### Podman-HPC + ### Share host environment variables By default, host variables are shared with the container: @@ -221,7 +219,7 @@ export HELLO="world" You can access that variable both with Singularity: ```bash -singularity exec ubuntu20.04_latest.sif bash -c 'echo $HELLO' +singularity exec ubuntu22.04_latest.sif bash -c 'echo $HELLO' ``` ```output world @@ -230,7 +228,7 @@ world and with Shifter: ```bash -shifter --image=ecpe4s/ubuntu20.04 bash -c 'echo $HELLO' +shifter --image=ecpe4s/ubuntu22.04 bash -c 'echo $HELLO' ``` ```output world @@ -239,7 +237,7 @@ world There are some additional user options worth discussing, for further tuning of the shell environment. In some cases, *e.g.* when using Python containers, you might need to isolate the container shell from the host. To this end, use `-e` with Singularity: ```bash -singularity exec -e ubuntu20.04_latest.sif bash -c 'echo $HELLO' +singularity exec -e ubuntu22.04_latest.sif bash -c 'echo $HELLO' ``` ```output @@ -248,7 +246,7 @@ singularity exec -e ubuntu20.04_latest.sif bash -c 'echo $HELLO' and `-E` with Shifter: ```bash -shifter -E --image=ecpe4s/ubuntu20.04 bash -c 'echo $HELLO' +shifter -E --image=ecpe4s/ubuntu22.04 bash -c 'echo $HELLO' ``` ```output @@ -258,7 +256,7 @@ If you need to pass variables to the container in this situation, you can use a ```bash export SINGULARITYENV_BYE="moon" -singularity exec -e ubuntu20.04_latest.sif bash -c 'echo $BYE' +singularity exec -e ubuntu22.04_latest.sif bash -c 'echo $BYE' ``` ```output moon @@ -267,7 +265,7 @@ moon or, from version `3.6.x` on: ```bash -singularity exec -e --env BYE="moon" ubuntu20.04_latest.sif bash -c 'echo $BYE' +singularity exec -e --env BYE="moon" ubuntu22.04_latest.sif bash -c 'echo $BYE' ``` ```output moon @@ -276,7 +274,7 @@ moon And with Shifter: ```bash -shifter -E --env BYE="moon" --image=ecpe4s/ubuntu20.04 bash -c 'echo $BYE' +shifter -E --env BYE="moon" --image=ecpe4s/ubuntu22.04 bash -c 'echo $BYE' ``` ```output moon @@ -289,7 +287,7 @@ By default and similar to Docker, Podman isolates host and container shell envir ```bash export HELLO="world" -podman run ecpe4s/ubuntu20.04 bash -c 'echo $HELLO' +podman run ecpe4s/ubuntu22.04 bash -c 'echo $HELLO' ``` ```output @@ -298,7 +296,7 @@ podman run ecpe4s/ubuntu20.04 bash -c 'echo $HELLO' You can pass specific variables to the container by using the flag `-e`: ```bash -podman run -e HELLO ecpe4s/ubuntu20.04 bash -c 'echo $HELLO' +podman run -e HELLO ecpe4s/ubuntu22.04 bash -c 'echo $HELLO' ``` ```output world @@ -307,12 +305,14 @@ world Or even redefine variables, with the same flag: ```bash -podman run -e HELLO=moon ecpe4s/ubuntu20.04 bash -c 'echo $HELLO' +podman run -e HELLO=moon ecpe4s/ubuntu22.04 bash -c 'echo $HELLO' ``` ```output moon ``` +#### What about Podman-HPC? + ### Use host directories @@ -353,14 +353,14 @@ Typically, HPC administrators will configure the container engine for you, so th In the unlikely scenario where you need to bind-mount additional paths, Singularity offers handy methods for users. For instance: ```bash -singularity exec ubuntu20.04_latest.sif ls /data2 +singularity exec ubuntu22.04_latest.sif ls /data2 ``` ```output ls: cannot access /data2: No such file or directory ``` ```bash -singularity exec -B /data2 ubuntu20.04_latest.sif ls /data2 +singularity exec -B /data2 ubuntu22.04_latest.sif ls /data2 ``` ```output file2 @@ -370,12 +370,30 @@ or ```bash export SINGULARITY_BINDPATH="/data2" -singularity exec ubuntu20.04_latest.sif ls /data2 +singularity exec ubuntu22.04_latest.sif ls /data2 ``` ```output file2 ``` +Additionally singularity can use SquashFS to create a filesystem overlay that can be added as a writable filesystem in the container. This compressed filesystem can then be moved as a single file on the host filesystem and even mounted. + +First, you create the overlay of a specific size + +```bash +# create the overlay +size=1000 #filesystem size in MB +filename=overlay.sqsh #filename of mountable filesystem +singularity overlay create -s ${size} ${filename} +``` + +then you load this filesystem into the root of the container, allowing you to create directories in which to store data + +```bash +singularity exec --overlay ${filename} ubuntu22.04_latest.sif mkdir -p /overlay/data +``` + +This will provide a writeable space that is isolated from the host filesystems, which can be useful if there are file quotas on the host filesystems. Files written within the container to this filesystem will not count against your quota. #### What about Podman? @@ -387,7 +405,7 @@ podman run marcodelapierre/ubuntu_workdir:18.04 pwd ``` ```bash -podman run ecpe4s/ubuntu20.04 pwd +podman run ecpe4s/ubuntu22.04 pwd ``` ```output / @@ -413,6 +431,7 @@ podman run -v $(pwd):$(pwd) -w $(pwd) marcodelapierre/ubuntu_workdir:18.04 pwd /home/tutorial/sc-tutorials/exercises ``` +#### What about Podman-HPC? ### Do It Yourself: BLAST example @@ -556,23 +575,42 @@ shpc show --versions -f blast ncbi/blast:2.11.0 ncbi/blast:2.12.0 ncbi/blast:latest +ncbi/blast:2.13.0 +ncbi/blast:2.14.0 +ncbi/blast:2.14.1 +quay.io/biocontainers/bioconductor-hicdatalymphoblast:1.30.0--r41hdfd78af_1 +quay.io/biocontainers/bioconductor-hicdatalymphoblast:1.33.0--r42hdfd78af_0 +quay.io/biocontainers/bioconductor-hicdatalymphoblast:1.36.0--r43hdfd78af_0 +quay.io/biocontainers/blast-legacy:2.2.26--h9ee0642_3 quay.io/biocontainers/blast:2.10.1--pl526he19e7b1_3 quay.io/biocontainers/blast:2.11.0--pl5262h3289130_1 quay.io/biocontainers/blast:2.12.0--pl5262h3289130_0 +... +quay.io/biocontainers/samblaster:0.1.26--h9f5acd7_2 +quay.io/biocontainers/samblaster:0.1.26--h4ac6f70_4 ``` And now let's install the latest BLAST biocontainer (copy-pasting the image and tag from the output above) with `shpc install`: ```bash -shpc install quay.io/biocontainers/blast:2.12.0--pl5262h3289130_0 +shpc install quay.io/biocontainers/blast:2.14.1--pl5321h6f7f691_0 ``` ```output -singularity pull --name /opt/singularity-hpc/containers/quay.io/biocontainers/blast/2.12.0--pl5262h3289130_0/quay.io-biocontainers-blast-2.12.0--pl5262h3289130_0-sha256:a7eb056f5ca6a32551bf9f87b6b15acc45598cfef39bffdd672f59da3847cd18.sif docker://quay.io/biocontainers/blast@sha256:a7eb056f5ca6a32551bf9f87b6b15acc45598cfef39bffdd672f59da3847cd18 +singularity pull --name /usr/local/lib/python3.10/dist-packages/singularity_hpc-0.1.26-py3.10.egg/containers/quay.io/biocontainers/blast/2.14.1--pl5321h6f7f691_0/quay.io-biocontainers-blast-2.14.1--pl5321h6f7f691_0-sha256:0fa116b90c6411d5b09cdda5ca81a857167d218c49915104e7e1588b16baedf7.sif docker://quay.io/biocontainers/blast@sha256:0fa116b90c6411d5b09cdda5ca81a857167d218c49915104e7e1588b16baedf7 INFO: Using cached SIF image INFO: Using cached SIF image -/opt/singularity-hpc/containers/quay.io/biocontainers/blast/2.12.0--pl5262h3289130_0/quay.io-biocontainers-blast-2.12.0--pl5262h3289130_0-sha256:a7eb056f5ca6a32551bf9f87b6b15acc45598cfef39bffdd672f59da3847cd18.sif -Module quay.io/biocontainers/blast:2.12.0--pl5262h3289130_0 was created. +Copying blob sha256:642efca944a099a40fb3c07af9503df633650ab4299bba89f04e2106bfe2d85e +Copying blob sha256:bd9ddc54bea929a22b334e73e026d4136e5b73f5cc29942896c72e4ece69b13d +Copying blob sha256:bfa1a70cade63da3b530cd6c74b1f60042ddadf4aab32369b885b55545d4156d +Copying config sha256:1db36fd7938c3efa43c779369231cee1a026b98d46cc82d2db324336bf89d8ef +Writing manifest to image destination +Storing signatures +2023/10/27 10:08:58 info unpack layer: sha256:642efca944a099a40fb3c07af9503df633650ab4299bba89f04e2106bfe2d85e +2023/10/27 10:08:58 info unpack layer: sha256:bd9ddc54bea929a22b334e73e026d4136e5b73f5cc29942896c72e4ece69b13d +2023/10/27 10:08:58 info unpack layer: sha256:bfa1a70cade63da3b530cd6c74b1f60042ddadf4aab32369b885b55545d4156d +INFO: Creating SIF file... +Module quay.io/biocontainers/blast:2.14.1--pl5321h6f7f691_0 was created. ``` That's it! We now have a BLAST module: @@ -584,33 +622,35 @@ module avail quay ```output -------------------------------------------- /opt/singularity-hpc/modules --------------------------------------------- -quay.io/biocontainers/blast/2.12.0--pl5262h3289130_0/module.tcl +quay.io/biocontainers/blast/2.14.1--pl5321h6f7f691_0/module.tcl ``` Which we can load and use (well, the newer version does not like the *-help* flag): ```bash -module load quay.io/biocontainers/blast/2.12.0--pl5262h3289130_0 +module load quay.io/biocontainers/blast/2.14.1--pl5321h6f7f691_0 blastp -help ``` ```output -BLAST query/options error: Either a BLAST database or subject sequence(s) must be specified -Please refer to the BLAST+ user manual. +... +DESCRIPTION + Protein-Protein BLAST 2.14.1+ +... ``` -We can also see that this command is indeed a bash function wrapping the singularity syntax: +We can also see that this command is indeed bash wrapper: ```bash -type blastp +more blastp ``` -```output -blastp is a function -blastp () -{ - singularity ${SINGULARITY_OPTS} exec ${SINGULARITY_COMMAND_OPTS} -B /opt/singularity-hpc/modules/quay.io/biocontainers/blast/2.12.0--pl5262h3289130_0/99-shpc.sh:/.singularity.d/env/99-shpc.sh /opt/singularity-hpc/containers/quay.io/biocontainers/blast/2.12.0--pl5262h3289130_0/quay.io-biocontainers-blast-2.12.0--pl5262h3289130_0-sha256:a7eb056f5ca6a32551bf9f87b6b15acc45598cfef39bffdd672f59da3847cd18.sif /usr/local/bin/blastp -} -``` +```bash +#!/bin/bash -NOTE: a limitation to this approach is that bash functions cannot be used as an argument for *mpirun* or *srun*, and hence are not usable for MPI applications. A simple workaround is to produce aliases for MPI applications in the form of wrapper bash scripts instead. +script=`realpath $0` +wrapperDir=`dirname $script`/.. + +singularity ${SINGULARITY_OPTS} exec ${SINGULARITY_COMMAND_OPTS} -B $wrapperDir/99-shpc.sh:/.singularity.d/env/99-shpc.sh /usr/local/lib/python3.10/dist-packages/singularity_hpc-0.1.26-py3.10.egg/containers/quay.io/biocontainers/blast/2.14.1--pl5321h6f7f691_0/quay.io-biocontainers-blast-2.14.1--pl5321h6f7f691_0-sha256:0fa116b90c6411d5b09cdda5ca81a857167d218c49915104e7e1588b16baedf7.sif /usr/local/bin/blastp "$@" +``` +