Running on an HPC cluster¶

Two ways to use a cluster¶

Inside an allocation. Request an interactive GPU node, activate the environment that provides nextflow and scfoundry, and run commands exactly as on a workstation. Every process executes on that node. This is the simplest route and the right one for a handful of runs.

Through the scheduler. Activate the slurm profile in the workspace nextflow.config, and scfoundry submits every process as its own job — GPU processes to a GPU queue, CPU processes elsewhere. Nextflow itself runs on the login node (or in a small long-lived job) and waits. This is the right route for a sweep over many datasets and methods.

The workspace nextflow.config ships with the second route commented out, because queue names, accounts and limits are yours to fill in.

The hook that already exists¶

Every process in the pipeline carries a label:

process embed_by_scgpt {
    label "gpu_task"
    container "housy17/scgpt:0.2.4"
    ...
}

Preprocessing, tokenisation, metric computation and the R integration steps are cpu_task; model inference and training are gpu_task. Two withLabel: blocks therefore cover the whole pipeline.

A Slurm profile¶

Uncomment and adapt the block at the end of nextflow.config:

nextflow.config (profiles section)¶

profiles {
    slurm {
        process.executor = 'slurm'

        process {
            withLabel: 'gpu_task' {
                queue          = 'gpu'
                clusterOptions = '--gres=gpu:1'
                cpus           = 8
                memory         = { 64.GB * task.attempt }
                time           = { 12.h * task.attempt }
            }
            withLabel: 'cpu_task' {
                queue  = 'day'
                cpus   = 4
                memory = { 32.GB * task.attempt }
                time   = { 4.h * task.attempt }
            }
            // Retry once on out-of-memory or timeout, with more headroom.
            errorStrategy = { task.exitStatus in [130, 137, 140, 143] ? 'retry' : 'terminate' }
            maxRetries    = 1
        }

        executor {
            queueSize       = 20        // jobs in the queue at once
            submitRateLimit = '10/1min'
        }
    }
}

Then:

scfoundry embed --method scgpt --data tissues/Liver.h5ad --profile slurm

Nextflow must keep running while jobs are queued, so launch it inside tmux, screen, or a long-lived batch job of its own.

Note

Closures like 64.GB * task.attempt let a retry ask for more than the first attempt: 64 GB, then 128 GB. That is usually cheaper than provisioning every job for the worst case.

Apptainer on a cluster¶

Apptainer is the default backend and the right choice here — it needs no daemon and no root privileges.

Pull images before you submit. Compute nodes often have no outbound network access, and a multi-gigabyte pull inside a job wastes an allocation. One CPU-only run on a login node populates the cache:

scfoundry download --method scgpt          # the scgpt image, and the checkpoint
scfoundry embed --method pca --data demo/colon_1000.h5ad

Put the cache and the weights on a shared filesystem. Every compute node must see the same image cache and the same checkpoints. Set it once in nextflow.config:

params {
    cache_dir         = "/shared/scfoundry/cache"
    model_weights_dir = "/shared/scfoundry/model_weights"
}

Export HOME. The Apptainer bind options include --bind ${params.host_side_home_dir}:${System.getenv('HOME')}, evaluated on the launching machine. Some schedulers start jobs without HOME set, which makes the bind target empty and produces a confusing mount failure. Set it explicitly in your submission script.

Filesystem layout¶

Nextflow is hard on shared filesystems. It creates a directory per task, each with several small files, and polls them.

Directory	Where to put it
run `work/`	Fast scratch. Pass `-- -work-dir /scratch/$USER/nxf-work` to move it off the workspace filesystem; never a quota-limited home directory.
`cache_dir`	Shared storage readable by every node. Written once, read many times.
`model_weights_dir`	Shared storage, read-only. Safe for several users to share.
`results/`	Project storage. Small — embeddings and tables only.

Concurrency and GPUs¶

Without a scheduler, every run executes on the machine you launch it from, and several runs launched at once will all target the same GPU unless you say otherwise. Three ways to control this:

# 1. One run at a time: a plain loop.
for f in tissues/*.h5ad; do scfoundry embed --method scgpt --data "$f"; done

# 2. Runs in parallel, each pinned to its own device.
scfoundry embed --method scgpt --data tissues/Liver.h5ad --gpu 0 &
scfoundry embed --method scgpt --data tissues/Lung.h5ad  --gpu 1 &
wait

# 3. Let the scheduler allocate — the Slurm profile above, with --gres=gpu:1.
scfoundry embed --method scgpt --data tissues/Liver.h5ad --profile slurm

The third is the right answer on a cluster. The first two are for a shared workstation.

Running many methods¶

--method and --data take one value each, so a full sweep is a nested shell loop. Each iteration is its own run, with its own run directory and log, so a failure in one method–tissue pair never disturbs the others:

run_all_methods.sh¶

#!/usr/bin/env bash
set -euo pipefail

METHODS=(pca scvi scgpt geneformer scfoundation scbert sccello uce scimilarity
         scprint langcell cellfm cellplm cellama genept c2s)

for m in "${METHODS[@]}"; do
  for f in tissues/*.h5ad; do
    scfoundry embed --method "$m" --data "$f" --profile slurm --quiet
  done
done

Because the runs are independent, the loop can also run several methods concurrently (& and wait, or one loop per terminal) when the queue has room.

Resuming¶

Note

Every launch gets its own run directory under runs/<task>/, and the workspace nextflow.config sets cleanup = true, so the task work directory is deleted once the run succeeds. After a failure it is kept: fix the cause, add --resume to the same command, and Nextflow reuses every task that already completed. scfoundry runs lists the run directories with their status; scfoundry runs --task <task> narrows the list.

On a cluster this matters more than anywhere else, because a single timed-out job should not cost you the whole sweep:

scfoundry embed --method scgpt --data tissues/Liver.h5ad --profile slurm --resume

Sizing a run before committing to it¶

Ask Nextflow to report what it actually used:

scfoundry embed --method scgpt --data tissues/Liver.h5ad --profile slurm \
  -- -with-report report.html -with-trace trace.txt

trace.txt (written in the run directory) gives peak RSS and wall time per task, which is the honest basis for the memory and time directives above. Run one tissue first, read the trace, then size the sweep.