The workspace

A workspace is the directory scfoundry init prepares. It holds everything a run needs and everything a run produces, so that a project can be moved, shared or deleted as one unit.

What init creates

my_project/
├── .scfoundry.json         marker: scfoundry version, pipeline path, container runtime
├── nextflow.config         container runtime, GPU pinning, paths — edit freely
├── data/model_weights/     checkpoints, filled by `scfoundry download`
├── cache/                  container images, the container's HOME, Nextflow caches
├── results/                published outputs
└── runs/                   one directory per launch (created on the first run)

nextflow.config is written from a template bundled with the package, with the container runtime init detected switched on. It is yours: nothing overwrites it later (rerun scfoundry init --force if you want a fresh copy).

How commands find the workspace

Every task command looks for a workspace in this order:

  1. --workspace DIR on the command line;

  2. the SCFOUNDRY_WORKSPACE environment variable;

  3. the current directory and its parents, stopping at the first one that contains .scfoundry.json — the way git finds a repository.

So cd my_project (or any directory below it) is all it takes. scfoundry info prints which workspace and configuration file are in effect.

Note

Several workspaces can share one copy of the weights and one image cache. Either pass --weights-dir and --cache-dir on each command, or set model_weights_dir and cache_dir in the workspace nextflow.config — see below.

nextflow.config

The file is annotated in full in the nextflow.config reference. The settings you are likely to touch:

Container runtime

Exactly one of the docker, singularity and apptainer blocks has enabled = true. init sets it from what it finds; to change your mind:

scfoundry init --force --runtime docker .

or flip the enabled flags by hand. Do not delete the blocks — they carry the bind-mount options each runtime needs.

GPU

gpu_id = null exposes every GPU on the machine. Pin one device per run with --gpu:

scfoundry embed --method scgpt --data demo/colon_1000.h5ad --gpu 0

or permanently, in the params block:

params {
    gpu_id = 0
}

Tip

On a multi-GPU machine, pin a run to one device with --gpu 0. Without it, Docker gets --gpus=all and Apptainer gets --nv with no CUDA_VISIBLE_DEVICES, so the job may land on a card someone else is using. To make it permanent, set gpu_id in the workspace nextflow.config.

Paths

Parameter

Default

What it controls

model_weights_dir

<workspace>/data/model_weights

Bind-mounted at /data/model_weights in every container. Point it at shared storage when several users or workspaces use the same checkpoints.

cache_dir

<workspace>/cache

Container images land in cache/.shared/nxf_singularity/. Images are large; put this somewhere with room.

host_side_home_dir

<workspace>/cache/.home

A stand-in $HOME for containers, so Hugging Face and PyTorch caches stay inside the workspace instead of filling your real home directory.

The per-run equivalents are --weights-dir and --cache-dir; they take precedence over the file.

Work directories and --resume

The template sets cleanup = true: Nextflow deletes a task’s work directory once the run has succeeded. Work directories are large (every task stages its inputs), and the outputs you want are already published under results/.

Note

Every launch gets its own run directory under runs/<task>/, and the workspace nextflow.config sets cleanup = true, so the task work directory is deleted once the run succeeds. After a failure it is kept: fix the cause, add --resume to the same command, and Nextflow reuses every task that already completed. scfoundry runs lists the run directories with their status; scfoundry runs --task <task> narrows the list.

Scheduler profiles

The template ends with a commented profiles { slurm { ... } } block. Uncomment it, adapt the queue names, and every process is submitted as its own job when you add --profile slurm to a command. Details on Running on an HPC cluster.

For one-off changes, keep a separate file and layer it on with --config extra.config rather than editing the workspace file.

Runs

Every launch gets its own directory:

runs/embed/20260828-162920_scgpt_colon_1000/
├── params.json      the parameters handed to Nextflow
├── command.sh       the exact command, re-runnable
├── run.json         task, method, input, timestamps, exit code, versions
├── nextflow.log     the full Nextflow log
├── .nextflow/       Nextflow's cache database (what -resume uses)
└── work/            task work directories, removed on success

The name is <timestamp>_<method>_<input>; --run-name NAME chooses it yourself. Because runs are isolated, several tasks can run in one workspace at the same time.

scfoundry runs --limit 5

started              task       method   status   exit  run directory
2026-08-28T20:33:41  benchmark  pca      ok       0     runs/benchmark/20260828-203341_pca_pca
2026-08-28T20:23:44  benchmark  scgpt    ok       0     runs/benchmark/20260828-202344_scgpt_colon_1000
2026-08-28T20:10:12  finetune   scgpt    ok       0     runs/finetune/20260828-201012_scgpt_colon_50
2026-08-28T18:19:09  transfer   scgpt    ok       0     runs/transfer/20260828-181909_scgpt_liver_1shot_query
2026-08-28T16:29:20  embed      scgpt    ok       0     runs/embed/20260828-162920_scgpt_colon_1000

A failed run keeps its work/. Rerun the same command with --resume and Nextflow reuses every task that already finished; --resume NAME targets a specific run directory.

--dry-run shows the parameter file and the Nextflow command without launching anything — the quickest way to check what a set of options resolves to:

scfoundry embed --method scgpt --data demo/colon_1000.h5ad --gpu 0 --batch_size 32 --dry-run

[scfoundry] warning: forwarding unrecognised option(s) to Nextflow as parameters: --batch_size=32
[scfoundry] dry run -- workspace /home/you/my_project
[scfoundry] would create /home/you/my_project/runs/embed/20260828-225931_scgpt_colon_1000
params.json:
{
  "task": "embed",
  "method": "scgpt",
  "data": "/home/you/my_project/demo/colon_1000.h5ad",
  "emb_results_dir": "/home/you/my_project/results",
  "gpu_id": "0",
  "batch_size": 32
}
command (cwd = run directory; env SCFOUNDRY_WORKSPACE=/home/you/my_project NXF_SYNTAX_PARSER=v1):
  /home/you/.conda/envs/nf-env/bin/nextflow -log nextflow.log run .../scfoundry/pipeline/main.nf -params-file params.json -work-dir work -ansi-log false

Results

Outputs are published under results/ by default; --outdir DIR redirects a run. The layout is the same for every method — see the output reference.

See also