Troubleshooting

Common failures, what causes them, and what to do. Errors that produce wrong results rather than a crash are collected at the end — those are the ones worth reading before you have a problem.

Reading a failure

scfoundry ends a failed run with the run directory and the Nextflow log:

[ERROR] ERROR ~ Error executing process > 'EMBED:embed_by_scgpt (colon_1000)'
...
[FAILED] completed=0 failed=1 cached=0
[scfoundry] warning: nextflow exited with code 1; see /home/you/my_project/runs/embed/20260828-162920_scgpt_colon_1000/nextflow.log

The log names the task’s work directory, which holds the whole story:

File	Contents
.command.sh	The exact script that ran, with all parameters interpolated. Check here first when a value looks wrong.
.command.err	stderr, including Python tracebacks.
.command.log	Combined output.
.exitcode	Exit status. 137 means killed, usually out of memory.

params.json and command.sh in the run directory show what was launched; scfoundry runs lists every run with its status.

Caution

cleanup = true deletes work directories on success only, so a failed task’s directory survives for inspection. Once you fix the problem and --resume succeeds, it is removed. Copy anything you want to keep before rerunning.

Setup

no workspace found

You are outside any workspace. cd into one, pass --workspace DIR, or export SCFOUNDRY_WORKSPACE. scfoundry init . turns the current directory into one.

nextflow: not found

scfoundry looks for nextflow on PATH. Activate the environment or module that provides it, or point at it with --nextflow /path/to/nextflow or SCFOUNDRY_NEXTFLOW. scfoundry info shows which one is in use.

Nextflow version ... does not match workflow required version: >=24.10

The pipeline declares a minimum Nextflow version. Upgrade (nextflow self-update, or a newer conda package).

Variable declarations cannot be mixed with config statements or other syntax errors from Nextflow 26

Nextflow 26 parses scripts strictly by default; the method modules use the classic DSL2 style. scfoundry sets NXF_SYNTAX_PARSER=v1 for every launch, so this only happens when you call nextflow directly — export the variable yourself.

Empty bind path / mount failure mentioning an empty target

The Apptainer bind options include ${System.getenv('HOME')}, evaluated on the launching machine. If HOME is unset — some batch schedulers do this — the bind target is an empty string. Export it in your submission script:

export HOME=/home/$USER

Unknown method 'X'. Allowed: ...

A typo, or a method that task does not support. Method ids are lowercase, and coverage differs by task — pca has no transfer, uce has no finetune. See the support matrix or scfoundry list methods.

run directory already exists

--run-name names a directory that exists. Choose another name, or --resume NAME to continue that run.

Container and GPU

Image pull times out or fails

pullTimeout is already set to 6 hours, so a timeout usually means the network, not the setting. Pull once on a machine with good connectivity; the cache is shared:

scfoundry embed --method scgpt --data demo/colon_1000.h5ad

On a cluster, do this on the login node before submitting — compute nodes frequently have no outbound access.

CUDA driver version is insufficient / no GPU visible

Check the driver on the host:

nvidia-smi

You need ≥ 525. Under Docker, also confirm the NVIDIA Container Toolkit works:

docker run --rm --gpus=all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

If that fails, the problem is the toolkit installation, not scFoundry. scfoundry init reports whether nvidia-smi sees a GPU — “no GPU visible” is expected on a login node and a problem on a compute node.

CUDA out of memory

Lower the batch size — which option depends on the task:

scfoundry embed --method uce --data your.h5ad --batch-size 8

Full guidance in GPU memory and runtime.

If several runs are sharing one GPU, that is often the real cause: run them one after another, or pin each to its own device with --gpu.

Exit code 137, or “bus error” under Docker

137 is the process being killed, almost always by the out-of-memory killer — request more host memory, or lower the batch size.

A “bus error” specifically points at shared memory exhaustion from PyTorch dataloader workers. The shipped Docker options already set --shm-size=16g; if you have overridden runOptions, put it back.

Data problems

KeyError: 'barcode'

obs["barcode"] is required and missing.

adata.obs_names_make_unique()
adata.obs["barcode"] = adata.obs_names
adata.write_h5ad("fixed.h5ad")

KeyError: 'ensembl_id' or 'gene_symbol'

Both var columns are required. Geneformer and scPRINT depend on ensembl_id specifically. See Input data format.

... cannot integrate batches with <= n_pcs=30 cells

Seurat CCA and RPCA cannot handle a batch with 30 or fewer cells. Merge or drop tiny batches, choose a coarser --batch-key, or use harmony.

--batch-key is ignored, or a single batch is reported

Only the integration methods read --batch-key in embed; other methods warn and ignore it. An integration method that cannot find the column treats the data as one batch and says so — check the spelling against adata.obs.columns.

Duplicate cell or gene names

adata.obs_names_make_unique()
adata.var_names_make_unique()
adata.obs["barcode"] = adata.obs_names
adata.var["gene_symbol"] = adata.var_names

Regenerate the barcode and gene_symbol columns after making names unique, or they will disagree with the index.

Transfer and fine-tuning

mlp needs a validation split

The MLP head holds out 20% of each class for early stopping and needs at least one class with five or more reference cells. Use prototype, knn or logreg on tiny references.

--fitted is refused

--fitted takes the model directory written by the fit stage: results/transfer/models/<method>/<classifier>/<reference> or results/finetune/finetuned_models/<method>/<reference>. For transfer, meta.json inside it records the method and classifier; a --method that disagrees with it is an error rather than a silent mismatch.

Predictions are all one class

Check the reference: a class that is absent cannot be predicted, and with knn a k larger than the smallest class can never give that class a majority. meta.json lists cells_per_class.

Reruns and output

--resume reruns everything

Expected after a successful run: cleanup = true removed its work directory, so there is nothing to reuse. Resume is for failed runs. Set cleanup = false in nextflow.config before a run whose tasks you may want to reuse later.

Results overwrote each other

Output filenames come from the input basename. Two input files both called Blood.h5ad produce one results/embeddings/scgpt/Blood.h5ad. Rename them, or give the second run its own --outdir.

No output at all, but the run succeeded

Check run.json for outdir, and params.json for the *_results_dir parameter — the run may have published somewhere else.

Failures that do not raise errors

These are the dangerous ones: the pipeline completes, files appear, and the numbers are wrong.

Warning

Log-normalised or scaled input. Models expect raw counts. Normalised input produces a plausible embedding with no warning. Verify before running:

import numpy as np
from scipy.sparse import issparse

v = adata.X.data if issparse(adata.X) else adata.X.ravel()
print("integer counts:", np.allclose(v[:10000], np.round(v[:10000])))

Warning

HVG-subset input. Every model matches your genes against its own vocabulary. Subsetting first changes which vocabulary entries are populated, differently for each model — so the comparison measures your gene selection, not the models. adata.n_vars below ~15,000 for a human dataset is a red flag.

Warning

Batch confounded with biology. If batch_id is donor-only in a single-assay dataset, donors differ biologically as well as technically, and integration removes signal you wanted. Prefer a genuine technical axis — assay, chemistry, sequencing run.

Warning

Query classes absent from the reference. A classifier can only emit classes it saw during fitting or training. Unseen cell types are silently assigned to some known class. Compare the label sets before you trust a prediction:

set(query.obs["cell_type"]) - set(reference.obs["cell_type"])

Warning

Comparing across label columns or clusterings. benchmark numbers depend on the label column (it sets the target cluster count) and on --clustering. Only compare methods scored with the same settings, and only compare with the published tables when using the default Leiden protocol.

Getting help

Report a bug at [https://github.com/Svvord/scFoundry](https://github.com/Svvord/scFoundry/issues) with:

1. The full command you ran, and command.sh from the run directory.

2. nextflow.log from the run directory, or at least the error it prints.

3. The contents of .command.err from the failed task’s work directory.

4. scfoundry info, your container runtime and version, and nvidia-smi output.

5. The shape and obs/var columns of your input — not the data itself.

See also

• GPU memory and runtime

• Input data format

• nextflow.config reference