Embed

What it does

Takes a count matrix and returns a cell embedding. This is the task the rest of the framework is built around: transfer consumes these embeddings, benchmark and geometry score them.

Three kinds of method produce one, and the difference matters when you read results:

  • zero-shot — a pretrained foundation model used exactly as published: no training, no labels, no batch information. Each model does something different internally (Geneformer ranks genes by expression, scBERT bins values, CELLama turns each cell into a sentence, GenePT-w averages gene text embeddings), but the interface and the output are identical.

  • reference — pca, the standard Scanpy workflow on 2,000 highly variable genes — the cheap yardstick; a foundation model that does not beat it on your data has not earned its GPU hours — and scvi, zero-shot projection onto the CELLxGENE Census scVI, a model pretrained on tens of millions of cells that the paper keeps apart from the transformer scFMs. The everyday scVI baseline, trained on your own data, is scvi_denovo below.

  • integration — scgpt_integrated, scvi_denovo, harmony, seurat_cca and seurat_rpca are trained on your data using its batch labels, and nothing else. They correct batch effects actively, which zero-shot methods can only do passively.

Supported methods

scfoundry list methods --task embed lists all 22:

zero-shot    c2s cellama cellfm cellplm geneformer genept langcell scbert sccello
             scfoundation scgpt scimilarity scprint uce novae
reference    pca scvi
integration  scgpt_integrated scvi_denovo harmony seurat_cca seurat_rpca

See also

Method reference — container images, checkpoints, and upstream versions for each of these.

Inputs

Warning

Input must be raw counts over the full transcriptome. Passing log-normalised values, or an object already subset to highly variable genes, does not raise an error — it produces a plausible-looking embedding that is silently wrong. See Input data format.

Zero-shot and reference methods need obs["barcode"] and nothing else — labels are not used. Geneformer and scPRINT additionally require var["ensembl_id"]; novae requires obsm["spatial"]. Integration methods read the batch column (batch_id, or --batch-key).

Every example on this page runs against the four demo files, downloaded once into a demo/ directory of the workspace as shown in Demo datasets:

demo/colon_1000.h5ad            1,000 human colon cells, labelled, 7 batches
demo/colon_50.h5ad                 50 human colon cells, labelled
demo/liver_1shot_support.h5ad   one labelled cell per class (5 cells)
demo/liver_1shot_query.h5ad     75 liver cells to annotate

Running it

Note

Run scfoundry commands from inside a workspace created by scfoundry init (any subdirectory works — the workspace is found by walking upwards, like a git repository), or pass --workspace DIR. Weights, the image cache, results and run logs all live there. See The workspace.

scfoundry embed --method scgpt --data demo/colon_1000.h5ad

[PROCESS 75/2315a7] EMBED:embed_by_scgpt (colon_1000)

[SUCCESS] completed=1 failed=0 cached=0
[scfoundry] done (ok).

scfoundry embed --method pca  --data demo/colon_1000.h5ad
scfoundry embed --method scvi --data demo/colon_1000.h5ad

pca runs on the CPU in seconds. scvi here means projection onto the pretrained Census model; training a fresh scVI is --method scvi_denovo.

scfoundry embed --method harmony --data demo/colon_1000.h5ad --batch-key batch_id

--batch-key defaults to batch_id, so it can be omitted when your column has that name. A missing batch column is treated as a single batch (with a warning): Seurat then returns the uncorrected PCA, scVI and scGPT train without their batch objectives.

--data takes one file per run. A tissue collection is a shell loop, and each iteration gets its own run directory:

for f in tissues/*.h5ad; do
  scfoundry embed --method scgpt --data "$f"
done

--method takes one value, so loop in the shell. Each iteration is a separate run with its own run directory:

for m in pca scgpt geneformer scfoundation; do
  scfoundry embed --method "$m" --data demo/colon_1000.h5ad --gpu 0
done

Note

The first run of any method pulls its container image, which can take several minutes and a few gigabytes. The image is cached under cache/.shared/nxf_singularity/ in the workspace (or wherever --cache-dir points) and reused by every later run, so this cost is paid once per method.

Parameters

Option

Type

Default

Description

--method

string

required

Which method to run. Also selects the container image.

--data

path

required

Input .h5ad, one file per run.

--model

string

per method

Checkpoint path under data/model_weights/, e.g. scGPT/scGPT_human.

--batch-size

integer

per method

Inference batch size. The first thing to lower when you hit out-of-memory.

--batch-key

string

batch_id

obs column with batch labels. Integration methods only; other methods ignore it with a warning.

Method-specific knobs are forwarded verbatim as pipeline parameters — --top_k 30 (CELLama), --scfoundation_pool_type max, --seurat_n_pcs 30, --integration_epoch 15 (scGPT integration), --scvi_n_latent 30 — and are listed with their defaults in the parameter reference.

Defaults for --batch-size and --model are per method, because a sensible batch size for SCimilarity (2048) would exhaust memory for Cell2Sentence (8). The full table is in the method reference.

Integration methods

Method

Container

How it integrates

scgpt_integrated

{{ registry }}/scgpt:0.2.4

Fine-tunes scGPT_human on the input with domain-specific batch norm, an adversarial batch discriminator, masked expression prediction and elastic cell similarity. The cell-type objective is switched off. By far the most expensive option here.

scvi_denovo

scverse/scvi-tools

Trains a fresh scVI variational autoencoder with setup_anndata(batch_key=...). The canonical deep-learning baseline.

harmony

satijalab/seurat:5.5.0

PCA followed by Harmony, through Seurat v5 IntegrateLayers.

seurat_cca

satijalab/seurat:5.5.0

Seurat v5 IntegrateLayers with canonical correlation analysis.

seurat_rpca

satijalab/seurat:5.5.0

Seurat v5 IntegrateLayers with reciprocal PCA. Faster and more conservative than CCA.

One constraint holds throughout: only batch information enters training. Cell-type labels are never used — they are carried through in obs so the same benchmark can score biological conservation afterwards. Without that discipline the evaluation would be circular.

Warning

Choose a batch column that represents a genuine technical axis — different assays, chemistries, or sequencing runs. On a single-assay dataset split only by donor, batch is confounded with biology, and aggressive integration removes the signal you were trying to measure.

Note

Seurat’s CCA and RPCA cannot integrate a batch with 30 or fewer cells (the default --seurat_n_pcs 30 is a hard floor in Seurat). scFoundry checks this before starting R and fails with a clear message. Merge or drop tiny batches, choose a coarser batch column, or use harmony — which is what happens on the demo colon file, whose smallest batch has 30 cells.

Outputs

results/embeddings/<method>/<dataset>.h5ad

For the commands above:

results/embeddings/scgpt/colon_1000.h5ad
results/embeddings/pca/colon_1000.h5ad
results/embeddings/harmony/colon_1000.h5ad

The output is an AnnData object whose matrix is the embedding:

  • adata.X — the embedding, cells × dimensions. Dense, float32.

  • adata.obs — your original cell metadata, carried through untouched.

  • adata.var — placeholder names V1, V2, … one per embedding dimension.

  • adata.obsm["spatial"] — preserved if it was present in the input.

Note

The embedding lives in X, not obsm. That is deliberate: it means every method’s output has the same shape and the benchmark can score all of them identically. It also means you cannot recover expression values from an embedding file — keep your input.

Reading results in Python

import anndata as ad

adata = ad.read_h5ad("results/embeddings/scgpt/colon_1000.h5ad")

embedding = adata.X                   # (n_cells, n_dims)
labels    = adata.obs["cell_type"]    # metadata came along

To visualise, treat the embedding as you would a PCA representation:

import scanpy as sc

adata.obsm["X_emb"] = adata.X
sc.pp.neighbors(adata, use_rep="X_emb")
sc.tl.umap(adata)
sc.pl.umap(adata, color=["cell_type", "batch_id"])

For an integrated embedding, what you are looking for is batches mixing while cell types stay apart. Both mixing is under-correction; both merging is over-correction, and the more dangerous failure. Benchmark puts numbers on both.

Gotchas

Tip

On a multi-GPU machine, pin a run to one device with --gpu 0. Without it, Docker gets --gpus=all and Apptainer gets --nv with no CUDA_VISIBLE_DEVICES, so the job may land on a card someone else is using. To make it permanent, set gpu_id in the workspace nextflow.config.

Output collisions. The output filename is the input basename, so embedding two different files both named Blood.h5ad writes one over the other. Give them distinct names or distinct --outdir values.

Zero-shot scVI and de novo scVI are different things. --method scvi projects your data onto a pretrained Census model and uses no batch labels; --method scvi_denovo trains a new model on your data alone. They answer different questions.

Novae needs coordinates. It is a spatial model and fails without obsm["spatial"].

Concurrency on one GPU. Runs are independent, so nothing stops you launching several embed commands at once — and nothing stops them all targeting the same device. Run them one after another, pin each to its own GPU with --gpu, or use a scheduler profile — see Running on an HPC cluster.

Note

Every launch gets its own run directory under runs/<task>/, and the workspace nextflow.config sets cleanup = true, so the task work directory is deleted once the run succeeds. After a failure it is kept: fix the cause, add --resume to the same command, and Nextflow reuses every task that already completed. scfoundry runs lists the run directories with their status; scfoundry runs --task <task> narrows the list.

Next steps