Input data format¶

Every task takes the same input: AnnData .h5ad files. The contract is short, but each part of it matters, because violations do not raise errors — they produce plausible-looking output that is quietly wrong.

The contract at a glance¶

Where	What	Requirement
`adata.X`	matrix	Raw integer counts, full transcriptome. Not normalised, not log-transformed, not subset to highly variable genes. Sparse (CSR) or dense both work.
`adata.var.index`	gene ids	HGNC gene symbols where one exists; Ensembl gene ID as a fallback for genes that have none. Must be unique.
`adata.var["gene_symbol"]`	column	The same identifier as the index — the value models tokenise against.
`adata.var["ensembl_id"]`	column	Ensembl gene ID. Required by Geneformer and scPRINT.
`adata.obs["barcode"]`	column	Unique cell identifier. Usually a copy of `adata.obs_names`.
`adata.obs["cell_type"]`	column	Labels. Read by `benchmark`, `geometry`, and on the reference set of `transfer` and `finetune`; the column name is configurable with `--label-key`. Not needed for `embed`.
`adata.obs["batch_id"]`	column	Batch labels. Read by the integration methods of `embed`, and by `benchmark` and `geometry` for the batch statistics; configurable with `--batch-key`.
`adata.obsm["spatial"]`	array	Spatial coordinates. Required by `novae` only.

Raw counts, full transcriptome¶

Warning

Input must be raw counts over the full transcriptome. Passing log-normalised values, or an object already subset to highly variable genes, does not raise an error — it produces a plausible-looking embedding that is silently wrong. See Input data format.

This is the requirement people most often break, so it is worth spelling out why.

Foundation models are pretrained on raw counts. Most of them rank or bin expression internally — Geneformer builds a rank-ordered gene list, scBERT bins values, scGPT tokenises binned expression. Handing them values that have already been normalised and log-transformed shifts the input distribution away from anything seen during pretraining, and the model has no way to detect it.

Subsetting to highly variable genes is worse still. Each model matches your genes against its own fixed vocabulary. Remove 90% of the transcriptome and you have not just reduced the information available — you have changed which vocabulary entries are populated, in a way that varies per model. The comparison then measures your HVG selection more than it measures the models.

Do your quality control — filter low-quality cells, remove doublets if you want to — then stop. Everything downstream of that is the framework’s job.

Gene identifiers¶

var.index should carry HGNC gene symbols. Genes that have no official symbol may use their Ensembl gene ID instead. This keeps every gene represented by a valid token rather than dropping it.

The demo file shows exactly this pattern:

>>> adata.var_names[:5].tolist()
['MIR1302-2HG', 'FAM138A', 'OR4F5', 'ENSG00000239945', 'ENSG00000239906']

Three named genes, then two that fall back to Ensembl IDs. Both var["gene_symbol"] and the index carry this same mixed identifier; var["ensembl_id"] always carries the Ensembl ID.

Tip

If you are starting from a CZ CELLxGENE download, the field you want is var["feature_name"] — but check it first. CELLxGENE sometimes puts an Ensembl ID in feature_name for unnamed genes, which is precisely the fallback behaviour you want, so it usually maps across directly.

Cell identifiers¶

obs["barcode"] must exist and must be unique. Prediction tables are indexed by it, and geometry aligns an embedding with its input by obs_names.

adata.obs_names_make_unique()
adata.obs["barcode"] = adata.obs_names

Labels and batches¶

cell_type and batch_id are the default column names, not hard requirements. Any discrete column works, as long as you name it:

scfoundry transfer  ... --label-key my_annotation
scfoundry finetune  ... --label-key my_annotation
scfoundry benchmark ... --label-key my_annotation --batch-key donor_id
scfoundry embed --method harmony ... --batch-key donor_id

For the integration methods, think about what your batch column actually represents. A genuine technical axis — different assays, different sequencing runs — is what these methods are designed to remove. A donor-only split in a single-assay dataset confounds batch with biology, and aggressive integration will erase signal you wanted to keep.

Checking a file before you run¶

scfoundry check reads a file and reports, line by line, whether it satisfies the contract — raw counts, full transcriptome, unique identifiers, the columns each task and method reads. It changes nothing and never blocks a run; it exists so that the silent failures above are caught before a multi-hour job rather than after it.

scfoundry check --data demo/colon_1000.h5ad

== colon_1000.h5ad ==
  [INFO] 1,000 cells x 35,461 genes
  [OK  ] adata.X is integer-valued (raw counts), sparse (csr_matrix), maximum 2852
  [OK  ] 35,461 genes: full transcriptome
  [OK  ] gene index is unique
  [INFO] 26.3% of gene names are Ensembl IDs (fallback for genes without a symbol)
  [OK  ] var['gene_symbol'] present and equal to the gene index
  [OK  ] var['ensembl_id'] present
  [OK  ] obs['barcode'] present, unique and equal to obs_names
  [OK  ] obs['cell_type'] present: 7 classes, smallest has 20 cells, largest 467 cells
  [OK  ] obs['batch_id'] present: 7 batches, smallest has 30 cells
  [INFO] the smallest batch has 30 cells: seurat_cca and seurat_rpca cannot integrate batches of 30 cells or fewer (harmony can)
  8 ok, 0 warnings, 0 problems
  ready for embed

FAIL marks a definite contract violation — the things every task needs: raw counts, a unique gene index with gene_symbol, a unique barcode. WARN is a heuristic worth a look, INFO a fact about the file. Labels, batches and Ensembl IDs are not required to embed, so their absence is only reported, together with what it rules out:

  [INFO] var['ensembl_id'] is missing
  [INFO] obs['cell_type'] is missing (not needed to embed; pass --label-key if the labels are in another column)
  [INFO] obs['batch_id'] is missing (not needed to embed; pass --batch-key if the batches are in another column)
  5 ok, 0 warnings, 0 problems
  ready for embed
    without var['ensembl_id'] you cannot run geneformer or scprint
    without obs['cell_type'] you cannot run benchmark or use the file as a transfer/finetune reference; geometry runs, but its cell-type statistics (within/between-type anisotropy, partial eta-squared) stay empty
    without obs['batch_id'] the integration methods, the batch-mixing metrics and the batch terms of geometry treat the data as one batch

Tell it what you intend to do and the corresponding column becomes a requirement:

scfoundry check --data reference.h5ad --role reference --label-key cell_type   # labels required
scfoundry check --data cells.h5ad --method geneformer                           # ensembl_id required
scfoundry check --data cells.h5ad --method seurat_cca --batch-key donor_id      # batch sizes
scfoundry check --data tissues/                                                 # every file in a directory

On a log-normalised, HVG-subset file with no barcode column the report reads:

  [FAIL] adata.X is not integer-valued and its maximum is 7.16: this looks log-normalised. Models expect raw counts in X; raw counts appear to be in layers['counts']
  [WARN] only 3,000 genes: models match genes against their own vocabulary, so an HVG-subset input is scored on your gene selection rather than on the model (fine for a targeted panel)
  [FAIL] obs['barcode'] is missing; set adata.obs['barcode'] = adata.obs_names
  ...
  5 ok, 1 warning, 2 problems
  fix the problems before running a task on this file

The report is also written to results/check/<file>_check.txt. The check runs in a container like every other task (it needs anndata), so the first call pulls the small scllms image.

Note

The gene-count and maximum-value checks are heuristics, not rules. A targeted panel legitimately has few genes, a shallow assay legitimately has low counts, and corrected counts (SoupX and the like) are legitimately non-integer. They are there to catch the common accident, so read what they say rather than treating them as a pass/fail gate.

Preparing a file from scratch¶

A minimal conversion from a raw CELLxGENE download:

import anndata as ad

adata = ad.read_h5ad("cellxgene_download.h5ad")

# CELLxGENE keeps raw counts in .raw; make sure X holds them.
if adata.raw is not None:
    adata = adata.raw.to_adata()

# Gene identifiers: symbol where available, Ensembl ID as fallback.
adata.var["ensembl_id"] = adata.var_names  # CELLxGENE indexes by Ensembl ID
adata.var["gene_symbol"] = adata.var["feature_name"].astype(str)
adata.var_names = adata.var["gene_symbol"]
adata.var_names_make_unique()
adata.var["gene_symbol"] = adata.var_names

# Cell identifiers.
adata.obs_names_make_unique()
adata.obs["barcode"] = adata.obs_names

# Batch axis: assay crossed with donor is a reasonable technical definition.
adata.obs["batch_id"] = (
    adata.obs["assay"].astype(str) + "|" + adata.obs["donor_id"].astype(str)
)

del adata.raw
adata.write_h5ad("prepared.h5ad")

Warning

Check where your raw counts actually live before writing. Depending on the source they may be in adata.X, adata.raw.X, or a layer such as adata.layers["counts"]. Copying the wrong one is the single easiest way to produce a silently wrong benchmark.

Many files at once¶

embed takes one file per run, so a tissue collection is a shell loop. Each run writes its own output under results/embeddings/scgpt/, named after the input’s basename, and leaves its own record under runs/:

for f in tissues/*.h5ad; do
  scfoundry embed --method scgpt --data "$f"
done

benchmark and geometry accept a whole directory of embeddings in one call. See Embed for how names propagate.