Benchmark design¶

The dataset¶

The zero-shot benchmark uses Tabula Sapiens v2 — released through CZ CELLxGENE Discover as collection 10df7690-6d10-4029-a47e-0f071bb2df83 under CELLxGENE schema 5.2.0, and distributed as per-tissue benchmarking files that can be downloaded with one command (below).

Property	Value
Tissues	26
Cells	548,977
Cells per tissue	2,044 – 72,290
Cell types per tissue	14 – 32
Genes per tissue	21,808 – 37,066
Tissues with multiple technical batches	19

Using 26 separate tissues rather than one pooled object is deliberate. Each tissue is an independent evaluation with its own cell-type composition and its own batch structure, which turns a single benchmark into 26 replicates — and lets the analysis ask whether a method’s advantage is consistent or tissue-specific.

Downloading it¶

The 26 per-tissue files are distributed through CZI’s public benchmark-data bucket, with anonymous access — about 25 GB in total (pip install awscli if aws is not on your machine):

aws s3 sync --no-sign-request \
  s3://cz-benchmarks-data/datasets/v1/cell_atlases/Homo_sapiens/Tabula_Sapiens_v2/ raw/

One .h5ad per tissue, named homo_sapiens_10df7690-…_<Tissue>_v2_curated.h5ad. The sample_id in every result table of this site is exactly this filename, so what you download is byte-for-byte what the benchmark ran on; the preparation below changes metadata only.

Tissues¶

Bladder          Ear      Heart             Liver        Muscle   Salivary_Gland   Spleen   Thymus    Uterus
Blood            Eye      Large_Intestine   Lung         Ovary    Skin             Stomach  Tongue    Vasculature
Bone_Marrow      Fat      Lymph_Node        Mammary      Prostate Small_Intestine  Testis   Trachea

Seven have only a single technical batch — Eye, Lung, Mammary, Prostate, Skin, Testis and Uterus — so batch-mixing metrics are undefined for them. The remaining 19 carry the batch comparison.

Preparing the inputs¶

Raw CELLxGENE files do not satisfy the input contract directly. The preparation is metadata-only: no quality control, no normalisation, no gene filtering, so the models receive the data as deposited.

Four transformations:

1. Gene identifiers. CELLxGENE indexes by Ensembl ID and carries display names in var["feature_name"]. The benchmark uses feature_name as the primary identifier where it is a usable HGNC symbol, and falls back to the Ensembl ID where it is not — an Ensembl-looking or version-suffixed feature_name is treated as no symbol at all. The result is written to the index, to var["gene_symbol"], and the Ensembl ID to var["ensembl_id"]. Preparation fails loudly if the derived identifiers are not unique, rather than silently deduplicating.

2. Cell identifiers. obs["barcode"] is set from the original observation index.

3. Batch definition. This is the substantive choice:

adata.obs["batch_id"] = (
    adata.obs["assay"].astype(str) + "|" + adata.obs["donor_id"].astype(str)
)

Batch is assay crossed with donor, not donor alone. A donor-only definition would confound batch with biology — donors differ genetically and clinically, not just technically — and any method that removed it would be removing signal. Crossing with assay means a tissue only counts as multi-batch when the same biological material was measured on more than one platform.

4. Raw slot. The AnnData raw slot is deleted; X is left untouched.

Generating embeddings¶

One run per method and tissue. Every method sees byte-identical input files — the whole point of the framework.

for m in pca scvi scgpt geneformer scfoundation scbert sccello uce scimilarity \
         scprint langcell cellfm cellplm cellama genept c2s; do
  for f in prepared/*.h5ad; do
    scfoundry embed --method "$m" --data "$f" --gpu 0
  done
done

for m in scgpt_integrated scvi_denovo harmony seurat_cca seurat_rpca; do
  for f in prepared/*.h5ad; do
    scfoundry embed --method "$m" --data "$f" --batch-key batch_id --gpu 0
  done
done

Defaults are used throughout — no per-method tuning, no per-tissue tuning. A benchmark where each method is tuned by a different amount measures the tuning effort, not the models.

Note

The full sweep is 20 methods × 26 tissues = 520 embedding runs. Every run has its own directory and log, so a failure costs one method-tissue pair rather than the sweep; on a cluster, add --profile slurm and let the scheduler place them. See Running on an HPC cluster.

pca is included as the reference point. It has no pretraining, no checkpoint, and no GPU requirement, and it is the standard every foundation model has to clear to justify its cost.

Computing metrics¶

One call per method scores all 26 embeddings:

for m in $(ls results/embeddings); do
  scfoundry benchmark --embedding results/embeddings/$m --batch-key batch_id
done

Biological conservation is scored on every tissue against obs["cell_type"]; batch mixing against obs["batch_id"] wherever it has more than one level, which leaves the seven single-batch tissues with NaN batch metrics rather than misleading ones. The wide tables in results/benchmark/ are, row for row, the tables offered for download.

Measuring geometry¶

Likewise one call per method, pairing each embedding with the prepared input of the same name:

for m in $(ls results/embeddings); do
  scfoundry geometry --embedding results/embeddings/$m --data prepared/
done

Tissues larger than 20,000 cells are subsampled once, with a seed derived from the tissue name, so every method is measured on the same cells. The defaults (--max-cells 20000, --seed 0) are the paper’s settings.

Aggregating¶

Within each tissue every method is ranked on every metric, the ranks are averaged within four families, the family ranks are weighted, and the per-tissue aggregate ranks are averaged over tissues. Definitions, weights and the code are on the metrics page.

Two properties of the aggregation are worth stating here, because they shape how results should be read.

Single-batch tissues are not penalised. When the batch-mixing family is undefined, it is dropped and the remaining three weights are renormalised. A tissue with one batch is scored on biology alone rather than being scored as if it had failed at batch correction.

Ranking is per tissue and per metric, then aggregated. Methods are ranked within each tissue on each metric and the ranks are combined, rather than averaging raw metric values. Raw values are not comparable between tissues — a tissue with 32 cell types yields systematically lower ARI than one with 14, regardless of method — so averaging them would let tissue difficulty dominate the result; and they are not comparable between metrics either, so ranking per metric keeps a wide-ranging metric from dominating its family.

Extending the benchmark to your own data¶

The protocol transfers directly. Three things to keep if you want your numbers to mean the same thing:

Prepare identically. Metadata only. Any quality control you apply must be applied before preparation and identically for every method.

Use defaults. Tuning one method and not others produces a comparison of your effort rather than of the models.

Include PCA. It is cheap, and without a reference a table of foundation-model scores has no scale. If nothing beats PCA on your data, that is the finding.