Benchmark results¶

Results of running every supported method over Tabula Sapiens v2 — 26 human tissues, 548,977 cells — with default settings throughout. This page reports the headline rankings; the explorer lets you drill into any tissue or metric, and What the geometry says shows the same embeddings measured rather than ranked.

Important

These are the results reported in the accompanying manuscript, which is under revision. Please cite the paper if you use them.

The headline¶

Most single-cell foundation models do not beat PCA. Ranking all 20 methods together within each tissue and averaging over tissues, only SCimilarity is significantly better than the PCA reference; scFoundation edges ahead of it on average without a significant margin, and the other eleven foundation models rank below it — seven of them significantly (CELLama, CellFM, GenePT-w, Geneformer, LangCell, scBERT, scPRINT).

Classical integration still wins on batch correction. Among methods that actively use batch labels, Harmony and Seurat RPCA take the second and third places overall, ahead of scVI trained de novo and well ahead of a scGPT fine-tuned for integration.

The ranking¶

Lower is better. The mean aggregate rank is the manuscript’s headline figure: within each tissue, every method is ranked on each of the twelve scored metrics, the ranks are averaged within the four metric families, the family ranks are weighted (0.325 / 0.10 / 0.325 / 0.25) and the result is averaged over the 26 tissues. The interval is a 95% tissue-level bootstrap (10,000 resamples). best and worst are the method’s best and worst per-tissue aggregate rank.

Zero-shot embedding¶

Method	Mean aggregate rank	95% CI	Best	Worst
SCimilarity	5.54	4.99 – 6.06	2.2	8.6
scFoundation	7.77	7.09 – 8.41	4.1	10.4
PCA (reference)	8.00	7.08 – 8.93	2.8	12.0
UCE	8.15	7.57 – 8.74	5.3	11.3
CellPLM	8.84	8.01 – 9.68	5.6	12.9
scGPT	9.53	8.81 – 10.18	5.3	12.2
scVI (pretrained Census model)	10.02	9.13 – 10.91	5.3	15.3
scCello	10.18	9.39 – 10.95	5.1	15.5
LangCell	10.48	9.76 – 11.24	7.2	14.2
Geneformer	13.17	12.52 – 13.77	10.0	16.2
scPRINT	13.26	12.67 – 13.82	9.8	15.9
CellFM	14.25	13.58 – 14.91	9.4	17.2
CELLama	15.94	15.51 – 16.39	14.0	17.9
GenePT-w	17.40	17.05 – 17.79	16.3	19.0
scBERT	18.89	18.64 – 19.17	18.0	20.0

The spread within a method matters as much as its mean. SCimilarity’s interval does not overlap any other method’s, and it is the top method in 99% of bootstrap resamples. PCA’s per-tissue aggregate rank runs from 2.8 to 12.0 — a strong reference on some tissues and a middling one on others — whereas scBERT never rises above 18.0 and is last outright on several tissues.

Active batch integration¶

Method	Mean aggregate rank	95% CI	Best	Worst
Harmony (baseline)	7.03	6.15 – 7.96	2.9	13.5
Seurat RPCA (baseline)	7.14	6.37 – 7.90	3.4	11.5
scVI (de novo) (baseline)	7.74	7.09 – 8.41	4.8	11.6
Seurat CCA (baseline)	7.95	7.10 – 8.81	3.6	14.4
scGPT (integrated)	8.72	8.02 – 9.51	5.3	14.9

These five are ranked in the same 20-method table as the zero-shot methods, so the numbers are comparable across the two lists. Harmony, Seurat RPCA and scVI trained de novo rank above every foundation model except SCimilarity — that is what consuming the batch labels buys — while scGPT fine-tuned for integration comes last of the five, behind UCE run zero-shot.

How to read these numbers¶

Ranks, not raw scores. Methods are ranked within each tissue, then ranks are averaged. Raw metric values are not comparable between tissues — a tissue with 32 cell types yields systematically lower ARI than one with 14, whatever the method — so averaging raw values would rank tissue difficulty as much as method quality.

Ranks per metric, then families. Within a tissue every method is ranked on each metric separately, so a single metric with a wide numeric range cannot dominate. The twelve metric ranks are averaged within four families — cluster-label concordance (ARI, NMI, HOM, COM, FMI), continuous cell-type separation (ASW), local cell-type neighbourhood structure (Acc@kNN, cLISI, GC) and batch mixing (kBET, BRAS, CiLISI) — and the family ranks are weighted 0.325, 0.10, 0.325 and 0.25, as defined in Evaluation metrics. On the seven single-batch tissues the batch family is undefined, so it is dropped and the remaining weights are renormalised.

Everyone in one table. Zero-shot and integration methods are ranked together, as in the manuscript; the two lists above are the same ranking split by category. Integration methods see batch labels that zero-shot methods never do, so a comparison across the lists is a comparison of what that extra information buys, not of model quality alone.

The same protocol you can run. Every metric behind these ranks is what scfoundry benchmark --batch-key batch_id computes by default — Leiden clustering with the resolution matched to the number of cell types, the same kNN settings, the same scib-metrics definitions — and the aggregation is the few lines of Python on the metrics page. A number you produce on your own data is comparable with a number here.

Defaults throughout. No per-method or per-tissue tuning. A benchmark where each method receives a different amount of tuning measures the tuning, not the models.

Explore the full results¶

Interactive explorer

Every method, every tissue, all thirteen metrics and the per-tissue aggregate ranks. Filter by tissue or method type, sort by any column, and drill into a single tissue.

Results explorer

What the geometry says

Effective dimension, anisotropy and expression-neighbourhood preservation of the same embeddings, and how they line up with the ranking.

What the geometry says

Download the data¶

Biological conservation metrics — 520 rows, 20 methods × 26 tissues, nine metrics
Batch mixing metrics — 380 rows, 20 methods × 19 multi-batch tissues, four metrics
Representation-geometry statistics — 520 rows, 20 methods × 26 tissues

All are wide-format CSVs keyed by sample_id and method. sample_id is the prepared Tabula Sapiens filename; the tissue is the segment between the collection ID and _v2_curated.