Reproducing the paper

The accompanying manuscript benchmarks thirteen single-cell foundation models, two reference methods and five batch-integration methods across 26 human tissues, and characterises the geometry of every embedding. This section documents that work at the level you need to reproduce or extend it: which data, which commands, how every metric and probe is defined, and how metrics are aggregated into the rankings the paper reports.

What this section covers

Benchmark design

The dataset, how it was prepared, how batches are defined, and the commands that regenerate every embedding and metric.

Benchmark design
Evaluation metrics

Precise definitions for all thirteen metrics, the four metric families and their weights, and how the overall score and the ranks are computed.

Evaluation metrics
Geometry probes

How effective dimension, anisotropy, R_NX, intrinsic dimension and partial η² are defined and computed.

Representation-geometry probes

Scope

Everything here is reproducible with the tasks documented on this site. The embeddings, all thirteen metrics under the paper’s Leiden protocol, and every geometry statistic are what scfoundry embed, scfoundry benchmark and scfoundry geometry compute by default.

Not covered here — the paper’s own analysis scripts: the bootstrap confidence intervals, the leave-one-tissue-out robustness analysis, the moderator meta-regressions and the figure code. These are research code tied to a specific compute environment and are described in the manuscript’s methods section rather than released as part of the framework. The aggregation from metrics to ranks is simple enough to give in full, and Evaluation metrics does.

The shape of a reproduction

Four stages, in order.

1. Prepare inputs. Download the tissue files from CZ CELLxGENE and convert them to the input contract — gene symbols with Ensembl fallback, unique barcodes, and a batch_id defined as assay crossed with donor. Details in Benchmark design.

2. Generate embeddings. One scfoundry embed run per method and tissue. This is the expensive stage — budget GPU days for the full sweep, and see Running on an HPC cluster for how to structure it.

3. Compute metrics and probes. One scfoundry benchmark --batch-key batch_id call per method scores all its tissues; one scfoundry geometry call per method measures them.

4. Aggregate and rank. Collapse metrics into four families, weight them, and rank methods per tissue and overall. The weights and the code are in Evaluation metrics.

Reproducing on a smaller scale

The full benchmark is a large undertaking. A subset is often enough to answer a specific question, and the design carries over unchanged:

  • One tissue, all methods — tests whether the method ranking you care about holds on your tissue of interest.

  • All tissues, three methods — the best foundation model, the worst, and PCA. Enough to establish whether the foundation-model premium is real for your data.

  • Your own data, the same protocol — the most useful variant. The metric definitions and aggregation are the transferable part; the specific dataset is not.

See also