Benchmark¶

What it does¶

Scores embedding files against known labels and writes metric tables. Against the cell-type labels it reports nine biological conservation metrics; with --batch-key it adds four batch mixing metrics. These are the thirteen metrics of the paper, computed with the paper’s protocol — Leiden clustering with the resolution matched to the number of cell types — so numbers produced here are directly comparable with the published results.

It does not need a model, a GPU or weights: only embedding files and their labels.

Inputs¶

Embedding .h5ad files as produced by embed:

adata.X — the embedding, cells × dimensions
adata.obs[label_key] — cell-type labels, cell_type by default
adata.obs[batch_key] — batch labels, only when --batch-key is given

--embedding accepts one file, a directory, or a quoted glob. All files of one call are scored under one method label — by default the name of the directory they are in, which is the method id when they came from embed.

Note

This is the one task that does not take raw counts. Pointing --embedding at an expression matrix will run and produce numbers, but they will describe your count matrix rather than any model.

Running it¶

Note

Run scfoundry commands from inside a workspace created by scfoundry init (any subdirectory works — the workspace is found by walking upwards, like a git repository), or pass --workspace DIR. Weights, the image cache, results and run logs all live there. See The workspace.

Biological conservation

scfoundry benchmark --embedding results/embeddings/scgpt

[PROCESS aa/c3c25c] BENCHMARK:benchmark_embeddings (scgpt)
[run] colon_1000: bio
[done] wrote out/scgpt_bio_conservation_leiden_metrics_long.csv

[SUCCESS] completed=1 failed=0 cached=0

With batch mixing

scfoundry benchmark --embedding results/embeddings/scgpt --batch-key batch_id

--batch-key switches the metric set to all. Batch metrics are only meaningful when the column has more than one level; files with a single batch get NaN for them.

One file, another label

scfoundry benchmark \
  --embedding results/embeddings/scgpt/colon_1000.h5ad \
  --label-key level_1_annot

Any discrete obs column works. The target cluster count follows the number of distinct values, so a coarse annotation and a fine one give genuinely different — and not directly comparable — scores.

Several methods

for m in pca scgpt geneformer; do
  scfoundry benchmark --embedding results/embeddings/$m --batch-key batch_id
done

Every method writes its own tables, named after the method label, so nothing collides.

Parameters¶

Option	Type	Default	Description
`--embedding`	path	required	Embedding `.h5ad`, a directory of them, or a quoted glob.
`--method`	string	directory name	Method label written into the tables and file names.
`--label-key`	string	`cell_type`	`obs` column with cell-type labels.
`--batch-key`	string	none	`obs` column with batch labels; enables the batch metrics.
`--metrics`	string	`bio`, or `all` with `--batch-key`	`bio`, `batch` or `all`.
`--clustering`	string	`leiden`	Clustering behind the five label-agreement metrics: `leiden` (the paper’s protocol) or `kmeans`.
`--batch-max-cells`	integer	`0`	Stratified subsample size for the batch metrics; `0` uses every cell.

What the thirteen metrics measure¶

All are oriented so that higher is better, and all are computed in {{ registry }}/biometrics:cu12 with scikit-learn and scib-metrics ≥ 0.5.6.

Metric	Group	What it captures
`ARI`	cluster agreement	Adjusted Rand index between clusters and true labels; chance-corrected.
`NMI`	cluster agreement	Normalised mutual information between clusters and labels.
`HOM`	cluster agreement	Homogeneity — does each cluster contain only one cell type?
`COM`	cluster agreement	Completeness — is each cell type confined to one cluster? The complement of `HOM`.
`FMI`	cluster agreement	Fowlkes–Mallows index, the geometric mean of pairwise precision and recall.
`ASW`	geometric separation	Average silhouette width over true labels, Euclidean. The only metric that looks at distances rather than neighbourhoods or partitions.
`cLISI`	local structure	Cell-type local inverse Simpson’s index at perplexity 30, scaled. High when a cell’s neighbours share its type.
`Acc@kNN`	local structure	Accuracy of a 5-nearest-neighbour classifier under 5-fold stratified cross-validation.
`GC`	local structure	Graph connectivity — is each cell type a single connected component in the kNN graph?
`kBET`	batch mixing	Is the batch composition of each neighbourhood consistent with the global one? Computed per cell type, 50 neighbours.
`BRAS`	batch mixing	Batch-removal-adapted silhouette: silhouette over batch labels, cosine distance, between-cluster distances taken as the mean over other clusters.
`CiLISI`	batch mixing	Cell-type-conditioned iLISI: batch LISI computed within each cell type, normalised by that type’s own batch count.
`iLISI`	batch mixing	Integration LISI, perplexity 30. Reported, but excluded from the paper’s overall score because it duplicates `CiLISI`.

Reporting all of them matters because they disagree in informative ways. An embedding can score well on HOM and poorly on COM by shattering one cell type across many pure clusters; high Acc@kNN with low ARI means the local structure is right but the global geometry does not partition cleanly; any batch metric can be maximised by destroying biology, which is why batch metrics are never read alone.

The clustering¶

The five cluster-agreement metrics need a partition, and the choice of clustering changes the numbers. The default reproduces the paper:

a 15-nearest-neighbour graph on the embedding (Euclidean), Leiden with flavor="igraph" and random_state=0;
resolutions 0.1, 0.2, …, 3.0, extended in steps of 0.5 up to 6.0 if no resolution yet reaches the number of cell types;
the resolution whose cluster count is closest to the number of cell types is selected.

--clustering kmeans instead runs k-means with k equal to the number of cell types (random_state=1, n_init=10). It is simpler and faster, but not comparable with the published tables — the metric names are the same; the partitions are not. The note column of every _long.csv records which protocol, which resolution and which cluster count produced each value.

ASW, cLISI, Acc@kNN, GC and the batch metrics use the true labels directly and are unaffected by this choice.

Outputs¶

results/benchmark/
├── <method>_bio_conservation_leiden_metrics_wide.csv     one row per file, one column per metric
├── <method>_bio_conservation_leiden_metrics_long.csv     one row per metric, with provenance
├── <method>_batch_mixing_leiden_metrics_wide.csv         with --batch-key
├── <method>_batch_mixing_leiden_metrics_long.csv
├── bio_conservation_leiden_metrics/<method>/<file>_<method>.csv        per-file tables
├── batch_mixing_leiden_metrics/<method>/<file>_<method>.csv
└── bio_conservation_leiden_predictions/<method>/<file>_<method>_leiden_labels.tsv

With --clustering kmeans the _leiden infix is dropped and the cluster labels are _kmeans_labels.tsv.

The wide table is what you will usually read:

sample_id,method,ARI,ASW,Acc@kNN,COM,FMI,GC,HOM,NMI,cLISI
colon_1000,scgpt,0.589,0.142,0.906,0.650,0.702,0.824,0.655,0.652,0.990

The long table carries the same values with their provenance:

sample_id,method,metric,value,note,metric_group,n_cells,n_dims,n_labels,n_batches
colon_1000,scgpt,NMI,0.652,clustering=leiden;selection=match_n_labels;resolution=0.3;n_clusters=6;target_n_labels=7;scan_n=30,bio_conservation,1000,512,7,

The cluster-label TSV lets you inspect the selected partition cell by cell:

sample_id	method	barcode	true_label	leiden_label
colon_1000	scgpt	GAGGTGAAGGATGGTC-HT-236_day132_colon	intestine goblet cell	0

Reading results in Python¶

Comparing several methods means stacking their wide tables:

from pathlib import Path
import pandas as pd

frames = [
    pd.read_csv(p) for p in Path("results/benchmark").glob("*_bio_conservation_leiden_metrics_wide.csv")
]
wide = pd.concat(frames)
print(wide.pivot_table(index="method", columns="sample_id", values="ARI").round(3))

To rank methods the way the paper does — per-metric ranks within each dataset, averaged within weighted metric families, then averaged over datasets — see Evaluation metrics.

Gotchas¶

Batch metrics need the right column. --batch-key donor_id on a single-donor file runs, but every batch metric is NaN. The n_batches column of the long table tells you what the scorer saw.

Scaling. Metric computation is CPU-bound and memory-hungry: X is densified, and ASW computes a full pairwise distance matrix. Hundreds of thousands of cells want a large-memory CPU node; --batch-max-cells 50000 bounds the batch metrics, which are the most expensive part.

Label cardinality drives the cluster count. Scoring the same embedding against a 20-type and a 60-type annotation gives two different resolution scans. Compare methods against the same label column only.

Note

Every launch gets its own run directory under runs/<task>/, and the workspace nextflow.config sets cleanup = true, so the task work directory is deleted once the run succeeds. After a failure it is kept: fix the cause, add --resume to the same command, and Nextflow reuses every task that already completed. scfoundry runs lists the run directories with their status; scfoundry runs --task <task> narrows the list.

Next steps¶

Evaluation metrics — precise definitions, metric families, and how the paper aggregates them.
Geometry — measure the embedding rather than score it.