Evaluation metrics¶

Thirteen metrics in four families. All are oriented so that higher is better, and all are bounded, which is what makes weighted aggregation across them meaningful. All are computed by scfoundry benchmark.

Biological conservation metrics¶

These ask whether cells of the same type end up near each other. Computed against obs["cell_type"] on every tissue.

Metric	Implementation	What it measures
`ARI`	`sklearn.metrics.adjusted_rand_score`	Agreement between clusters and true labels over all cell pairs, corrected for chance.
`NMI`	`sklearn.metrics.normalized_mutual_info_score`	Shared information between the cluster partition and the label partition.
`HOM`	`sklearn.metrics.homogeneity_score`	Does each cluster contain only one cell type? Rewards splitting.
`COM`	`sklearn.metrics.completeness_score`	Is each cell type confined to one cluster? Rewards merging. The counterweight to `HOM`.
`FMI`	`sklearn.metrics.fowlkes_mallows_score`	Geometric mean of pairwise precision and recall.
`ASW`	`sklearn.metrics.silhouette_score`	Average silhouette width over true labels, Euclidean. Compares within-type to between-type distance — the only metric that uses distances directly rather than neighbourhoods or partitions.
`cLISI`	`scib_metrics.clisi_knn`	Cell-type local inverse Simpson’s index, perplexity 30, scaled. High when a cell’s neighbours share its type.
`Acc@kNN`	`sklearn` 5-NN, 5-fold stratified CV	Can a cell’s type be predicted from its neighbours? The most directly interpretable metric here.
`GC`	`scib_metrics.graph_connectivity`	Is each cell type a single connected component in the 15-nearest-neighbour graph, or fragmented?

Reporting all nine matters because they fail differently. HOM and COM pull in opposite directions, so an embedding can score well on one by doing badly at the other. High Acc@kNN with low ARI means local structure is right while global geometry does not partition cleanly. A single summary metric hides all of this.

Batch-mixing metrics¶

These ask whether cells from different technical batches are interleaved. Computed against obs["batch_id"] on the 19 multi-batch tissues.

Metric	Implementation	What it measures
`kBET`	`scib_metrics.kbet_per_label`	Is the batch composition of each local neighbourhood consistent with the global composition? Computed per cell type, α = 0.05, 50 neighbours.
`BRAS`	`scib_metrics.bras`	Batch-removal-adapted silhouette: silhouette over batch labels, cosine distance, with between-cluster distances taken as the mean over other clusters. Requires `scib-metrics` ≥ 0.5.6.
`CiLISI`	built on `scib_metrics.lisi_knn`	Cell-type-conditioned iLISI: batch LISI computed within each cell type, normalised by that cell type’s own batch count, averaged over cell types with at least 10 cells and at least 2 batches.
`iLISI`	`scib_metrics.ilisi_knn`	Integration LISI, perplexity 30, scaled. Computed but excluded from the reported score.

Important

iLISI is computed and stored but does not enter the batch-mixing family. It and CiLISI measure nearly the same thing, and including both would double-weight LISI-style mixing relative to kBET and BRAS. CiLISI is preferred because conditioning on cell type prevents a method from scoring well simply by mixing everything together.

The conditioning point generalises. Any batch-mixing metric can be maximised by destroying biological structure, which is why batch metrics are never reported alone — the weighted score always includes the three biological families.

Metric families and weights¶

The twelve scored metrics are correlated in blocks. Averaging them equally would give the five cluster-agreement metrics five times the influence of ASW, purely because there happen to be five of them. Grouping into families and weighting the families fixes that.

Family	Weight	Metrics
Cluster-label concordance	0.325	`ARI`, `NMI`, `HOM`, `COM`, `FMI`
Continuous cell-type separation	0.10	`ASW`
Local cell-type neighbourhood structure	0.325	`Acc@kNN`, `cLISI`, `GC`
Batch mixing	0.25	`kBET`, `BRAS`, `CiLISI`

Three quarters of the weight sits on biology, split evenly between partition-level agreement (0.325) and neighbourhood-level structure (0.325), with continuous geometry deliberately light at 0.10 — ASW is the most sensitive to embedding scale and dimensionality, and so the least comparable across methods. Batch mixing takes the remaining 0.25.

Missing families¶

When a family cannot be computed — the batch family on a single-batch tissue — it is dropped and the remaining weights are renormalised to sum to 1. A single-batch tissue is scored on its three biological families, not penalised for lacking a fourth.

Clustering¶

The five cluster-agreement metrics need a partition, and the choice of clustering changes the numbers. The paper’s protocol, and the default of scfoundry benchmark:

a 15-nearest-neighbour graph on the embedding, Euclidean; Leiden with flavor="igraph" and random_state=0;
resolutions 0.1, 0.2, …, 3.0, extended in steps of 0.5 up to 6.0 while no resolution yet reaches the number of true labels;
the resolution whose cluster count is closest to the number of true labels is selected.

Leiden is graph-based, matching how single-cell data is normally clustered, and does not assume the isotropic clusters k-means implies. scfoundry benchmark --clustering kmeans is offered as the simpler alternative — k-means with k equal to the number of true labels, random_state=1, n_init=10 — and its numbers are not interchangeable with the Leiden ones. Say which protocol you used; the note column of every long-format table records it.

The four non-clustering metrics — ASW, cLISI, Acc@kNN, GC — and the batch metrics use true labels directly and are unaffected by this choice.

Ranking across tissues¶

Raw metric values are not comparable between tissues. A tissue with 32 cell types produces systematically lower ARI than one with 14, whatever the method. Averaging raw values across tissues therefore ranks tissue difficulty as much as method quality — and averaging raw values across metrics lets the metric with the widest numeric range dominate.

The paper therefore works with ranks throughout. Within each tissue:

metric rank      rank of the method on that metric among all methods (1 = best; ties averaged)
family rank      mean of the metric ranks within the family, over the metrics observed
aggregate rank   Σ weight_f × family_rank_f  /  Σ weight_f     over the families observed

and the mean aggregate rank is the mean of a method’s per-tissue aggregate ranks. A method that places second in every tissue beats one that wins three tissues and comes last in the rest — which is the right notion of a generally useful method. All 20 methods, zero-shot and integration alike, are ranked together.

from pathlib import Path
import pandas as pd

# One wide table per method and metric group, as written by `scfoundry benchmark`.
bio = pd.concat(pd.read_csv(p) for p in Path("results/benchmark").glob("*_bio_conservation_leiden_metrics_wide.csv"))
batch = pd.concat(pd.read_csv(p) for p in Path("results/benchmark").glob("*_batch_mixing_leiden_metrics_wide.csv"))
wide = bio.merge(batch, on=["sample_id", "method"], how="left").set_index(["sample_id", "method"])

FAMILIES = {
    "cluster_label_concordance":             ["ARI", "NMI", "HOM", "COM", "FMI"],
    "continuous_cell_type_separation":       ["ASW"],
    "local_cell_type_neighborhood_structure": ["Acc@kNN", "cLISI", "GC"],
    "batch_mixing":                          ["kBET", "BRAS", "CiLISI"],
}
WEIGHTS = pd.Series({
    "cluster_label_concordance": 0.325,
    "continuous_cell_type_separation": 0.10,
    "local_cell_type_neighborhood_structure": 0.325,
    "batch_mixing": 0.25,
})

# 1. Rank every method on every metric within each tissue (higher value = better rank).
metric_rank = wide.groupby(level="sample_id").rank(ascending=False, method="average")

# 2. Average the metric ranks within each family; a family with no observed metric is NaN.
family_rank = pd.DataFrame({f: metric_rank[cols].mean(axis=1) for f, cols in FAMILIES.items()})

# 3. Weighted mean of the family ranks, weights renormalised over the families observed.
present = family_rank.notna()
aggregate = family_rank.mul(WEIGHTS, axis=1).sum(axis=1) / present.mul(WEIGHTS, axis=1).sum(axis=1)

# 4. Average over tissues.
print(aggregate.groupby(level="method").mean().sort_values())   # lower is better

On the Tabula Sapiens v2 tables this reproduces the published ranking to the last digit; the manuscript adds tissue-level bootstrap confidence intervals, Friedman and paired Wilcoxon tests against PCA and scVI, and leave-one-tissue-out stability on top of it.

Reproducing metric values¶

scfoundry benchmark --embedding results/embeddings/scgpt --batch-key batch_id

reproduces every value of the published tables for that method: Leiden protocol, kNN settings, scib-metrics ≥ 0.5.6 with kbet_per_label at 50 neighbours, bras with cosine distance and between_cluster_distances="mean_other", and the cell-type-conditioned LISI for CiLISI. Leiden is deterministic given the graph, so reruns agree to the last digit on the same software; across scib-metrics or igraph versions expect differences in the third decimal.

Note

BRAS changed definition across scib-metrics releases. Values computed with anything older than 0.5.6 are not comparable, and the note column of the long table records the definition used alongside each value for exactly this reason.