Geometry

What it does

Measures the geometry of an embedding rather than scoring it. Where benchmark asks how well does this embedding separate known cell types, geometry asks what shape is it — how many directions it really uses, whether every cell points the same way, how much of the input’s expression neighbourhood it preserves, and how much of its variance cell type and batch explain.

These are the representation-geometry probes introduced in the paper’s revision. They are descriptive statistics with no notion of better or worse, but they turned out to explain a good deal of why some models rank where they do — see What the geometry says.

Inputs

Each embedding file is paired with the raw-count input it was computed from, matched by file name and aligned by obs_names:

  • --embedding — embedding .h5ad files from embed: one file, a directory, or a quoted glob.

  • --data — the corresponding input .h5ad files, with raw counts in X: one file, or a directory that contains a file of the same name for every embedding.

Labels come from obs[label_key] (cell_type by default) and batches from obs[batch_key] (batch_id). Neither is required: without labels the label-based statistics (within/between-type anisotropy, partial η² for cell type) are left empty and everything else — effective dimension, spectral and mixed-pair anisotropy, R_NX, intrinsic dimension — is computed; a missing batch column means one batch.

Running it

Note

Run scfoundry commands from inside a workspace created by scfoundry init (any subdirectory works — the workspace is found by walking upwards, like a git repository), or pass --workspace DIR. Weights, the image cache, results and run logs all live there. See The workspace.

scfoundry geometry \
  --embedding results/embeddings/scgpt/colon_1000.h5ad \
  --data demo/colon_1000.h5ad

[PROCESS 77/7d41c7] GEOMETRY:geometry_probes (scgpt)

[SUCCESS] completed=1 failed=0 cached=0

scfoundry geometry --embedding results/embeddings/scgpt --data demo

Every embedding under the directory is scored, each against the file of the same name under --data. All of them share one method label — the directory name, or --method.

for m in pca scgpt scfoundation; do
  scfoundry geometry --embedding results/embeddings/$m --data tissues/
done

Parameters

Option

Type

Default

Description

--embedding

path

required

Embedding .h5ad, a directory of them, or a quoted glob.

--data

path

required

The raw-count input file, or a directory matched by file name.

--method

string

directory name

Method label written into the tables.

--label-key

string

cell_type

obs column with cell-type labels.

--batch-key

string

batch_id

obs column with batch labels.

--max-cells

integer

20000

Subsample datasets larger than this.

--seed

integer

0

Seed for the subsample and the pair sampling.

Subsampling is seeded by the dataset name, so every method scored on the same dataset sees the same cells — a requirement for comparing statistics across methods.

What the statistics measure

Probe

Columns

What it captures

Effective dimension

pr, npr

Participation ratio (tr C)² / tr(C²) of the coordinate-standardised embedding — the number of directions that carry variance — and its ratio to the embedding dimension. npr near 1 means variance spread evenly; near 0 means collapse onto a few axes.

Spectral anisotropy

aniso_spec

Share of the uncentred embedding’s squared Frobenius norm on its first singular direction. Near 1 means one common direction dominates every cell.

Cell-pair anisotropy

aniso_cos, aniso_cos_within_ct, aniso_cos_between_ct, aniso_cos_ct_gap

Mean cosine similarity over 100,000 sampled cell pairs — mixed, within a cell type, between cell types (types sampled uniformly), and within minus between. The gap is how much more alike same-type cells are than different-type cells.

Expression-neighbourhood preservation

rnx_k15, rnx_k30, rnx_k50, rnx_mean

Chance-corrected R_NX between within-batch k-nearest-neighbour graphs built on the embedding and on the input’s analytic Pearson residuals. Batches with 100 cells or fewer are skipped; rnx_eligible_cell_fraction reports how many cells entered.

Intrinsic dimension

id_twonn_raw, id_twonn_z

TwoNN estimate in the embedding as stored and after standardising every coordinate.

Variance decomposition

partial_eta2_celltype, partial_eta2_batch

Partial η² of cell type and batch in the per-coordinate model z ~ cell type + batch, averaged over coordinates. partial_eta2_batch is NaN on single-batch data.

Every statistic is defined precisely, with the reference construction for R_NX, on Representation-geometry probes.

Outputs

results/geometry/<method>/<dataset>_<method>.csv               one row per dataset
results/geometry/<method>/<dataset>_<method>_rnx_batches.csv   R_NX per batch and k
results/<method>_geometry.csv                                  all datasets of the call

For the first command above, results/scgpt_geometry.csv:

dataset_id,method,embedding_file,n_cells_total,n_cells_used,embedding_dim,zero_var_dims,pr,npr,aniso_spec,aniso_cos,aniso_cos_within_ct,aniso_cos_between_ct,aniso_cos_ct_gap,...,rnx_mean,...,id_twonn_raw,...,id_twonn_z,...,partial_eta2_celltype,partial_eta2_batch,...
colon_1000,scgpt,colon_1000.h5ad,1000,1000,512,0,7.10,0.0139,0.853,0.851,0.911,0.822,0.0888,...,0.434,...,10.65,...,10.90,...,0.430,0.169,...

So on a thousand colon cells, scGPT’s 512-dimensional embedding has an effective dimension of about 7, the average cell pair has a cosine similarity of 0.85 — every cell points roughly the same way — and same-type pairs are only 0.09 more similar than different-type pairs. The trailing notes column records the backends used (exact or approximate nearest neighbours, and so on), and probe_status / failure_reason say whether every probe completed.

The per-batch table:

dataset_id,method,batch,n_cells,k,RNX,embedding_knn_backend
colon_1000,scgpt,HT-236_day132_colon,363,15,0.3595,sklearn_brute_exact
colon_1000,scgpt,HT-236_day132_colon,363,30,0.4570,sklearn_brute_exact
colon_1000,scgpt,HT-236_day132_colon,363,50,0.5471,sklearn_brute_exact

Reading results in Python

from pathlib import Path
import pandas as pd

geo = pd.concat(pd.read_csv(p) for p in Path("results").glob("*_geometry.csv"))
cols = ["npr", "aniso_cos", "aniso_cos_ct_gap", "rnx_mean", "id_twonn_z", "partial_eta2_celltype"]
print(geo.set_index(["dataset_id", "method"])[cols].round(3))

Gotchas

R_NX needs batches of more than 100 cells. The reference graph is built within each batch, so a dataset of many tiny batches yields no eligible cells and rnx_* is NaN. rnx_eligible_cell_fraction says how much of the data was used; on the demo colon file it is 0.7, from two of seven batches.

The input must be raw counts. The Pearson-residual reference is computed from X of the --data file; normalised input makes it meaningless. The probe checks for integer counts and fails loudly if it finds none.

Cells are matched by obs_names. If an embedding and its input disagree — cells filtered between the two, or renamed — the probe refuses rather than guess.

Note

Every launch gets its own run directory under runs/<task>/, and the workspace nextflow.config sets cleanup = true, so the task work directory is deleted once the run succeeds. After a failure it is kept: fix the cause, add --resume to the same command, and Nextflow reuses every task that already completed. scfoundry runs lists the run directories with their status; scfoundry runs --task <task> narrows the list.

Next steps