Extending scFoundry

scFoundry is designed to grow: the benchmark is only useful if new models can be evaluated under the same protocol as the published ones. There are three ways in, in increasing order of effort.

Evaluate without adding

Write your model’s embeddings in scFoundry’s output format and run benchmark and geometry on them. No code, no container.

Evaluating a model without adding it

Add a method

Wrap a model as a method: a container, a weight download, a Nextflow module, a registry entry. It then works with every task.

Adding a method

Contribute it

Open a pull request so that everyone gets the method, and it can appear in the published comparison.

Contributing a method

Evaluating a model without adding it

benchmark and geometry do not care where an embedding came from. They read an .h5ad whose X is the embedding and whose obs carries the labels — exactly what embed writes — so a model you run yourself, by any means, can be scored under the paper’s protocol in two commands.

Write the embedding in the output format:

import anndata as ad
import numpy as np
import pandas as pd

adata = ad.read_h5ad("cells.h5ad")            # the raw-count input you embedded
embedding = np.load("my_model_embedding.npy")  # (n_cells, n_dims), same cell order

out = ad.AnnData(
    X=np.asarray(embedding, dtype=np.float32),
    obs=adata.obs.copy(),                      # labels and batches travel with it
    var=pd.DataFrame(index=[f"V{i + 1}" for i in range(embedding.shape[1])]),
)
if "spatial" in adata.obsm:
    out.obsm["spatial"] = adata.obsm["spatial"]
out.write_h5ad("embeddings/my_model/cells.h5ad")

Then, inside a workspace:

scfoundry benchmark --embedding embeddings/my_model --batch-key batch_id
scfoundry geometry  --embedding embeddings/my_model --data inputs/

The method label in the tables is the directory name (my_model), or --method. Every number is computed exactly as for the published results, so the comparison is fair as long as your embedding was produced from the same raw-count inputs with the model’s default settings.

What this does not give you is transfer and finetune, which need to run the model themselves — for those, add the method.

What a method is

A method is one entry in the registry conf/methods.json plus one Nextflow module workflows/methods/<id>.nf. The module runs the model’s own code inside its own container image and honours two contracts:

• input — the raw-count .h5ad of the input contract. The module does whatever preprocessing the model’s authors prescribe;

• output — an embedding .h5ad with the matrix in X, obs carried through and placeholder var names; for fine-tuning, a model directory and two prediction tables indexed by barcode.

Everything else — the workspace, run records, benchmark, geometry, the classifiers of transfer — is shared and needs no change. Adding a method walks through the pieces.

Ground rules

Three principles keep the benchmark meaningful, and they apply to every method, yours included:

The authors’ recipe, with defaults. A method should do what its authors’ documentation or example code does for zero-shot embedding, with their default checkpoint and settings. scFoundry is not the place to tune a model, and a method that is tuned when the others are not is not comparable with them.

Official weights, pinned. Checkpoints come from the authors’ release — Hugging Face, Zenodo, figshare, the authors’ own share — at a fixed revision, verified where possible. Nothing re-trained, nothing re-uploaded.

Pinned containers, buildable from source. A method’s image is built from a Dockerfile that lives in the repository and is tagged with a version. latest is acceptable while a method is being developed, not for a published comparison.