Adding a method¶

How to wrap a model so that scfoundry --method <id> runs it. Expect a day for a model with a clean Python API, and longer for one that ships as research scripts.

Before you start¶

Work in a development checkout: git clone, pip install -e ., scfoundry init . — see Developer install. The scfoundry command then runs the pipeline from your checkout, and data/demo/ is in place.
Read the model’s own documentation for zero-shot embedding and note the exact preprocessing, tokenisation, pooling and checkpoint it uses. That recipe is what you implement — not what you think works better.
Check the licence of the code and of the weights. Some checkpoints are restricted to non-commercial use; the method’s registry entry should say so.
Pick a short lowercase id (newmodel). It becomes the --method value, the module name, the output directory and the registry key.

1. The container¶

Every method runs in its own image, so the model’s dependency pins never meet anyone else’s. Write containers/newmodel/Dockerfile:

containers/newmodel/Dockerfile¶

FROM nvidia/cuda:12.1.1-cudnn8-runtime-ubuntu22.04

RUN apt-get update && apt-get install -y --no-install-recommends python3 python3-pip git \
 && rm -rf /var/lib/apt/lists/*

# The model's own code at a pinned revision, plus what the module needs to read and
# write AnnData. Pin everything the model pins.
RUN pip3 install --no-cache-dir \
      torch==2.3.1 --index-url https://download.pytorch.org/whl/cu121 \
 && pip3 install --no-cache-dir \
      "newmodel @ git+https://github.com/lab/newmodel.git@a1b2c3d" \
      anndata==0.10.8 scanpy==1.10.2

docker build -t yourname/newmodel:0.1.0 containers/newmodel
docker push yourname/newmodel:0.1.0

Requirements the pipeline relies on:

python3 (or python) on PATH, with anndata and scanpy importable, because the module’s scripts are Python;
a CUDA runtime matching the driver floor (525) for GPU methods;
no assumption about $HOME: containers run with --no-home --cleanenv, and a stand-in home is mounted. Code that writes caches to ~/.cache works; code that reads your dotfiles does not.

Test it before writing any Nextflow: apptainer exec --nv docker://yourname/newmodel:0.1.0 python3 -c "import newmodel, scanpy".

2. The weights¶

Add a process to workflows/utils/download.nf that fetches the official checkpoint into /data/model_weights/<Name>/, which is the workspace’s data/model_weights/ bind-mounted:

workflows/utils/download.nf¶

process download_newmodel_checkpoints {

    container "housy17/scfm_download:latest"

    output:
    stdout

    script:
    """
    cd /data/model_weights
    mkdir -p NewModel && cd NewModel
    hf download lab/newmodel model.ckpt --revision a1b2c3d4e5f6 --local-dir ./
    echo "NewModel checkpoints downloaded!"
    """
}

The download image has hf (Hugging Face), gdown, curl and tar. Pin the revision, and verify a checksum when the source cannot pin one for you — the CellPLM and scPRINT processes in the same file show both patterns. Then add the process to the runners map of workflows/tasks/download.nf.

The path under data/model_weights/ becomes the module’s params.model default.

3. The module¶

workflows/methods/newmodel.nf declares the method’s parameters and its processes. The minimal module is one process; pca.nf (55 lines) is the simplest real one, and scimilarity.nf shows the common two-step shape — a preprocessing process and an embedding process, wrapped in a named workflow.

workflows/methods/newmodel.nf¶

params.model = "NewModel/model.ckpt"      // under /data/model_weights
params.batch_size = 64
params.emb_results_dir = "results"

process embed_by_newmodel {

    tag "${id}"

    label 'gpu_task'                        // or 'cpu_task'

    container "yourname/newmodel:0.1.0"

    publishDir "${params.emb_results_dir}/embeddings/newmodel", mode: 'copy',
               saveAs: { filename -> "${id}.h5ad" }, enabled: params.emb_results_dir as boolean

    input:
    tuple val(id), path(raw_h5ad)

    output:
    tuple val(id), val("NewModel"), path("*_embeddings.h5ad")

    script:
    """
    #!/usr/bin/env python3
    import numpy as np
    import pandas as pd
    import scanpy as sc
    import newmodel

    adata = sc.read_h5ad("${raw_h5ad}")
    obs, spatial = adata.obs.copy(), adata.obsm.get("spatial")

    # --- the authors' recipe, exactly ---------------------------------------
    model = newmodel.load("/data/model_weights/${params.model}")
    embedding = model.embed(adata, batch_size=${params.batch_size})   # (n_cells, n_dims)
    # -------------------------------------------------------------------------

    out = sc.AnnData(
        X=np.asarray(embedding, dtype=np.float32),
        obs=obs,
        var=pd.DataFrame(index=[f"V{i + 1}" for i in range(embedding.shape[1])]),
    )
    if spatial is not None:
        out.obsm["spatial"] = spatial
    out.write_h5ad("newmodel_embeddings.h5ad")
    """
}

The parts that are not negotiable:

Element	Contract
`input:`	`tuple val(id), path(raw_h5ad)` — the dataset id (input basename) and the raw-count file.
`output:`	`tuple val(id), val("<Display name>"), path("*_embeddings.h5ad")`.
The embedding file	`X` dense `float32`, cells in the input order; `obs` copied unchanged; `var` index `V1…Vn`; `obsm["spatial"]` preserved if present.
`publishDir`	`<emb_results_dir>/embeddings/<id>/`, file renamed to `<dataset>.h5ad`, guarded by `enabled:` so that `transfer` can switch publication off.
`label`	`gpu_task` for anything that runs the model; `cpu_task` for preprocessing. Scheduler profiles bind resources to these two labels.
`params.<name>`	Every knob a user may want, with the authors’ default. Method-specific names get the method as prefix (`newmodel_pool_type`), shared ones keep the shared name (`model`, `batch_size`).
Weights and code paths	Weights under `/data/model_weights/`; helper scripts, if any, under `bin/newmodel/`, which is mounted at `/code/newmodel/`. Keep long Python out of the module and in `bin/`.

If the model needs a separate preprocessing step (tokenisation, gene alignment), make it its own process — it can then run on the CPU and be retried independently — and expose a workflow embed_by_newmodel that chains the two, as scimilarity.nf does. The task maps call embed_by_newmodel(ch) either way.

Fine-tuning¶

Only add fine-tuning if the authors publish a recipe that updates the model’s parameters. It is two more processes:

finetune_by_newmodel — input: tuple val(id), path(raw_h5ad) with labels in obs[params.finetune_label_key]; output: tuple val(id), path("<model dir>"), published by copy to <finetune_results_dir>/finetune/finetuned_models/newmodel/<id>/. Declare params.finetune_epoch, params.finetune_batch_size, params.finetune_eval_size = 0.2 and params.predict_batch_size with the authors’ values.
predict_by_newmodel — input: tuple val(id), path(query_h5ad), path(model_dir); writes <id>_predicted_labels.tsv (columns: barcode index, predicted_label) and <id>_predicted_probs.tsv (one column per class), published to <finetune_results_dir>/finetune/prediction/newmodel/.

Look at scgpt.nf for a complete native example.

4. The registry¶

One line in conf/methods.json:

"newmodel": {"name": "NewModel", "category": "zero-shot", "container": "yourname/newmodel:0.1.0",
             "gpu": true, "tasks": ["download", "embed", "transfer"], "notes": ""}

category is zero-shot, reference or integration; tasks lists what the module implements. scfoundry list methods and the launcher’s argument checks read this file, so a method that is not registered does not exist as far as the command is concerned.

5. The task maps¶

Nextflow includes are static, so each task workflow names the modules it can call. Add an include line and a map entry to every task in the registry entry’s tasks:

File	What to add
`workflows/tasks/download.nf`	`include { download_newmodel_checkpoints } from '../utils/download'` and `'newmodel': { download_newmodel_checkpoints() }` in `runners`.
`workflows/tasks/embed.nf`	`include { embed_by_newmodel } from '../methods/newmodel'` and `'newmodel': { ch -> embed_by_newmodel(ch) }` in `runners`.
`workflows/tasks/transfer.nf`	The same include, and the same entry in `embedders`.
`workflows/tasks/finetune.nf`	Include `finetune_by_newmodel` and `predict_by_newmodel`; entries in `fitters` and `predictors`.

The registry test enforces that every map agrees with conf/methods.json, so a forgotten entry fails immediately rather than at runtime:

python -m unittest tests/test_registry.py tests/test_launcher.py

6. Verify on the demo data¶

scfoundry list methods --task embed          # newmodel is listed
scfoundry download --method newmodel
scfoundry embed --method newmodel --data data/demo/colon_1000.h5ad

Check the output against the contract:

import anndata as ad, numpy as np

a = ad.read_h5ad("results/embeddings/newmodel/colon_1000.h5ad")
raw = ad.read_h5ad("data/demo/colon_1000.h5ad")
assert a.shape[0] == raw.n_obs and (a.obs_names == raw.obs_names).all()
assert a.X.dtype == np.float32 and not np.isnan(a.X).any()
assert list(a.var_names[:2]) == ["V1", "V2"]
print(a.shape)

Then score it, and read the numbers next to PCA’s — an embedding that scores far below PCA on the demo is more often a preprocessing mistake than a bad model:

scfoundry benchmark --embedding results/embeddings/newmodel --batch-key batch_id
scfoundry benchmark --embedding results/embeddings/pca --batch-key batch_id

If transfer is declared, run the one-shot liver pair; if finetune is, the colon pair. Finally, confirm you reproduce something the authors published — the embedding of their tutorial dataset, a reported accuracy — so that “the authors’ recipe” is a checked claim.

7. Document it¶

In the documentation repository: a row in docs/_data/methods.csv and docs/_data/support_matrix.csv, a row on the citation page, the weight size and source on Downloading model weights, and the method-specific parameters on the method reference.