Clustering Pipeline

Groups immune receptor sequences into clusters across a whole cohort. It runs in two stages: first it pools every subject’s sequences and collapses near-identical ones into sub-graphs, then it builds a hierarchical tree over those sub-graphs to merge related ones into final clusters — so you can see which sequence groups are shared across people.

Web app slug: clustering-pipeline

---

When to use

Use this pipeline when you have a multi-subject cohort in AIRR format and want to find sequence clusters that recur across subjects

---

How it works

Clustering happens at two levels of granularity, both over the pooled sequences of the entire cohort:

Stage 1  — tight grouping              Stage 2  — loose grouping
──────────────────────────────        ──────────────────────────────
ALL subjects' sequences pooled    →    distances between sub-graphs
into one similarity graph; near-       computed with the chosen metric,
identical sequences (sid + cov)        agglomerated into a tree, then
become connected "sub-graphs"          cut at height_cutoff → clusters
(≈ clonal families)                    (≈ shared specificity groups)

1. Validate & ingest — the cohort sheet and every referenced AIRR file are validated up front (before any heavy compute).

2. Stage 1 — Sub-graph (subG) clustering: all subjects’ paratope sequences are pooled into a single similarity graph. Two sequences are linked when they clear both the sequence-identity (sid) and coverage (cov) thresholds, computed position-by-position over the alignment. The connected components of that graph are the sub-graphs. Because the default thresholds are high (sid 0.9 / cov 0.95), a sub-graph is essentially a clonal family. The Sub-graph Method selects how sub-graphs are formed: gsc (connected components of the similarity graph, the default), clonotype (exact-clonotype grouping, no similarity graph), or louvain (community detection).

3. Stage 2 — Hierarchical clustering of sub-graphs A distance is computed between sub-graphs using the chosen metrics, agglomerated with the chosen linkage method , and the resulting dendrogram is cut at height_cutoff to produce the final flat clusters. Because everything was pooled in Stage 1, a single cluster can contain sequences contributed by many subjects — that is what makes a cluster “public.”

4. Stage 2b — Map clusters back Cluster membership (stored as global sequence indices) is joined back to subjects and sequences, yielding the key cohort-level table: which subjects populate which cluster.

Note

Two similarity knobs, two different jobs. sid/cov in Stage 1 decide which sequences are near-identical enough to collapse into the same sub-graph. The Stage 2 metrics + method + height_cutoff decide how those sub-graphs are merged into final clusters. Tuning the wrong one is the most common source of “clusters look wrong” — tighten/loosen sid/cov to change sub-graph granularity, change metrics/height_cutoff to change how aggressively sub-graphs merge.

Tip

Clustering runs on the aligned pseudo-sequence — the full IMGT-numbered variable domain, not just CDR3. That sequence is what the TCR to Amino Acid Sequence and Pseudo-Sequence Alignment pipelines produce, which is why they feed naturally into clustering.

Choosing a Stage 2 metric

The distance metric controls what “similar” means when building the cohort tree:

• Identity-based — sid, sid_cov, hamming, hamming_norm: fast, count identical/mismatched aligned positions (the _norm variants divide by alignment length).
• Substitution-aware — blosum_ave, blosum_norm, blosum_ave_norm, blosum_dist, blosum_dist_centrality, blosum_RPD: score amino-acid similarity with BLOSUM62 so conservative substitutions cost less than disruptive ones — better for grouping functionally related sequences rather than just literal-identical ones.
• TCR-specialized — tcrdist: a position-weighted, capped BLOSUM62-based distance (the TCRdist metric).

blosum_RPD is a reciprocal BLOSUM62 distance and additionally requires the RPD_method parameter (an integer 1–4, selecting among alternative distance-decay formulas); setting RPD_method with any other metric is rejected at validation time.

Optional side branches

These run alongside the main path and upload their own outputs — if one fails it surfaces as a DAG failure (so you’re alerted) but it does not block the main results.zip:

• Threshold diagnostics (run_threshold_diagnostics) — plots the pairwise sid/cov distributions from Stage 1 so you can pick sensible thresholds.
• HTML report (generate_html_report) — a self-contained interactive report (cluster network, CLR/norm-transformed abundance charts, optional Wilcoxon filtering between two arms). Uploaded separately as report.html, not included in results.zip.

---

Submitting a job

Navigate to Dashboard → Clustering Pipeline → New Job.

Required inputs

Field	Description
Job name	A label for this run
Input bundle (zip)	A zip containing the cohort sheet plus one AIRR TSV per subject

The cohort sheet (default filename cohort.tsv, set via Cohort Sheet Filename) needs at minimum a subject column and an airr_file column; any extra metadata columns pass through. If no cohort sheet is found in the bundle, a minimal one is auto-generated from the discovered *.tsv files (one subject per file). Each referenced AIRR TSV must contain sequence_id, v_call, j_call, and sequence_aa (or the alternative set cell_id, v_gene, j_gene, sequence_aa). airr_file paths resolve against the sheet’s own directory inside the bundle and must point at a plain TSV present there — remote s3:// URIs and zips fail validation. Full contract: Cohort TSV.

This pipeline supports input bundle preflight — the web app can check the bundle against the requirements above in your browser, before it uploads, and flag any AIRR file that would be rejected or quietly dropped.

Caution

Add a condition column to the cohort sheet if you plan to enable Generate HTML report. Each distinct value becomes a condition:<value>.* column triple in cluster_list.tsv, and the report generator aborts with no 'condition:*.counts' columns found when none exist. The main clustering output does not need it. Any column whose name contains condition counts, so multiple groupings can coexist.

Optional inputs

Stage 1 — sub-graph clustering

Field	Default	Description
Receptor chain	TRB	One of IGH, IGL, IGK, TRB, TRA
Species	human	human, mouse, or none (affects germline V/J calls)
Sequence identity (sid)	0.9	Identity cutoff for pairwise similarity, 0 < sid ≤ 1. Ignored when auto-tune is on
Coverage (cov)	0.95	Alignment coverage cutoff, 0 < cov ≤ 1. Ignored when auto-tune is on
Clustering method	gsc	Sub-graph algorithm: gsc, louvain, or clonotype
Minimum clones	1	Skip AIRR files with fewer clones than this
Keep singletons	false	Retain sequences that fall in no sub-graph
CPU cores	0	Threads for the pairwise similarity computation; 0 uses all available cores
Auto-tune sid/cov	false	Detect thresholds from the data (valley search over a random pair sample) instead of using the fixed values
Auto-tune sample size	100000	Random pairs sampled for valley detection. Only used when auto-tune is on

Stage 2 — hierarchical clustering

Field	Default	Description
Linkage method	average	single, complete, average, weighted, centroid, median, or ward
Distance metric	sid_cov	See Choosing a Stage 2 metric for the full list
Height cutoff	0.1	Tree-cut height (> 0); units depend on the chosen metric
RPD method	—	Required only when metric is blosum_RPD (integer 1–4); must be unset otherwise
fastcluster backend	true	Faster linkage computation
Use sparse linkage backend	false	Cluster very large sub-graphs that the default path cannot handle. Changes results — see Sparse linkage
Enable S1 lookup table	false	No effect. Every run already takes the lookup-table path, so this flag is accepted and ignored. Kept only so existing presets keep loading

Sparse linkage

By default Stage 2 enumerates the distance between every pair of members in a sub-graph. That is exact, but the work grows with the square of the sub-graph size, and a sufficiently large sub-graph simply will not fit.

Use sparse linkage backend clusters from the edge list Stage 1 already produced instead, treating any pair that Stage 1 never recorded as maximally distant. This is what lets a sub-graph of a few hundred thousand members complete at all.

Caution

This is an approximation, and it is biased toward splitting clusters — pairs that are genuinely close but were not captured as edges in Stage 1 are treated as far apart. Leave it off unless a run is failing or infeasible on the exact path, and do not compare sparse-linkage results against dense-path results as though they were produced the same way.

It applies only to metrics sid and sid_cov combined with linkage methods single, complete, average, or weighted. Any other combination silently falls back to the exact path, so the flag is safe to leave set on a run where it does not apply — but check your run parameters if you expected it to take effect.

Reports & lifecycle

Field	Default	Description
Run threshold diagnostics	false	Plot sid/cov distributions (side branch)
Generate HTML report	false	Emit the interactive report.html (side branch, supports two-arm contrasts)
Cleanup temporary files	true	Remove the run’s temp directory after upload

HTML report contents — all ignored unless Generate HTML report is on

Field	Default	Description
Positive arm	—	condition value naming the positive arm of the primary contrast (e.g. the responder label)
Negative arm	—	condition value naming the negative arm
Serology positive / negative arm	—	Optional second contrast, layered on top of the primary one
Count metric	UMIs	Count column driving the bar and heatmap panels: UMIs, Reads, or Clones (Clones weights every clonotype equally)
Transform	clr	clr (centered log-ratio, compositional) or norm (relative abundance)
Top-N clusters in network	25	How many clusters the membership network view draws
Patient sample key / Healthy-control sample key	—	Substrings identifying the two arms of the Wilcoxon differential-abundance filter
Wilcoxon confidence	0	Confidence threshold for that filter — the p-value cutoff is 1 − conf. Requires both sample keys; 0 disables filtering
Reference sample key	—	Substring picking the reference samples for the CLR scatter panels
Pre-transform cluster filter	0	Drop clusters seen in fewer than N subjects (integer) or fewer than f × n_subjects (fraction 0 < f < 1), before transforming
Post-transform cluster filter	0	Same, applied after the transform
Dataset label / GSC label	—	Strings shown in the report title. Left unset, both are derived automatically

Note

The arm and sample-key fields are matched against the condition values and subject column names in your cohort — which is why the report needs a condition column at all. Set Positive arm and Negative arm to two distinct values of that column to get the contrast panels.

Note

Auto-tune adds an extra sampling/optimization pass and can noticeably increase runtime on large cohorts. Disable it if you already know good sid/cov values.

---

Output files

results.zip (under the run’s output prefix) is the complete clustering output tree, not a curated subset — it also carries the intermediate airr/ and db/ directories, the stage-1 sub-graph reports, and a logs/ directory. The files you will actually reach for:

File	Description
cluster_list.tsv	One row per cluster — size, subject count, diversity, BLOSUM/Hamming means, condition splits, members
cluster_index.tsv	Flat expansion of cluster membership: one row per member (cluster_id, Rep_idx, Mem_idx)
subject_cluster_map.tsv	One row per sequence, joining cluster membership back to subjects and AIRR fields
lngs_trees.tsv	The hierarchical tree(s) produced by Stage 2
config.json	Stage-1 sentinel with the resolved run config (including the real sid/cov when auto-tuned)
info_hcluster.txt	Run metadata (chain, species, resolved parameters, subject counts)

Column-level schemas for the three cluster tables: Clustering data formats.

Uploaded separately to the same run prefix (not inside results.zip):

File	Description
report.html	Interactive cluster report — only when Generate HTML report is enabled
threshold_plots/	sid/cov diagnostic plots — only when Run threshold diagnostics is enabled

---

FAQ

What does the diversity column in cluster_list.tsv mean, and how is it computed?

It scores how many distinct sequences a cluster holds — not how far apart they are. The metric is the Gini–Simpson index (Simpson’s index of diversity, 1 − D):

diversity = 1 − Σ pᵢ²

where pᵢ is the frequency of the i-th distinct member sequence within the cluster. Equivalently: the probability that two members drawn at random from the cluster are different sequences.

How to read a value:

Value	Meaning
0	Every member is the same sequence. Also the value reported for singleton clusters (size = 1).
1 − 1/size	Every member is unique — the maximum attainable for a cluster of that size (e.g. 0.923 at size = 13). Most clusters land exactly here.
Between the two	Some sequence occurs in more than one copy — typically the same pseudo-sequence recovered from several subjects.

Caution

The ceiling depends on cluster size, so raw diversity is not comparable between clusters of very different sizes. Compare each value against its own 1 − 1/size ceiling, or use the Diversity vs cluster size scatter (plot 3) in the HTML report, which plots the two together.

Details worth knowing:

• Not Shannon entropy. Normalized Shannon entropy and the inverse-Simpson (Hill D₂) form were both evaluated; the pipeline ships the Gini–Simpson form.
• Identity classes only. Two members differing at one position count as “different” exactly as much as two differing at twenty. For how far apart members are, use the companion columns: ave_blosum (mean pairwise BLOSUM62 score — higher means more similar) and ave_hamming (mean pairwise Hamming distance over the alignment — higher means more different).
• Computed on the aligned pseudo-sequence, the same IMGT-numbered representation used for clustering, gap characters included. Two sequences with identical amino acids but different gap placement therefore count as distinct.
• One count per sequence row. Frequencies are not weighted by duplicate_count or consensus_count.

diversity is a cluster-level column: it appears in cluster_list.tsv only, not in the per-sequence subject_cluster_map.tsv.

---

Troubleshooting

Empty or tiny cluster list Stage 1 sid/cov may be too strict, so few sequences connect into sub-graphs. Lower them (e.g. sid toward 0.8) or enable Auto-tune. If Stage 2 produces a tree but no flat clusters, your height_cutoff may be cutting above every merge — lower it.

metrics='blosum_RPD' but the run fails validation blosum_RPD requires RPD_method set to one of 1–4. Conversely, leaving RPD_method set while choosing any other metric is also rejected — clear it.

Cohort sheet rejected The sheet must have subject and airr_file columns, unique non-empty subjects, and every airr_file must resolve on disk with the required AIRR columns present. The validation error names the exact missing column or unreadable file.

Stage 1 slow for large cohorts Pairwise similarity scales steeply with sequences per subject. Use Minimum clones to drop tiny repertoires, set CPU cores to 0 (all cores), or pre-filter your AIRR files before upload.

The HTML report / diagnostics failed but I still got results.zip That’s by design — those side branches upload independently and only fail the DAG to alert you; they never block the main clustering output.