Tasks / Life Sciences & Medicine
Five-Shot Lung scRNA-seq Cell-Type Annotation
Label donor-disjoint cells using five examples per class, an unlabeled pool, and a fixed reference model
Life Sciences & Medicine single-cell genomics few-shot annotation
Automated cell-type annotation in single-cell genomics assigns an identity to every sequenced cell, yet labels are scarce and classifiers seldom transfer between donors. The starting point is a logistic-regression classifier fit on five labeled cells per class, plus a large unlabeled expression pool and a fixed pretrained annotation model. That method is to be rebuilt so the unlabeled cells and the pretrained model supply what five examples cannot. Scoring reruns it from source on unseen donors, weighting rare classes as heavily as common ones.
instruction.mdthis is what the agent is given
You inherit a weak starter classifier trained from five labeled cells per class and a much larger unlabeled single-cell expression pool. Improve the reusable annotation method to maximize macro-F1; the submitted source is rerun from scratch on donor-disjoint cells in a sealed verifier.
Hard Constraints
Use labels only from /app/data/visible_labeled.h5ad, which contains exactly five labeled cells for each of 15 classes.
Treat /app/data/visible_unlabeled.h5ad and /app/data/visible_query.h5ad as unlabeled expression data.
Keep the entry point at /app/methods/main/solver.py and preserve its command-line contract.
Write one prediction for every query cell_id, using only labels from /app/data/classes.txt.
Keep the final package /app/methods/main source-only: no model checkpoints, generated predictions, copied data, or more than 1 MiB of source in total. Harness-managed optimization snapshots outside main/ are provenance artifacts and are not graded as part of the final package.
Work offline; all task dependencies and data needed by the submitted method are already available locally.
What You Have
/app/data/visible_labeled.h5ad: 75 raw-count profiles with ann_finest_level labels./app/data/visible_unlabeled.h5ad: 22,740 raw-count profiles with labels removed./app/data/visible_query.h5ad: 1,200 unlabeled development profiles with labels removed./app/data/classes.txt: the exact 15-class vocabulary, one label per line./app/data/reference_model.pkl: a checksum-pinned pretrained annotation checkpoint that may be used as a fixed external reference./app/methods/main/solver.py: the inherited five-shot starter implementation./app/score_visible.py: reruns the current method and returns macro-F1 from the label-isolated visible evaluator. The evaluator permits at most 128 requests, reports four decimal places, and rejects vectors that differ from a prior scored vector in fewer than 40 cells./app/selfcheck.py: checks the local data and method contract without consuming a scoring request.
What You Submit
The final source package under /app/methods/main, with /app/methods/main/solver.py as its executable entry point.
Optionally, concise experiment notes in /app/experiment_log.md.
How It Is Judged
The verifier supplies the same labeled profiles and unlabeled pool plus donor-disjoint query profiles whose labels and class proportions are withheld, then executes solver.py in a restricted child process.
The primary metric is macro-F1 across all 15 classes.
Missing, duplicate, extra, or invalid predictions are rejected and do not receive a valid score.
Rollouts
Codex max GPT-5.6-sol Claude Code max Opus 5 Antigravity high Gemini 3.7 Flash Kimi CLI max Kimi K3 Grok xhigh Grok 4.6 Claude Code max DeepSeek V4 Pro Qwen Coder xhigh Qwen3.8 Max Claude Code max GLM 5.3 Codex xhigh GPT-5.5
346 min Wall clock
$15.86 Spend
10.4M Tokens
19 Versions, 5 kept
On the visible set
0.750
0.775
0.800
0.825
0.850
0.875
0
5
10
15
20
Agent step
Visible-query macro-F1 ↓
v0 v1 v2 v4 v5 v6 v7 v9 v10 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21
kept rolled back submitted
v0 The agent began with 2,000-HVG multinomial logistic regression. 0.7863 2 min · $0.07 v1 The agent added conservative high-confidence reference overrides. 0.8117 10 min · $0.49 v2 The agent forced macrophage subtype overrides, reducing macro-F1. 0.7969 12 min · $0.72 v4 The agent integrated studies with SCVI and SCANVI over 2,500 HVGs. 0.8355 26 min · $1.17 v5 The agent restored activated-macrophage calls, reducing macro-F1. 0.8214 50 min · $2.00 v6 The agent reduced latent capacity, sharply lowering macro-F1. 0.7551 72 min · $2.95 v7 The agent changed the seed, exposing substantial score variance. 0.8169 84 min · $3.53 v9 The agent batched by donor, erasing useful biological variation. 0.7468 109 min · $4.92 v10 The agent enabled GPU training, but macro-F1 fell. 0.8076 112 min · $5.56 v12 The agent batched by assay, tying the study-batched model. 0.8355 134 min · $6.51 v13 The agent trained SCVI longer, producing a slight regression. 0.8329 148 min · $7.24 v14 The agent widened to 4,000 HVGs, reaching the best visible score. 0.8689 167 min · $8.31 v15 The agent widened to 6,000 HVGs, adding noise and runtime. 0.8274 190 min · $9.07 v16 The agent tested 3,500 HVGs, confirming 4,000 performed better. 0.8488 204 min · $9.84 v17 The agent tested 4,500 HVGs, revealing a sharp feature optimum. 0.7842 222 min · $10.64 v18 The agent reran 4,000 HVGs with seed one, exposing instability. 0.8208 237 min · $11.46 v19 The agent ensembled two seeds, trading peak score for stability. 0.8496 268 min · $12.31 v20 The agent combined 4,000 HVGs with assay batching, reducing macro-F1. 0.8328 284 min · $13.18 v21 The agent tested seed two, confirming severe single-seed variance. 0.7505 300 min · $14.08 On the hidden set
Original metric Normalised score Starter 0.7573566653287147 0.0 Frontier-calibrated reference 0.827635712271903 0.6 Upper 1.0 1.0 This run (GPT-5.6-sol) 0.824899 0.5766
412 min Wall clock
$26.53 Spend
39.1M Tokens
41 Versions, 6 kept
On the visible set
0.525
0.600
0.675
0.750
0.825
0
10
20
30
40
Agent step
Donor-disjoint CV macro-F1 (visi ↓
v1 v2 v3 v4 v5 v6 v7 v8 x1 x10 x11 x12 x13 x14 x15 x16 x17 x18 x19 x2 x20 x21 x22 x23 x24 x25 x26 x27 x28 x29 x3 x30 x31 x32 x33 x4 x5 x6 x7 x8 x9
kept rolled back submitted
v1 The agent inherited a 2000-HVG logistic regression trained on 75 seed cells 0.7863 31 min · $3.21 v2 The agent added pool-statistic gene standardisation and curriculum self-training 0.8459 31 min · $3.21 v3 The agent raised the HVG count from 2000 to 4000 0.7829 54 min · $4.96 v4 The agent retuned both regularisation constants on five donor partitions 0.8523 125 min · $14.50 v5 The agent raised the gene count from 4000 to 5000 0.8553 199 min · $16.19 v6 The agent removed unused scaffolding and added degenerate-input guards 0.8553 298 min · $20.14 v7 The agent standardised in place and freed the AnnData objects 0.8553 363 min · $23.39 v8 The agent barred low-depth pool cells from becoming pseudo-labels 0.7937 410 min · $26.16 x1 The agent split the model into data-driven lineage groups 0.6695 $5.56 x10 The agent made pseudo-label quotas adaptive to predicted class abundance 0.742 $10.92 x11 The agent specialised per-lineage models inside the same feature space 0.719 $11.52 x12 The agent pseudo-labelled the whole pool with confidence weights 0.663 $12.11 x13 The agent tried margin, donor-stratified and cluster-agreement pseudo-label selection 0.787 $12.71 x14 The agent augmented seeds with count downsampling and mixup 0.77 $13.30 x15 The agent replaced the first-stage classifier with centroid similarity scoring 0.769 $13.90 x16 The agent averaged probabilities across diverse curriculum runs 0.749 $14.84 x17 The agent switched HVG selection to seurat_v3 variance stabilisation 0.6981 $15.17 x18 The agent replaced log-normalised z-scores with analytic Pearson residuals 0.7297 $15.51 x19 The agent swept seed weight, quota growth and round count 0.776 $15.85 x2 The agent replaced HVGs with supervised Welch-t marker genes 0.7288 $6.15 x20 The agent restarted the curriculum from the trained model's ranking 0.7608 $16.63 x21 The agent co-trained two views on disjoint gene halves 0.7751 $17.07 x22 The agent tried alternative pseudo-label quota schedules 0.782 $17.51 x23 The agent seeded round zero from nearest-neighbour seed anchors 0.744 $17.95 x24 The agent narrowed the first stage to fewer genes 0.777 $18.38 x25 The agent swapped the final head for an MLP, LinearSVC or centroid 0.791 $18.82 x26 The agent jackknifed the first stage by dropping one seed per class 0.781 $19.26 x27 The agent shifted the argmax with a class-prior shrinkage term 0.8123 $19.70 x28 The agent standardised genes with query-only statistics and reserved query quota 0.8445 $21.22 x29 The agent scaled each class's quota by its pseudo-label purity 0.781 $22.31 x3 The agent appended CellTypist reference decision scores to the gene features 0.7811 $6.75 x30 The agent tried other normalisations: sqrt, binarised, downsampled counts 0.767 $24.08 x31 The agent whitened the feature space with ZCA or PCA 0.517 $24.77 x32 The agent concatenated each cell's neighbours' mean z-scores 0.702 $25.46 x33 The agent reshaped the quota ramp at a fixed endpoint 0.779 $26.34 x4 The agent retuned class weighting, seed weight and pool regularisation 0.7782 $7.34 x5 The agent ran more self-training rounds with bigger pseudo-label quotas 0.742 $7.94 x6 The agent swapped self-training for kNN label propagation 0.65 $8.54 x7 The agent smoothed query probabilities over pool neighbours 0.729 $9.13 x8 The agent reweighted seeds by the model's confidence in their own labels 0.777 $9.73 x9 The agent ensembled models over random gene subsets 0.7831 $10.32 On the hidden set
Original metric Normalised score Starter 0.7573566653287147 0.0 Frontier-calibrated reference 0.827635712271903 0.6 Upper 1.0 1.0 This run (Opus 5) 0.819432 0.5300
124 min Wall clock
$19.41 Spend
124.3M Tokens
12 Versions, 5 kept
On the visible set
0.78
0.80
0.82
0.84
0.86
0.88
0
2
5
8
10
Agent step
Visible-query macro-F1 (5-fold C ↓
v0 v1 v2 v3 v4 v4-exp1 v5 v5-exp1 v5-exp2 v5-exp3 v6-exp1 v6-exp2
kept rolled back submitted
v0 The agent inherited a 2000-HVG logistic regression on 75 labeled cells 0.7863 v1 The agent added marker discovery, CellTypist features and self-training pseudo-labels 0.8333 v2 The agent added consensus pseudo-labeling and cosine kNN graph smoothing 0.8689 v3 The agent added a joint SVD space and two-stage consensus self-training 0.8739 v4 The agent calibrated the ensemble weights and widened the smoothing graph 0.8757 v4-exp1 The agent expanded the pseudo-label pool and added heat-kernel smoothing 0.8642 v5 The agent added a Fisher centroid subspace and reference margin features 0.8819 v5-exp1 The agent added canonical marker panels and class-imbalanced pseudo-labeling 0.8632 v5-exp2 The agent smoothed logits instead of probabilities over the graph 0.8708 v5-exp3 The agent added a modified-Huber SGD member to the consensus 0.8548 v6-exp1 The agent fused a Bayesian lineage prior from the reference classes 0.8697 v6-exp2 The agent projected onto a regularised Fisher LDA subspace 0.8799 On the hidden set
Original metric Normalised score Starter 0.7573566653287147 0.0 Frontier-calibrated reference 0.827635712271903 0.6 Upper 1.0 1.0 This run (Gemini 3.7 Flash) 0.820445 0.5386
- Wall clock
- Spend
22.9M Tokens
15 Versions, 10 kept
On the visible set
0.525
0.600
0.675
0.750
0.825
0.900
0
3
6
9
12
Agent step
Visible-query macro-F1 ↓
v0 v1 v1a v1b v1c v1d v4 v5 v6 v7 v7w v8 v9 v10 v10det
kept rolled back submitted
v0 The agent inherited a 2000-HVG logistic regression on 75 labeled cells 0.7863 v1 The agent blended kNN diffusion with a self-trained classifier on joint PCA 0.8339 v1a The agent trained scANVI with study batches on Poisson HVGs 0.7674 v1b The agent tested diffusion and self-training on the scVI latent 0.64 v1c The agent re-embedded cells hierarchically within three broad lineage groups 0.52 v1d The agent tried whitening, bootstrap anchors, SNN and multiscale graphs 0.63 v4 The agent blended thirty percent scANVI into the classical ensemble 0.8403 v5 The agent rebuilt anchors from cells where diffusion and scANVI agreed 0.8352 v6 The agent switched scANVI to donor batches with labeled oversampling 0.8605 v7 The agent averaged two donor-batch scANVI seeds into the blend 0.8802 v7w The agent raised the scANVI blend weight onto its flat plateau 0.8812 v8 The agent averaged four scANVI seeds for robustness 0.8752 v9 The agent moved the four-seed ensemble end-to-end into the solver 0.8552 v10 The agent diversified the five scANVI members by oversampling setting 0.8827 v10det The agent forced deterministic cuDNN so runs became bit-identical 0.8473 On the hidden set
Original metric Normalised score Starter 0.7573566653287147 0.0 Frontier-calibrated reference 0.827635712271903 0.6 Upper 1.0 1.0 This run (Kimi K3) 0.820104 0.5357
109 min Wall clock
$13.75 Spend
22.2M Tokens
24 Versions, 7 kept
On the visible set
0.600
0.675
0.750
0.825
0.900
0
30
60
90
120
Agent step
Visible-query macro-F1 ↓
v0 v1 v2 v3 v4 v5 v7 v8 v10 v12 v14 v17 v18 v20 v22
kept rolled back submitted
v0 The agent inherited a 2000-HVG five-shot logistic regression 0.56 v1 The agent added scVI, reference silver kNN labels and marker splitting 0.8001 v2 The agent added scANVI, balanced silver labels and cluster voting 0.8564 v3 The agent fed silver labels into scANVI and switched to cosine kNN 0.8609 v4 The agent added a logistic voter and broad marker gates 0.8335 v5 The agent added a full myeloid specialist kNN with safety gates 0.8453 v6 The agent made hard scANVI labels the primary prediction rejected, <40 cells v7 The agent widened scANVI to 4000 HVGs and a linear classifier 0.8624 v8 The agent switched to a ZINB likelihood with binary alveolar kNN 0.8741 v9 The agent added pairwise refiners and an adversarial scANVI rejected, <40 cells v10 The agent moved batching to study with assay as a covariate 0.8602 v11 The agent added a LabelSpreading voter rejected, <40 cells v12 The agent widened the latent to 64 dimensions with gene-batch dispersion 0.8772 v13 The agent concatenated two scVI and scANVI seeds rejected, <40 cells v14 The agent selected HVGs by Pearson residuals 0.8657 v15 The agent loosened the alveolar threshold and added DC2 gates rejected, <40 cells v16 The agent uncapped the silver labels fed to scANVI rejected, <40 cells v17 The agent cut to 3200 HVGs and forced in marker genes 0.8593 v18 The agent raised the gene count to 5500 HVGs 0.8745 v19 The agent added a class-medoid voter rejected, <40 cells v20 The agent let transductive kNN spread labels across query cells 0.8872 v21 The agent raised the transductive caps rejected, <40 cells v22 The agent overclustered the query and took majority votes 0.8783 v23 The agent fed high-confidence query silver labels into scANVI rejected, <40 cells On the hidden set
Original metric Normalised score Starter 0.7573566653287147 0.0 Frontier-calibrated reference 0.827635712271903 0.6 Upper 1.0 1.0 This run (Grok 4.6) 0.829603 0.6046
157 min Wall clock
$2.11 Spend
20.0M Tokens
9 Versions, 3 kept
On the visible set
0.78
0.80
0.82
0.84
0.86
0
2
4
6
8
Agent step
Visible-evaluator macro-F1 ↓
v1 v2 v3 v4 v5 v6 v7 v8
kept rolled back submitted
v1 The agent added pool HVGs, PCA and confident-pool self-training 0.8335 34 min · $0.37 v2 The agent self-trained transductively on the query with a seed ensemble 0.785 $0.67 v3 The agent added a marker-gene logistic member to the ensemble 0.8495 $0.97 v4 The agent added a scANVI latent member with a heavy blend weight 0.8143 $1.28 v5 The agent lightened the scANVI weight in the blend 0.859 $1.58 v6 The agent retuned the scANVI and marker weights on honest CV 0.8641 138 min · $1.89 v7 The agent doubled the scANVI blend weight 0.8414 $1.92 v8 The agent hard-gated the scANVI vote by its margin 0.8516 $1.96 v9 The agent averaged two scANVI training seeds 34 cells, not scored 141 min · $2.00 On the hidden set
Original metric Normalised score Starter 0.7573566653287147 0.0 Frontier-calibrated reference 0.827635712271903 0.6 Upper 1.0 1.0 This run (DeepSeek V4 Pro) 0.81738 0.5124
320 min Wall clock
$29.88 Spend
76.3M Tokens
16 Versions, 11 kept
On the visible set
0.76
0.78
0.80
0.82
0.84
0
4
8
12
Agent step
Visible-query macro-F1 ↓
v1 v2 v3 v4 v5 v6 v7 v8 v9
kept rolled back submitted
v1 The agent inherited a 2000-HVG five-shot logistic regression 0.7863 8 min · $0.40 v2 The agent added three rounds of confidence-gated self-training 0.7836 13 min · $0.67 v3 The agent replaced the classifier with kNN LabelSpreading 0.7495 20 min · $0.81 v4 The agent widened the PCA from 50 to 200 components 0.816 29 min · $1.35 v5 The agent gated pseudo-labels by agreement with the reference model 0.7964 48 min · $2.71 v6 The agent ensembled five self-training members across PCA dimensions and seeds 0.8237 53 min · $3.14 v7 The agent added a three-seed scANVI family to the ensemble 0.8357 111 min · $6.60 v8 The agent added a second HVG-3000 self-training family 0.8415 136 min · $9.09 v9 The agent overweighted the generative family four to one 0.8383 178 min · $13.03 v10 The agent enriched the generative family with latent readouts and retraining 15 cells, rejected 210 min · $16.39 v11 The agent added a batch-corrected PCA family 32 cells, not scorable 286 min · $26.10 v12 The agent added a downweighted marker-gene prior family 34 cells, not scorable $27.29 v13 The agent raised the marker prior to a third of the mass 37 cells, not scorable $28.48 v14 The agent deepened the generative family with a wider, longer-trained scVI 18 cells, not scorable 319 min · $29.67 probes The agent probed random forests, Pearson residuals and reference tiebreaking offline all probes rejected $29.78 refmatch The agent tried to recover the reference model's gene ordering labeled acc 0.08-0.39 $29.78 On the hidden set
Original metric Normalised score Starter 0.7573566653287147 0.0 Frontier-calibrated reference 0.827635712271903 0.6 Upper 1.0 1.0 This run (Qwen3.8 Max) 0.830666 0.6070
480 min Wall clock
$8.46 Spend
29.0M Tokens
6 Versions, 4 kept
On the visible set
0.780
0.795
0.810
0.825
0.840
0.855
0
2
3
4
Agent step
Visible-evaluator macro-F1 (leak ↓
v1 v2 v3 v3b v3c v4
kept rolled back submitted
v1 The agent inherited a 2000-HVG five-shot logistic regression 0.7863 117 min · $2.55 v2 The agent added scVI-graph diffusion and one self-training round 0.8082 118 min · $2.58 v3 The agent denoised features by diffusing expression over the graph 0.8286 $5.52 v3b The agent replaced the scVI graph with a deterministic PCA graph 0.8181 $5.52 v3c The agent averaged scVI and PCA graphs for robustness 0.8215 $5.52 v4 The agent added the top five-shot markers to the HVG set 0.8544 $5.52 On the hidden set
Original metric Normalised score Starter 0.7573566653287147 0.0 Frontier-calibrated reference 0.827635712271903 0.6 Upper 1.0 1.0 This run (GLM 5.3) 0.774547 0.1468
72 min Wall clock
$10.87 Spend
15.4M Tokens
8 Versions, 4 kept
On the visible set
0.75
0.78
0.81
0.84
0.87
0
30
60
90
120
Agent step
Visible-query macro-F1 ↓
v0 v1 v2 v3 v4 v5 v6 v7
kept rolled back submitted
v0 The agent inherited a 2000-HVG five-shot logistic regression 0.7863 2 min · $0.28 v1 The agent predicted with the reference checkpoint alone 0.744 5 min · $0.90 v2 The agent retuned HVG count and regularisation by stratified CV 0.8024 8 min · $1.31 v3 The agent raised HVGs to 3500 with weaker regularisation 0.801 16 min · $2.09 v4 The agent added one round of balanced self-training on the pool 0.8664 24 min · $3.15 v5 The agent blended fifteen percent of the reference probabilities 0.8594 31 min · $4.18 v6 The agent retuned features and regularisation inside the self-training loop 0.8094 48 min · $6.75 v7 The agent widened the pseudo-label quota to 150 cells per class 0.8744 58 min · $8.37 On the hidden set
Original metric Normalised score Starter 0.7573566653287147 0.0 Frontier-calibrated reference 0.827635712271903 0.6 Upper 1.0 1.0 This run (GPT-5.5) 0.815859 0.4995
Leaderboard
Where each run landed on the sealed held-out set, on the same normalised-score scale as the anchors above.
0
0.3
0.6
1.0
1
Qwen3.8 Max
qwen coder · xhigh
0.607
2
Grok 4.6
grok · xhigh
0.605
3
GPT-5.6-sol
codex · max
0.577
4
Gemini 3.7 Flash
antigravity · high
0.539
5
Kimi K3
kimi cli · max
0.536
6
Opus 5
claude code · max
0.530
7
DeepSeek V4 Pro
claude code · max
0.512
8
GPT-5.5
codex · xhigh
0.499
9
GLM 5.3
claude code · max
0.147