RSI-Exam
Tasks/AI Models & Agents

Video Point-Track Candidate Routing

Select candidate trajectories and predict visibility under a four-switch budget on unseen videos

AI Models & Agentsvideo point trackingconstrained routing
Background

Video point tracking — following a chosen physical point across frames as it moves and disappears behind objects — has no single tracker that is best everywhere. The starting system is a router that sees ten aligned candidate trajectories per query, drawn from two tracker variants at five refinement depths, and commits to one for the entire clip. The work is to redesign that selection and the accompanying visibility decision. Gains must survive a tight cap on route changes, and hold on unseen footage.

instruction.mdthis is what the agent is given

You inherit a weak persistent router over ten aligned candidate tracks for each queried point. Improve the general routing and visibility method to maximize mean per-video Average Jaccard while using at most four semantic-state changes after each point's query frame. The verifier re-runs your submitted method on a disjoint sealed split for final scoring.

Hard Constraints

  • Put your final method in /app/methods/main/predict.py and keep this exact entry point:

python def predict( query_points, candidate_tracks, occlusion_logits, expected_dist_logits, candidate_model_id, candidate_stage, ) -> tuple[numpy.ndarray, numpy.ndarray]: ...

  • Return (state_token, occluded). Both arrays must have shape [Q,T]; state_token must have a non-boolean integer dtype with values in 0..9, and occluded must have boolean dtype.
  • A state token is semantic: state_token = 5 * candidate_model_id + candidate_stage. It selects the corresponding candidate track at that query and frame.
  • For every query, the state-token sequence on frames strictly after the query frame may contain at most four changes. Frames at or before the query frame are not scored and do not count toward this switch budget.
  • Use only the six arrays passed to predict. Do not read visible labels, case identifiers, index order, filenames, verifier state, process state, credentials, or files outside /app/methods from prediction code.
  • Implement one reusable method. Do not encode visible answers, specialize to individual sequences or queries, infer hidden identities, or branch on array fingerprints, exact shapes, ordering, filenames, or case counts.
  • Treat candidate order as arbitrary and independently permuted for every query. Use candidate_model_id and candidate_stage as semantic metadata; never assume a fixed candidate-axis position.
  • Preserve query equivariance: reordering queries must only reorder outputs, and a query evaluated alone must receive the same prediction as it does in a batch.
  • Prediction must be deterministic for identical inputs. Do not access sealed data, another container, Docker, network resources, or the verifier.
  • Keep runtime and memory practical on CPU. Invalid output, more than four scored-frame switches for any query, timeout, excess resource use, or an exception fails the submission.

Run the complete visible evaluation and public contract audit with:

python /app/selfcheck.py --audit-contract

Use python /app/selfcheck.py --case-limit 3 --audit-contract only for quick smoke tests; the subset score is not comparable to the complete visible score.

What You Have

  • /app/methods/main/predict.py: a weak route that chooses one persistent native-confidence state after each query.
  • /app/data/visible/inputs/: 53 visible candidate-lattice cases.
  • /app/data/visible/labels/: visible labels used by selfcheck.py only. They are development targets, never prediction inputs.
  • /app/data/visible/index.json: shapes and integrity commitments for the visible cases.
  • /app/selfcheck.py: the visible Average Jaccard evaluator, switch-budget validator, and deterministic/query/candidate/singleton audit.

For one case, Q is the number of query tracks, T the number of frames, and K=10 the candidate count. Inputs have these shapes:

  • query_points: float32 [Q,3] in (query_frame, y, x) order;
  • candidate_tracks: float16 [Q,T,K,2] in (x,y) pixel coordinates;
  • occlusion_logits and expected_dist_logits: float16 [Q,T,K];
  • candidate_model_id and candidate_stage: uint8 [Q,K], containing each (model,stage) pair from {0,1} x {0,1,2,3,4} exactly once per query.

Candidate permutations and opaque case identifiers differ between visible and sealed cases. The candidates differ systematically in quality, so the visible evaluator provides an edit-run-measure signal without prescribing how routing or visibility should be modeled.

What You Submit

Harbor collects /app/methods and /app/experiment_log.md. Helper Python files and a compact learned artifact may be placed under /app/methods/main; prediction must not require installation, training, labels, network access, or writable caches at grading time.

Keep every graded helper or learned artifact beside predict.py under /app/methods/main. The staged bundle may contain at most 64 regular files and 64 MiB total; each file may be at most 32 MiB and must end in .py, .json, .joblib, .npz, or .npy.

Keep a concise experiment record in /app/experiment_log.md. Record complete visible scores, the change tested, and whether you kept or reverted it.

How It Is Judged

The verifier maps every predicted semantic state to its candidate track and computes Average Jaccard in first-query mode at pixel thresholds 1, 2, 4, 8, 16; higher is better. Frames at or before each query frame are excluded. Scores are computed per video and macro-averaged across videos.

On a disjoint sealed split, the verifier privately permutes candidate and query order, invokes the submitted method without labels or case identity, validates the four-switch budget, and checks deterministic repeat, semantic candidate-permutation invariance, query equivariance, and singleton equivalence. These are the same behaviors exercised by selfcheck.py --audit-contract.

Rollouts

229 minWall clock
$47.11Spend
77.3MTokens
8Versions, 6 kept

On the visible set

0.60 0.62 0.64 0.66 0 150 300 450 Agent step Video AJ ↑ v1 v2 v3 v4 v5 v6 v7 v8
keptrolled backsubmitted
  1. v1The agent started from the unmodified zero-switch confidence route.0.5904841 min · $0.24
  2. v2The agent found one candidate that beat the router on almost every video.0.6296396 min · $0.70
  3. v3The agent learned what a frame is worth and spent its four switches by exact DP.0.66027767 min · $8.96
  4. v4The agent stacked a route-aware calibrator and decoded visibility over time.0.6668598 min · $15.98
  5. v5The agent weighted training toward the frames where routing actually matters.0.667902111 min · $20.39
  6. v6The agent tried a second blended utility model and paid runtime for nothing.0.667865163 min · $38.07
  7. v7The agent made it cheap to declare occlusion and expensive to come back.0.667979174 min · $38.96
  8. v8The agent tried feeding the meta-model more frames from its longest videos.0.667726196 min · $41.61

On the hidden set

Original metricNormalised score
Starter0.57086524873934990.0
Upper1.01.0
This run (GPT-5.6-sol)0.6352410.1500
320 minWall clock
$18.14Spend
24.1MTokens
4Versions, 3 kept

On the visible set

0.585 0.600 0.615 0.630 0.645 0.660 0 1 2 2 3 Agent step Video AJ ↑ v0 v1 v2 v3
keptrolled backsubmitted
  1. v0The agent inherited a zero-switch persistent native-confidence route0.590484$2.12
  2. v1The agent rewrote the solver as a calibrated MLP with Viterbi routingTurn a calibrated per-candidate hit probability into expected AJ, then decode the route under the budget.0.644240 min · $4.25
  3. v2The agent added a frame-level occlusion head and ensembled three seedsRead occlusion once per frame from the whole candidate set instead of from each candidate alone.0.6555133 min · $11.72
  4. v3The agent grew the ensemble to five and fixed a standardisation bug0.6561319 min · $17.82

On the hidden set

Original metricNormalised score
Starter0.57086524873934990.0
Upper1.01.0
This run (Opus 5)0.6408770.1631
44 minWall clock
$7.93Spend
54.7MTokens
5Versions, 4 kept

On the visible set

0.59 0.60 0.61 0.62 0.63 0 1 2 3 4 Agent step Video AJ ↑ v0 v1 iter1 iter2 iter3
keptrolled backsubmitted
  1. v0The agent inherited a zero-switch persistent native-confidence route0.590484
  2. v1The agent added consensus visibility and temporal Gaussian smoothing of logitsHold the strongest candidate and flag drift by its divergence from the median of the others.0.630699
  3. iter1The agent built a DP router with unary and transition penalties0.613754
  4. iter2The agent fixed candidate 9 and calibrated a linear visibility boundary0.627623
  5. iter3The agent added a consensus-divergence drift filter to the visibility gate0.6306

On the hidden set

Original metricNormalised score
Starter0.57086524873934990.0
Upper1.01.0
This run (Gemini 3.7 Flash)0.6202830.1152
185 minWall clock
$3.64Spend
7.6MTokens
2Versions, 1 kept

On the visible set

0.59 0.60 0.61 0.62 0.63 0 20 40 60 80 Agent step Video AJ ↑ v0 v1
keptrolled backsubmitted
  1. v0The agent inherited a zero-switch persistent native-confidence route0.590484$0.54
  2. v1The agent learned a ridge utility and routed it by budgeted DPLearn one per-frame utility from track and occlusion features, then route by exact DP under the budget.0.628338$3.21

On the hidden set

Original metricNormalised score
Starter0.57086524873934990.0
Upper1.01.0
This run (Kimi K3)0.61780.1094
48 minWall clock
$9.30Spend
14.6MTokens
7Versions, 6 kept

On the visible set

0.59 0.60 0.61 0.62 0.63 0 2 3 4 6 Agent step Video AJ ↑ v0 v1 v2 v3 v4 v5 v6
keptrolled backsubmitted
  1. v0The agent inherited a zero-switch persistent native-confidence route0.5905
  2. v1The agent pinned semantic state 9 and gated visibility on two logitsNative confidence routes badly, so hold the strongest state and spend the effort on the visibility rule.0.629202
  3. v2The agent replaced the two gates with one linear occlusion-distance rule0.630725
  4. v3The agent added a ridge ranker allowing switches to states 4/7/8Spend a switch only where a tune-fit ranker says another state beats state 9 by a wide margin.0.631485
  5. v4The agent kept ranker switches only when the alternative's expected distance improved0.631558
  6. v5The agent tightened the visibility cutoffs per semantic state0.631762
  7. v6The agent refit the ridge on the AJ gap, overriding only to state 40.631992

On the hidden set

Original metricNormalised score
Starter0.57086524873934990.0
Upper1.01.0
This run (Grok 4.6)0.621350.1176
233 minWall clock
$1.90Spend
20.5MTokens
12Versions, 9 kept

On the visible set

0.600 0.615 0.630 0.645 0 2 5 8 10 Agent step Video AJ ↑ v0 v1 v2 v3 v4 v5 v6 v6b v7 v7b v8 v9
keptrolled backsubmitted
  1. v0The agent inherited a zero-switch persistent native-confidence route0.590555 min · $0.60
  2. v1The agent learned logistic hit models and routed them by budgeted DP0.612355 min · $0.59
  3. v2The agent enriched features with interactions, consensus and query-point distance0.614880 min · $0.94
  4. v3The agent consolidated a 91-feature set with model/stage interactions0.616380 min · $0.95
  5. v4The agent swapped the logistic hit models for 64-unit MLPs0.6219880 min · $0.94
  6. v5The agent widened the MLPs to 96 units and doubled training rows0.6234788 min · $1.08
  7. v6The agent added rank and gap features and box-blurred the DP surface0.62879142 min · $1.52
  8. v6bThe agent trained joint visible-and-accurate targets instead of the product decomposition0.62193$1.60
  9. v7The agent added query-anchor frame features to the visibility model0.63244175 min · $1.68
  10. v7bThe agent widened the hidden layer to 128 and overfit0.6353$1.72
  11. v8The agent trained on two million rows instead of one million0.63489213 min · $1.77
  12. v9The agent refit the final model on all 53 visible cases0.64215230 min · $1.85

On the hidden set

Original metricNormalised score
Starter0.57086524873934990.0
Upper1.01.0
This run (DeepSeek V4 Pro)0.6155630.1042
173 minWall clock
$12.17Spend
40.8MTokens
5Versions, 4 kept

On the visible set

0.59 0.60 0.61 0.62 0.63 0 40 80 120 Agent step Video AJ ↑ v1 v2 v3 v4 v5
keptrolled backsubmitted
  1. v1The agent inherited a zero-switch native-confidence baseline0.59048431 min · $1.71
  2. v2The agent rewrote routing with calibrated logits and an exact five-segment DP0.61675783 min · $3.89
  3. v3The agent added a model/stage quality prior to the reward0.627491108 min · $4.92
  4. v4The agent dropped switching entirely and kept one constant route0.630575125 min · $5.82
  5. v5The agent blended cohort occlusion into the visibility gate0.631248143 min · $7.97

On the hidden set

Original metricNormalised score
Starter0.57086524873934990.0
Upper1.01.0
This run (Qwen3.8 Max)0.6195510.1134
676 minWall clock
$13.90Spend
47.2MTokens
8Versions, 7 kept

On the visible set

0.585 0.600 0.615 0.630 0.645 0 75 150 225 Agent step Video AJ ↑ v0 v1 v2 v4 v5 v6 v7 v8
keptrolled backsubmitted
  1. v0The agent inherited a zero-switch persistent native-confidence route0.590547 min · $1.60
  2. v1The agent fit GBM reward and visibility models feeding a segment DP0.647217 min · $6.32
  3. v2The agent added temporal and consensus features and fixed the forest export0.6472373 min · $8.75
  4. v4The agent trained the gate on out-of-fold routes to match deployment0.6463462 min · $10.13
  5. v5The agent fixed a caching bug that leaked in-sample routes into CV0.6463518 min · $10.76
  6. v6The agent cut the gate inputs from 45 to 350.6478578 min · $12.06
  7. v7The agent chunked queries to hold peak memory near 215 MB0.6478619 min · $12.83
  8. v8The agent also marked frames occluded below 0.3 visibility probability0.648666 min · $13.59

On the hidden set

Original metricNormalised score
Starter0.57086524873934990.0
Upper1.01.0
This run (GLM 5.3)0.6350060.1495
58 minWall clock
$8.24Spend
9.6MTokens
10Versions, 9 kept

On the visible set

0.59 0.60 0.61 0.62 0.63 0.64 0 25 50 75 100 Agent step Video AJ ↑ v0 v1 v2 v3 v4 v5 v6 v7 v8 v9
keptrolled backsubmitted
  1. v0The agent inherited a zero-switch persistent native-confidence route0.5904841 min · $0.16
  2. v1The agent replaced native-confidence routing with fixed semantic state 90.6206392 min · $0.37
  3. v2The agent calibrated a single global native visibility threshold at 0.2850.6277168 min · $0.83
  4. v3The agent picked one persistent state per query using a semantic prior0.63349322 min · $2.15
  5. v4The agent gave each semantic state its own visibility threshold0.63455523 min · $2.54
  6. v5The agent added a native override over the first three frames0.63463327 min · $3.12
  7. v6The agent scored visibility by weighted occlusion and distance probabilities0.63569734 min · $4.12
  8. v7The agent added a native-margin term to the visibility risk0.63586740 min · $4.95
  9. v8The agent added candidate-disagreement and speed features to visibility0.63604647 min · $5.95
  10. v9The agent refined two thresholds after matching float16 prediction rounding0.63605155 min · $7.31

On the hidden set

Original metricNormalised score
Starter0.57086524873934990.0
Upper1.01.0
This run (GPT-5.5)0.6215870.1182

Leaderboard

Where each run landed on the sealed held-out set, on the same normalised-score scale as the anchors above.

0 0.3 0.6 1.0 1 Opus 5 claude code · max 0.163 2 GPT-5.6-sol codex · max 0.150 3 GLM 5.3 claude code · max 0.149 4 GPT-5.5 codex · xhigh 0.118 5 Grok 4.6 grok · xhigh 0.118 6 Gemini 3.7 Flash antigravity · high 0.115 7 Qwen3.8 Max qwen coder · xhigh 0.113 8 Kimi K3 kimi cli · max 0.109 9 DeepSeek V4 Pro claude code · max 0.104
PreviousScientific Discovery Agent Harness DesignNextSmall-Model Math Reasoning Post-Training