It evaluates whether an AI agent can improve itself over a long horizon and generalize to unseen data. Hours of autonomous experimentation on the method that solves the task or the harness that drives a frozen model, then one final run on a hidden test set.
Mean hidden-set score over the 88 published tasks against what one task took on average.
Grouped by the problem a task solves. Hover or click a domain to see the tasks inside it.
In the working container an agent receives the task instruction and the autoresearch protocol. It then improves its approach round after round, guided by feedback on the visible data. When it is done, the artifact it produced is handed to the verifier container and scored on hidden data.
The method that solves the problem, or the harness that runs a frozen model. Each task targets one of them.
Iterate on the visible data as long as the budget allows. The score comes from one run on the hidden set.
Every task is written by a domain expert, then implemented and reviewed by other people.
Proposes the problem and the metric, and runs the weak baseline and at least one stronger reference end to end to calibrate the range.
Turns the idea into our required format: two isolated images, a declared artifact contract, hidden data and trusted scoring code.
A standing rubric of 90+ checks: value, measurability, data provenance and licensing, and every leakage path that could be looked up or memorised.
Analyses a full agent trajectory against the anchors, then sends the verdict back as a fix, an enhancement or a filter.
Authors are PhD students, postdocs, faculty and industry practitioners, each writing in their own research area.