RSI-Exam

What RSI-Exam measures

RSI-Exam measures whether a language-model agent can improve the executable method that solves a task or the harness that drives a frozen model. It studies improvement of that working artifact through sustained experimentation, not modification of the model's weights.

• Long-horizon improvement. Agents inherit a functioning starting artifact and spend hours proposing, measuring, retaining, and discarding changes.
• Hidden-set re-execution. The deliverable is executable code rather than a reported score or a file of predictions, and the verifier runs it unchanged on the hidden set.
• Domain-native evaluation. Tasks retain the metrics used in their source fields, while each task's frozen scoring definition places measured progress on a common scale.
• Inspectable research trajectories. Saved method versions, experiment descriptions, resource use, and hidden-set outcomes remain available for analysis.

The first release applies these principles across six domains. RSI-Exam 0.1 contains 88 tasks: 35 public and 53 private. They span AI models and agents, systems and hardware, physical sciences and engineering, optimization, life sciences and medicine, and finance, law, and business, while retaining the metric and executable workflow of each field.

The RSI-Exam benchmark

How an evaluation works

Every RSI-Exam task begins with a working, executable artifact that produces a valid but improvable result. Depending on the task, that artifact may be a method that solves the problem directly or a harness that directs a frozen model. Examples include a query optimizer restricted to nested-loop joins, a gate-sizing method that assigns maximum drive strength to every instance, and a wind-retrieval method that solves only part of the governing system. Starting from a functioning artifact rather than an empty repository requires the agent to diagnose and improve an existing approach.

That inherited artifact becomes the starting point for one fixed-budget evaluation run, or rollout. Within its time and resource limits, the agent modifies the artifact, evaluates candidate versions on the visible set, and saves numbered versions as its approach evolves. At the end, it submits one executable artifact rather than an answer or a precomputed prediction file. The verifier then runs that artifact unchanged on the hidden set in a sealed evaluation environment.

The two sets serve different purposes. The visible set guides experimentation and method selection; the hidden set tests whether the final improvement transfers beyond the data or cases available during development. Using the same set for both would conflate generalizable progress with adaptation to the development cases. The task score therefore comes from the hidden-set rerun, separating transferable improvement from a high visible-set score.

Construction and quality control

That protocol is trustworthy only if a better hidden-set result represents a genuine, generalizable improvement rather than leakage, execution noise, or a defect in the benchmark. RSI-Exam therefore builds and reviews every task through four stages: problem selection, environment construction, score calibration, and end-to-end review.

Problem selection. We begin from published research problems with an executable baseline and a quantitative objective. A candidate must leave meaningful headroom beyond the inherited method, support multiple rounds of substantive experimentation, and admit evaluation on private data. Only problems that satisfy these requirements are converted into benchmark tasks, and candidates already solved by the pilot agent are excluded.

Environment construction. Each selected problem is packaged into two isolated runtime environments: a development environment visible to the agent and a sealed evaluation environment used by the verifier. Any author-only reference implementations and calibration records remain outside the agent-visible surface; they are supporting artifacts, not a third runtime environment. Resource limits and the submission interface are made explicit. The required deliverable is an executable method, not a stored answer or prediction file.

Within the sealed evaluation environment, a trusted parent process loads the protected targets and keeps them outside the child environment before importing the submission. The child receives only the permitted evaluation inputs and a narrow prediction interface; network access and retraining inside the verifier are disabled. Score calibration begins only after this isolation boundary is fixed.

Score calibration. Landmarks are measured reference points used to map a task's native metric onto the common scale. Every landmark must be supported by the frozen task definition. A landmark derived from a method must also correspond to a concrete run in the evaluation environment. Reference methods use the same verifier as submissions; the task's aggregation rule, landmarks, and score map are then frozen for the release. The mapping is described under Scoring and normalization below.

End-to-end review. Review covers the problem definition, constraints, data split, leakage surface, reference methods, and verifier implementation. The completed task then undergoes a long-horizon pilot that checks four properties:

• Headroom: the inherited method leaves room for meaningful improvement.
• Difficulty: progress requires substantive experiments rather than a local patch.
• Distribution consistency: visible-set gains transfer to the hidden set.
• Horizon: the task sustains a long sequence of consequential decisions.

Only tasks that pass all four stages enter the release bank. A failed task returns to the relevant stage for revision, and an unresolved candidate is removed.

Task bank

The RSI-Exam 0.1 release bank contains 88 tasks across six domains, comprising 35 public and 53 private tasks. Together they span AI, science, engineering, systems, medicine, and professional decision-making while retaining the native metric and executable workflow of each field.

Virtual cellsSix-Channel Cell Morphology TPU kernelsTPU v6e Masked GQA Kernel Optimization Chip designEDA Flip-Flop Banking and Placement Optimization

Quant financeCross-Sectional Stock Return Ranking Harness designLLM Agent Memory Architecture Design Model distillationSmall-Model Math Reasoning Post-Training

Click a card to view task details.

Scoring and normalization

Anchors. The inherited Starter is fixed at 0.00. When a genuine mathematical, theoretical, or oracle Upper bound exists, it is fixed at 1.00. When the task author provides a strong frontier solution, we verify its quality with the same sealed verifier, correct the calibration when necessary, and use the resulting measured performance as an optional Frontier-calibrated SOTA anchor at 0.60.

Mappings between anchors. We use linear or log-linear interpolation between finite anchors. When no finite upper bound exists, we use an exponential tail so further gains still receive credit while the score approaches, but never reaches, 1.00. Each task’s anchors and mapping are fixed before comparison.

Experimental setup

We hold the tasks, verifiers, and scoring procedure fixed and vary the model-agent system. The experiment measures both the final method produced by each system and the research process that produced it.

Models and agent harnesses

A rollout begins with a single agent working alone in a fresh container that holds the task environment and inherited method. When the run ends, the verifier re-executes the final method on the hidden split and computes its score. Each model receives one rollout per released task, so the results compare aggregate capability but do not estimate within-model run-to-run variance.

Execution and records

Execution isolation. Every rollout runs in its own Docker environment under the resource and time budgets declared by the task. The agent and verifier stages remain separate, and the container is destroyed after evaluation.

Network isolation. Container egress was restricted to the model provider API and, where required, its authentication host. Literature, external leaderboards, and package indexes were inaccessible. The Claude harness additionally ran with WebSearch and WebFetch disabled. No run used a skill library or MCP server, and all models received an instruction rendered from the same prompt template.

Interrupted runs. If an agent terminates early because of a timeout or another error, the verifier still evaluates the final method left in the container. A score of 0 therefore means that the method did not improve on the inherited baseline; it does not necessarily indicate a runtime failure.

Inspectable records. Each rollout preserves its resolved model configuration, timing, verifier reward, and complete version history. The history records each attempted method, the agent’s description of the change, and which versions were retained or discarded. Task pages are generated from these underlying artifacts, making every leaderboard result traceable to its research trajectory.

Analysis plan

The analysis proceeds from outcomes to mechanisms. We first compare mean normalised scores, then use within-task relative performance to ask whether model strengths vary by domain. Next, we measure how much visible-set performance transfers to the hidden set and describe how runtime, spend, and step count relate to outcomes. Finally, selected trajectories show how retained and discarded experiments accumulate into a final method. Aggregate comparisons use every task in the stated panel; analyses that require additional records report their smaller sample explicitly.

Conclusion and limitations

RSI-Exam turns recursive self-improvement from an abstract capability claim into an executable, reproducible research loop. An agent inherits a working method, repeatedly proposes changes, tests hypotheses, retains useful versions, and rolls back failed ones. The final method is then re-executed on the hidden set; only improvements that survive beyond the development environment contribute to its score. In this way, RSI-Exam measures improvement within a bounded, executable research setting.

Limitations and next steps

• Evaluation scope. RSI-Exam measures method improvement within bounded research environments. It does not evaluate models modifying their own weights, nor does it constitute unrestricted recursive self-improvement. Future releases can extend the protocol to additional forms of adaptation while preserving executable, hidden-set evaluation.
• Model–harness pairs. Each result reflects a model operating through a particular agent harness. Tool use, context management, and stopping behaviour are part of that evaluated system, so leaderboard differences should not be attributed to the underlying model alone.
• Starting conditions. Every rollout begins from a task-specific inherited artifact, although the agent may modify or replace it completely. Because the research budget is fixed, that starting artifact can still shape the search path and which alternatives are reached. Scores therefore describe improvement from the specified starting condition.
• Single rollouts. Each model has at most one graded rollout per task. Because the leaderboard averages results across 88 tasks, idiosyncratic variation in any individual rollout is diluted across the bank; the aggregate leaderboard is therefore expected to have low run-to-run variance. At the task level, however, each score and trajectory remains a single observation and cannot show how much the same model–task pair would change on rerun. The next minor release will include repeated runs to measure this variation directly.

Data availability

Each RSI-Exam 0.1 task has a page carrying the instruction given to the agent, the saved method versions and their change descriptions, and the hidden-set results available for the evaluated models.

Those pages are derived from the underlying task definitions and rollout records. The release artifacts store the resolved model configuration, run record, version history, and verifier output for each available model-task pair. Every figure in this post is computed from the RSI-Exam 0.1 records and the pages generated from them.

The contribution guide explains how researchers can propose a task from their own field, review an existing task, or contribute domain expertise to a future release.

How to cite RSI-Exam

RSI-Exam Team. (2026). RSI-Exam: Benchmarking Recursive Self-Improvement through Executable Research.

@misc{rsi-exam-2026,
  author = {{RSI-Exam Team}},
  title  = {RSI-Exam: Benchmarking Recursive Self-Improvement through Executable Research},
  year   = {2026},
  url    = {https://github.com/aiming-lab/RSI-Exam}
}