A coding agent can sound convincing after reading the wrong files. RepoContextBench measures whether it found enough of the right repository context to answer correctly.
Explore the interactive RepoContextBench results →
Version 1 uses a pinned snapshot of microsoft/agent-framework: 512,954 source lines, 4,181 files, and 20 practical repository questions. The public dataset contains 82 weighted gold claims and 99 exact source-evidence spans.
What Score means
RepoContextBench does not grade answers by exact text matching. Its Score runs from 0 to 100 and combines:
- coverage of weighted, source-backed gold claims;
- faithfulness to the repository;
- use of supporting source evidence.
This gives partial credit to an answer that correctly explains part of a workflow while missing another required part. Judge time, tokens, and cost are excluded from answerer efficiency metrics.
The matched CodeAlive result
We compared Claude Sonnet 4.6 on the same 20 questions, with maximum reasoning effort and the same fixed judge contract. The only material change was the path used to research repository context.
CodeAlive raised the Score by 6.5 points.
This is the clearest v1 result: better repository context produced more complete, source-grounded answers from the same model. The comparison is a complete archival matched pair from the same 20-task line, rejudged under the publication judge contract.
Scope and limits
V1 covers one repository snapshot and 20 tasks. The public questions, answers, claims, trajectories, and evidence make the result auditable, but also allow benchmark-specific tuning after publication. Agents perform static repository research only: they cannot use the internet, run tests, execute code, or modify the repository.
The interactive Benchmark, dataset and runner, methodology, and task-level run dashboard are public.
We are now working on the next RepoContextBench release. It will add a dataset built on a codebase with more than one million lines of code.