The task
Given a repository at a specific commit and one failing test, make that test pass.
Proposed by PIG-DemoRank 7 of 12In the first set
Code Fix-the-Test
Who it is for
VP of Engineering
Every vendor shows me a pass rate on a public benchmark my team has never run on code my team has never seen.
The reward
The test suite is the grader. It pays 1 when the target test passes and everything that passed before still passes.
The counterweight
Editing the test, weakening its assertion or marking it skipped is caught by diffing the test files, and scores zero. This is the vertical where the grader argues back the least, which is exactly why it is the cheapest one to trust.
The counterweight is the part worth arguing about. A reward with nothing pulling against it is a target, and a target is what a model learns to hit instead of doing the job.
What exists today
Nothing. There is no environment for this vertical, no recorded run, and no number. It is a proposal, written to the same shape as the one that is built so you can judge it on the same terms.
The environment that does exist is Word Five. It is a word game, and it is deliberately not one of these twelve: it is there to show the machinery, not to imply we have built yours.