The task
Redline a counterparty contract against our own negotiation playbook and route each deviation to accept, negotiate, or escalate.
Proposed by PIG-DemoRank 11 of 12Not in the first set
Legal Playbook Redline
Who it is for
General Counsel
First-pass review of a mutual NDA is not legal work, and it is still what my team does on a Thursday night.
The reward
Pays for finding every clause the playbook flags and putting it in the right one of the three buckets. That is checkable against the playbook itself.
The counterweight
Escalating everything finds every clause and reviews nothing, so the escalation bucket has a budget and overspending it is penalised.
The counterweight is the part worth arguing about. A reward with nothing pulling against it is a target, and a target is what a model learns to hit instead of doing the job.
Where this stops
Finding the clause is checkable. Whether the replacement language is an acceptable redline is judgment, and grading judgment collapses to an LLM judge — the exact thing a verifiable reward is meant to replace. We would ship the detection half with a real verifier and say plainly that the drafting half is unverified. We would not put a judge behind a bar chart and call it a score.
What exists today
Nothing. There is no environment for this vertical, no recorded run, and no number. It is a proposal, written to the same shape as the one that is built so you can judge it on the same terms.
The environment that does exist is Word Five. It is a word game, and it is deliberately not one of these twelve: it is there to show the machinery, not to imply we have built yours.