CVB
The Constraint-Violation Benchmark. Teams record a decision once — “we standardized on httpx after the March socket-exhaustion incident” — and expect every future AI generation to honor it. CVB measures whether that actually happens.
The Gap It Measures
Constraints reach a coding agent two ways. They can arrive as explicit instructions in the prompt — orders. Or they can sit in ambient context: a memory layer, a CLAUDE.md, a RAG retrieval. Every team building on agents assumes the second works like the first.
IFEval and AgentIF only measure the first. That is the gap: the entire premise of persistent project memory rests on an assumption nobody had benchmarked.
Design
Three arms, identical tasks, different context:
- cold — task only. Establishes the baseline violation rate.
- mandated — task plus the constraints as explicit orders. Establishes the instruction-following ceiling.
- incentivized — the same constraints embedded in an ambient project-memory narrative. Measures memory adherence.
The headline number is the distance between the last two: mandated strict accuracy minus incentivized strict accuracy. A small gap means ambient memory is as good as an order. A large gap means every memory layer in production is quietly failing, and the industry has been measuring the wrong thing.
Results
Groq API, temperature 0, 3 runs per arm, 35 scenarios, 1260 records. Strict accuracy is the share of runs where every constraint was honored.
- llama-3.1-8b-instant — cold 0.229 · mandated 0.914 · incentivized 0.857 · gap +0.057
- llama-3.3-70b-versatile — cold 0.229 · mandated 0.952 · incentivized 0.981 · gap −0.029
- gpt-oss-120b — cold 0.638 · mandated 0.962 · incentivized 0.952 · gap +0.010
- qwen3.6-27b — cold 0.476 · mandated 1.000 · incentivized 1.000 · gap 0.000
The result went against the hypothesis. The expectation was that constraints demoted to ambient memory would be widely ignored. They were not. Only the weakest model loses meaningfully (5.7 points); the 70B model adheres better to memory framing than to direct orders, and qwen is perfect under both.
What the run does establish, decisively, is the thing nobody was arguing about: cold accuracy of 0.229 means that without context these models default to the violating pattern — naive datetime.now(), shell=True, missing locks. Getting constraints into context at all is worth 60–75 points. Which framing you use is worth close to zero.
Why It Exists
CVB is the adversarial counterpart to HCR. HCR's thesis is that ambient memory has to be enforced rather than hoped for. A benchmark that could disprove that thesis is more valuable than another feature that assumes it — which is exactly what happened, for single-turn generation at these model sizes.
The design and harness went public before the numbers, deliberately. A benchmark whose methodology appears only alongside favourable results is a marketing asset, not an evaluation. The honest limitations are published with it: regex checks are conservative, so a pass means “no detected violation” rather than proof of compliance, and the run measures single-turn generation — not the long-horizon, many-turn setting where memory decay is actually claimed to bite.