evalrust

noun  /ˈiːvəlrʌst/

The slow decay of an evaluation set as the system it measures moves on — where passing tells you less and less, and nobody notices because the number stays green.

Symptoms

• Your suite has passed 100% for three months, and nobody on the team trusts it anymore.
• A regression ships. You go back and find the eval covered it — with the wrong expected output.
• The set was written against a model you no longer run.
• You can say when a case was last run. You cannot say when it was last reviewed.

Why it happens

Evals get written at the moment of maximum uncertainty — early, fast, under pressure to ship something measurable. Every expected output is a judgment frozen at that moment. The judgment ages; the assertion does not.

Green then does the real damage. A passing suite reads as evidence, but a suite that has stopped tracking reality passes for the same reason a broken thermometer reads room temperature: nothing is connected to anything. The failure is silent by construction, because the only signal a suite has for "I am wrong" is the one thing it can no longer produce.

What to do about it

• Date every case. Not when it was added — when a human last agreed the expected output was correct.
• Review on a schedule, not on failure. Waiting for a red run means waiting for the one event rust prevents.
• Keep a small hostile set you rewrite from scratch every time the model or the prompt changes. If the old suite passes and the fresh one fails, you have found your rust.
• Track the age of the set as a metric. Median case age belongs on the dashboard next to the pass rate — one number is meaningless without the other.