Hash the questions before any model sees them
A benchmark is only worth something if nobody could have tuned it after seeing the results, including the person who wrote it. Here's how Max-Q makes that checkable.
Every model release comes with claims about what it’s good at. Some are true. The trouble is that most ways of checking them have a quiet flaw: the person running the test can see the answers, and then change the test.
Usually nobody does it on purpose. A question looks ambiguous after one model gets it wrong, so it gets reworded. A rubric seems too harsh, so it gets loosened. Do that a few times and the benchmark measures how the author felt about the results.
Max-Q is an aerospace engineering benchmark: propulsion, orbital mechanics, structures, guidance and control. It was built around one rule: the questions can’t move once a model has seen them.
Freeze, then publish the hash
Questions are written in waves. When a wave is done, it gets frozen and hashed, and the hash is posted publicly before a single model runs.
The wave itself stays private. Only the fingerprint is public. Later, when the wave is published, anyone can hash it and check that it matches what was posted before the runs. If one word changed, the hash won’t match.
That turns “trust me” into something a stranger can verify in a few seconds.
Run every model the same way
Fairness comes down to boring details:
- One route. Every model goes through the same API gateway, with provider fallbacks turned off. When you ask for a particular model, you get that model, not a substitute that happened to be available.
- Pinned upstream. Where a model is served by more than one provider, the first-party provider is pinned, and the harness records which one actually answered each question.
Machines grade the numbers, a person grades the rest
Numeric answers are scored by the harness against the reference. Open-ended answers get a first pass from an LLM judge, and the judge model is recorded too. Then a person confirms every rubric row by hand. The judge speeds things up, but it doesn’t get the final word.
It’s slower than letting the judge decide. It’s also the only version where a reported score means what it says.
Keep results back until there’s something to compare
The first wave’s results aren’t public yet, on purpose. A single snapshot of today’s models turns into a leaderboard, and leaderboards get gamed. The plan is to publish after the next model generation is run against the same frozen wave. Then the comparison is between a fixed baseline and whatever came next, which is the question people actually care about.
Why this matters outside aerospace
Any team about to trust a model with real work has the same problem, in miniature. Someone runs a few test prompts, likes what comes back, and ships. The fix is the same too:
- Write the questions before you look at any answers.
- Freeze them, and keep a fingerprint.
- Run every candidate the same way, and record exactly what ran.
- Let machines grade what can be graded, and have a person sign off on the rest.
None of it needs clever tooling. It just needs doing in the right order.