Intermundia Labs
← All work

IL-05 / Building

Max-Q

An aerospace engineering benchmark for LLMs — propulsion, orbital mechanics, structures, GNC.

10203040kPathrottleT+0sT+40sT+80sT+120sT+160sMAX Q32 kPa · T+58s · 8.4 kmaltitude

How it runs

  1. Questionswritten in waves
  2. Freezehash posted first
  3. Runevery model the same way
  4. Scoreharness + LLM judge
  5. Hand checkevery rubric row
  6. Publishafter the next generation
  • source
  • step
  • check: nothing passes unless it holds
  • output

Model vendors claim a lot about domain expertise. Max-Q is how I check those claims for aerospace engineering. It’s a set of engineering questions, a harness that runs every model the same way, and a scoring process that’s hard to game.

How it works

  • Questions are written and then frozen in waves. Each wave is hashed, and the hash is posted publicly before any model sees it. That way nobody can quietly tweak questions after seeing results, including me.
  • Every model goes through OpenRouter with fallbacks turned off and the first-party provider pinned. The harness records which upstream actually answered.
  • Numeric answers are scored by the harness. Open-ended answers get an LLM judge first pass, and then I confirm every rubric row by hand.

Status

Wave 1 is frozen. The runs are on hold until the next generation of models is out, and results stay private until that generation is scored. The point is to compare against a fixed baseline, not to chase a leaderboard.

esc