Tasks are generated from a random seed at run time and never published. There is nothing to train on — the set does not exist until the run happens.
A model scores 92% on MMLU. Impressive — until you learn that some of those questions were in its training data. How many? What is the real score? Nobody knows. This is not hypothetical: benchmark contamination is a recognized problem (CoDeC, ICLR 2026; DCR, EMNLP 2025; LiveBench, ICLR 2025).
Static benchmarks leak. Their questions live on GitHub, arXiv and Stack Overflow for years. LiveBench rotates questions monthly — but even a monthly set can be partly memorized.
225 worlds deterministically generate tasks from a seed. Nothing is stored in advance — the set is created the moment the run starts.
Reports contain aggregates only. No individual question is revealed — there is nothing to memorize.
sha256(seed) is published before the run, the seed after. This proves the set was not swapped.
Old tasks are mixed back in. The fresh-vs-old gap directly measures memorization.
Tasks depend on the seed, and a solver must solve them for any seed. So we can re-mix tasks from a previous run (regenerated byte-for-byte from a ref) into the current set and compare:
| candidate | fresh | old | gap | verdict |
|---|---|---|---|---|
| honest solver | 100% | 100% | +0.0 pp | solves |
| memorizer | 0% | 100% | +100.0 pp | recalls |
A gap > 15 pp is a reliable sign the model is optimized for the benchmark rather than solving it.
Tasks are generated from a seed at run time and never published. θ is a Rasch ability estimate (comparable across rounds). Real models are called directly through provider APIs; the reference is our own deterministic code (0 LLM calls).
| candidate | accuracy | θ (IRT) | notes |
|---|---|---|---|
| script:all | 99.7% | +5.4 | deterministic solver (reference) |
| openai/gpt-5.6-terra | 96.9% | −0.15 | 31/32, crypto-paid run (Nano) |
| anthropic/claude-sonnet-4.5 | 96.8% | −0.15 | 30/32, crypto-paid run (Nano) |
| deepseek-v4-flash | 95.8% | −0.04 | 3 runs |
| gpt-oss-120b (groq) | 93.7% | +0.19 | 6 runs |
| memorizer (test) | 50.0% | −0.66 | gap +100 pp → flagged |
Technical errors (HTTP 402/429 — rate limits, insufficient balance) are counted separately and excluded from accuracy: a model that was rate-limited is not a model that failed. A candidate needs ≥5 valid tasks to be ranked, so a quota-limited 1/1 is not shown as 100%.
Math, algorithms, graphs, SQL, compilers, OS, crypto, physics, chemistry, biology, geography, linguistics, economics, finance, logistics, marketing, medicine, law, games, puzzles, probability, electronics, music, cooking, sports…
A Rasch model yields a candidate θ that is comparable across rounds even when the task sets differ.
Anyone who has seen a gap between the leaderboard and production. If a model promises 92% and then fails in the real world, you need a way to check whether it learned the skill or memorized the answers.
The same benchmark is a pay-per-call API for agents: generate fresh tasks from a seed and grade answers, no signup, no key. 225 domains, ~$0.01 a call, settled in USDC on Base or Nano via x402.
Discovery: /.well-known/x402 · this is how a contamination-resistant eval reaches an agent that is already paying for APIs.
Example report card (deepseek-v4-flash) — a shareable per-model card: accuracy, θ, contamination gap, by-world breakdown.