Honest measurement

The question a score has to survive

A benchmark is shortcut-solvable when a policy can reach a good score on it while lacking the capability the benchmark claims to measure. So the question for this competition is not "did the model score well". It is "could it have scored well without reading the demonstration".

Four things close the obvious shortcuts.

Identical units, both sides

The crown is decided by comparing two models on the same units — same task, same initial state, same demonstration, same seed — never by putting two leaderboard numbers side by side. Some units are simply harder than others; pairing removes that variance, and what is left is the difference between the models. The per-unit outcomes are published so the pairing can be checked row by row.

The demonstration starts somewhere else

If the scored episode began where the demonstration began, replaying the demonstration's actions would solve it, and the competition would be measuring playback. So every unit's demonstration is recorded from a different initial state than the one the model is scored from, and the perturbation on every axis moves the scene further still. The draw refuses a unit whose demonstration and scored instance coincide.

No language

The task is never named. There is no instruction, no task id, no caption. The task names are public — the benchmarks are open source — but the only channel that tells the model what to do is the one demonstration. A model that scores well here did so by reading it.

Arithmetic you can redo

Every duel page recomputes both sides' per-axis scores from the published unit list, shows the average row the rule reads, and states the comparison:

challenger average ≥ reigning average + +3.0 pts

The page shows the published verdict beside its own. If the two ever disagreed, the page would say so rather than pick one — a ranking that cannot be checked is not worth publishing.

The diagnostic that does not decide

Because both sides ran the same units, every unit can be labelled won, lost or tied. Those counts sit on every duel page under the heading diagnostic, does not decide the crown. They are there so a reader can see where the two models differed — which axis, which task — not to rank them. Only the average does that.

What is published about your entry

The repository and revision, both sides' per-unit outcomes on every axis with the perturbation each unit carried, three clips per unit, the per-axis scores and the average, the diagnostic counts, the protocol fingerprint and the duel identifier — all of it, as soon as the duel finishes, win or lose. The record is signed by the validator and mirrored to a public dataset, so a third party can verify it without trusting this site.