Honest measurement

The question a score has to survive

A benchmark is shortcut-solvable when a policy can reach a good score on it while lacking the capability the benchmark claims to measure. So the question for this competition is not "did the model score well". It is "could it have scored well without reading the demonstration".

Four things close the obvious shortcuts.

Identical units, both sides

The crown is decided by comparing two models on the same units — same task, same initial state, same demonstration, same seed — never by putting two leaderboard numbers side by side. Some units are simply harder than others; pairing removes that variance, and what is left is the difference between the models. The per-unit outcomes are published so the pairing can be checked row by row.

Replaying the demonstration

If a policy can simply play the demonstration's actions back and solve the episode, a score partly measures playback. This contest does not hide that: the policy is shown the frames, the action trajectory and the proprioception, with nothing withheld, and is scored from a different initial state. Can the demonstration be replayed? no — the actions do not fit a different initial state.

Where a replay can solve a unit, it is the floor of what a score can mean, and a published one: a replay policy winning a duel is not a bug. Where it cannot, a score measures what a policy carries over from the demonstration, not playback. Either way the crown rule measures one thing — how much better than the reigning model a policy does on the same units, a paired comparison on identical demonstration bytes, never two leaderboard numbers side by side — and every duel is published with every unit and every clip.

No language

The task is never named. There is no instruction, no task id, no caption. The task names are public — the benchmarks are open source — but the only channel that tells the model what to do is the one demonstration. A model that scores well here did so by reading it.

Arithmetic you can redo

Every duel page recomputes both sides' per-skill scores from the published unit list, shows the average row the rule reads, and states the comparison:

challenger average ≥ reigning average + +3.0 pts

The page shows the published verdict beside its own. If the two ever disagreed, the page would say so rather than pick one — a ranking that cannot be checked is not worth publishing.

The diagnostic that does not decide

Because both sides ran the same units, every unit can be labelled won, lost or tied. Those counts sit on every duel page under the heading diagnostic, does not decide the crown. They are there so a reader can see where the two models differed — which skill, which task — not to rank them. Only the average does that.

What is published about your entry

Nothing about the inside of it: your code runs in its own sandbox, and the orchestrator never imports, unpickles or executes anything from it — see what you submit. What is published is what came out: the repository and revision, both sides' per-unit outcomes on every skill with each unit's initial state, three clips per unit, the per-skill scores and the average, the diagnostic counts, the protocol fingerprint and the duel identifier — all of it, as soon as the duel finishes, win or lose. The record is signed by the orchestrator and mirrored to a public dataset, so a third party can verify it without trusting this site.