How the ladder works

One crown

There is one crown per contest, and every ladder is entirely separate: its own king, its own queue, its own lineage. The crown has a reign, a public lineage of predecessors, and a queue of challengers. At genesis it is held by the contest's baseline when it has one, crowned unopposed so that the first real entrant has something to beat; the baseline is marked as such wherever it appears. This contest's baseline: robotensor/vector-base.

Submit, queue, duel

You submit a repository on Hugging Face — not this contest: submissions are weights, and nothing of them runs at its root and the policy code it names — at a pinned revision. It joins the queue. There is one entry per submission key: re-submitting replaces your entry and sends it to the back of the queue, which is what rate limiting looks like here instead of a daily quota.

When your turn comes, your model fights a duel with whoever holds the crown at that moment.

What a duel is

Both sides run the identical unit list. Not similar units, not a comparable sample — the same tasks, the same initial states, the same demonstration for each unit, the same seeds. The list is drawn evenly across the skills, and within a skill evenly across its tasks: at the default launch size that is 160 units per skill, 160 per side.

The challenger runs the whole list first, then the reigning model runs the same list. Each side's result on each skill is a success rate. The score is the mean over the skills.

The crown rule

The crown transfers if and only if

challenger average ≥ reigning average + +3.0 pts

That is the whole rule. The margin is deliberate: an incumbent should not be displaced by noise, so a challenger has to show evidence of improvement, not evidence of parity.

Why copying the reigning model can never win

Submit an exact copy of the policy that currently holds the crown. Both sides run identical units on identical seeds with identical code, so they produce identical outcomes everywhere. The two averages are equal, the difference is zero points, and zero is short of the margin. The crown does not move — however good the copied model is.

No plagiarism detector is needed and none is used. The rule does the work. The same asymmetry is why a challenger cannot win by being merely similar: it has to be measurably better, on the same units, by the margin.

Void units

An infrastructure fault on either side — a simulator crash, a scene that could not be built, a timeout the orchestrator caused — marks the unit void. A void unit is excluded from both sides' scores, so neither policy is penalised for the orchestrator's problem. A duel with more than 20% of its units void is itself void.

Fresh units every time

Duel units are not drawn from a pinned suite. They are derived per duel: a deterministic identifier is computed from the specification version, the contest, and both submissions' keys and revisions; the tasks, scenes and seeds follow from that, and the demonstrations are materialized once per duel and published with the record by content hash.

Three consequences. Every duel is a fresh draw, so there is nothing to overfit across duels. There is no secret test set to leak or to age out. And the draw is reproducible after the fact from published fields, so anyone can reconstruct exactly which units a duel ran.

What is published

Everything, as soon as a duel finishes:

• per-unit outcomes for both sides on every skill, with each unit's scene and the hash of the demonstration both were shown;
• three clips per unit — the demonstration, the reigning model's attempt, the challenger's;
• both sides' per-skill scores and the average, recomputed on the page from the unit list;
• the paired won/lost/tied counts, as a diagnostic;
• the protocol fingerprint the duel was judged under.

Next: the skills.