A round, end to end

A round is one week of entries. It opens on a secret, it derives its units from that secret, it builds one demonstration for each of them, and it closes by revealing the secret and publishing everything it scored. Nothing in the middle changes what was fixed at the start.

Rounds run back to back: the moment one round stops taking entries the next one opens, so every entry falls in exactly one round's window — from the instant it opens up to, but not including, the instant it closes. The round that has just closed is then scored while the next one takes entries, and each model's result is published as soon as that model's scoring finishes. A round has no name: it is Round 1, Round 2 and so on, and its id is only the address of its page.

Open

A round is sealed before it runs: the organiser draws a secret, writes sha256 of it — the commitment — into the round directory, and freezes the register at that instant, so what the week will be scored on and who it will be scored for are both fixed before either is known.

A round that signs a record then — the commitment, the cutoff and the entries the register held at it — is shown here as open from that moment, under that record. A round whose store holds no such record appears only once it closes, and its commitment is read from the record that closed it.

Derive

Every unit follows from the secret: unit_seed = H(secret ‖ axis ‖ task ‖ k), and from it the scene candidates and the policy seed.

Build the pool

Each unit gets one demonstration, built by the benchmark from the first candidate scene the expert can solve.

Verify the pool

Every bundle is rebuilt and checked against its fingerprint before anything is evaluated.

Screen

The screening pass runs one unit per task, so a model that cannot run at all costs one episode, not a week. Who made it through is the round's shortlist, and a version 2 record carries it — with the submitted_at each entry was stamped with, which is what breaks a tie when two submissions come out on the same score.

Full evaluation

The full evaluation runs the whole pool: the top-K of the screen, and the previous round's winner when it entered again, are evaluated on every unit of it, not the one per task the screen ran.

Live results

Models are scored one after another, and each one's result is published the moment its scoring finishes, as a signed record of its own: its score on each axis over the units it ran, and the mean of those. The dashboard shows them as they arrive, in the order they finished. They are provisional — a unit found void on a later model is dropped for everyone — so the round's close record, published when every model is scored, is the result.

Score

Every submission that ran is scored on every axis, and its score for the round is the mean of those axis scores. The rank is by that score, highest first; a tie goes to whoever entered first. Nothing is paid. Scores is the whole rule, and how the rates underneath it are formed.

Close

The secret is revealed, and the round's manifest with its commitment, the secret, the pool summary and the scores are published as one signed record. From version 2 the record also carries what the scores were computed from — the shortlist, the per-task rates, the per-axis void counts and the config the round was run under — and a digest of every bundle and every result, though not the bundles and results themselves.

A dry run

A dry run is a rehearsal of the close, signed as one: the competition publishes it under the round's name with -dry-run after it, marked as a dry run at the top of the record. It is a rehearsal and not a week the competition ran, so it is in no score series and is never the latest round. Its numbers are shown on a page of its own because they were published, and this site lists it apart and never as a round's result.

Score series

Rounds are grouped into score series. A run ends wherever the config sha256, a benchmark pin or an axis's task config changes, because the numbers either side of that are not measured on the same thing — and a line drawn through the two would be a line through two different measurements. The frozen pool is what a bundle records, and so what a change to one is.

A series is not a rule the scorer applies: no submission is ranked across one. It says only which rounds' numbers may be read as a sequence and which may not.

The king

The winner of the newest closed round — the first submission in the rank its record published — is the king, and holds the crown until a round crowns someone else. Nothing carries over otherwise: a round scores only the entries made inside its own window.

Two records, one round

round open publishes one record and round close publishes another, and this site shows them as one round: open until the close record is there, and closed after it. The close record's fields win. What the open one adds is that the commitment was signed before any submission was seen — and the two are compared before they are merged, so a close record that reveals a secret for another commitment, or under another config, than the open record signed makes the round unverified even though both signatures hold.