An epoch, end to end

An epoch is one week. It opens on a secret, it derives its units from that secret, it builds one demonstration for each of them, and it closes by revealing the secret and publishing everything it scored. Nothing in the middle changes what was fixed at the start.

Open

An epoch is sealed before it runs: the organiser draws a secret, writes sha256 of it — the commitment — into the epoch directory, and freezes the register at that instant, so what the week will be scored on and who it will be scored for are both fixed before either is known.

An epoch that signs a record then — the commitment, the cutoff and the entries the register held at it — is shown here as open from that moment, under that record. An epoch whose store holds no such record appears only once it closes, and its commitment is read from the record that closed it.

Derive

Every unit follows from the secret: unit_seed = H(secret ‖ axis ‖ task ‖ k), and from it the scene candidates and the policy seed.

Build the pool

Each unit gets one demonstration, built by the benchmark from the first candidate scene the expert can solve.

Verify the pool

Every bundle is rebuilt and checked against its fingerprint before anything is evaluated.

Screen

The screening round runs one unit per task, so a model that cannot run at all costs one episode, not a week. Who made it through is the epoch's shortlist, and a version 2 record carries it — with the submitted_at each entry was stamped with, which is what breaks a tie when two submissions come out on the same score.

Full evaluation

The full round runs the whole pool: the base model, the leader and the top-K are evaluated on every unit of it, not the one per task the screen ran.

Score

Every submission that ran is scored on every axis, and its score for the epoch is the mean of those axis scores. The rank is by that score, highest first; a tie goes to whoever entered first. Nothing is paid. Scores is the whole rule, and how the rates underneath it are formed.

Close

The secret is revealed, and the epoch's manifest with its commitment, the secret, the pool summary and the scores are published as one signed record. From version 2 the record also carries what the scores were computed from — the shortlist, the per-task rates, the per-axis void counts and the config the epoch was run under — and a digest of every bundle and every result, though not the bundles and results themselves.

A dry run

A dry run is a rehearsal of the close, signed as one: the competition publishes it under the epoch's name with -dry-run after it, marked as a dry run at the top of the record. It is a rehearsal and not a week the competition ran, so it is in no score series and is never the latest epoch. Its numbers are shown on a page of its own because they were published, and this site lists it apart and never as an epoch's result.

Score series

Epochs are grouped into score series. A run ends wherever the config sha256, a benchmark pin or an axis's task config changes, because the numbers either side of that are not measured on the same thing — and a line drawn through the two would be a line through two different measurements. The frozen pool is what a bundle records, and so what a change to one is.

A series is not a rule the scorer applies: no submission is ranked across one. It says only which epochs' numbers may be read as a sequence and which may not.

Two records, one epoch

epoch open publishes one record and epoch close publishes another, and this site shows them as one epoch: open until the close record is there, and closed after it. The close record's fields win. What the open one adds is that the commitment was signed before any submission was seen — and the two are compared before they are merged, so a close record that reveals a secret for another commitment, or under another config, than the open record signed makes the epoch unverified even though both signatures hold.