The frozen pool

Built once, for everybody

The pool is built once and frozen, before any model runs, and it is not rebuilt for anyone: every submission in the epoch is evaluated against these exact demonstrations. Each one sees the identical bundles in the identical scenes, with the same policy seed per unit, so two models differ by what they do and not by what they were asked to do.

The secret, and the commitment

The seeds come from a secret the organiser draws when the epoch opens. Its hash is meant to be signed at once, so that the week's scenes cannot be chosen after seeing what was submitted. Either the hash was signed in its own record when the epoch opened, or it appears first in the record that closed the epoch, in the same file as the secret. Only the first is a commitment anyone can hold the organiser to, and which of the two an epoch is, is in the store: an epoch with a record of kind epoch-open was committed to before it saw a submission.

Re-deriving the units

When the epoch closes anyone can re-derive the units from the secret and rebuild the same demonstrations. The record carries the units recipe the derivation takes — each axis's resolved task list, the episode counts, the seed candidates — so re-deriving needs this package and the record, and no benchmark, no config and no fork checkout. epoch derive takes the published record and the secret it reveals, and writes the epoch's units.jsonl back out.

An epoch published before the competition recorded that recipe cannot be re-derived from its record alone, and epoch derive refuses it rather than deriving other units.

What a bundle records

A bundle is more than its demonstration. Each one records the benchmark that built it, its task config's name and sha256, the fork commit that benchmark was checked out at, where the demonstration came from and the camera it was recorded through. That is what makes an epoch describable after the fact: what an axis ran under can be read back out of the bundles the pool holds, not out of anything typed elsewhere.

It is also what ends a score series. A change to a task config's name or sha256, or to the fork commit, means an axis is no longer measuring what it measured the week before, so the epochs either side of the change are not comparable and are shown as separate runs rather than one line. Within a single epoch the same test applies to one axis: bundles that disagree on any of those values are not one score series either.

What the epoch directory holds

The bundles themselves, the per-unit results and the clips are in an epoch directory beside the store, not in the record. Nothing from that directory is shown as the epoch's unless it agrees with the record on the secret commitment, the config's sha256 and the units. From record version 2 the record also signs a digest of each bundle and each result, and every file this site shows is hashed here and held to it. Verification is that check in full.