Glossary

The competition

Axis — a family of tasks scored as one number: each task a success rate over its units, the axis the mean over its tasks. What each one measures.

Unit — one axis, one task and one episode index. It has its own demonstration and its own policy seed, and it is the smallest thing anyone is scored on. Its id is written axis/task/k, and on this site the / is doubled to __ so a URL and a directory read the same.

Bundle — the demonstration a unit holds, built by the benchmark from the first candidate scene the expert could solve. It lives in the epoch directory, not in the record.

Pool — every unit's bundle for one epoch, built once before any model runs and not rebuilt for anyone. Why it is frozen.

Epoch — one week: one pool, one set of submissions, one signed record. Open once epoch open has published its commitment; closed once epoch close has published the scores.

Secret — what an epoch's units are derived from. Drawn at open, published at close.

Commitment — sha256 of the secret, published when the epoch opens so the week's scenes cannot be chosen after seeing what was submitted.

Dry run — a rehearsal of a close, published as a record of its own and marked as one. It is no week the competition ran, and this site lists it apart and never as an epoch's result.

Scoring

Base model — the reference every submission is measured against, evaluated in the epoch under its own plain key, base.

Shortlist — who the screening round put through to the full round. A version 2 record carries it.

Void — a unit that produced no score. A harness void is dropped for every submission on the shortlist, which keeps the comparison paired; a runtime void that outlived its retries counts as that submission's own failure.

Dropped axis — an axis whose voids were over max_void_fraction after one re-run, left out of that epoch's scoring with the reason published.

Overall score — a submission's score for the epoch: the mean of its axis scores over the axes the epoch scored. Scores.

Rank — the submissions in overall-score order, best first, with a tie going to whoever entered first. The base model is not in it: it is the reference, not an entrant.

The store

Store — the directory the competition publishes: index.json and records/<name>.json, each record signed. This site reads one and writes none.

Record — one signed JSON document. An epoch has up to two — the one epoch open published and the one epoch close published — and a submission has one.

Record version — the close record's own version. Version 2 carries the per-task detail and the digests; version 4 is the one that carries an overall score and a rank. An epoch whose record is older than that shows no overall score, because there is none to show.

Key — two things, kept apart. The organiser's ed25519 public key, which records are verified against and which a deployment may pin; and a submission key, the name a submission goes by, <owner>__<name>@<commit>-<hash>, which is also its directory under an epoch's results/ and the name it is shown under everywhere.

Epoch directory — where an epoch's units, bundles and results live, beside the store and unsigned. Nothing in it is shown as the epoch's unless it agrees with the record.

Digest — the sha256 a version 2 record signs for a bundle, a result or a rollout. This site hashes the file it is about to show and holds it to that.