Horizon · The guide
The axes
An axis is a family of tasks scored as one number, and a submission’s score for the epoch is the mean of the axis scores it was given.
2 min read
What an axis is
An axis is a family of tasks, scored as one number: each task is a success rate over its units, and the axis is the mean over its tasks, so a task with many episodes does not outweigh one with few. That mean is also what hides things, which is why the per-task rates are published beside it: a submission strong on one task and absent on another reads the same as one that is even, and only the rates underneath say which of the two you are reading.
An epoch scores every submission on every axis it ran, and a submission's score for the epoch is the mean of those axis scores — so an axis is a third or a half of a result, never a result on its own. Scores is that rule.
Which axes an epoch scored is in its record, and is on the epoch's page. An axis that went too badly wrong to measure is dropped from that epoch's scoring, with the reason published, so the list is not the same every week.
What each one measures
The competition's axes differ in where the demonstration comes from and where the rollout is run.
| Axis | Demonstration | Scene |
|---|---|---|
| RoboTwin-HumanGen | a generated human video of the task | a scene of its own, as the official protocol does |
| RoboTwin-Simulation | the scripted expert, through the robot's head camera | the demonstration's own scene |
| RoboCasa-Simulation | a generated demonstration, replayed | the demonstration's own scene |
The difference is the point of having more than one. A model that only replays what it was shown does well where the rollout starts in the demonstration's own scene and badly where it does not; a model that generalises from a human video is being asked something else again.
A dropped axis
An axis that went too badly wrong to measure is dropped from the epoch's scoring, and an epoch's page gives the reason. A dropped axis is in nobody's mean: it is not one of the epoch's axes, so losing it costs every submission exactly the same, which is nothing.
What an axis is not
An axis is not a leaderboard of its own. Nothing is decided by an axis score by itself: a submission is ranked on the mean of all of them, so being best on one axis and weak on the others is a middling result, and being even across them all is a good one. That is the whole reason there is more than one axis.