The tasks

Sixteen tasks, scored separately

Every duel plays the same 16 tasks of RoboTwin at level 1: Beat block hammer, Move stapler pad, Place bread skillet, Place burger fries, Place can basket, Place cans plasticbox, Place container plate, Place dual shoes, Place empty cup, Place fan, Place mouse pad, Place object basket, Place object scale, Place object stand, Place phone stand, Place shoe.

Each task is scored on its own, as a success rate over its units, and a model's score is the mean of its task rates. Every task gets 10 units, so none counts for more than another.

One unit

A unit is one task, one scene and one demonstration.

1. The demonstration. The benchmark's expert does the task in one scene, recorded by three cameras (the head and both wrists) with both arms' poses at every step.
2. The scene. The policy is placed in a different scene of the same task, with the objects elsewhere.
3. The rollout. The policy is shown the demonstration once, then drives both arms from the live cameras and poses until the task is done or the unit runs out of steps.

Replaying the demonstration's actions does not work: they belong to the other scene. What the policy is handed, array by array, and what it returns is on The model.

The step limit

A unit ends when the task is done or at its step limit, 2× the expert’s steps in the scene and never more than the task's own limit; a policy that has not finished by then has failed the unit. The ratio comes down as models get faster.

The difficulty level

RoboTwin's randomisation is gated in four levels, and the competition runs at level 1. At level 1 what changes between the demonstration's scene and the policy's is where the task's objects are (position and yaw) and, where a task allows either, which arm does the job; the lighting, the table and the objects themselves stay as they are. Level 2 adds lighting, textures and table height, level 3 other object instances and categories, and level 4 clutter.

What is never evaluated

The scenes and demonstrations of a duel are dealt from a chain block's hash, from seeds above the benchmark's public training and evaluation ranges, so a policy cannot have seen them.