Questions

Entering

Can I submit code? A submission is code: a Hugging Face repository at a commit sha, holding not this contest: submissions are weights, and nothing of them runs and the policy class it names. It runs only inside its own sandbox, with network not this contest: submissions are weights, and nothing of them runs. Both sides of a duel still receive exactly the same prompt, because the orchestrator materializes it once and hands it to each.

Is there a model or architecture check? No. There is no converter, no template and no configuration to match. A submission is held to the manifest, the policy protocol and the sandbox: if it serves, answers hello and returns actions of the contest's shape in time, it is queued. What you submit lists all three.

My weights are a .ckpt file, a pickle, or something else unusual. Is that allowed? Yes. The orchestrator never reads your weights; your own code loads them, inside the sandbox. Name the file in the manifest's kwargs or open it from the repository root, and keep the whole repository under 8 GiB. Nothing outside it — no download at evaluation time — is reachable.

Can I fine-tune at evaluation time? Nothing stops your code from doing what it likes with the one demonstration, a gradient step included: the orchestrator does not inspect what happens inside the sandbox, only what comes out of it and when. What bounds it is the budget — 60 s per action, 1,800 s per unit, not this contest: submissions are weights, and nothing of them runs GPU — and what is reachable: the demonstration and your own observations, nothing else. No network, no store, no other side.

Can my policy compile code at run time? Yes, into not this contest: submissions are weights, and nothing of them runs: it is writable and executable, a tmpfs of not this contest: submissions are weights, and nothing of them runs that starts empty and counts against the sandbox's not this contest: submissions are weights, and nothing of them runs of memory. HOME and the compile caches — Triton's, TorchInductor's, torch.utils.cpp_extension's — point into it, so torch.compile, Triton, cffi and C, C++ or CUDA extensions need no configuration. The base image has gcc, g++ and make, Python's headers and CUDA's nvcc. It does not have ninja, which torch.utils.cpp_extension refuses to build without: list it in your requirements. The rest of the root filesystem is still read-only, so nothing compiles anywhere else, and the network is still not this contest: submissions are weights, and nothing of them runs, so a build that downloads anything fails. Compiling takes your time: in the constructor it counts against 600 s to answer hello, on the first act against 60 s.

Do I have to submit every skill? Yes. A duel runs every skill of the contest — Sixteen tabletop tasks (two Franka arms, level 1) — with the same policy, and the score is the average over them. There is no per-skill entry and no way to sit one out; a policy that cannot do a skill still runs it, and scores what it scores.

How often can I resubmit? As often as you like, but there is one entry per submission key: a new commit is a new key, and re-submitting sends you to the back of the queue. That is the rate limit.

Can I submit a branch name as the revision? No. A revision is a commit sha, in full: an abbreviation is refused too, because the key is a hash of repo@revision and a shorter spelling would be a different key for the same code. A mutable name would let the code that holds a crown quietly stop being the code that was scored.

The rules

How does the crown move? The challenger's average over the skills must be at least the reigning model's average plus +3.0 pts, on an identical unit list. Nothing else decides it.

Is anything below a skill scored separately? No. Every unit's task and initial state — on RoboTwin the scene seed and the embodiment — are published, so you can see where a model struggles, but each skill is one success rate over all of its units.

What are the won/lost/tied counts on a duel page? A diagnostic: on which units the two models differed. They do not decide the crown, and the page says so beside them.

Why can't a copy of the reigning model win? An exact copy of the reigning repository runs identical code and weights on identical units with identical seeds, so the two averages are equal and the difference is zero points — short of the margin.

What happens if a unit crashes? If the fault was the orchestrator's — a scene that could not be built, a unit past its wall clock — the unit is void and excluded from both scores. How a unit is scored when your policy raises or runs past its per-action budget is not yet stated by the contract; this answer will say once the orchestrator's duel runner does.

Is there language? No. none. The demonstration is the whole prompt.

How many demonstrations? 1.

Evaluation

Can I just replay the demonstration? no — the actions do not fit a different initial state. The rollout is scored from a different initial state — honest measurement says what that lets a score tell you.

Which tasks are hidden? None of them — the tasks are RoboTwin's, and the benchmark is open source. What is not fixed in advance is which units a duel draws: the tasks, the scenes and the seeds of a duel are derived from its identifier, and the demonstrations are produced by the benchmark's own expert at evaluation time and published with the record by hash.

Can I see the evaluation suite? There isn't one to see. Every duel derives its own units from a deterministic identifier, so there is no fixed test set to leak or age out — and the draw is reconstructible after the fact from published fields.

What is published about my submission, and when? The repository and revision, both sides' per-unit outcomes on every skill, three clips per unit, the per-skill scores and the average, and the diagnostic counts — all of it, as soon as the duel finishes, win or lose.

Who holds the crown at the start? The contest's baseline, when it has one — robotensor/vector-base. A baseline is crowned unopposed at genesis and marked Baseline wherever it appears; with none, the first accepted entrant is crowned by genesis, and every entrant after it has to win a duel.

The competition

Is there a private phase, a finalist round, or withheld results? No, no and no.

What would count as a real result here? A submission that beats the reigning policy's average by the margin on a fresh draw across every skill, on the same units, with every per-unit outcome and every clip published for anyone to check.