Vector · The guide
Start here
A King-of-the-Hill competition for one-demonstration in-context imitation: show a policy one demonstration of a task, then measure it skill by skill on an identical list of units against the reigning model.
2 min read
The competition in one paragraph
A policy is shown 1 demonstration of a task — one recording, no language, nothing else — and then has to do the task from a different initial state: the demonstration is the whole prompt, and the rollout starts where it started. Every submitted policy is measured on the same 1 skills — Sixteen tabletop tasks (two Franka arms, level 1) — each as one success rate over its units, and the score is the average over skills. A challenger takes the crown by beating the reigning model's average by +3.0 pts on an identical list of units, and every duel is published in full.
Three words you need
The skills
The competition, read from spec.json
- Contests
- 1
- The contests
- Vector: one demonstration from another scene, level 1
- Demonstrations in the prompt
- 1
- Language input
- none
- Scored skills
- 1
- Score margin
- +3.0 pts
Rendered from spec.json at build time — the same file the orchestrator reads.
Where the crown stands
Vector: one demonstration from another scene, level 1
38.3%average over the skillsReigning model
robotensor/vector-update1
Revision c9e5372 · 0 successful defences
1 duel published · the dashboard
How to read this
Four short pages, in order, and you will know what the competition is and how to enter.
- How the ladder works — duels, the average rule, and who holds the crown at genesis.
- The skills — what each skill asks, and how its units are drawn.
- What you submit — what an entry is, how it is pinned, and what it is asked to do.
- Honest measurement — the replay floor, why a copy of the reigning model cannot win, and what is published so you can check our arithmetic.
What we measure, and what we do not
We measure one-demonstration adaptation, skill by skill: how well a policy uses one demonstration of a task to do it from a different initial state. That is narrower than "how good is this robot policy", and the narrowness is the point.
We do not measure language following — there is no language. We do not measure how good a policy is after training on the task — nothing but the one demonstration, inside one unit's budget of 1,800 s, tells it what to do. And we say plainly whether the demonstration can simply be replayed: no — the actions do not fit a different initial state. Honest measurement says what that lets a score tell you.
Three things that make this different
Every result is published in full. Per-unit outcomes for both sides of every duel, on every skill, with three clips per unit: the demonstration the model was shown, the reigning model's attempt, and the challenger's attempt.
The rule is arithmetic you can redo. The duel page recomputes both sides' scores from the published unit list and shows the comparison line by line. If the page's arithmetic ever disagreed with the published verdict, the page would say so.
Nothing is held back. No private phase, no finalist round, no results withheld. A duel the reigning model wins is published in exactly the same detail as one it loses.
