Skip to content

Under maintenance. Some results may be incomplete or inaccurate.

Robotensor
All pages21

Vector · The guide

Start here

A King-of-the-Hill competition for one-demonstration in-context imitation: show a policy one demonstration of a task, then measure it skill by skill on an identical list of units against the reigning model.

2 min read

The competition in one paragraph

A policy is shown 1 demonstration of a task — one recording, no language, nothing else — and then has to do the task from a different initial state: the demonstration is the whole prompt, and the rollout starts where it started. Every submitted policy is measured on the same 1 skills — Sixteen tabletop tasks (two Franka arms, level 1) — each as one success rate over its units, and the score is the average over skills. A challenger takes the crown by beating the reigning model's average by +3.0 pts on an identical list of units, and every duel is published in full.

Three words you need

The skills

Vector level 1SHOWNframes, actions, proprioceptionSCORED FROMa different initial stateREPLAYING THE DEMONSTRATIONthe actions do not fit
One demonstration and no language. The policy is shown the whole demonstration and scored in the scene it was recorded in, so replaying its actions can solve a unit; the contract accepts that, and a replay policy winning is not a bug.
Sixteen tabletop tasks (two Franka arms, level 1)one demonstration;scored from another initial staterobotwin · vector_level1
Every duel draws the same number of units on each skill for both sides, spread evenly over the skill's tasks. Each skill is scored as one success rate; the crown rule reads their average. Nothing below a skill is scored on its own.

The competition, read from spec.json

Contests
1
The contests
Vector: one demonstration from another scene, level 1
Demonstrations in the prompt
1
Language input
none
Scored skills
1
Score margin
+3.0 pts

Rendered from spec.json at build time — the same file the orchestrator reads.

Where the crown stands

Vector: one demonstration from another scene, level 1

Vector: one demonstration from another scene, level 1

38.3%average over the skills

Reigning model

robotensor/vector-update1

Revision c9e5372 · 0 successful defences

1 duel published · the dashboard

How to read this

Four short pages, in order, and you will know what the competition is and how to enter.

  1. How the ladder works — duels, the average rule, and who holds the crown at genesis.
  2. The skills — what each skill asks, and how its units are drawn.
  3. What you submit — what an entry is, how it is pinned, and what it is asked to do.
  4. Honest measurement — the replay floor, why a copy of the reigning model cannot win, and what is published so you can check our arithmetic.

What we measure, and what we do not

We measure one-demonstration adaptation, skill by skill: how well a policy uses one demonstration of a task to do it from a different initial state. That is narrower than "how good is this robot policy", and the narrowness is the point.

We do not measure language following — there is no language. We do not measure how good a policy is after training on the task — nothing but the one demonstration, inside one unit's budget of 1,800 s, tells it what to do. And we say plainly whether the demonstration can simply be replayed: no — the actions do not fit a different initial state. Honest measurement says what that lets a score tell you.

Three things that make this different

Every result is published in full. Per-unit outcomes for both sides of every duel, on every skill, with three clips per unit: the demonstration the model was shown, the reigning model's attempt, and the challenger's attempt.

The rule is arithmetic you can redo. The duel page recomputes both sides' scores from the published unit list and shows the comparison line by line. If the page's arithmetic ever disagreed with the published verdict, the page would say so.

Nothing is held back. No private phase, no finalist round, no results withheld. A duel the reigning model wins is published in exactly the same detail as one it loses.