Robotics / world models / in progress
Teaching a robot arm to notice its own mistakes
I'm recording a real SO-101 arm failing on purpose, to test whether a video world model catches failed grasps better than a vision-language model does.
- Hardware
- SO-101 leader and follower (LeRobot)
- Models
- V-JEPA 2 probe, Qwen2.5-VL judge
- Status
- Recording, 15 of 100 episodes
Robots fail quietly. A grasp closes on nothing, a block slips out during the lift, something knocks the arm mid-reach. A monitor that notices in time lets the robot retry or ask for help. I want to know whether a video world model already sees the difference: whether a small probe on V-JEPA 2's embeddings can tell a failed pick from a good one, and how that compares with asking a vision-language model, Qwen2.5-VL, to judge the same clip.
The dataset
- 100 teleoperated pick-and-place episodes of one block, recorded at 30 frames a second from a top camera and a wrist camera.
- 25 of them fail on purpose, in three ways: a missed grasp, a slip during the lift, and a nudge mid-reach.
- The evaluation is fixed before any results: leave-one-cell-out folds by where the block sits on a grid, so every test is on a position the probe hasn't seen.
So far
- 15episodes recorded on the real arm on October 29, 2026: 5 pilot runs (2 empty) and 10 of the first batch of 50
- 0.5 sfor V-JEPA 2 ViT-L to embed an episode, at 0.9 GiB peak on an RTX 5080
- 1.7 sper call for Qwen2.5-VL-7B in 4-bit, which passed its self-test on synthetic clips
- 3BQwen2.5-VL-3B failed the same self-test
Not done yet: the other 85 episodes, the labels, the embeddings and the judge's outputs. There are no accuracy numbers, and there won't be until all 100 episodes are in. They'll go here, per fold.
The stack around the arm
Alongside the dataset I'm building armlab, software for one SO-101: inverse kinematics, a 3D voxel map from several cameras, a motion planner, and a web gateway with a hardware stop. Against a MuJoCo twin and simulated servos, the inverse kinematics round trip is within 0.0015 mm over 10,000 poses, the planner's paths had no collisions across 400 start and goal pairs (median planning time 9 ms), and the stop command lands in 16 to 98 ms against a 100 ms budget. None of armlab has run on the real arm yet; the recordings above go through LeRobot directly.