When a robot uses a drill, the motion of the drill head relative to the target matters more for control than the color of the work surface. In Compact-WAM, we use demonstrations to teach a policy which parts of the scene matter for the task and how those parts move in three dimensions over time.
From a demonstration to a training target
A reasoning model identifies the objects involved and breaks the demonstration into subtasks. Vision models then mark the relevant parts of each frame and track points on them in 3D. These annotations let us train a smaller model to predict where the hand and objects should end up after the next subtask. That predicted arrangement helps it choose the next movement.
Testing on a physical robot
Compact-WAM succeeded in 90% of drill trials and 85% of trials that required removing objects from a container in sequence. Across the two tasks, its average success rate was 87.5%, compared with 60% for the strongest baseline. In simulation, it reached 98.6% on LIBERO and averaged 86.0% across seven categories of perturbations in LIBERO-Plus.
