A reasoning agent proposes corrections from failed robot trials. A bar chart compares success rate and throughput across three training methods.

A failed trial tells a robot that its policy needs work. The reward alone says little about which movement to change. We gave a reasoning agent the recorded trajectory and asked it to find a moment when a different action could have helped.

Using a correction during training

While the robot policy collects more experience, the agent inspects earlier failures and proposes replacement actions for particular states. We train the policy to imitate those suggestions alongside its usual reinforcement learning objective. The suggestions guide exploration, and the rewards from subsequent trials let the policy improve on an imperfect correction.

We tested the method on ten simulated LIBERO manipulation tasks with a budget of 400 episodes per task. The agent-guided policy achieved 6.9 times the normalized success throughput of the strongest baseline trained with reinforcement learning alone. The difference between advice and direct imitation was clearest with a weaker agent. Learning from both rewards and that agent’s proposals yielded a 58% average success rate, compared with 26% from imitating the same proposals alone.