Ziyuan Robotics released and open-sourced the HOST framework today (Human-to-robot One-Shot Skill AcquisiTion), claiming that it enables robots to learn new skills simply by observing a human video lasting several dozen seconds, while retaining the skills they have already mastered.
HOST adopts a groundbreaking approach: instead of translating human actions into robot actions and attempting to imitate them, it first has the robot imagine the result of executing the action, then breaks that result down into the actions that need to be performed. The results of this new approach are encouraging: the success rate for a robot learning a skill from a 29-second human video reaches as high as 62%, surpassing the zero-shot baseline of 45%. Compared with the Pi-0.5 + fine-tuning approach, HOST reduces the required data by 50 times and increases the speed of skill learning by 500 times.
HOST’s approach is inspired by the “observational learning” theory in cognitive science: when humans face unfamiliar tasks, they do not need to learn through prolonged practice or exhaustive trial and error. Instead, they use prior abilities to simulate the expected effects of actions in their brains before performing the corresponding actions. Inspired by this, Ziyuan researchers changed HOST’s learning approach from a “training-time fine-tuning loop” to “inference-time skill acquisition,” enabling robots to learn new skills by observing humans perform tasks.
The first problem the robot must solve is: how can it align its own task progress with the progress shown in the video?
For example, after 10 seconds of performing the same task, the person in the video may have finished chopping vegetables and started stir-frying, while the robot is still chopping vegetables. If alignment is based only on time, the robot will lose track of task progress.
HOST addresses this with “task-progress alignment.” It converts every video frame and every step of the robot’s operation into vectors in the same vector space, then uses a “dynamic time warping” algorithm to automatically align “the Xth frame of the human video with the Yth step of the robot’s operation.” As a result, whenever the robot performs a given step, it always sees the frame aligned with the task progress.
With this method, HOST reduces the time error in task-progress alignment by an order of magnitude, enabling precise synchronization between the robot’s execution and the video demonstration.
The core of HOST is a cascaded policy model with visual prediction capabilities. It uses a dual-expert MoT architecture, like two clearly divided brains: the “visual brain” (the video expert) specializes in processing video frames, locating task progress, and predicting future scenes; the “action brain” (the action expert) is responsible for converting the predicted scenes into the actions that need to be performed and controlling their execution.
When training the model, the Ziyuan team divided the process into two stages: “same-embodiment pretraining” and “human-robot video training.” During same-embodiment pretraining, the robot learns from videos of itself performing tasks to predict the state after completing a task and generate the corresponding actions. During human-robot video training, the robot further learns from human demonstration videos, converting the process of a human performing a task into the task result from the robot’s perspective, then reasoning from the target result to determine the actions needed to complete the task, thereby transferring skills from human demonstrations to the robot.
At present, HOST’s research paper and code have been fully open-sourced. In the future, household robots may be able to learn new skills through intuitive human demonstrations, without the need for specialized data collection and training.
References
Project homepage
Paper link
GitHub link
Hugging Face link