# OmniHOI turns human hand videos into robot motions

> The research pipeline converts single-camera videos of people handling objects into robot-hand trajectories. Its authors report higher success rates than earlier transfer methods.

Oossa · 2026-10-09 · https://oossa.com/en/omnihoi-turns-human-hand-videos-into-robot-motions

Event date: 2026-10-07

A research team has introduced OmniHOI, a system that converts ordinary one-camera videos of people manipulating objects into motion trajectories for dexterous robot hands. The work addresses a practical problem: a video shows only one view of a hand, and directly translating that movement to a robot can produce motions that fail to keep hold of an object.

The authors report that OmniHOI succeeded on 53% of 60 video clips, compared with 28% for the strongest prior video-to-robot pipeline they tested. In a separate set of tests, they transferred 150 motion-capture trajectories to each of five robot hands, which had between 6 and 22 degrees of freedom—a measure of how many ways a hand can move. Success rates ranged from 39% to 89%, while earlier transfer methods reached no more than 31%.

## How the pipeline works

OmniHOI has three stages. It first reconstructs hand and object motion using clues in the video. It then adjusts the motion to fit the robot hand’s contact with the object, before refining it in a physics simulation so the resulting movement better accounts for dynamics.

The paper says the resulting trajectories also ran on a real robot with two arms across a range of tasks. The abstract does not name the robot or give success rates for those real-robot runs, so the reported percentages above should not be read as real-world deployment results. The work is a research paper; the abstract does not specify whether the system or code is publicly available.

## What the results show

The results suggest that a single-camera human video can provide useful demonstrations for robot hands, without requiring clean motion-capture data or task-specific reinforcement-learning training. The authors’ tests are reported in the paper abstract; it does not describe independent validation.

## The facts

- OmniHOI was submitted to arXiv on October 7, 2026.
- In tests on 60 monocular video clips, the authors report 53% success, versus 28% for the best prior video-to-robot pipeline.
- For 150 motion-capture trajectories transferred to each of five hands, reported success ranged from 39% to 89%; prior transfer methods reached at most 31%.
- The five tested robot hands had between 6 and 22 degrees of freedom.

## Why it matters

For robotics teams, the reported results point to a possible way to turn videos of people handling objects into robot-hand demonstrations, rather than relying on motion-capture trajectories. But the abstract does not establish how well the system works beyond the reported tests or whether other teams can access the tools.

## Sources & references

1. [OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video](https://arxiv.org/abs/2610.10855) – arXiv cs.RO, 2026-10-09

Last updated: 2026-10-09