From：Internet Info Agency 2026-09-04 11:30:00
Recently, DaXiao Robotics, in collaboration with the Multimedia Laboratory at The Chinese University of Hong Kong, Nanyang Technological University, Shanghai Jiao Tong University, and other institutions, has open-sourced ACE-Ego-Hand—a 3D two-hand shape reconstruction model tailored for first-person perspective videos—alongside approximately 5,000 hours of training data. This model pioneers the use of a video generation model as an encoder to enable end-to-end training, requiring only a single pass through an entire video clip to output hand poses, shapes, and 3D positions. Unlike existing hand models that rely on frame-by-frame detection, window-based temporal regression, or multi-step diffusion sampling, ACE-Ego-Hand repurposes a video diffusion model into a geometric encoder. It recovers continuous 3D hand trajectories via a single forward pass, reducing inference latency and error accumulation while leveraging contextual information from the full video segment to reconstruct hand trajectories even when hands are occluded or temporarily out of view. Traditional video diffusion models typically require multi-step denoising sampling to generate intermediate videos or pixel-level outputs before estimating hand poses—an approach that incurs high computational costs and propagates errors. In contrast, ACE-Ego-Hand eliminates the multi-step denoising process entirely. Instead of using the video diffusion model for pixel-space generation, it treats the model as a deterministic geometric encoder. After compressing the input video via a VAE, the model performs a single forward pass over clean latent representations, extracting spatiotemporal features from intermediate layers of a video diffusion Transformer and feeding them into a bidirectional spatiotemporal decoder to produce end-to-end outputs of hand pose, shape, visibility, and 3D spatial location. On a single A100 GPU, ACE-Ego-Hand achieves approximately 63.1 fps—about 33 times more efficient than ViDiHand under its high-accuracy configuration. The model’s ability to infer and recover hand motion trajectories after occlusion or temporary disappearance stems from its bidirectional spatiotemporal reasoning mechanism. Unlike causal models that predict current results solely based on past frames, ACE-Ego-Hand jointly analyzes the entire video without applying causal masking. When processing any given time step, it simultaneously leverages information both before the hand disappears and after it reappears, enabling accurate inference of trajectories during invisible intervals while preserving hand identity and trajectory continuity. Moreover, during inference, ACE-Ego-Hand does not require explicit camera intrinsics as input. It recovers metric 3D hand positions directly in the camera coordinate system using only head-mounted first-person videos and natively supports ultra-wide-angle fisheye inputs. To handle radial distortion inherent in fisheye imagery, the model bypasses the need for prior camera calibration or image undistortion. Instead, it directly predicts observation rays from distorted pixels to 3D space, thereby recovering metric 3D positions. ACE-Ego-Hand is applicable to scenarios such as production-line operation data collection, robot teaching, manual assembly process analysis, dexterous hand motion retargeting, and imitation learning dataset construction. Particularly in real-world environments where hands are frequently occluded by workpieces, temporarily exit the field of view, or experience rapid camera motion, the model significantly reduces trajectory interruptions and hand loss, providing a robust data foundation for robots to learn complete action sequences from human demonstration videos.
