Ep#80: LATENT: Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data

In this talk

1. 1:08

   Why tennis is a hard test

   Lu and Lian frame tennis as a test of whether a humanoid can combine learned movement with fast perception and control. The robot must move across the court, react to incoming balls, and strike accurately, rather than replaying a fixed sequence.

2. 3:32

   Learning from imperfect motion

   The team used human motion clips recorded in an area over 17 times smaller than a tennis court. The data lack reliable wrist detail and full-court position context, so the researchers trained motion trackers and compressed their skills into a conditional VAE latent space. During policy learning, the robot adjusts its right wrist directly while the rest of its body draws on those learned skills.

3. 12:10

   A policy guided by a motion prior

   In simulation, the robot learns to return eight balls in a row toward target areas, with randomized ball positions and velocities. Its policy combines latent actions for 26 whole-body controls with three direct right-wrist controls. A motion prior supplies an initial skill choice, while a latent action barrier limits exploration to a two-sigma range around the prior, helping avoid awkward and potentially damaging movements.

4. 23:13

   Transfer and evaluation

   For real-world deployment, the team used domain randomization, noisy and delayed observations, and motion capture markers on the racket to correct its estimated position. In simulations, results averaged over more than 10,000 trials favored LATENT over the tested baselines; the team also evaluated 20 human-robot rallies, counting a return as successful when it landed within the opponent’s court. They described removing motion capture and extending the system toward competitive or cooperative play as future work.

5. 32:47

   Where the approach could go next

   The researchers said adapting the system to pickleball may be relatively straightforward with suitable racket contact modeling, while badminton poses harder aerodynamic and collision challenges. They see human demonstrations as useful priors for reinforcement learning in humanoid control, where unrestricted exploration can produce unsafe reward-seeking behavior. They also discussed applying the general pipeline beyond tennis and using lighter perception systems in place of motion capture.