ECE 290 Seminar: “Learning Dexterous Manipulation: From Simulation to Human Demonstrations”

Presented by: Joe Dong, Co-Founder & CTO @ Chestnut Robotics

Description: “Learning dexterous manipulation requires robot hardware, training environments, and data to work together. This talk explores two complementary approaches to making robotic hands learnable: training in simulation and learning from human demonstrations.
First, I will present Aero Hand Open, an open-source tendon-driven hand developed with a simulation and reinforcement learning stack. I will discuss its design, transmission modeling in MuJoCo, the actuation mapping between simulation and hardware, and experimental validation, including an in-hand cube rotation policy transferred to the physical hand without policy fine-tuning.

Second, I will introduce the newer Aero Hand and Aero UMI, a robotic hand and wearable demonstration interface developed as a matched pair. I will discuss how co-designing the hand and capture interface connects human demonstrations to robot actions, with particular attention to motion, contact sensing, calibration, and data alignment. Through experimental results and demonstrations, the talk will examine the engineering tradeoffs and open research questions in connecting robot embodiment with the simulation and data used to train it.”

Bio: Xu (Joe) Dong is cofounder and CTO of Chestnut Robotics, where he develops robotic hands and the data, simulation, and learning infrastructure for physical AI. He previously worked on autonomous driving at Waymo and XPeng Motors, including learning-based motion planning and production end-to-end driving models. He holds a Ph.D. in Biomedical Engineering and an M.S. in Computer Science from Virginia Tech, and a B.S. in Physics from the University of Science and Technology of China. Personal Website: http://joedong.ai/

Hosted by: Professor Zhicong Yu, Electrical & Computer Engineering Department

When: Monday, October 26th from 10:40 to 11:45 AM PST

Location: Classroom E2-194