Sundae Robotics 08: Skillful Manipulation, Simulation & Precise Assembly

🤖🍨 Grab an ice cream sundae and join Sundae Robotics, a private, invitation-only Sunday series that brings together robotics researchers, entrepreneurs, and developers at the forefront of physical intelligence. Sundae Robotics 07 Skillful Manipulation, Simulation & Sim-to-Real Learning Featured Talk: Scaling skillful manipulation with simulation — From general tool use to precise assembly. Scaling skillful manipulation with simulation: From general tool use to precise assembly. Opening Talk: Tyler Lum, PhD Candidate in his 5th year of Computer Science at Stanford University · The Movement Lab · Interactive Perception and Robot Learning Lab. Skillful robotic hands have the potential to perform a variety of manipulation tasks, but learning policies that are both general and precise remains a challenge. In this talk, Tyler will present two projects that explore how large-scale reinforcement learning in simulations can help close this gap. First, Tyler will discuss SimToolReal, which learns skillful tool use policies in simulations that generalize to unknown tools and tasks and transfer to the real world in a zero-shot manner. Next, he will introduce Play2Perfect, which explores how broad, task-agnostic pre-training games can provide reusable manipulation skills that can be efficiently fine-tuned for precise, contact-rich assembly. Together, these projects explore how simulation can be used to learn broadly capable skillful behaviors while enabling the precision and specialization required for challenging tasks in the real world. Tyler Lum is a 5th year PhD candidate in Computer Science at Stanford University, studying artificial intelligence and robotics. He is advised by Prof. C. Karen Liu in The Movement Lab (TML) and Prof. Jeannette Bohg in the Interactive Perception and Robot Learning (IPRL) Lab, and is supported by an NSERC Postgraduate Scholarship (PGS-D). Tyler's research focuses on building robots that can move and act gracefully and efficiently in dynamic real-world environments. He develops methods for integrating learning-based perception and control and investigates the inductive biases that can improve the efficiency and reliability of these systems. His recent work has a strong emphasis on skillful manipulation and sim-to-real robot learning. His projects include Play2Perfect, which investigates skillful game pre-training for precise assembly; SimToolReal, which develops object-centric policies for zero-shot skillful tool manipulation; and Human2Sim2Robot, which explores sim-to-real reinforcement learning from a single human demonstration. Before his PhD, Tyler studied Engineering Physics at the University of British Columbia (UBC), where he graduated as a Wesbrook Scholar. He worked with Prof. Michiel van de Panne on reinforcement learning and motion planning for quadrupedal robots and with Prof. Purang Abolmaesumi on deep learning for medical image analysis. Tyler also worked as a research intern in robotics at RAI and as a robotics research and development intern at NVIDIA, where he worked on reinforcement learning for humanoid locomotion and skillful robot control. His broader research interests include intelligent robots, graceful movement, interactive perception, dexterous bimanual manipulation, reasoning under uncertainty, long-term planning, creative problem solving, common sense, and exploration strategies. Tyler's long-term goal is to create robots that can reason through uncertainties, continuously learn from their environment, adapt to new challenges, and become reliable enough to provide immense value to humans in daily life. Topics • Scaling skillful manipulation with large-scale reinforcement learning in simulation. • Learning policies that combine broad generalization with precise control. • Skillful tool usage across unknown tools and tasks. • SimToolReal and zero-shot transfer from simulation to the real world. • Object-centric policies for skillful manipulation. • Sim-to-real reinforcement learning for multi-fingered robotic hands. • Broad, task-agnostic pre-training for games. • Learning reusable manipulation skills through simulated interaction. • Play2Perfect and efficient fine-tuning afterwards. • Precise, contact-rich robotic assembly. • Bridging general manipulation and specialized precision. • Learning-based perception and control for dynamic real-world environments. • Inductive biases for more efficient and reliable robot learning. • Scaling robot learning through simulated experiences. • Building skillful systems that are both general and precise. Open Discussion + Q&A • How can skillful policies remain general across tools and tasks without sacrificing precision? • Which aspects of skillful manipulation are the hardest to transfer from simulation to the real world? • How much simulation diversity is needed for robust zero-shot transfers? • What representations or inductive biases enable generalization across unknown tools? • Can a single policy learn reusable concepts of grasping, tool usage, and contact-rich interaction? • What does a task-agnostic game teach a robot that task-specific demonstrations do not? • How should broad manipulation skills be fine-tuned for tasks that require extremely precise control? • What fundamentally makes contact-rich assembly different from general object manipulation? • How much real data is still necessary after large-scale simulation pre-training? • Can game pre-training become a general foundation for skillful manipulation? • When does reinforcement learning in simulation outperform imitation learning from real demonstrations? • How should simulation environments be designed to promote transferable rather than simulator-specific behaviors? • Can the same pre-trained policy support both open tool usage and highly constrained assembly? • How should learning-based perception and control be integrated for reliable skillful manipulation? • What would be needed to scale simulated skillful learning to achieve truly general robotic hands?