ASC: Adaptive Skill Coordination for Robotic Mobile Manipulation

• Published in: IEEE robotics and automation letters Vol. 9; no. 1; pp. 1 - 8
• Main Authors: Yokoyama, Naoki, Clegg, Alex, Truong, Joanne, Undersander, Eric, Yang, Tsung-Yen, Arnaud, Sergio, Ha, Sehoon, Batra, Dhruv, Rai, Akshara
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01-01-2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: AI-Enabled Robotics; Barriers; Coordination; Courtyards; Deep Learning Methods; Grippers; Indoor environments; Layouts; Navigation; Reinforcement Learning; Robot kinematics; Robot sensing systems; Robots; Sensors; Skills; Task analysis
• ISSN: 2377-3766; 2377-3766
• DOI: 10.1109/LRA.2023.3336109

We present Adaptive Skill Coordination (ASC) - an approach for accomplishing long-horizon tasks like mobile pick-and-place ( i.e. , navigating to an object, picking it, navigating to another location, and placing it). ASC consists of three components - (1) a library of basic visuomotor skills (navigation, pick, place), (2) a skill coordination policy that chooses which skill to use when, and (3) a corrective policy that adapts pre-trained skills in out-of-distribution states. All components of ASC rely only on onboard visual and proprioceptive sensing, without requiring detailed maps with obstacle layouts or precise object locations, easing real-world deployment. We train ASC in simulated indoor environments, and deploy it zero-shot (without any real-world experience or fine-tuning) on the Boston Dynamics Spot robot in eight novel real-world environments (one apartment, one lab, two microkitchens, two lounges, one office space, one outdoor courtyard). In rigorous quantitative comparisons in two environments, ASC achieves near-perfect performance (59/60 episodes, or 98%), while sequentially executing skills succeeds in only 44/60 (73%) episodes. Extensive perturbation experiments show that ASC is robust to hand-off errors, changes in the environment layout, dynamic obstacles ( e.g. , people), and unexpected disturbances. Supplementary videos at adaptiveskillcoordination.github.io.