Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills

• Main Authors: Chebotar, Yevgen, Hausman, Karol, Lu, Yao, Xiao, Ted, Kalashnikov, Dmitry, Varley, Jake, Irpan, Alex, Eysenbach, Benjamin, Julian, Ryan, Finn, Chelsea, Levine, Sergey
• Format: Journal Article
• Language: English
• Published: 15-04-2021
• Subjects: Computer Science - Learning; Computer Science - Robotics
• DOI: 10.48550/arxiv.2104.07749

We consider the problem of learning useful robotic skills from previously
collected offline data without access to manually specified rewards or
additional online exploration, a setting that is becoming increasingly
important for scaling robot learning by reusing past robotic data. In
particular, we propose the objective of learning a functional understanding of
the environment by learning to reach any goal state in a given dataset. We
employ goal-conditioned Q-learning with hindsight relabeling and develop
several techniques that enable training in a particularly challenging offline
setting. We find that our method can operate on high-dimensional camera images
and learn a variety of skills on real robots that generalize to previously
unseen scenes and objects. We also show that our method can learn to reach
long-horizon goals across multiple episodes through goal chaining, and learn
rich representations that can help with downstream tasks through pre-training
or auxiliary objectives. The videos of our experiments can be found at
https://actionable-models.github.io