Open X-Embodiment: Robotic Learning Datasets and RT-X Models

• Main Authors: Collaboration, Open X-Embodiment, O'Neill, Abby, Rehman, Abdul, Pooley, Acorn, Gupta, Agrim, Irpan, Alex, Khazatsky, Alexander, Rai, Anant, Singh, Anikait, Garg, Animesh, Kembhavi, Aniruddha, Xie, Annie, Raffin, Antonin, Burgess-Limerick, Ben, Wulfe, Blake, Ichter, Brian, Xu, Charles, Le, Charlotte, Chi, Cheng, Huang, Chenguang, Chan, Christine, Morton, Daniel, Driess, Danny, Kalashnikov, Dmitry, Sadigh, Dorsa, Stulp, Freek, Zhou, Gaoyue, Salhotra, Gautam, Feng, Gilbert, Schiavi, Giulio, Su, Hao, Kim, Jaehyung, Drake, Jaimyn, Peters, Jan, Vakil, Jay, Bohg, Jeannette, Wu, Jeffrey, Wu, Jiajun, Luo, Jianlan, Wu, Jimmy, Yang, Jingyun, Silvério, João, Booher, Jonathan, Lim, Joseph J, Wang, Kaiyuan, Hausman, Karol, Byrne, Kendra, Kawaharazuka, Kento, Black, Kevin, Zhang, Kevin, Ellis, Kirsty, Singh, Kunal Pratap, Zeng, Kuo-Hao, Chen, Lawrence Yunliang, Pinto, Lerrel, Fei-Fei, Li, Fan, Linxi "Jim", Memmel, Marius, Tomizuka, Masayoshi, Spero, Max, Kanazawa, Naoaki, Liu, Ning, Di Palo, Norman, Mees, Oier, Bastani, Osbert, Sanketi, Pannag R, Yin, Patrick, Xu, Peng, Fagan, Peter David, Sermanet, Pierre, Sundaresan, Priya, Vuong, Quan, Rafailov, Rafael, Hendrix, Rose, Lin, Roy, Zhang, Ruohan, Shah, Rutav, Bustamante, Samuel, Bahl, Shikhar, Song, Shuran, Xu, Sichun, Haldar, Siddhant, Guist, Simon, Nasiriany, Soroush, Schaal, Stefan, Tian, Stephen, Dasari, Sudeep, Zhao, Tony Z, Chung, Trinity, Jain, Vidhi, Vanhoucke, Vincent, Zhan, Wei, Chen, Xi, Chen, Xiangyu, Liu, Xiyuan, Zhu, Yifeng, Bisk, Yonatan, Dou, Yongqiang, Cao, Yue, Zhu, Yuke
• Format: Journal Article
• Language: English
• Published: 13-10-2023
• Subjects: Computer Science - Robotics
• DOI: 10.48550/arxiv.2310.08864

Large, high-capacity models trained on diverse datasets have shown remarkable
successes on efficiently tackling downstream applications. In domains from NLP
to Computer Vision, this has led to a consolidation of pretrained models, with
general pretrained backbones serving as a starting point for many applications.
Can such a consolidation happen in robotics? Conventionally, robotic learning
methods train a separate model for every application, every robot, and even
every environment. Can we instead train generalist X-robot policy that can be
adapted efficiently to new robots, tasks, and environments? In this paper, we
provide datasets in standardized data formats and models to make it possible to
explore this possibility in the context of robotic manipulation, alongside
experimental results that provide an example of effective X-robot policies. We
assemble a dataset from 22 different robots collected through a collaboration
between 21 institutions, demonstrating 527 skills (160266 tasks). We show that
a high-capacity model trained on this data, which we call RT-X, exhibits
positive transfer and improves the capabilities of multiple robots by
leveraging experience from other platforms. More details can be found on the
project website https://robotics-transformer-x.github.io.