Learning Multi-Objective Curricula for Robotic Policy Learning

• Main Authors: Kang, Jikun, Liu, Miao, Gupta, Abhinav, Pal, Chris, Liu, Xue, Fu, Jie
• Format: Journal Article
• Language: English
• Published: 06-10-2021
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Learning; Computer Science - Robotics; Computer Science - Systems and Control; Statistics - Machine Learning
• DOI: 10.48550/arxiv.2110.03032

Various automatic curriculum learning (ACL) methods have been proposed to
improve the sample efficiency and final performance of deep reinforcement
learning (DRL). They are designed to control how a DRL agent collects data,
which is inspired by how humans gradually adapt their learning processes to
their capabilities. For example, ACL can be used for subgoal generation, reward
shaping, environment generation, or initial state generation. However, prior
work only considers curriculum learning following one of the aforementioned
predefined paradigms. It is unclear which of these paradigms are complementary,
and how the combination of them can be learned from interactions with the
environment. Therefore, in this paper, we propose a unified automatic
curriculum learning framework to create multi-objective but coherent curricula
that are generated by a set of parametric curriculum modules. Each curriculum
module is instantiated as a neural network and is responsible for generating a
particular curriculum. In order to coordinate those potentially conflicting
modules in unified parameter space, we propose a multi-task hyper-net learning
framework that uses a single hyper-net to parameterize all those curriculum
modules. In addition to existing hand-designed curricula paradigms, we further
design a flexible memory mechanism to learn an abstract curriculum, which may
otherwise be difficult to design manually. We evaluate our method on a series
of robotic manipulation tasks and demonstrate its superiority over other
state-of-the-art ACL methods in terms of sample efficiency and final
performance.