Introducing Symmetries to Black Box Meta Reinforcement Learning

• Main Authors: Kirsch, Louis, Flennerhag, Sebastian, van Hasselt, Hado, Friesen, Abram, Oh, Junhyuk, Chen, Yutian
• Format: Journal Article
• Language: English
• Published: 22-09-2021
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Learning; Computer Science - Neural and Evolutionary Computing; Statistics - Machine Learning
• DOI: 10.48550/arxiv.2109.10781

Meta reinforcement learning (RL) attempts to discover new RL algorithms
automatically from environment interaction. In so-called black-box approaches,
the policy and the learning algorithm are jointly represented by a single
neural network. These methods are very flexible, but they tend to underperform
in terms of generalisation to new, unseen environments. In this paper, we
explore the role of symmetries in meta-generalisation. We show that a recent
successful meta RL approach that meta-learns an objective for
backpropagation-based learning exhibits certain symmetries (specifically the
reuse of the learning rule, and invariance to input and output permutations)
that are not present in typical black-box meta RL systems. We hypothesise that
these symmetries can play an important role in meta-generalisation. Building
off recent work in black-box supervised meta learning, we develop a black-box
meta RL system that exhibits these same symmetries. We show through careful
experimentation that incorporating these symmetries can lead to algorithms with
a greater ability to generalise to unseen action & observation spaces, tasks,
and environments.