IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

• Main Authors: Espeholt, Lasse, Soyer, Hubert, Munos, Remi, Simonyan, Karen, Mnih, Volodymir, Ward, Tom, Doron, Yotam, Firoiu, Vlad, Harley, Tim, Dunning, Iain, Legg, Shane, Kavukcuoglu, Koray
• Format: Journal Article
• Language: English
• Published: 05-02-2018
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Learning
• DOI: 10.48550/arxiv.1802.01561

In this work we aim to solve a large collection of tasks using a single
reinforcement learning agent with a single set of parameters. A key challenge
is to handle the increased amount of data and extended training time. We have
developed a new distributed agent IMPALA (Importance Weighted Actor-Learner
Architecture) that not only uses resources more efficiently in single-machine
training but also scales to thousands of machines without sacrificing data
efficiency or resource utilisation. We achieve stable learning at high
throughput by combining decoupled acting and learning with a novel off-policy
correction method called V-trace. We demonstrate the effectiveness of IMPALA
for multi-task reinforcement learning on DMLab-30 (a set of 30 tasks from the
DeepMind Lab environment (Beattie et al., 2016)) and Atari-57 (all available
Atari games in Arcade Learning Environment (Bellemare et al., 2013a)). Our
results show that IMPALA is able to achieve better performance than previous
agents with less data, and crucially exhibits positive transfer between tasks
as a result of its multi-task approach.