Deep Reinforcement Learning with Plasticity Injection

• Main Authors: Nikishin, Evgenii, Oh, Junhyuk, Ostrovski, Georg, Lyle, Clare, Pascanu, Razvan, Dabney, Will, Barreto, André
• Format: Journal Article
• Language: English
• Published: 24-05-2023
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Learning
• DOI: 10.48550/arxiv.2305.15555

A growing body of evidence suggests that neural networks employed in deep
reinforcement learning (RL) gradually lose their plasticity, the ability to
learn from new data; however, the analysis and mitigation of this phenomenon is
hampered by the complex relationship between plasticity, exploration, and
performance in RL. This paper introduces plasticity injection, a minimalistic
intervention that increases the network plasticity without changing the number
of trainable parameters or biasing the predictions. The applications of this
intervention are two-fold: first, as a diagnostic tool $\unicode{x2014}$ if
injection increases the performance, we may conclude that an agent's network
was losing its plasticity. This tool allows us to identify a subset of Atari
environments where the lack of plasticity causes performance plateaus,
motivating future studies on understanding and combating plasticity loss.
Second, plasticity injection can be used to improve the computational
efficiency of RL training if the agent has to re-learn from scratch due to
exhausted plasticity or by growing the agent's network dynamically without
compromising performance. The results on Atari show that plasticity injection
attains stronger performance compared to alternative methods while being
computationally efficient.