Model-based Lifelong Reinforcement Learning with Bayesian Exploration

• Main Authors: Fu, Haotian, Yu, Shangqun, Littman, Michael, Konidaris, George
• Format: Journal Article
• Language: English
• Published: 20-10-2022
• Subjects: Computer Science - Learning
• DOI: 10.48550/arxiv.2210.11579

We propose a model-based lifelong reinforcement-learning approach that
estimates a hierarchical Bayesian posterior distilling the common structure
shared across different tasks. The learned posterior combined with a
sample-based Bayesian exploration procedure increases the sample efficiency of
learning across a family of related tasks. We first derive an analysis of the
relationship between the sample complexity and the initialization quality of
the posterior in the finite MDP setting. We next scale the approach to
continuous-state domains by introducing a Variational Bayesian Lifelong
Reinforcement Learning algorithm that can be combined with recent model-based
deep RL methods, and that exhibits backward transfer. Experimental results on
several challenging domains show that our algorithms achieve both better
forward and backward transfer performance than state-of-the-art lifelong RL
methods.