Ensemble Successor Representations for Task Generalization in Offline-to-Online Reinforcement Learning

• Main Authors: Wang, Changhong, Yu, Xudong, Bai, Chenjia, Zhang, Qiaosheng, Wang, Zhen
• Format: Journal Article
• Language: English
• Published: 12-05-2024
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Learning
• DOI: 10.48550/arxiv.2405.07223

In Reinforcement Learning (RL), training a policy from scratch with online
experiences can be inefficient because of the difficulties in exploration.
Recently, offline RL provides a promising solution by giving an initialized
offline policy, which can be refined through online interactions. However,
existing approaches primarily perform offline and online learning in the same
task, without considering the task generalization problem in offline-to-online
adaptation. In real-world applications, it is common that we only have an
offline dataset from a specific task while aiming for fast online-adaptation
for several tasks. To address this problem, our work builds upon the
investigation of successor representations for task generalization in online RL
and extends the framework to incorporate offline-to-online learning. We
demonstrate that the conventional paradigm using successor features cannot
effectively utilize offline data and improve the performance for the new task
by online fine-tuning. To mitigate this, we introduce a novel methodology that
leverages offline data to acquire an ensemble of successor representations and
subsequently constructs ensemble Q functions. This approach enables robust
representation learning from datasets with different coverage and facilitates
fast adaption of Q functions towards new tasks during the online fine-tuning
phase. Extensive empirical evaluations provide compelling evidence showcasing
the superior performance of our method in generalizing to diverse or even
unseen tasks.