Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation

• Main Authors: Davchev, Todor, Sushkov, Oleg, Regli, Jean-Baptiste, Schaal, Stefan, Aytar, Yusuf, Wulfmeier, Markus, Scholz, Jon
• Format: Journal Article
• Language: English
• Published: 01-12-2021
• Subjects: Computer Science - Robotics; Statistics - Machine Learning
• DOI: 10.48550/arxiv.2112.00597

International Conference on Learning Representations (ICLR 2022) Complex sequential tasks in continuous-control settings often require agents
to successfully traverse a set of "narrow passages" in their state space.
Solving such tasks with a sparse reward in a sample-efficient manner poses a
challenge to modern reinforcement learning (RL) due to the associated
long-horizon nature of the problem and the lack of sufficient positive signal
during learning. Various tools have been applied to address this challenge.
When available, large sets of demonstrations can guide agent exploration.
Hindsight relabelling on the other hand does not require additional sources of
information. However, existing strategies explore based on task-agnostic goal
distributions, which can render the solution of long-horizon tasks impractical.
In this work, we extend hindsight relabelling mechanisms to guide exploration
along task-specific distributions implied by a small set of successful
demonstrations. We evaluate the approach on four complex, single and dual arm,
robotics manipulation tasks against strong suitable baselines. The method
requires far fewer demonstrations to solve all tasks and achieves a
significantly higher overall performance as task complexity increases. Finally,
we investigate the robustness of the proposed solution with respect to the
quality of input representations and the number of demonstrations.