Learning where to learn: Gradient sparsity in meta and continual learning

• Main Authors: von Oswald, Johannes, Zhao, Dominic, Kobayashi, Seijin, Schug, Simon, Caccia, Massimo, Zucchet, Nicolas, Sacramento, João
• Format: Journal Article
• Language: English
• Published: 27-10-2021
• Subjects: Computer Science - Learning; Computer Science - Neural and Evolutionary Computing
• DOI: 10.48550/arxiv.2110.14402

Finding neural network weights that generalize well from small datasets is
difficult. A promising approach is to learn a weight initialization such that a
small number of weight changes results in low generalization error. We show
that this form of meta-learning can be improved by letting the learning
algorithm decide which weights to change, i.e., by learning where to learn. We
find that patterned sparsity emerges from this process, with the pattern of
sparsity varying on a problem-by-problem basis. This selective sparsity results
in better generalization and less interference in a range of few-shot and
continual learning problems. Moreover, we find that sparse learning also
emerges in a more expressive model where learning rates are meta-learned. Our
results shed light on an ongoing debate on whether meta-learning can discover
adaptable features and suggest that learning by sparse gradient descent is a
powerful inductive bias for meta-learning systems.