INGENIOUS: Using Informative Data Subsets for Efficient Pre-Training of Language Models

• Main Authors: Renduchintala, H S V N S Kowndinya, Killamsetty, Krishnateja, Bhatia, Sumit, Aggarwal, Milan, Ramakrishnan, Ganesh, Iyer, Rishabh, Krishnamurthy, Balaji
• Format: Journal Article
• Language: English
• Published: 11-05-2023
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Learning
• DOI: 10.48550/arxiv.2305.06677

A salient characteristic of pre-trained language models (PTLMs) is a
remarkable improvement in their generalization capability and emergence of new
capabilities with increasing model capacity and pre-training dataset size.
Consequently, we are witnessing the development of enormous models pushing the
state-of-the-art. It is, however, imperative to realize that this inevitably
leads to prohibitively long training times, extortionate computing costs, and a
detrimental environmental impact. Significant efforts are underway to make PTLM
training more efficient through innovations in model architectures, training
pipelines, and loss function design, with scant attention being paid to
optimizing the utility of training data. The key question that we ask is
whether it is possible to train PTLMs by employing only highly informative
subsets of the training data while maintaining downstream performance? Building
upon the recent progress in informative data subset selection, we show how we
can employ submodular optimization to select highly representative subsets of
the training corpora and demonstrate that the proposed framework can be applied
to efficiently train multiple PTLMs (BERT, BioBERT, GPT-2) using only a
fraction of data. Further, we perform a rigorous empirical evaluation to show
that the resulting models achieve up to $\sim99\%$ of the performance of the
fully-trained models. We made our framework publicly available at
https://github.com/Efficient-AI/ingenious.