Algorithmic progress in language models

• Main Authors: Ho, Anson, Besiroglu, Tamay, Erdil, Ege, Owen, David, Rahman, Robi, Guo, Zifan Carl, Atkinson, David, Thompson, Neil, Sevilla, Jaime
• Format: Journal Article
• Language: English
• Published: 09-03-2024
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Computation and Language
• DOI: 10.48550/arxiv.2403.05812

We investigate the rate at which algorithms for pre-training language models
have improved since the advent of deep learning. Using a dataset of over 200
language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we
find that the compute required to reach a set performance threshold has halved
approximately every 8 months, with a 95% confidence interval of around 5 to 14
months, substantially faster than hardware gains per Moore's Law. We estimate
augmented scaling laws, which enable us to quantify algorithmic progress and
determine the relative contributions of scaling models versus innovations in
training algorithms. Despite the rapid pace of algorithmic progress and the
development of new architectures such as the transformer, our analysis reveals
that the increase in compute made an even larger contribution to overall
performance improvements over this time period. Though limited by noisy
benchmark data, our analysis quantifies the rapid progress in language
modeling, shedding light on the relative contributions from compute and
algorithms.