Approximating Two-Layer Feedforward Networks for Efficient Transformers

• Main Authors: Csordás, Róbert, Irie, Kazuki, Schmidhuber, Jürgen
• Format: Journal Article
• Language: English
• Published: 16-10-2023
• Subjects: Computer Science - Learning; Computer Science - Neural and Evolutionary Computing
• DOI: 10.48550/arxiv.2310.10837

How to reduce compute and memory requirements of neural networks (NNs)
without sacrificing performance? Many recent works use sparse Mixtures of
Experts (MoEs) to build resource-efficient large language models (LMs). Here we
introduce several novel perspectives on MoEs, presenting a general framework
that unifies various methods to approximate two-layer NNs (e.g., feedforward
blocks of Transformers), including product-key memories (PKMs). Leveraging
insights from this framework, we propose methods to improve both MoEs and PKMs.
Unlike prior work that compares MoEs with dense baselines under the
compute-equal condition, our evaluation condition is parameter-equal, which is
crucial to properly evaluate LMs. We show that our MoEs are competitive with
the dense Transformer-XL on both the WikiText-103 and enwiki8 datasets at two
different scales, while being much more resource efficient. This demonstrates
that MoEs are relevant not only to extremely large LMs but also to any-scale
resource-efficient LMs. Our code is public.