Expressivity of Neural Networks with Random Weights and Learned Biases

• Main Authors: Williams, Ezekiel, Ryoo, Avery Hee-Woon, Jiralerspong, Thomas, Payeur, Alexandre, Perich, Matthew G, Mazzucato, Luca, Lajoie, Guillaume
• Format: Journal Article
• Language: English
• Published: 01-07-2024
• Subjects: Computer Science - Neural and Evolutionary Computing; Quantitative Biology - Neurons and Cognition; Statistics - Machine Learning
• DOI: 10.48550/arxiv.2407.00957

Landmark universal function approximation results for neural networks with
trained weights and biases provided impetus for the ubiquitous use of neural
networks as learning models in Artificial Intelligence (AI) and neuroscience.
Recent work has pushed the bounds of universal approximation by showing that
arbitrary functions can similarly be learned by tuning smaller subsets of
parameters, for example the output weights, within randomly initialized
networks. Motivated by the fact that biases can be interpreted as biologically
plausible mechanisms for adjusting unit outputs in neural networks, such as
tonic inputs or activation thresholds, we investigate the expressivity of
neural networks with random weights where only biases are optimized. We provide
theoretical and numerical evidence demonstrating that feedforward neural
networks with fixed random weights can be trained to perform multiple tasks by
learning biases only. We further show that an equivalent result holds for
recurrent neural networks predicting dynamical system trajectories. Our results
are relevant to neuroscience, where they demonstrate the potential for
behaviourally relevant changes in dynamics without modifying synaptic weights,
as well as for AI, where they shed light on multi-task methods such as bias
fine-tuning and unit masking.