Search Results - "Launay, Julien"

Effect of viscous dissipation in the prediction of thermal behavior of an elastomer cylindrical flow by Launay, Julien, Allanic, Nadine, Mousseau, Pierre, Deterre, Rémi, Plot, Christophe

“…In this work, the thermal behavior of an elastomer flow all along a cylindrical runner placed at the outlet of an extruder is studied. A cylindrical runner…”
Get full text

Development of a thermal Reduced Order Model with explicit dependence on viscosity for a generalized Newtonian fluid by Girault, Manuel, Launay, Julien, Allanic, Nadine, Mousseau, Pierre, Deterre, Rémi

“…This work falls within the general framework of melted polymers flows characterization. It deals with the development of a thermo-rheological Reduced Order…”
Get full text

Adversarial Robustness by Design Through Analog Computing And Synthetic Gradients by Cappelli, Alessandro, Ohana, Ruben, Launay, Julien, Meunier, Laurent, Poli, Iacopo, Krzakala, Florent

“…We propose a new defense mechanism against adversarial at-tacks inspired by an optical co-processor, providing robustness without compromising natural accuracy…”
Get full text

Scaling Laws Beyond Backpropagation by Filipovich, Matthew J, Cappelli, Alessandro, Hesslow, Daniel, Launay, Julien

“…Alternatives to backpropagation have long been studied to better understand how biological brains may learn. Recently, they have also garnered interest as a…”
Get full text

Is the Number of Trainable Parameters All That Actually Matters? by Chatelain, Amélie, Djeghri, Amine, Hesslow, Daniel, Launay, Julien, Poli, Iacopo

“…Recent work has identified simple empirical scaling laws for language models, linking compute budget, dataset size, model size, and autoregressive modeling…”
Get full text

ROPUST: Improving Robustness through Fine-tuning with Photonic Processors and Synthetic Gradients by Cappelli, Alessandro, Launay, Julien, Meunier, Laurent, Ohana, Ruben, Poli, Iacopo

“…Robustness to adversarial attacks is typically obtained through expensive adversarial training with Projected Gradient Descent. Here we introduce ROPUST, a…”
Get full text

Principled Training of Neural Networks with Direct Feedback Alignment by Launay, Julien, Poli, Iacopo, Krzakala, Florent

“…The backpropagation algorithm has long been the canonical training method for neural networks. Modern paradigms are implicitly optimized for it, and numerous…”
Get full text

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only by Penedo, Guilherme, Malartic, Quentin, Hesslow, Daniel, Cojocaru, Ruxandra, Cappelli, Alessandro, Alobeidli, Hamza, Pannier, Baptiste, Almazrouei, Ebtesam, Launay, Julien

“…Large language models are commonly trained on a mixture of filtered web data and curated high-quality corpora, such as social media conversations, books, or…”
Get full text

Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures by Launay, Julien, Poli, Iacopo, Boniface, François, Krzakala, Florent

“…Advances in Neural Information Processing Systems, v33, pages 9346--9360, 2020 Despite being the workhorse of deep learning, the backpropagation algorithm is…”
Get full text

The Falcon Series of Open Language Models by Almazrouei, Ebtesam, Alobeidli, Hamza, Alshamsi, Abdulaziz, Cappelli, Alessandro, Cojocaru, Ruxandra, Debbah, Mérouane, Goffinet, Étienne, Hesslow, Daniel, Launay, Julien, Malartic, Quentin, Mazzotta, Daniele, Noune, Badreddine, Pannier, Baptiste, Penedo, Guilherme

“…We introduce the Falcon series: 7B, 40B, and 180B parameters causal decoder-only models trained on a diverse high-quality corpora predominantly assembled from…”
Get full text

Optical training of large-scale Transformers and deep neural networks with direct feedback alignment by Wang, Ziao, Müller, Kilian, Filipovich, Matthew, Launay, Julien, Ohana, Ruben, Pariente, Gustave, Mokaadi, Safa, Brossollet, Charles, Moreau, Fabien, Cappelli, Alessandro, Poli, Iacopo, Carron, Igor, Daudet, Laurent, Krzakala, Florent, Gigan, Sylvain

“…Modern machine learning relies nearly exclusively on dedicated electronic hardware accelerators. Photonic approaches, with low consumption and high operation…”
Get full text

PAGnol: An Extra-Large French Generative Model by Launay, Julien, Tommasone, Elena, Pannier, Baptiste, Boniface, François, Chatelain, Amélie, Cappelli, Alessandro, Poli, Iacopo, Seddah, Djamé

“…Access to large pre-trained models of varied architectures, in many different languages, is central to the democratization of NLP. We introduce PAGnol, a…”
Get full text

Artificial Neural Network Training on an Optical Processor via Direct Feedback Alignment by Muller, Kilian, Launay, Julien, Poli, Iacopo, Filipovich, Matthew, Capelli, Alessandro, Hesslow, Daniel, Carron, Igor, Daudet, Laurent, Krzakala, Florent, Gigan, Sylvain

“…Artificial Neural Networks (ANN) are habitually trained via the back-propagation (BP) algorithm. This approach has been extremely successful: Current models…”
Get full text

Photonic Differential Privacy with Direct Feedback Alignment by Ohana, Ruben, Ruiz, Hamlet J. Medina, Launay, Julien, Cappelli, Alessandro, Poli, Iacopo, Ralaivola, Liva, Rakotomamonjy, Alain

“…NeurIPS 2021 Optical Processing Units (OPUs) -- low-power photonic chips dedicated to large scale random projections -- have been used in previous work to…”
Get full text

What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? by Wang, Thomas, Roberts, Adam, Hesslow, Daniel, Scao, Teven Le, Chung, Hyung Won, Beltagy, Iz, Launay, Julien, Raffel, Colin

“…Large pretrained Transformer language models have been shown to exhibit zero-shot generalization, i.e. they can perform a wide variety of tasks that they were…”
Get full text

Adversarial Robustness by Design through Analog Computing and Synthetic Gradients by Cappelli, Alessandro, Ohana, Ruben, Launay, Julien, Meunier, Laurent, Poli, Iacopo, Krzakala, Florent

“…ICASSP 2022 - IEEE International Conference on Acoustics, Speech and Signal Processing, We propose a new defense mechanism against adversarial attacks inspired…”
Get full text

Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment by Launay, Julien, Poli, Iacopo, Müller, Kilian, Pariente, Gustave, Carron, Igor, Daudet, Laurent, Krzakala, Florent, Gigan, Sylvain

“…The scaling hypothesis motivates the expansion of models past trillions of parameters as a path towards better performance. Recent significant developments,…”
Get full text

Light-in-the-loop: using a photonics co-processor for scalable training of neural networks by Launay, Julien, Poli, Iacopo, Müller, Kilian, Carron, Igor, Daudet, Laurent, Krzakala, Florent, Gigan, Sylvain

“…As neural networks grow larger and more complex and data-hungry, training costs are skyrocketing. Especially when lifelong learning is necessary, such as in…”
Get full text

What Language Model to Train if You Have One Million GPU Hours? by Scao, Teven Le, Wang, Thomas, Hesslow, Daniel, Saulnier, Lucile, Bekman, Stas, Bari, M Saiful, Biderman, Stella, Elsahar, Hady, Muennighoff, Niklas, Phang, Jason, Press, Ofir, Raffel, Colin, Sanh, Victor, Shen, Sheng, Sutawika, Lintang, Tae, Jaesung, Yong, Zheng Xin, Launay, Julien, Beltagy, Iz

“…The crystallization of modeling methods around the Transformer architecture has been a boon for practitioners. Simple, well-motivated architectural variations…”
Get full text

LightOn Optical Processing Unit : Scaling-up AI and HPC with a Non von Neumann co-processor by BROSSOLLET, Charles, Cappelli, Alessandro, Carron, Igor, Chaintoutis, Charidimos, Chatelain, Amelie, Daudet, Laurent, Gigan, Sylvain, Hesslow, Daniel, Krzakala, Florent, Launay, Julien, Mokaadi, Safa, Moreau, Fabien, Muller, Kilian, Ohana, Ruben, Pariente, Gustave, Poli, Iacopo, Tommasone, Elena

“…Beyond pure Von Neumann processing Scalability of AI / HPC models is limited by the Von Neumann bottleneck for accessing massive amounts of memory, driving up…”
Get full text