Search Results - "Child, Rewon"

Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images by Child, Rewon

“…We present a hierarchical VAE that, for the first time, generates samples quickly while outperforming the PixelCNN in log-likelihood on all natural image…”
Get full text

Exploring neural transducers for end-to-end speech recognition by Battenberg, Eric, Jitong Chen, Child, Rewon, Coates, Adam, Li, Yashesh Gaur Yi, Hairong Liu, Satheesh, Sanjeev, Sriram, Anuroop, Zhenyao Zhu

“…In this work, we perform an empirical comparison among the CTC, RNN-Transducer, and attention-based Seq2Seq models for end-to-end speech recognition. We show…”
Get full text

Generating Long Sequences with Sparse Transformers by Child, Rewon, Gray, Scott, Radford, Alec, Sutskever, Ilya

“…Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse…”
Get full text

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model by Smith, Shaden, Patwary, Mostofa, Norick, Brandon, LeGresley, Patrick, Rajbhandari, Samyam, Casper, Jared, Liu, Zhun, Prabhumoye, Shrimai, Zerveas, George, Korthikanti, Vijay, Zhang, Elton, Child, Rewon, Aminabadi, Reza Yazdani, Bernauer, Julie, Song, Xia, Shoeybi, Mohammad, He, Yuxiong, Houston, Michael, Tiwary, Saurabh, Catanzaro, Bryan

“…Pretrained general-purpose language models can achieve state-of-the-art accuracies in various natural language processing domains by adapting to downstream…”
Get full text

Scaling Laws for Neural Language Models by Kaplan, Jared, McCandlish, Sam, Henighan, Tom, Brown, Tom B, Chess, Benjamin, Child, Rewon, Gray, Scott, Radford, Alec, Wu, Jeffrey, Amodei, Dario

“…We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the…”
Get full text

PaLM: Scaling Language Modeling with Pathways by Chowdhery, Aakanksha, Narang, Sharan, Devlin, Jacob, Bosma, Maarten, Mishra, Gaurav, Roberts, Adam, Barham, Paul, Chung, Hyung Won, Sutton, Charles, Gehrmann, Sebastian, Schuh, Parker, Shi, Kensen, Tsvyashchenko, Sasha, Maynez, Joshua, Rao, Abhishek, Barnes, Parker, Tay, Yi, Shazeer, Noam, Prabhakaran, Vinodkumar, Reif, Emily, Du, Nan, Hutchinson, Ben, Pope, Reiner, Bradbury, James, Austin, Jacob, Isard, Michael, Gur-Ari, Guy, Yin, Pengcheng, Duke, Toju, Levskaya, Anselm, Ghemawat, Sanjay, Dev, Sunipa, Michalewski, Henryk, Garcia, Xavier, Misra, Vedant, Robinson, Kevin, Fedus, Liam, Zhou, Denny, Ippolito, Daphne, Luan, David, Lim, Hyeontaek, Zoph, Barret, Spiridonov, Alexander, Sepassi, Ryan, Dohan, David, Agrawal, Shivani, Omernick, Mark, Dai, Andrew M, Pillai, Thanumalayan Sankaranarayana, Pellat, Marie, Lewkowycz, Aitor, Moreira, Erica, Child, Rewon, Polozov, Oleksandr, Lee, Katherine, Zhou, Zongwei, Wang, Xuezhi, Saeta, Brennan, Diaz, Mark, Firat, Orhan, Catasta, Michele, Wei, Jason, Meier-Hellstern, Kathy, Eck, Douglas, Dean, Jeff, Petrov, Slav, Fiedel, Noah

“…Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically…”
Get full text

Active Learning for Speech Recognition: the Power of Gradients by Huang, Jiaji, Child, Rewon, Rao, Vinay, Liu, Hairong, Satheesh, Sanjeev, Coates, Adam

“…In training speech recognition systems, labeling audio clips can be expensive, and not all data is equally valuable. Active learning aims to label only the…”
Get full text

Language Models are Few-Shot Learners by Brown, Tom B, Mann, Benjamin, Ryder, Nick, Subbiah, Melanie, Kaplan, Jared, Dhariwal, Prafulla, Neelakantan, Arvind, Shyam, Pranav, Sastry, Girish, Askell, Amanda, Agarwal, Sandhini, Herbert-Voss, Ariel, Krueger, Gretchen, Henighan, Tom, Child, Rewon, Ramesh, Aditya, Ziegler, Daniel M, Wu, Jeffrey, Winter, Clemens, Hesse, Christopher, Chen, Mark, Sigler, Eric, Litwin, Mateusz, Gray, Scott, Chess, Benjamin, Clark, Jack, Berner, Christopher, McCandlish, Sam, Radford, Alec, Sutskever, Ilya, Amodei, Dario

“…Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific…”
Get full text

Convolutional Recurrent Neural Networks for Small-Footprint Keyword Spotting by Arik, Sercan O, Kliegl, Markus, Child, Rewon, Hestness, Joel, Gibiansky, Andrew, Fougner, Chris, Prenger, Ryan, Coates, Adam

“…Keyword spotting (KWS) constitutes a major component of human-technology interfaces. Maximizing the detection accuracy at a low false alarm (FA) rate, while…”
Get full text

Exploring Neural Transducers for End-to-End Speech Recognition by Battenberg, Eric, Chen, Jitong, Child, Rewon, Coates, Adam, Gaur, Yashesh, Li, Yi, Liu, Hairong, Satheesh, Sanjeev, Seetapun, David, Sriram, Anuroop, Zhu, Zhenyao

“…In this work, we perform an empirical comparison among the CTC, RNN-Transducer, and attention-based Seq2Seq models for end-to-end speech recognition. We show…”
Get full text

Reducing Bias in Production Speech Models by Battenberg, Eric, Child, Rewon, Coates, Adam, Fougner, Christopher, Gaur, Yashesh, Huang, Jiaji, Jun, Heewoo, Kannan, Ajay, Kliegl, Markus, Kumar, Atul, Liu, Hairong, Rao, Vinay, Satheesh, Sanjeev, Seetapun, David, Sriram, Anuroop, Zhu, Zhenyao

“…Replacing hand-engineered pipelines with end-to-end deep learning systems has enabled strong results in applications like speech and object recognition…”
Get full text