Unsupervised Data Augmentation with Naive Augmentation and without Unlabeled Data

• Main Authors: Lowell, David, Howard, Brian E, Lipton, Zachary C, Wallace, Byron C
• Format: Journal Article
• Language: English
• Published: 22-10-2020
• Subjects: Computer Science - Computation and Language; Computer Science - Learning
• DOI: 10.48550/arxiv.2010.11966

Unsupervised Data Augmentation (UDA) is a semi-supervised technique that
applies a consistency loss to penalize differences between a model's
predictions on (a) observed (unlabeled) examples; and (b) corresponding
'noised' examples produced via data augmentation. While UDA has gained
popularity for text classification, open questions linger over which design
decisions are necessary and over how to extend the method to sequence labeling
tasks. This method has recently gained traction for text classification. In
this paper, we re-examine UDA and demonstrate its efficacy on several
sequential tasks. Our main contribution is an empirical study of UDA to
establish which components of the algorithm confer benefits in NLP. Notably,
although prior work has emphasized the use of clever augmentation techniques
including back-translation, we find that enforcing consistency between
predictions assigned to observed and randomly substituted words often yields
comparable (or greater) benefits compared to these complex perturbation models.
Furthermore, we find that applying its consistency loss affords meaningful
gains without any unlabeled data at all, i.e., in a standard supervised
setting. In short: UDA need not be unsupervised, and does not require complex
data augmentation to be effective.