More power to you: Using machine learning to augment human coding for more efficient inference in text-based randomized trials

• Main Authors: Mozer, Reagan, Miratrix, Luke
• Format: Journal Article
• Language: English
• Published: 24-09-2023
• Subjects: Statistics - Methodology
• DOI: 10.48550/arxiv.2309.13666

For randomized trials that use text as an outcome, traditional approaches for
assessing treatment impact require that each document first be manually coded
for constructs of interest by trained human raters. This process, the current
standard, is both time-consuming and limiting: even the largest human coding
efforts are typically constrained to measure only a small set of dimensions
across a subsample of available texts. In this work, we present an inferential
framework that can be used to increase the power of an impact assessment, given
a fixed human-coding budget, by taking advantage of any "untapped" observations
-- those documents not manually scored due to time or resource constraints --
as a supplementary resource. Our approach, a methodological combination of
causal inference, survey sampling methods, and machine learning, has four
steps: (1) select and code a sample of documents; (2) build a machine learning
model to predict the human-coded outcomes from a set of automatically extracted
text features; (3) generate machine-predicted scores for all documents and use
these scores to estimate treatment impacts; and (4) adjust the final impact
estimates using the residual differences between human-coded and
machine-predicted outcomes. This final step ensures any biases in the modeling
procedure do not propagate to biases in final estimated effects. Through an
extensive simulation study and an application to a recent field trial in
education, we show that our proposed approach can be used to reduce the scope
of a human-coding effort while maintaining nominal power to detect a
significant treatment impact.