The distillation method: A novel approach for analyzing randomized trials when exposure to the intervention is diluted

• Published in: Health services research Vol. 57; no. 6; pp. 1361 - 1369
• Main Authors: Adams, John L., Davis, Anna C., Schneider, Eric C., Hull, Michaela M., McGlynn, Elizabeth A.
• Format: Journal Article
• Language: English
• Published: Oxford, UK Blackwell Publishing Ltd 01-12-2022; Health Research and Educational Trust
• Subjects: Averages; biostatistical methods; Case studies; Clinical trials; Data; Demographics; Distillation; Emergency medical care; Emergency medical services; Emergency Service, Hospital; Emergency services; Enrollments; Evaluation; Extraction; Health aspects; Health care industry; Health Services; Health services utilization; Hospitalization; Humans; Innovations; Internal validity; Intervention; learning health system; Medical research; Methods; Methods Corner; participant responsiveness; Participation; Patients; Performance prediction; Probability; Randomization; randomized controlled trial; Randomized Controlled Trials as Topic; Regression analysis; Regression models; Research Design; Simulation; Social services delivery; Statistical analysis; Statistical methods; Subpopulations; Subsets; Utilization; United States; biostatistical methods; learning health system; internal validity; participant responsiveness; randomized controlled trial
• ISSN: 0017-9124; 1475-6773
• DOI: 10.1111/1475-6773.14014

Objective
To introduce a novel analytical approach for randomized controlled trials that are underpowered because of low participant enrollment or engagement.
Data Sources
Reanalysis of data for 805 patients randomized as part of a pilot complex care intervention in 2015–2016 in a large delivery system. In the pilot randomized trial, only 64.6% of patients assigned to the intervention group participated.
Study Design
A case study and simulation. The “Distillation Method” capitalizes on the frequently observed correlation between the probability of subjects' participation or engagement in the intervention and the magnitude of benefit they experience. The novel method involves three stages: first, it uses baseline covariates to generate predicted probabilities of participation. Next, these are used to produce nested subsamples of the randomized intervention and control groups that are more concentrated with subjects who were likely to participate/engage. Finally, for the outcomes of interest, standard statistical methods are used to re‐evaluate intervention effectiveness in these concentrated subsets.
Data Extraction Methods
We assembled secondary data on patients who were randomized to the pilot intervention for one year prior to randomization and two follow‐up years. Data included program enrollment status, membership data, demographics, utilization, costs, and clinical data.
Principal Findings
Using baseline covariates only, Generalized Boosted Regression Models predicting program enrollment performed well (AUC 0.884). We then distilled the full randomized sample to increasing levels of concentration and reanalyzed program outcomes. We found statistically significant differences in outpatient utilization and emergency department utilization (both follow‐up years), and in total costs (follow‐up year two only) at select levels of population concentration.
Conclusions
By offering an internally valid analytic framework, the Distillation Method can increase the power to detect effects by redefining the estimand to subpopulations with higher enrollment probabilities and stronger average treatment effects while maintaining the original randomization.