Bias for Treatment Effect by Measurement Error in Pretest in ANCOVA Analysis

• Published in: Educational and psychological measurement Vol. 82; no. 6; pp. 1130 - 1152
• Main Authors: Miyazaki, Yasuo, Kamata, Akihito, Uekawa, Kazuaki, Sun, Yizhi
• Format: Journal Article
• Language: English
• Published: Los Angeles, CA SAGE Publications 01-12-2022; SAGE PUBLICATIONS, INC
• Subjects: Analysis of covariance; Bias; Correlation; Educational tests & measurements; Error of Measurement; Prediction; Pretesting; Pretests Posttests; Regression (Statistics); Scores; Simulation; Statistical Analysis; Statistical Bias; Structural equation modeling; Structural Equation Models; bias in treatment effect; analysis of covariance; measurement error in pretest; structural equation model
• DOI: 10.1177/00131644211068801
• EISSN: 1552-3888
• ISSN: 0013-1644

This paper investigated consequences of measurement error in the pretest on the estimate of the treatment effect in a pretest–posttest design with the analysis of covariance (ANCOVA) model, focusing on both the direction and magnitude of its bias. Some prior studies have examined the magnitude of the bias due to measurement error and suggested ways to correct it. However, none of them clarified how the direction of bias is affected by measurement error. This study analytically derived a formula for the asymptotic bias for the treatment effect. The derived formula is a function of the reliability of the pretest, the standardized population group mean difference for the pretest, and the correlation between pretest and posttest true scores. It revealed a concerning consequence of ignoring measurement errors in pretest scores: treatment effects could be overestimated or underestimated, and positive treatment effects can be estimated as negative effects in certain conditions. A simulation study was also conducted to verify the derived bias formula.