Estimating the Accuracy of Relative Growth Measures Using Empirical Data

• Published in: Journal of educational measurement Vol. 57; no. 1; pp. 92 - 123
• Main Authors: Castellano, Katherine E., McCaffrey, Daniel F.
• Format: Journal Article
• Language: English
• Published: Madison Wiley-Blackwell 01-03-2020; Wiley Subscription Services, Inc
• Subjects: Accuracy; Bias; Computation; Conditioning; Educational tests & measurements; Error of Measurement; Evaluation Methods; Inferences; Item Response Theory; Measurement; Measurement Techniques; Scores; Statistical Analysis; Statistical Bias; Student Development; Student Evaluation; Test Reliability
• ISSN: 0022-0655; 1745-3984
• DOI: 10.1111/jedm.12243

The residual gain score has been of historical interest, and its percentile rank has been of interest more recently given its close correspondence to the popular Student Growth Percentile. However, these estimators suffer from low accuracy and systematic bias (bias conditional on prior latent achievement). This article explores three alternatives—using the expected a posterior (EAP), conditioning on an additional lagged score, and correcting for measurement error bias from the prior score (Corrected‐Observed)—evaluated in terms of their systematic bias, squared correlation with their target (R2), and proportional reduction in mean squared error (PRMSE). Both analytic results (under model assumptions) and empirical results (found using item response data to calculate the growth estimators) reveal that the EAP estimators are the most accurate, whereas the Corrected‐Observed removes systematic bias, but reduces overall accuracy. Adding another prior year often decreases accuracy but only slightly reduces systematic bias at realistic test reliabilities. For all estimators, R2 and PRMSE are substantially below levels that are considered necessary for reporting educational measurements with moderate to high stakes. For all but the EAP, the raw residual gain estimators have negative PRMSE, indicating that inferences about a student's latent growth would be more accurate if students were assigned the average residual rather than estimating their residual.