The Impact of Measurement Error on the Accuracy of Individual and Aggregate SGP

• Published in: Educational measurement, issues and practice Vol. 34; no. 1; pp. 15 - 21
• Main Authors: McCaffrey, Daniel F., Castellano, Katherine E., Lockwood, J. R.
• Format: Journal Article
• Language: English
• Published: Washington Blackwell Publishing Ltd 01-03-2015; Wiley-Blackwell; Wiley Subscription Services, Inc
• Subjects: Academic achievement; Accuracy; Achievement Gains; Bias; bias-variance trade off; Comparative Analysis; Computation; Error of Measurement; Estimation bias; growth measures; Inferences; measurement error bias; Measurement errors; Scores; Simulation; Student Development; student growth percentiles; Students; teacher accountability
• ISSN: 0731-1745; 1745-3992
• DOI: 10.1111/emip.12062

Student growth percentiles (SGPs) express students' current observed scores as percentile ranks in the distribution of scores among students with the same prior‐year scores. A common concern about SGPs at the student level, and mean or median SGPs (MGPs) at the aggregate level, is potential bias due to test measurement error (ME). Shang, vanIwaarden, and Betebenner (SVB; this issue) develop a simulation‐extrapolation (SIMEX) approach to adjust SGPs for test ME. In this paper, we use a tractable example in which different SGP estimators, including SVB's SIMEX estimator, can be computed analytically to explain why ME is detrimental to both student‐level and aggregate‐level SGP estimation. A comparison of the alternative SGP estimators to the standard approach demonstrates the common bias‐variance tradeoff problem: estimators that decrease the bias relative to the standard SGP estimator increase variance, and vice versa. Even the most accurate estimator for individual student SGP has large errors of roughly 19 percentile points on average for realistic settings. Those estimators that reduce bias may suffice at the aggregate level but no single estimator is optimal for meeting the dual goals of student‐ and aggregate level inferences.