The Sensitivity of Value-Added Modeling to the Creation of a Vertical Score Scale

• Published in: Education finance and policy Vol. 4; no. 4; pp. 384 - 414
• Main Authors: Briggs, Derek C., Weeks, Jonathan P.
• Format: Journal Article
• Language: English
• Published: One Rogers Street, Cambridge, MA 02142-1209, USA MIT Press 01-10-2009
• Subjects: Calibration; Colorado; Educational Assessment; Educational Indicators; Employee assistance programs; Evaluation Methods; Grade 3; Grade 4; Grade 5; Item Response Theory; Longitudinal Studies; Maximum likelihood estimation; Measurement; Measurement Techniques; Parametric models; Psychometrics; Rating Scales; School Effectiveness; Student Evaluation; Test scores; Colorado
• ISSN: 1557-3060; 1557-3079
• DOI: 10.1162/edfp.2009.4.4.384

The purpose of this study was to evaluate the sensitivity of growth and value-added modeling to the way an underlying vertical score scale has been created. Longitudinal item-level data were analyzed with both student- and school-level identifiers for the entire state of Colorado between 2003 and 2006. Eight different vertical scales were established on the basis of choices made for three key variables: the item response theory modeling approach, the calibration approach, and the student proficiency estimation approach. Each scale represented a methodological approach that was psychometrically defensible. Longitudinal values from each scale were used as the outcome in a commonly used value-added model (the “layered model” popularized by William Sanders) as a means of estimating school effects. Our findings suggest that while the ordering of estimating school effects is insensitive to the underlying vertical scale, the precision of such value-added estimates can be quite sensitive to the combinations of choices made in the creation of the scale.