Designing Reading Comprehension Assessments for Reading Interventions: How a Theoretically Motivated Assessment Can Serve as an Outcome Measure

• Published in: Educational psychology review Vol. 26; no. 3; pp. 403 - 424
• Main Authors: O'Reilly, Tenaha, Weeks, Jonathan, Sabatini, John, Halderman, Laura, Steinberg, Jonathan
• Format: Journal Article
• Language: English
• Published: Boston Springer 01-09-2014; Springer US; Springer Nature B.V
• Subjects: Apprenticeships; Child and School Psychology; Correlation; Correlations; Education; Educational Psychology; Educational research; Factor Analysis; High School Students; History; Instructional design; Intervention; Language comprehension; Learning and Instruction; Literacy; Literature; Outcome Measures; Psychometrics; Reading ability; Reading Achievement; Reading Comprehension; Reading Instruction; Reading research; Reading Skills; Reading Tests; RESEARCH INTO PRACTICE; Sciences; Scores; Standardized Tests; Statistical variance; Summative Evaluation; Test Reliability; California; Intervention; Outcome measures; Disciplinary literacy; Reading comprehension; Assessment
• ISSN: 1040-726X; 1573-336X
• DOI: 10.1007/s10648-014-9269-z

When designing a reading intervention, researchers and educators face a number of challenges related to the focus, intensity, and duration of the intervention. In this paper, we argue there is another fundamental challenge—the nature of the reading outcome measures used to evaluate the intervention. Many interventions fail to demonstrate significant improvements on standardized measures of reading comprehension. Although there are a number of reasons to explain this phenomenon, an important one to consider is misalignment between the nature of the outcome assessment and the targets of the intervention. In this study, we present data on three theoretically driven summative reading assessments that were developed in consultation with a research and evaluation team conducting an intervention study. The reading intervention, Reading Apprenticeship, involved instructing teachers to use disciplinary strategies in three domains: literature, history, and science. Factor analyses and other psychometric analyses on data from over 12,000 high school students revealed the assessments had adequate reliability, moderate correlations with state reading test scores and measures of background knowledge, a large general reading factor, and some preliminary evidence for separate, smaller factors specific to each form. In this paper, we describe the empirical work that motivated the assessments, the aims of the intervention, and the process used to develop the new assessments. Implications for intervention and assessment are discussed.