Accuracy of Automated Written Expression Curriculum-Based Measurement Scoring

• Published in: Canadian journal of school psychology Vol. 36; no. 4; pp. 304 - 317
• Main Authors: Mercer, Sterett H., Cannon, Joanna E., Squires, Bonita, Guo, Yue, Pinco, Ella
• Format: Journal Article
• Language: English
• Published: Los Angeles, CA SAGE Publications 01-12-2021; SAGE PUBLICATIONS, INC
• Subjects: Accuracy; Automation; Business writing; Curricula; Curriculum Based Assessment; Educational psychology; Educational tests & measurements; Elementary School Students; Grade 1; Learning Problems; Progress Monitoring; Scoring; Secondary School Students; Student writing; Writing Tests; progress monitoring; automated text evaluation; screening; curriculum-based measurement; written expression
• ISSN: 0829-5735; 2154-3984
• DOI: 10.1177/0829573520987753

We examined the extent to which automated written expression curriculum-based measurement (aWE-CBM) can be accurately used to computer score student writing samples for screening and progress monitoring. Students (n = 174) with learning difficulties in Grades 1 to 12 who received 1:1 academic tutoring through a community-based organization completed narrative writing samples in the fall and spring across two academic years. The samples were evaluated using four automated and hand-calculated WE-CBM scoring metrics. Results indicated automated and hand-calculated scores were highly correlated at all four timepoints for counts of total words written (rs = 1.00), words spelled correctly (rs = .99–1.00), correct word sequences (CWS; rs = .96–.97), and correct minus incorrect word sequences (CIWS; rs = .86–.92). For CWS and CIWS, however, automated scores systematically overestimated hand-calculated scores, with an unacceptable amount of error for CIWS for some types of decisions. These findings provide preliminary evidence that aWE-CBM can be used to efficiently score narrative writing samples, potentially improving the feasibility of implementing multi-tiered systems of support in which the written expression skills of large numbers of students are screened and monitored.