Predicting mastery level on a large-scale standardized patient test : A comparison of case and instrument score-based models using discriminant function analysis

• Published in: Advances in health sciences education : theory and practice Vol. 6; no. 2; pp. 151 - 158
• Main Authors: DE CHAMPLAIN, André F, MARGOLIS, Melissa J, MACMILLAN, Mary K, KLASS, Daniel J
• Format: Journal Article
• Language: English
• Published: Heidelberg Springer 01-01-2001; Springer Nature B.V
• Subjects: Clinical Competence; Curriculum subjects: programmes and methods; Discriminant Analysis; Educational Measurement - methods; Educational Measurement - standards; Educational sciences; Humans; Medical and paramedical education; Models, Educational; Multivariate Analysis; Psychometrics; Teaching methods; United States; United States; Medicine; Diagnosis; Psychometry; Performance; Evaluative Behaviour; Examination
• ISSN: 1382-4996; 1573-1677
• DOI: 10.1023/A:1011421706300

Clinical skills assessments have traditionally been scored via experts' ratings of examinee performance. However, this approach to scoring may be impractical in a large-scale context due to logistical and cost considerations as well as the increased probability of rater error. The purpose of this investigation was therefore to identify, using discriminant analysis, weighted score-based models that maximize the accuracy with which mastery level can be estimated for examinees taking a nationally administered standardized patient test. Additionally, the accuracy with which the resulting classification functions can be applied to predict mastery level for a cross-validation sample of examinees was also examined. Results suggest that it might be feasible to implement an automated scoring procedure in a cost-effective manner while still retaining the important facets of the decision-making process of expert raters. Cost-benefit, test development and psychometric implications of these results are important and discussed in the full paper.