Measuring and correcting staff variability in large-scale OSCEs

Objective Structured Clinical Examinations (OSCEs) are an increasingly popular evaluation modality for medical students. While the face-to-face interaction allows for more in-depth assessment, it may cause standardization problems. Methods to quantify, limit or adjust for examiner effects are needed...

Full description

Saved in:

Bibliographic Details
Published in:	BMC medical education Vol. 24; no. 1; pp. 817 - 11
Main Authors:	Haviari, Skerdi, de Tymowski, Christian, Burnichon, Nelly, Lemogne, Cédric, Flamant, Martin, Ruszniewski, Philippe, Bensaadi, Saja, Mercier, Gregory, Hamaoui, Hasséne, Mirault, Tristan, Faye, Albert, Bouzid, Donia
Format:	Journal Article
Language:	English
Published:	England BioMed Central Ltd 29-07-2024 BioMed Central BMC
Subjects:	Clinical Competence - standards Down Syndrome Education, Medical, Undergraduate - standards Educational Measurement - methods Educational Measurement - standards Evaluation Familiarity Fractures Grading Humans Inter-rater variability Likert Scales Medical education Medical schools Medical students Methods Objective tests Observer Variation OSCE Paris Psychometrics Reproducibility of Results Score variability Skills Statistical analysis Students Students, Medical Summative Evaluation Teachers Validity Wages & salaries Paris France United Kingdom > UK Inter-rater variability OSCE Score variability
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Description
Summary:	Objective Structured Clinical Examinations (OSCEs) are an increasingly popular evaluation modality for medical students. While the face-to-face interaction allows for more in-depth assessment, it may cause standardization problems. Methods to quantify, limit or adjust for examiner effects are needed. Data originated from 3 OSCEs undergone by 900-student classes of 5 - and 6 -year medical students at Université Paris Cité in the 2022-2023 academic year. Sessions had five stations each, and one of the three sessions was scored by consensus by two raters (rather than one). We report OSCEs' longitudinal consistency for one of the classes and staff-related and student variability by session. We also propose a statistical method to adjust for inter-rater variability by deriving a statistical random student effect that accounts for staff-related and station random effects. From the four sessions, a total of 16,910 station scores were collected from 2615 student sessions, with two of the sessions undergone by the same students, and 36, 36, 35 and 20 distinct staff teams in each station for each session. Scores had staff-related heterogeneity (p<10 ), with staff-level standard errors approximately doubled compared to chance. With mixed models, staff-related heterogeneity explained respectively 11.4%, 11.6%, and 4.7% of station score variance (95% confidence intervals, 9.5-13.8, 9.7-14.1, and 3.9-5.8, respectively) with 1, 1 and 2 raters, suggesting a moderating effect of consensus grading. Student random effects explained a small proportion of variance, respectively 8.8%, 11.3%, and 9.6% (8.0-9.7, 10.3-12.4, and 8.7-10.5), and this low amount of signal resulted in student rankings being no more consistent over time with this metric, rather than with average scores (p=0.45). Staff variability impacts OSCE scores as much as student variability, and the former can be reduced with dual assessment or adjusted for with mixed models. Both are small compared to unmeasured sources of variability, making them difficult to capture consistently.
Bibliography:	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23
ISSN:	1472-6920 1472-6920
DOI:	10.1186/s12909-024-05803-6