Milestone-Based Assessments Are Superior to Likert-Type Assessments in Illustrating Trainee Progression

• Published in: Journal of graduate medical education Vol. 7; no. 1; pp. 75 - 80
• Main Authors: Bartlett, Kathleen W, Whicker, Shari A, Bookman, Jack, Narayan, Aditee P, Staples, Betty B, Hering, Holly, McGann, Kathleen A
• Format: Journal Article
• Language: English
• Published: United States The Accreditation Council for Graduate Medical Education 01-03-2015
• Subjects: Clinical Competence; Delphi Technique; Education, Medical, Graduate; Educational Measurement - methods; Female; Humans; Internal Medicine - education; Internship and Residency; Male; North Carolina; Original Research; Pediatrics - education; United States; North Carolina; United States
• ISSN: 1949-8349; 1949-8357
• DOI: 10.4300/JGME-D-14-00389.1

The Pediatrics Milestone Project uses behavioral anchors, narrative descriptions of observable behaviors, to describe learner progression through the Accreditation Council for Graduate Medical Education competencies. Starting June 2014, pediatrics programs were required to submit milestone reports for their trainees semiannually. Likert-type scale assessment tools were not designed to inform milestone reporting, creating a challenge for Clinical Competency Committees.
To determine if milestone-based assessments better stratify trainees by training level compared to Likert-type assessments.
We compared assessment results for 3 subcompetencies after changing from a 5-point Likert scale to milestone-based behavioral anchors in July 2013. Program leadership evaluated the new system by (1) comparing PGY-1 mean scores on Likert-type versus milestone-based assessments; and (2) comparing mean scores on the Likert-type versus milestone-based assessments across PGY levels.
Mean scores for PGY-1 residents were significantly higher on the prior year's Likert-type assessments than milestone-based assessments for all 3 subcompetencies (P < .01). Stratification by PGY level was not observed with Likert-type assessments (eg, interpersonal and communication skills 1 [ICS1] mean score for PGY-1, 3.99 versus PGY-3, 3.98; P  =  .98). In contrast, milestone-based assessments demonstrated stratification by PGY level (eg, the ICS1 mean score was 3.06 for PGY-1, 3.83 for PGY-2, and 3.99 for PGY-3; P < .01 for PGY-1 versus PGY-3). Significantly different means by trainee level were noted across 21 subcompetencies on milestone-based assessments (P < .01 for PGY-1 versus PGY-3).
Initial results indicate milestone-based assessments stratify trainee performance by level better than Likert-type assessments. Average PGY-level scores from milestone-based assessments may ultimately provide guidance for determining whether trainees are progressing at the expected pace.