Incorporating aspects of programmatic assessment into examinations: Aggregating rich information to inform decision-making

• Published in: Medical teacher Vol. 43; no. 5; pp. 567 - 574
• Main Authors: Pearce, Jacob, Reid, Katharine, Chiavaroli, Neville, Hyam, Dylan
• Format: Journal Article
• Language: English
• Published: England Taylor & Francis 04-05-2021; Taylor & Francis Ltd
• Subjects: Candidates; Curricula; Decision making; Evaluation; Evaluation methods; examination reform; Examiners; Feedback; Higher education; Medical education; Performance standards; Programmatic assessment; programmatic thinking; Rating Scales; Ratings & rankings; rich information; Student assessment; Test construction; Tests; Programmatic assessment; programmatic thinking; decision-making; examination reform; rich information
• ISSN: 0142-159X; 1466-187X
• DOI: 10.1080/0142159X.2021.1878122

A programmatic approach to assessment entails gathering and aggregating 'rich information' on candidates to inform progress decisions. However, there is little guidance on how such an approach might be implemented in practice.
We describe an approach to aggregating rich information across assessment formats to inform committee decision-making in a specialist medical college.
Each item (n = 272) for every examination was blueprinted to 15 curriculum modules and 7 proficiencies. We developed a six-point holistic rating scale with detailed rubrics outlining expected performance standards for every item. Examiners used this rating scale in making judgements for each item, generating rich performance data for each candidate.
A colour-coded 'mosaic' of patterns of performance across modules and proficiencies was generated along with frequency distributions of ratings. These data allowed examiners to easily visualise candidate performance and to use these data to inform deliberations on borderline candidates. Committee decision-making was facilitated by maintaining the richness of assessment information throughout the process. Moreover, the data facilitated detailed and useful feedback to candidates.
Our study demonstrates that incorporating aspects of programmatic thinking into high-stakes examinations by using a novel approach to aggregating information is a useful first step in reforming an assessment program.