Programmatic assessment and Kane's validity perspective

• Published in: Medical education Vol. 46; no. 1; pp. 38 - 48
• Main Authors: Schuwirth, Lambert W T, van der Vleuten, Cees P M
• Format: Journal Article
• Language: English
• Published: Oxford, UK Blackwell Publishing Ltd 01-01-2012; Wiley-Blackwell
• Subjects: Biological and medical sciences; Clinical Competence - standards; Education, Medical - standards; Educational Measurement - methods; Educational Measurement - standards; Health participants; Humans; Medical sciences; Miscellaneous; Psychometrics - standards; Public health. Hygiene; Public health. Hygiene-occupational medicine; Reproducibility of Results; Physician; Validity; Perspective; Public health
• ISSN: 0308-0110; 1365-2923
• DOI: 10.1111/j.1365-2923.2011.04098.x

Medical Education 2012: 46: 38–48
Context  Programmatic assessment is a notion that implies that the strength of the assessment process results from a careful combination of various assessment instruments. Accordingly, no single instrument is superior to another, but each has its own strengths, weaknesses and purpose in a programme. Yet, in terms of psychometric methods, a one‐size‐fits‐all approach is often used. Kane’s views on validity as represented by a series of arguments provide a useful framework from which to highlight the value of different widely used approaches to improve the quality and validity of assessment procedures.
Methods  In this paper we discuss four inferences which form part of Kane’s validity theory: from observations to scores; from scores to universe scores; from universe scores to target domain, and from target domain to construct. For each of these inferences, we provide examples and descriptions of approaches and arguments that may help to support the validity inference.
Conclusions  As well as standard psychometric methods, a programme of assessment makes use of various other arguments, such as: item review and quality control, structuring and examiner training; probabilistic methods, saturation approaches and judgement processes, and epidemiological methods, collation, triangulation and member‐checking procedures. In an assessment programme each of these can be used.