Test-retest reliability on the Cambridge Neuropsychological Test Automated Battery: Comment on Karlsen et al. (2020)

• Published in: Applied neuropsychology. Adult Vol. 29; no. 5; pp. 889 - 892
• Main Authors: Skirrow, Caroline, Cashdollar, Nathan, Granger, Kiri, Jennings, Sally, Baker, Elizabeth, Barnett, Jennifer, Cormack, Francesca
• Format: Journal Article
• Language: English
• Published: United States Routledge 03-09-2022
• Subjects: CANTAB; Cognition; measurement/statistics; neuropsychology; test-retest reliability; tests; CANTAB; neuropsychology; Cognition; measurement/statistics; tests; test-retest reliability
• ISSN: 2327-9095; 2327-9109
• DOI: 10.1080/23279095.2020.1860987

Test-retest reliability is essential to the development and validation of psychometric tools. Here we respond to the article by Karlsen et al. (Applied Neuropsychology: Adult, 2020), reporting test-retest reliability on the Cambridge Neuropsychological Test Automated Battery (CANTAB), with results that are in keeping with prior research on CANTAB and the broader cognitive assessment literature. However, after adopting a high threshold for adequate test-retest reliability, the authors report inadequate reliability for many measures. In this commentary we provide examples of stable, trait-like constructs which we would expect to remain highly consistent across longer time periods, and contrast these with measures which show acute within-subject change in response to contextual or psychological factors. Measures characterized by greater true within-subject variability typically have lower test-retest reliability, requiring adequate powering in research examining group differences and longitudinal change. However, these measures remain sensitive to important clinical and functional outcomes. Setting arbitrarily elevated test-retest reliability thresholds for test adoption in cognitive research limits the pool of available tools and precludes the adoption of many well-established tests showing consistent contextual, diagnostic, and treatment sensitivity. Overall, test-retest reliability must be balanced with other theoretical and practical considerations in study design, including test relevance and sensitivity.