Scoring reading parameters: An inter-rater reliability study using the MNREAD chart

• Published in: PloS one Vol. 14; no. 6; p. e0216775
• Main Authors: Baskaran, Karthikeyan, Macedo, Antonio Filipe, He, Yingchen, Hernandez-Moreno, Laura, Queirós, Tatiana, Mansfield, J Stephen, Calabrèse, Aurélie
• Format: Journal Article
• Language: English
• Published: United States Public Library of Science 07-06-2019; Public Library of Science (PLoS)
• Subjects: Acuity; Algorithms; Analysis; Automation; Biology and Life Sciences; Blindness; Cognitive science; Computer and Information Sciences; Computer based research; Correlation coefficient; Correlation coefficients; Human performance; Humans; Low vision; Measurement methods; Medicine and Health Sciences; Methods; MNREAD acuity chart; Observer Variation; Optometri; Optometry; Parameters; Physical Sciences; Reading; Rehabilitation; Reliability analysis; Research and Analysis Methods; Scoring; Social Sciences; Standard deviation; Test reliability; Vision; Vision Disorders - physiopathology; Vision Tests - methods; Visual impairment; New York; Sweden; France; United States > US; Portugal
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0216775

First, to evaluate inter-rater reliability when human raters estimate the reading performance of visually impaired individuals using the MNREAD acuity chart. Second, to evaluate the agreement between computer-based scoring algorithms and compare them with human rating.
Reading performance was measured for 101 individuals with low vision, using the Portuguese version of the MNREAD test. Seven raters estimated the maximum reading speed (MRS) and critical print size (CPS) of each individual MNREAD curve. MRS and CPS were also calculated automatically for each curve using two different algorithms: the original standard deviation method (SDev) and a non-linear mixed effects (NLME) modeling. Intra-class correlation coefficients (ICC) were used to estimate absolute agreement between raters and/or algorithms.
Absolute agreement between raters was 'excellent' for MRS (ICC = 0.97; 95%CI [0.96, 0.98]) and 'moderate' to 'good' for CPS (ICC = 0.77; 95%CI [0.69, 0.83]). For CPS, inter-rater reliability was poorer among less experienced raters (ICC = 0.70; 95%CI [0.57, 0.80]) when compared to experienced ones (ICC = 0.82; 95%CI [0.76, 0.88]). Absolute agreement between the two algorithms was 'excellent' for MRS (ICC = 0.96; 95%CI [0.91, 0.98]). For CPS, the best possible agreement was found for CPS defined as the print size sustaining 80% of MRS (ICC = 0.77; 95%CI [0.68, 0.84]). Absolute agreement between raters and automated methods was 'excellent' for MRS (ICC = 0.96; 95% CI [0.88, 0.98] for SDev; ICC = 0.97; 95% CI [0.95, 0.98] for NLME). For CPS, absolute agreement between raters and SDev ranged from 'poor' to 'good' (ICC = 0.66; 95% CI [0.3, 0.80]), while agreement between raters and NLME was 'good' (ICC = 0.83; 95% CI [0.76, 0.88]).
For MRS, inter-rater reliability is excellent, even considering the possibility of noisy and/or incomplete data collected in low-vision individuals. For CPS, inter-rater reliability is lower. This may be problematic, for instance in the context of multisite investigations or follow-up examinations. The NLME method showed better agreement with the raters than the SDev method for both reading parameters. Setting up consensual guidelines to deal with ambiguous curves may help improve reliability. While the exact definition of CPS should be chosen on a case-by-case basis depending on the clinician or researcher's motivations, evidence suggests that estimating CPS as the smallest print size sustaining about 80% of MRS would increase inter-rater reliability.