The Prevalence Effect in a Laboratory Environment

• Published in: Academic radiology Vol. 14; no. 1; pp. 49 - 53
• Main Authors: Gur, David, Sc.D, Bandos, Andriy I., Ph.D, Fuhrman, Carl R., M.D, Klym, Amy H., B.S, King, Jill L., M.S, Rockette, Howard E., Ph.D
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 2007
• Subjects: confidence ratings; laboratory effect; observer performance; Radiology; ROC; laboratory effect; confidence ratings; observer performance; ROC
• ISSN: 1076-6332; 1878-4046
• DOI: 10.1016/j.acra.2006.10.003

Rationale and Objectives We sought to assess whether or not prevalence levels affected the confidence ratings of readers during the interpretation of cases in a laboratory receiver operating characteristic−type observer performance study. Materials and Methods We reanalyzed a previously conducted observer performance study that included 14 readers and 5 different levels of prevalence. The previous study yielded the observation that in the laboratory we could not detect a “prevalence effect” in terms of differences in areas under the receiver operating characteristic curves. The detection ratings (for presence or absence) of lung nodules, interstitial disease, and pneumothorax for the five prevalence levels were compared, and a test for trend in averaged ratings as a function of abnormality prevalence was performed within a mixed-model setting that accounts for different sources of variability and correlations induced by the study design. Results The ratings of the cases in terms of confidence that the specific abnormality in question is present tend, on average, to be larger when actual disease prevalence is lower. The rate of the increase of the average confidence ratings with the decreasing prevalence of a specific abnormality is very similar for actually positive and actually negative cases for every considered abnormality. The observed trend in the changes of the average confidence ratings as a function of prevalence levels was statistically significant (p < 0.01). Conclusion Expectations of disease prevalence in the case mix during a laboratory observer performance study may systematically affect the behavior of observers in terms of their actual confidence ratings.