Interrater Agreement and Diagnostic Accuracy of a Novel Computer-Aided Detection Process for the Detection and Prevention of Retained Surgical Instruments

• Published in: American journal of roentgenology (1976) Vol. 210; no. 4; pp. 709 - 714
• Main Authors: Marentis, Theodore C, Davenport, Matthew S, Dillman, Jonathan R, Sanchez, Ramon, Kelly, Aine M, Cronin, Paul, DeFreitas, Mariana R, Hadjiiski, Lubomir, Chan, Heang-Ping
• Format: Journal Article
• Language: English
• Published: United States 01-04-2018
• Subjects: Aged, 80 and over; Cadaver; Diagnosis, Computer-Assisted; Foreign Bodies - diagnostic imaging; Humans; Radiography, Abdominal - methods; Sensitivity and Specificity; patient safety; computer-aided detection; gossypiboma; retained surgical instrument; never-event
• ISSN: 0361-803X; 1546-3141
• DOI: 10.2214/AJR.17.18576

The purpose of this study is to evaluate the diagnostic accuracy of a process incorporating computer-aided detection (CAD) for the detection and prevention of retained surgical instruments using a novel nondeformable radiopaque μTag.
A high-specificity CAD system was developed iteratively from a training set (n = 540 radiographs) and a validation set (n = 560 radiographs). A novel test set composed of 700 thoracoabdominal radiographs (410 with a randomly placed μTag and 290 without a μTag) was obtained from 10 cadavers embedded with confounding iatrogenic objects. Data were analyzed first by the blinded CAD system; radiographs coded as negative (n = 373) were then independently reviewed by five blinded radiologists. The reference standard was the presence of a μTag. Sensitivity and specificity were calculated. Interrater agreement was assessed with Cohen kappa values. Mean (± SD) image analysis times were calculated.
The high-specificity CAD system had one false-positive (sensitivity, 79.5% [326/410]; specificity, 99.7% [289/290]). A combination of the CAD system and one failsafe radiologist had superior sensitivity (98.5% [404/410] to 100% [410/410]) and specificity (99.0% [287/290] to 99.7% [289/290]), with 327 (47%) radiographs not requiring immediate radiologist review. Interrater agreement was almost perfect for all radiologist pairwise comparisons (κ = 0.921-0.992). Cumulative mean image analysis time was less than one minute (CAD, 29 ± 2 seconds; radiologists, 26 ± 16 seconds).
The combination of a high-specificity CAD system with a failsafe radiologist had excellent diagnostic accuracy in the rapid detection of a nondeformable radiopaque μTag.