Revisiting performance metrics for prediction with rare outcomes

• Published in: Statistical methods in medical research Vol. 30; no. 10; pp. 2352 - 2366
• Main Authors: Adhikari, Samrachana, Normand, Sharon-Lise, Bloom, Jordan, Shahian, David, Rose, Sherri
• Format: Journal Article
• Language: English
• Published: London, England SAGE Publications 01-10-2021; Sage Publications Ltd
• Subjects: Accuracy; Algorithms; Aorta; Benchmarking; False Positive Reactions; False positive results; Graphical representations; Heart valves; Humans; Machine Learning; Medical research; Performance evaluation; Performance measurement; Postoperative care; Postoperative Complications; Predictive Value of Tests; ROC Curve; Surgery; mortality; classification; machine learning; Prediction; ensembles
• ISSN: 0962-2802; 1477-0334
• DOI: 10.1177/09622802211038754

Machine learning algorithms are increasingly used in the clinical literature, claiming advantages over logistic regression. However, they are generally designed to maximize the area under the receiver operating characteristic curve. While area under the receiver operating characteristic curve and other measures of accuracy are commonly reported for evaluating binary prediction problems, these metrics can be misleading. We aim to give clinical and machine learning researchers a realistic medical example of the dangers of relying on a single measure of discriminatory performance to evaluate binary prediction questions. Prediction of medical complications after surgery is a frequent but challenging task because many post-surgery outcomes are rare. We predicted post-surgery mortality among patients in a clinical registry who received at least one aortic valve replacement. Estimation incorporated multiple evaluation metrics and algorithms typically regarded as performing well with rare outcomes, as well as an ensemble and a new extension of the lasso for multiple unordered treatments. Results demonstrated high accuracy for all algorithms with moderate measures of cross-validated area under the receiver operating characteristic curve. False positive rates were
<
1%, however, true positive rates were
<
7%, even when paired with a 100% positive predictive value, and graphical representations of calibration were poor. Similar results were seen in simulations, with the addition of high area under the receiver operating characteristic curve (
>
90%) accompanying low true positive rates. Clinical studies should not primarily report only area under the receiver operating characteristic curve or accuracy.