Impact of random oversampling and random undersampling on the performance of prediction models developed using observational health data

• Published in: Journal of big data Vol. 11; no. 1; pp. 7 - 17
• Main Authors: Yang, Cynthia, Fridgeirsson, Egill A., Kors, Jan A., Reps, Jenna M., Rijnbeek, Peter R.
• Format: Journal Article
• Language: English
• Published: Cham Springer International Publishing 01-12-2024; Springer Nature B.V; SpringerOpen
• Subjects: Big Data; Calibration; Class Imbalance Problem; Clinical decision support; Clinical prediction model; Communications Engineering; Computational Science and Engineering; Computer Science; Data Mining and Knowledge Discovery; Database Management; External validation; Information Storage and Retrieval; Machine learning; Mathematical Applications in Computer Science; Networks; Oversampling; Patient-level prediction; Performance prediction; Prediction models; Patient-level prediction; Clinical prediction model; Class Imbalance Problem; Clinical decision support; External validation; Machine learning
• ISSN: 2196-1115; 2196-1115
• DOI: 10.1186/s40537-023-00857-7

Background
There is currently no consensus on the impact of class imbalance methods on the performance of clinical prediction models. We aimed to empirically investigate the impact of random oversampling and random undersampling, two commonly used class imbalance methods, on the internal and external validation performance of prediction models developed using observational health data.
Methods
We developed and externally validated prediction models for various outcomes of interest within a target population of people with pharmaceutically treated depression across four large observational health databases. We used three different classifiers (lasso logistic regression, random forest, XGBoost) and varied the target imbalance ratio. We evaluated the impact on model performance in terms of discrimination and calibration. Discrimination was assessed using the area under the receiver operating characteristic curve (AUROC) and calibration was assessed using calibration plots.
Results
We developed and externally validated a total of 1,566 prediction models. On internal and external validation, random oversampling and random undersampling generally did not result in higher AUROCs. Moreover, we found overestimated risks, although this miscalibration could largely be corrected by recalibrating the models towards the imbalance ratios in the original dataset.
Conclusions
Overall, we found that random oversampling or random undersampling generally does not improve the internal and external validation performance of prediction models developed in large observational health databases. Based on our findings, we do not recommend applying random oversampling or random undersampling when developing prediction models in large observational health databases.