Learning patient-level prediction models across multiple healthcare databases: evaluation of ensembles for increasing model transportability

• Published in: BMC medical informatics and decision making Vol. 22; no. 1; p. 142
• Main Authors: Reps, Jenna Marie, Williams, Ross D, Schuemie, Martijn J, Ryan, Patrick B, Rijnbeek, Peter R
• Format: Journal Article
• Language: English
• Published: England BioMed Central Ltd 25-05-2022; BioMed Central; BMC
• Subjects: Algorithms; Calibration; Communication; Datasets; Decision making; Ensemble learning; Federated learning; Generalized linear models; Health care; Information sharing; Insurance claims; Learning; Machine learning; Medical informatics; Medical prognosis; Model transportability; Observational data; Patient-level prediction; Patients; Prediction models; Privacy; Probability; Prognostic model; Questions; Stacking; Netherlands; United States > US; Observational data; Prognostic model; Patient-level prediction; Ensemble learning; Model transportability
• ISSN: 1472-6947; 1472-6947
• DOI: 10.1186/s12911-022-01879-6

Prognostic models that are accurate could help aid medical decision making. Large observational databases often contain temporal medical data for large and diverse populations of patients. It may be possible to learn prognostic models using the large observational data. Often the performance of a prognostic model undesirably worsens when transported to a different database (or into a clinical setting). In this study we investigate different ensemble approaches that combine prognostic models independently developed using different databases (a simple federated learning approach) to determine whether ensembles that combine models developed across databases can improve model transportability (perform better in new data than single database models)?
For a given prediction question we independently trained five single database models each using a different observational healthcare database. We then developed and investigated numerous ensemble models (fusion, stacking and mixture of experts) that combined the different database models. Performance of each model was investigated via discrimination and calibration using a leave one dataset out technique, i.e., hold out one database to use for validation and use the remaining four datasets for model development. The internal validation of a model developed using the hold out database was calculated and presented as the 'internal benchmark' for comparison.
In this study the fusion ensembles generally outperformed the single database models when transported to a previously unseen database and the performances were more consistent across unseen databases. Stacking ensembles performed poorly in terms of discrimination when the labels in the unseen database were limited. Calibration was consistently poor when both ensembles and single database models were applied to previously unseen databases.
A simple federated learning approach that implements ensemble techniques to combine models independently developed across different databases for the same prediction question may improve the discriminative performance in new data (new database or clinical setting) but will need to be recalibrated using the new data. This could help medical decision making by improving prognostic model performance.