Clinical Screening Prediction in the Portuguese National Health Service: Data Analysis, Machine Learning Models, Explainability and Meta-Evaluation

• Published in: Future internet Vol. 15; no. 1; p. 26
• Main Authors: Gonçalves, Teresa, Veladas, Rute, Yang, Hua, Vieira, Renata, Quaresma, Paulo, Infante, Paulo, Sousa Pinto, Cátia, Oliveira, João, Cortes Ferreira, Maria, Morais, Jéssica, Pereira, Ana Raquel, Fernandes, Nuno, Gonçalves, Carolina
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01-01-2023
• Subjects: Active learning; Algorithms; Artificial intelligence; Classification; Clinical decision making; clinical pathways; clinical triage; Comparative analysis; Computational linguistics; Data analysis; Datasets; Decision analysis; Decision support systems; Design; Error analysis; Experiments; Health services; Internet; Language processing; Machine learning; Medical research; Natural language interfaces; Neural networks; Nurses; Patient care planning; SNS24; Support vector machines; Text categorization; Portugal
• ISSN: 1999-5903; 1999-5903
• DOI: 10.3390/fi15010026

This paper presents an analysis of the calls made to the Portuguese National Health Contact Center (SNS24) during a three years period. The final goal was to develop a system to help nurse attendants select the appropriate clinical pathway (from 59 options) for each call. It examines several aspects of the calls distribution like age and gender of the user, date and time of the call and final referral, among others and presents comparative results for alternative classification models (SVM and CNN) and different data samples (three months, one and two years data models). For the task of selecting the appropriate pathway, the models, learned on the basis of the available data, achieved F1 values that range between 0.642 (3 months CNN model) and 0.783 (2 years CNN model), with SVM having a more stable performance (between 0.743 and 0.768 for the corresponding data samples). These results are discussed regarding error analysis and possibilities for explaining the system decisions. A final meta evaluation, based on a clinical expert overview, compares the different choices: the nurse attendants (reference ground truth), the expert and the automatic decisions (2 models), revealing a higher agreement between the ML models, followed by their agreement with the clinical expert, and minor agreement with the reference.