Efficacy and safety of artificial intelligence-based large language models for decision making support in herniology: evaluation by experts and general surgeons

• Published in: Hirurgija (Moskva) no. 8; p. 6
• Main Authors: Nechay, T V, Sazhin, A V, Loban, K M, Bogomolova, A K, Suglob, V V, Beniia, T R
• Format: Journal Article
• Language: Russian
• Published: Russia (Federation) 2024
• Subjects: Artificial Intelligence; Clinical Decision-Making - methods; Decision Support Systems, Clinical; Hernia, Inguinal - surgery; Herniorrhaphy - methods; Humans; Surgeons; evidence level; ChatGPT; guidelines; large language model; artificial intelligence; clinical decision making support tool; inguinal hernia
• ISSN: 0023-1207
• DOI: 10.17116/hirurgia20240816

To evaluate the quality of recommendations provided by ChatGPT regarding inguinal hernia repair.
ChatGPT was asked 5 questions about surgical management of inguinal hernias. The chat-bot was assigned the role of expert in herniology and requested to search only specialized medical databases and provide information about references and evidence. Herniology experts and surgeons (non-experts) rated the quality of recommendations generated by ChatGPT using 4-point scale (from 0 to 3 points). Statistical correlations were explored between participants' ratings and their stance regarding artificial intelligence.
Experts scored the quality of ChatGPT responses lower than non-experts (2 (1-2) vs. 2 (2-3),
<0.001). The chat-bot failed to provide valid references and actual evidence, as well as falsified half of references. Respondents were optimistic about the future of neural networks for clinical decision-making support. Most of them were against restricting their use in healthcare.
We would not recommend non-specialized large language models as a single or primary source of information for clinical decision making or virtual searching assistant.