Efficacy and safety of artificial intelligence-based large language models for decision making support in herniology: evaluation by experts and general surgeons

To evaluate the quality of recommendations provided by ChatGPT regarding inguinal hernia repair. ChatGPT was asked 5 questions about surgical management of inguinal hernias. The chat-bot was assigned the role of expert in herniology and requested to search only specialized medical databases and prov...

Full description

Saved in:

Bibliographic Details
Published in:	Hirurgija (Moskva) no. 8; p. 6
Main Authors:	Nechay, T V, Sazhin, A V, Loban, K M, Bogomolova, A K, Suglob, V V, Beniia, T R
Format:	Journal Article
Language:	Russian
Published:	Russia (Federation) 2024
Subjects:	Artificial Intelligence Clinical Decision-Making - methods Decision Support Systems, Clinical Hernia, Inguinal - surgery Herniorrhaphy - methods Humans Surgeons evidence level ChatGPT guidelines large language model artificial intelligence clinical decision making support tool inguinal hernia
Online Access:	Get more information
Tags:	Add Tag No Tags, Be the first to tag this record!

Description
Summary:	To evaluate the quality of recommendations provided by ChatGPT regarding inguinal hernia repair. ChatGPT was asked 5 questions about surgical management of inguinal hernias. The chat-bot was assigned the role of expert in herniology and requested to search only specialized medical databases and provide information about references and evidence. Herniology experts and surgeons (non-experts) rated the quality of recommendations generated by ChatGPT using 4-point scale (from 0 to 3 points). Statistical correlations were explored between participants' ratings and their stance regarding artificial intelligence. Experts scored the quality of ChatGPT responses lower than non-experts (2 (1-2) vs. 2 (2-3), <0.001). The chat-bot failed to provide valid references and actual evidence, as well as falsified half of references. Respondents were optimistic about the future of neural networks for clinical decision-making support. Most of them were against restricting their use in healthcare. We would not recommend non-specialized large language models as a single or primary source of information for clinical decision making or virtual searching assistant.
ISSN:	0023-1207
DOI:	10.17116/hirurgia20240816