Assessing the research landscape and clinical utility of large language models: a scoping review

• Published in: BMC medical informatics and decision making Vol. 24; no. 1; pp. 72 - 14
• Main Authors: Park, Ye-Jean, Pillai, Abhinav, Deng, Jiawen, Guo, Eddie, Gupta, Mehul, Paget, Mike, Naugler, Christopher
• Format: Journal Article
• Language: English
• Published: England BioMed Central Ltd 12-03-2024; BioMed Central; BMC
• Subjects: Bone surgery; Case reports; Chatbots; ChatGPT; Clinical Decision-Making; Clinical settings; Computational linguistics; Decision making; Ethical aspects; Ethics; Evidence-Based Medicine; Health care; Health Facilities; Humans; Language; Language processing; Large language models; Medicine; MEDLINE; Natural language interfaces; Natural language processing; Patients; Preprints; Privacy; Scoping review; Socioeconomics; Scoping review; Natural language processing; Large language models; ChatGPT; Clinical settings
• ISSN: 1472-6947; 1472-6947
• DOI: 10.1186/s12911-024-02459-6

Large language models (LLMs) like OpenAI's ChatGPT are powerful generative systems that rapidly synthesize natural language responses. Research on LLMs has revealed their potential and pitfalls, especially in clinical settings. However, the evolving landscape of LLM research in medicine has left several gaps regarding their evaluation, application, and evidence base.
This scoping review aims to (1) summarize current research evidence on the accuracy and efficacy of LLMs in medical applications, (2) discuss the ethical, legal, logistical, and socioeconomic implications of LLM use in clinical settings, (3) explore barriers and facilitators to LLM implementation in healthcare, (4) propose a standardized evaluation framework for assessing LLMs' clinical utility, and (5) identify evidence gaps and propose future research directions for LLMs in clinical applications.
We screened 4,036 records from MEDLINE, EMBASE, CINAHL, medRxiv, bioRxiv, and arXiv from January 2023 (inception of the search) to June 26, 2023 for English-language papers and analyzed findings from 55 worldwide studies. Quality of evidence was reported based on the Oxford Centre for Evidence-based Medicine recommendations.
Our results demonstrate that LLMs show promise in compiling patient notes, assisting patients in navigating the healthcare system, and to some extent, supporting clinical decision-making when combined with human oversight. However, their utilization is limited by biases in training data that may harm patients, the generation of inaccurate but convincing information, and ethical, legal, socioeconomic, and privacy concerns. We also identified a lack of standardized methods for evaluating LLMs' effectiveness and feasibility.
This review thus highlights potential future directions and questions to address these limitations and to further explore LLMs' potential in enhancing healthcare delivery.