Suitability of ChatGPT as a Source of Patient Information for Screening Mammography

• Published in: Health promotion practice p. 15248399241285060
• Main Authors: Spuur, Kelly, Currie, Geoff, Al-Mousa, Dana, Pape, Ruth
• Format: Journal Article
• Language: English
• Published: United States 11-10-2024
• Subjects: breast screening; readability; actionability; understandability; generative AI; mammography; ChatGPT; patient education; language model; artificial intelligence
• ISSN: 1524-8399; 1552-6372
• DOI: 10.1177/15248399241285060

ChatGPT3.5 and ChatGPT4 were released publicly in late November 2022 and March 2023, respectively, and have emerged as convenient sources of patient health education and information, including for screening mammography. ChatGPT4 offers enhanced capabilities; however, it is only available by paid subscription. The purported benefits of ChatGPT for health education need to be objectively evaluated. To assess performance differences, ChatGPT3.5 and GPT4 were used between 13 April and 29 May 2023 to generate breast screening patient information sheets, which were evaluated using the Patient Education Materials Assessment Tool for printed materials (PEMAT-P) and the CDC Clear Communication Index (CDC Index) Score Sheet; and benchmarked against gold standard content in BreastScreen NSW's patient information sheet. Mean scores were reported for comparison. GPT3.5 provided the appropriate tone and currency of information but lacked accuracy, omitting key insights: PEMAT-P understandability 68.0% (SD = 6.56) and actionability 36.7% (SD=20.4); CDC Index 58.8% (SD = 15.3). GPT4 was deemed superior to GPT3.5 but included several key omissions: PEMAT-P understandability 75.0% (SD = 17) and actionability 53.3% (SD = 11.54); CDC Index 66.0% (SD = 4.1). Both ChatGPT versions exhibited poor understandability and actionability and were unclear in their messaging. Those with poor health literacy will not benefit from accessing current versions of ChatGPT and may be further disadvantaged if they do not have access to a paid subscription. ChatGPT is evidenced to be an unreliable and inaccurate source of information concerning breast screening that may undermine participation and risk increased morbidity and mortality from breast cancer. ChatGPT may increase the demand on health care educators to rectify misinformation.