Evaluation of validity and reliability of AI Chatbots as public sources of information on dental trauma

This study aimed to assess the validity and reliability of AI chatbots, including Bing, ChatGPT 3.5, Google Gemini, and Claude AI, in addressing frequently asked questions (FAQs) related to dental trauma. A set of 30 FAQs was initially formulated by collecting responses from four AI chatbots. A pane...

Full description

Saved in:

Bibliographic Details
Published in:	Dental traumatology
Main Authors:	Johnson, Ashish J, Singh, Tarun Kumar, Gupta, Aakash, Sankar, Hariram, Gill, Ikroop, Shalini, Madhav, Mohan, Neeraj
Format:	Journal Article
Language:	English
Published:	Denmark 17-10-2024
Subjects:	Claude AI ChatGPT Artificial Intelligence Google Gemini Dental trauma Traumatic dental injury AI chatbots Bing
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Description
Summary:	This study aimed to assess the validity and reliability of AI chatbots, including Bing, ChatGPT 3.5, Google Gemini, and Claude AI, in addressing frequently asked questions (FAQs) related to dental trauma. A set of 30 FAQs was initially formulated by collecting responses from four AI chatbots. A panel comprising expert endodontists and maxillofacial surgeons then refined these to a final selection of 20 questions. Each question was entered into each chatbot three times, generating a total of 240 responses. These responses were evaluated using the Global Quality Score (GQS) on a 5-point Likert scale (5: strongly agree; 4: agree; 3: neutral; 2: disagree; 1: strongly disagree). Any disagreements in scoring were resolved through evidence-based discussions. The validity of the responses was determined by categorizing them as valid or invalid based on two thresholds: a low threshold (scores of ≥ 4 for all three responses) and a high threshold (scores of 5 for all three responses). A chi-squared test was used to compare the validity of the responses between the chatbots. Cronbach's alpha was calculated to assess the reliability by evaluating the consistency of repeated responses from each chatbot. The results indicate that the Claude AI chatbot demonstrated superior validity and reliability compared to ChatGPT and Google Gemini, whereas Bing was found to be less reliable. These findings underscore the need for authorities to establish strict guidelines to ensure the accuracy of medical information provided by AI chatbots.
Bibliography:	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23
ISSN:	1600-4469 1600-9657 1600-9657
DOI:	10.1111/edt.13000