Artificial intelligence-based data extraction for next generation risk assessment: Is fine-tuning of a large language model worth the effort?

• Published in: Toxicology (Amsterdam) Vol. 508; p. 153933
• Main Authors: Sonnenburg, Anna, van der Lugt, Benthe, Rehn, Johannes, Wittkowski, Paul, Bech, Karsten, Padberg, Florian, Eleftheriadou, Dimitra, Dobrikov, Todor, Bouwmeester, Hans, Mereu, Carla, Graf, Ferdinand, Kneuer, Carsten, Kramer, Nynke I., Blümmel, Tilmann
• Format: Journal Article
• Language: English
• Published: Ireland Elsevier B.V 01-11-2024
• Subjects: Artificial Intelligence; Automated data extraction; Benzhydryl Compounds - toxicity; Data Mining - methods; Fine-tuning; Humans; Large Language models; Phenols - toxicity; Risk Assessment; Risk Assessment - methods; Systematic literature review; Risk Assessment; Large Language models; Automated data extraction; AI; GPT; LLM; AI4NAMS; EFSA; BPA; OECD; ECHA; SLR; RAG; SVHC; NER; OHT; NAMs; PMID; Artificial intelligence; Fine-tuning; 3R; Systematic literature review
• ISSN: 0300-483X; 1879-3185; 1879-3185
• DOI: 10.1016/j.tox.2024.153933

To underpin scientific evaluations of chemical risks, agencies such as the European Food Safety Authority (EFSA) heavily rely on the outcome of systematic reviews, which currently require extensive manual effort. One specific challenge constitutes the meaningful use of vast amounts of valuable data from new approach methodologies (NAMs) which are mostly reported in an unstructured way in the scientific literature. In the EFSA-initiated project ‘AI4NAMS’, the potential of large language models (LLMs) was explored. Models from the GPT family, where GPT refers to Generative Pre-trained Transformer, were used for searching, extracting, and integrating data from scientific publications for NAM-based risk assessment. A case study on bisphenol A (BPA), a substance of very high concern due to its adverse effects on human health, focused on the structured extraction of information on test systems measuring biologic activities of BPA. Fine-tuning of a GPT-3 model (Curie base model) for extraction tasks was tested and the performance of the fine-tuned model was compared to the performance of a ready-to-use model (text-davinci-002). To update findings from the AI4NAMS project and to check for technical progress, the fine-tuning exercise was repeated and a newer ready-to-use model (text-davinci-003) served as comparison. In both cases, the fine-tuned Curie model was found to be superior to the ready-to-use model. Performance improvement was also obvious between text-davinci-002 and the newer text-davinci-003. Our findings demonstrate how fine-tuning and the swift general technical development improve model performance and contribute to the growing number of investigations on the use of AI in scientific and regulatory tasks.
[Display omitted]
•Fine-tuning improves large language model performance for text-extraction tasks.•Swift technical development enhances ready-to-use model performance.•Task complexity is the main limiting factor for model performance.•Proper prompt engineering is essential for successful use of large language models.