A comparative study of pre-trained language models for named entity recognition in clinical trial eligibility criteria from multiple corpora

• Published in: BMC medical informatics and decision making Vol. 22; no. 1; pp. 1 - 235
• Main Authors: Li, Jianfu, Wei, Qiang, Ghiasvand, Omid, Chen, Miao, Lobanov, Victor, Weng, Chunhua, Xu, Hua
• Format: Journal Article
• Language: English
• Published: London BioMed Central Ltd 06-09-2022; BioMed Central; BMC
• Subjects: Alliances; Alzheimer's disease; Analysis; Annotations; Clinical trial; Clinical trials; Comparative studies; Computational linguistics; Criteria; Data augmentation; Datasets; Deep learning; Electronic health records; Electronic records; Eligibility criteria; Embedding; Feasibility; Health informatics; Language; Language processing; Machine learning; Management; Medical records; Medical science; Named entity recognition; Natural language; Natural language interfaces; Natural language processing; Performance evaluation; Pre-trained language model; Recognition; Transformers; Unstructured data; United States
• ISSN: 1472-6947; 1472-6947
• DOI: 10.1186/s12911-022-01967-7

Background Clinical trial protocols are the foundation for advancing medical sciences, however, the extraction of accurate and meaningful information from the original clinical trials is very challenging due to the complex and unstructured texts of such documents. Named entity recognition (NER) is a fundamental and necessary step to process and standardize the unstructured text in clinical trials using Natural Language Processing (NLP) techniques. Methods In this study we fine-tuned pre-trained language models to support the NER task on clinical trial eligibility criteria. We systematically investigated four pre-trained contextual embedding models for the biomedical domain (i.e., BioBERT, BlueBERT, PubMedBERT, and SciBERT) and two models for the open domains (BERT and SpanBERT), for NER tasks using three existing clinical trial eligibility criteria corpora. In addition, we also investigated the feasibility of data augmentation approaches and evaluated their performance. Results Our evaluation results using tenfold cross-validation show that domain-specific transformer models achieved better performance than the general transformer models, with the best performance obtained by the PubMedBERT model (F1-scores of 0.715, 0.836, and 0.622 for the three corpora respectively). The data augmentation results show that it is feasible to leverage additional corpora to improve NER performance. Conclusions Findings from this study not only demonstrate the importance of contextual embeddings trained from domain-specific corpora, but also shed lights on the benefits of leveraging multiple data sources for the challenging NER task in clinical trial eligibility criteria text. Keywords: Clinical trial, Eligibility criteria, Named entity recognition, Pre-trained language model