Deep learning with language models improves named entity recognition for PharmaCoNER

• Published in: BMC bioinformatics Vol. 22; no. Suppl 1; p. 602
• Main Authors: Sun, Cong, Yang, Zhihao, Wang, Lei, Zhang, Yin, Lin, Hongfei, Wang, Jian
• Format: Journal Article
• Language: English
• Published: England BioMed Central Ltd 17-12-2021; BioMed Central; BMC
• Subjects: Analysis; BERT; Computational linguistics; Data mining; Datasets; Deep Learning; Domain names; Domains; Drug discovery; Knowledge representation; Language; Language model; Language processing; Methods; Multilingualism; Named entity recognition; Natural language interfaces; Natural language processing; NER; Neural networks; Pharmacology; Proteins; Recognition; Semantics; Text mining; Texts; China; Text mining; BERT; NER; Language model; Named entity recognition
• ISSN: 1471-2105; 1471-2105
• DOI: 10.1186/s12859-021-04260-y

The recognition of pharmacological substances, compounds and proteins is essential for biomedical relation extraction, knowledge graph construction, drug discovery, as well as medical question answering. Although considerable efforts have been made to recognize biomedical entities in English texts, to date, only few limited attempts were made to recognize them from biomedical texts in other languages. PharmaCoNER is a named entity recognition challenge to recognize pharmacological entities from Spanish texts. Because there are currently abundant resources in the field of natural language processing, how to leverage these resources to the PharmaCoNER challenge is a meaningful study.
Inspired by the success of deep learning with language models, we compare and explore various representative BERT models to promote the development of the PharmaCoNER task.
The experimental results show that deep learning with language models can effectively improve model performance on the PharmaCoNER dataset. Our method achieves state-of-the-art performance on the PharmaCoNER dataset, with a max F1-score of 92.01%.
For the BERT models on the PharmaCoNER dataset, biomedical domain knowledge has a greater impact on model performance than the native language (i.e., Spanish). The BERT models can obtain competitive performance by using WordPiece to alleviate the out of vocabulary limitation. The performance on the BERT model can be further improved by constructing a specific vocabulary based on domain knowledge. Moreover, the character case also has a certain impact on model performance.