AFR-BERT: Attention-based mechanism feature relevance fusion multimodal sentiment analysis model

• Published in: PloS one Vol. 17; no. 9; p. e0273936
• Main Authors: Mingyu, Ji, Jiawei, Zhou, Ning, Wei
• Format: Journal Article
• Language: English
• Published: San Francisco Public Library of Science 09-09-2022; Public Library of Science (PLoS)
• Subjects: Ablation; Analysis; Audio data; Biology and Life Sciences; Coders; Cognition & reasoning; Computational linguistics; Computer and Information Sciences; Consumer preferences; Data analysis; Data integration; Data mining; Data processing; Deep learning; Emotion recognition; Emotions; Experiments; Feature extraction; Language; Language processing; Long short-term memory; Methods; Modelling; Natural language interfaces; Natural language processing; Neural networks; Performance measurement; Product reviews; Semantics; Sentiment analysis; Social Sciences; China
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0273936

Multimodal sentiment analysis is an essential task in natural language processing which refers to the fact that machines can analyze and recognize emotions through logical reasoning and mathematical operations after learning multimodal emotional features. For the problem of how to consider the effective fusion of multimodal data and the relevance of multimodal data in multimodal sentiment analysis, we propose an attention-based mechanism feature relevance fusion multimodal sentiment analysis model (AFR-BERT). In the data pre-processing stage, text features are extracted using the pre-trained language model BERT (Bi-directional Encoder Representation from Transformers), and the BiLSTM (Bi-directional Long Short-Term Memory) is used to obtain the internal information of the audio. In the data fusion phase, the multimodal data fusion network effectively fuses multimodal features through the interaction of text and audio information. During the data analysis phase, the multimodal data association network analyzes the data by exploring the correlation of fused information between text and audio. In the data output phase, the model outputs the results of multimodal sentiment analysis. We conducted extensive comparative experiments on the publicly available sentiment analysis datasets CMU-MOSI and CMU-MOSEI. The experimental results show that AFR-BERT improves on the classical multimodal sentiment analysis model in terms of relevant performance metrics. In addition, ablation experiments and example analysis show that the multimodal data analysis network in AFR-BERT can effectively capture and analyze the sentiment features in text and audio.