E-MFNN: an emotion-multimodal fusion neural network framework for emotion recognition

Emotional recognition is a pivotal research domain in computer and cognitive science. Recent advancements have led to various emotion recognition methods, leveraging data from diverse sources like speech, facial expressions, electroencephalogram (EEG), electrocardiogram, and eye tracking (ET). This...

Full description

Saved in:

Bibliographic Details
Published in:	PeerJ. Computer science Vol. 10; p. e1977
Main Authors:	Guo, Zhuen, Yang, Mingqing, Lin, Li, Li, Jisong, Zhang, Shuyue, He, Qianbo, Gao, Jiaqi, Meng, Heling, Chen, Xinran, Tao, Yuehao, Yang, Chen
Format:	Journal Article
Language:	English
Published:	United States PeerJ. Ltd 19-04-2024 PeerJ Inc
Subjects:	Artificial Intelligence Computational linguistics Data Science Deep learning EEG Electrocardiogram Electrocardiography Electroencephalography Emotion recognition Human-Computer Interaction Language processing Multimodal Natural language interfaces Neural networks Sentiment Analysis Deep learning Emotion recognition Multimodal EEG ET
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Description
Summary:	Emotional recognition is a pivotal research domain in computer and cognitive science. Recent advancements have led to various emotion recognition methods, leveraging data from diverse sources like speech, facial expressions, electroencephalogram (EEG), electrocardiogram, and eye tracking (ET). This article introduces a novel emotion recognition framework, primarily targeting the analysis of users' psychological reactions and stimuli. It is important to note that the stimuli eliciting emotional responses are as critical as the responses themselves. Hence, our approach synergizes stimulus data with physical and physiological signals, pioneering a multimodal method for emotional cognition. Our proposed framework unites stimulus source data with physiological signals, aiming to enhance the accuracy and robustness of emotion recognition through data integration. We initiated an emotional cognition experiment to gather EEG and ET data alongside recording emotional responses. Building on this, we developed the Emotion-Multimodal Fusion Neural Network (E-MFNN), optimized for multimodal data fusion to process both stimulus and physiological data. We conducted extensive comparisons between our framework's outcomes and those from existing models, also assessing various algorithmic approaches within our framework. This comparison underscores our framework's efficacy in multimodal emotion recognition. The source code is publicly available at https://figshare.com/s/8833d837871c78542b29.
Bibliography:	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23
ISSN:	2376-5992 2376-5992
DOI:	10.7717/peerj-cs.1977