A Multimodal Framework for the Assessment of the Schizophrenia Spectrum

• Main Authors: Premananth, Gowtham, Siriwardena, Yashish M, Resnik, Philip, Bansal, Sonia, Kelly, Deanna L, Espy-Wilson, Carol
• Format: Journal Article
• Language: English
• Published: 14-06-2024
• DOI: 10.48550/arxiv.2406.09706

This paper presents a novel multimodal framework to distinguish between
different symptom classes of subjects in the schizophrenia spectrum and healthy
controls using audio, video, and text modalities. We implemented Convolution
Neural Network and Long Short Term Memory based unimodal models and
experimented on various multimodal fusion approaches to come up with the
proposed framework. We utilized a minimal Gated multimodal unit (mGMU) to
obtain a bi-modal intermediate fusion of the features extracted from the input
modalities before finally fusing the outputs of the bimodal fusions to perform
subject-wise classifications. The use of mGMU units in the multimodal framework
improved the performance in both weighted f1-score and weighted AUC-ROC scores.