Extracting Speaker and Emotion Information from Self-Supervised Speech Models via Channel-Wise Correlations

• Published in: 2022 IEEE Spoken Language Technology Workshop (SLT) pp. 1136 - 1143
• Main Authors: Stafylakis, Themos, Mosner, Ladislav, Kakouros, Sofoklis, Plchot, Oldrich, Burget, Lukas, Cernocky, Jan
• Format: Conference Proceeding
• Language: English
• Published: IEEE 09-01-2023
• Subjects: Codes; Conferences; Correlation; Data mining; Emotion recognition; Self-supervised learning; self-supervised models; Speaker identification; speaker verification; Speech recognition
• DOI: 10.1109/SLT54892.2023.10023345

Self-supervised learning of speech representations from large amounts of unlabeled data has enabled state-of-the-art results in several speech processing tasks. Aggregating these speech representations across time is typically approached by using descriptive statistics, and in particular, using the first - and second-order statistics of representation coefficients. In this paper, we examine an alternative way of extracting speaker and emotion information from self-supervised trained models, based on the correlations between the coefficients of the representations - correlation pooling. We show improvements over mean pooling and further gains when the pooling methods are combined via fusion. The code is available at github.com/Lamomal/s3prl_correlation.