SCORE: Self-supervised Correspondence Fine-tuning for Improved Content Representations

• Main Authors: Meghanani, Amit, Hain, Thomas
• Format: Journal Article
• Language: English
• Published: 10-03-2024
• Subjects: Computer Science - Computation and Language; Computer Science - Sound
• DOI: 10.48550/arxiv.2403.06260

There is a growing interest in cost-effective self-supervised fine-tuning
(SSFT) of self-supervised learning (SSL)-based speech models to obtain
task-specific representations. These task-specific representations are used for
robust performance on various downstream tasks by fine-tuning on the labelled
data. This work presents a cost-effective SSFT method named Self-supervised
Correspondence (SCORE) fine-tuning to adapt the SSL speech representations for
content-related tasks. The proposed method uses a correspondence training
strategy, aiming to learn similar representations from perturbed speech and
original speech. Commonly used data augmentation techniques for content-related
tasks (ASR) are applied to obtain perturbed speech. SCORE fine-tuned HuBERT
outperforms the vanilla HuBERT on SUPERB benchmark with only a few hours of
fine-tuning (< 5 hrs) on a single GPU for automatic speech recognition, phoneme
recognition, and query-by-example tasks, with relative improvements of 1.09%,
3.58%, and 12.65%, respectively. SCORE provides competitive results with the
recently proposed SSFT method SPIN, using only 1/3 of the processed speech
compared to SPIN.