Fine-Tuning Self-Supervised Learning Models for End-to-End Pronunciation Scoring

Automatic pronunciation assessment models are regularly used in language learning applications. Common methodologies for pronunciation assessment use feature-based approaches, such as the Goodness-of-Pronunciation (GOP) approach, or deep learning speech recognition models to perform speech assessmen...

Full description

Saved in:

Bibliographic Details
Published in:	IEEE access Vol. 11; pp. 112650 - 112663
Main Authors:	Zahran, Ahmed I., Fahmy, Aly A., Wassif, Khaled T., Bayomi, Hanaa
Format:	Journal Article
Language:	English
Published:	Piscataway IEEE 2023 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:	Automatic pronunciation assessment Computational modeling Correlation coefficients Deep learning Feature extraction Hidden Markov models HuBERT Machine learning Neural networks Phonemes pre-trained speech representations pronunciation scoring Representations Scoring models Self-supervised learning self-supervised speech representation learning Speech recognition Task analysis Training Transformers wav2vec 20 Waveforms WavLM
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Description
Summary:	Automatic pronunciation assessment models are regularly used in language learning applications. Common methodologies for pronunciation assessment use feature-based approaches, such as the Goodness-of-Pronunciation (GOP) approach, or deep learning speech recognition models to perform speech assessment. With the rise of transformers, pre-trained self-supervised learning (SSL) models have been utilized to extract contextual speech representations, showing improvements in various downstream tasks. In this study, we propose the end-to-end regressor (E2E-R) model for pronunciation scoring. E2E-R is trained using a two-step training process. In the first step, the pre-trained SSL model is fine-tuned on a phoneme recognition task to obtain better representations for the pronounced phonemes. In the second step, transfer learning is used to build a pronunciation scoring model that uses a Siamese neural network to compare the pronounced phoneme representations to embeddings of the canonical phonemes and produce the final pronunciation scores. E2E-R achieves a Pearson correlation coefficient (PCC) of 0.68, which is almost similar to the state-of-the-art GOPT-PAII model while eliminating the need for training on additional native speech data, feature engineering, or external forced alignment modules. To our knowledge, this work presents the first utilization of a pre-trained SSL model for end-to-end phoneme-level pronunciation scoring on raw speech waveforms. The code is available at https://github.com/ai-zahran/E2E-R .
ISSN:	2169-3536 2169-3536
DOI:	10.1109/ACCESS.2023.3317236