Developing phoneme‐based lip‐reading sentences system for silent speech recognition

Developing phoneme‐based lip‐reading sentences system for silent speech recognition

Lip‐reading is a process of interpreting speech by visually analysing lip movements. Recent research in this area has shifted from simple word recognition to lip‐reading sentences in the wild. This paper attempts to use phonemes as a classification schema for lip‐reading sentences to explore an alte...

Full description

Saved in:
Bibliographic Details
Published in:CAAI Transactions on Intelligence Technology Vol. 8; no. 1; pp. 129 - 138
Main Authors: El‐Bialy, Randa, Chen, Daqing, Fenghour, Souheil, Hussein, Walid, Xiao, Perry, Karam, Omar H., Li, Bo
Format: Journal Article
Language:English
Published: Beijing John Wiley & Sons, Inc 01-03-2023
Wiley
Subjects:
Accuracy
Acoustics
Classification
Datasets
deep learning
deep neural networks
Dictionaries
Lip reading
Lipreading
Neural networks
Phonemes
phoneme‐based lip‐reading
Recurrent neural networks
Sentences
spatial‐temporal convolution
Speech
Speech recognition
transformers
Voice recognition
Words (language)
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Lip‐reading is a process of interpreting speech by visually analysing lip movements. Recent research in this area has shifted from simple word recognition to lip‐reading sentences in the wild. This paper attempts to use phonemes as a classification schema for lip‐reading sentences to explore an alternative schema and to enhance system performance. Different classification schemas have been investigated, including character‐based and visemes‐based schemas. The visual front‐end model of the system consists of a Spatial‐Temporal (3D) convolution followed by a 2D ResNet. Transformers utilise multi‐headed attention for phoneme recognition models. For the language model, a Recurrent Neural Network is used. The performance of the proposed system has been testified with the BBC Lip Reading Sentences 2 (LRS2) benchmark dataset. Compared with the state‐of‐the‐art approaches in lip‐reading sentences, the proposed system has demonstrated an improved performance by a 10% lower word error rate on average under varying illumination ratios.
ISSN:2468-2322
2468-2322
DOI:10.1049/cit2.12131