Real-time Informatized caption enhancement based on speaker pronunciation time database

• Published in: Multimedia tools and applications Vol. 79; no. 47-48; pp. 35667 - 35688
• Main Authors: Choi, Yong-Sik, Kang, Jin-Gu, Joo, Jong Wha J., Jung, Jin-Woo
• Format: Journal Article
• Language: English
• Published: New York Springer US 01-12-2020; Springer Nature B.V
• Subjects: Algorithms; Animation; Artificial intelligence; Computer Communication Networks; Computer Science; Data Structures and Information Theory; Multimedia; Multimedia Information Systems; Music; Natural language processing; Real time; Sound effects; Special Purpose and Application-Based Systems; Special sound effects; Speech; Speech recognition; Voice recognition; Speaker pronunciation time; Informatized caption; IBM Watson API; Speech to text translation
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-020-09590-2

IBM Watson is one of the representative tools for speech recognition system which can automatically generate not only speech-to-text information but also speaker ID and timing information, which is called as Informatized Caption. However, if there is some noise in the voice signal to the IBM Watson API, the recognition performance is significantly decreased. It can be easily found in movies with background music and special sound effects. This paper aims to improve the inaccuracy problem of current Informatized Captions in noisy environments. In this paper, a method of modifying incorrectly recognized words and a method of enhancing timing accuracy while updating database in real time are suggested based on the original caption and Informatized Caption information. Experimental results shows that the proposed method can give 81.09% timing accuracy for the case of 10 representative animation, horror and action movies.