Advancing Total Communication in SIBI: A Proposed Conceptual Framework for Sign Language Translation

Total communication in sign language involves hand gestures as manual signs and facial expressions as non-manual signs. Both types of signs have an important role in communication. Research related to sign language translators focuses more on manual signs. Ignoring the non-manual components causes t...

Full description

Saved in:

Bibliographic Details
Published in:	2023 International Conference on Smart-Green Technology in Electrical and Information Systems (ICSGTEIS) pp. 23 - 28
Main Authors:	Darmawan, I Dewa M. Bayu A., Linawati, Linawati, Sukadarmika, Gede, Wirastuti, Ni M. Ary Esta Dewi, Pulungan, Reza, Mulyanto, Hariyanti, Ni Kadek Dessy
Format:	Conference Proceeding
Language:	English
Published:	IEEE 02-11-2023
Subjects:	Assistive technologies computer vision deep learning Face recognition facial expressions Gesture recognition hand gestures Integrated optics Manuals SIBI sign language translation Time measurement total communication transformer Transformers
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Description
Summary:	Total communication in sign language involves hand gestures as manual signs and facial expressions as non-manual signs. Both types of signs have an important role in communication. Research related to sign language translators focuses more on manual signs. Ignoring the non-manual components causes the translation of spoken language to be less precise because of the loss of information from the signer's facial emotions. Indonesian Sign Language (SIBI) is different from other sign languages. SIBI follows the rules of the Indonesian language, which consist of structure and syntax. This paper proposes a framework for sign language translation composed of manual and non-manual components. Two propositions will be tested and contribute to this research. The first proposition is related to the use of sign gesture detection and segmentation methods using optical flow-based methods, and the second proposition is associated with the use of STMC, Facial Expression Recognition, and Transformer methods at the stage of recognition and translation of sign language. The proposed work arrangement is expected to improve performance measured by WER, BLEU, and METEOR and reduce computational load as measured by processing time.
ISSN:	2831-400X
DOI:	10.1109/ICSGTEIS60500.2023.10424020