Modeling Future Cost for Neural Machine Translation

• Published in: IEEE/ACM transactions on audio, speech, and language processing Vol. 29; pp. 770 - 781
• Main Authors: Duan, Chaoqun, Chen, Kehai, Wang, Rui, Utiyama, Masao, Sumita, Eiichiro, Zhu, Conghui, Zhao, Tiejun
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptation models; Computational modeling; Context modeling; Future cost; future cost representation; Machine translation; Maximum likelihood decoding; neural machine translation; Neural networks; Representations; Speech processing; transformer
• ISSN: 2329-9290; 2329-9304
• DOI: 10.1109/TASLP.2020.3042006

Existing neural machine translation (NMT) systems utilize sequence-to-sequence neural networks to generate target translation word by word, and then make the generated word at each time-step and the counterpart in the references as consistent as possible. However, the trained translation model tends to focus on ensuring the accuracy of the generated target word at the current time-step and does not consider its future cost which means the expected cost of generating the subsequent target translation (i.e., the next target word). To respond to this issue, in this article, we propose a simple and effective method to model the future cost of each target word for NMT systems. In detail, a future cost representation is learned based on the current generated target word and its contextual information to compute an additional loss to guide the training of the NMT model. Furthermore, the learned future cost representation at the current time-step is used to help the generation of the next target word in the decoding. Experimental results on three widely-used translation datasets, including the WMT14 English-to-German, WMT14 English-to-French, and WMT17 Chinese-to-English, show that the proposed approach achieves significant improvements over strong Transformer-based NMT baseline.