Evaluating MT for massive open online courses A multifaceted comparison between PBSMT and NMT systems

• Published in: Machine translation Vol. 32; no. 3; pp. 255 - 278
• Main Authors: Castilho, Sheila, Moorkens, Joss, Gaspari, Federico, Sennrich, Rico, Way, Andy, Georgakopoulou, Panayota
• Format: Journal Article
• Language: English
• Published: Dordrecht Springer Netherlands 01-09-2018
• Subjects: Artificial Intelligence; Computational Linguistics; Computer Science; Natural Language Processing (NLP); Statistical MT; Neural MT; Human MT evaluation; MOOCs
• ISSN: 0922-6567; 1573-0573
• DOI: 10.1007/s10590-018-9221-y

This article reports a multifaceted comparison between statistical and neural machine translation (MT) systems that were developed for translation of data from massive open online courses (MOOCs). The study uses four language pairs: English to German, Greek, Portuguese, and Russian. Translation quality is evaluated using automatic metrics and human evaluation, carried out by professional translators. Results show that neural MT is preferred in side-by-side ranking, and is found to contain fewer overall errors. Results are less clear-cut for some error categories, and for temporal and technical post-editing effort. In addition, results are reported based on sentence length, showing advantages and disadvantages depending on the particular language pair and MT paradigm.