Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation

• Main Authors: Toral, Antonio, Castilho, Sheila, Hu, Ke, Way, Andy
• Format: Journal Article
• Language: English
• Published: 30-08-2018
• Subjects: Computer Science - Computation and Language
• DOI: 10.48550/arxiv.1808.10432

We reassess a recent study (Hassan et al., 2018) that claimed that machine
translation (MT) has reached human parity for the translation of news from
Chinese into English, using pairwise ranking and considering three variables
that were not taken into account in that previous study: the language in which
the source side of the test set was originally written, the translation
proficiency of the evaluators, and the provision of inter-sentential context.
If we consider only original source text (i.e. not translated from another
language, or translationese), then we find evidence showing that human parity
has not been achieved. We compare the judgments of professional translators
against those of non-experts and discover that those of the experts result in
higher inter-annotator agreement and better discrimination between human and
machine translations. In addition, we analyse the human translations of the
test set and identify important translation issues. Finally, based on these
findings, we provide a set of recommendations for future human evaluations of
MT.