Automating Text Naturalness Evaluation of NLG Systems

• Main Authors: Çano, Erion, Bojar, Ondřej
• Format: Journal Article
• Language: English
• Published: 23-06-2020
• Subjects: Computer Science - Computation and Language; Computer Science - Learning
• DOI: 10.48550/arxiv.2006.13268

Automatic methods and metrics that assess various quality criteria of
automatically generated texts are important for developing NLG systems because
they produce repeatable results and allow for a fast development cycle. We
present here an attempt to automate the evaluation of text naturalness which is
a very important characteristic of natural language generation methods. Instead
of relying on human participants for scoring or labeling the text samples, we
propose to automate the process by using a human likeliness metric we define
and a discrimination procedure based on large pretrained language models with
their probability distributions. We analyze the text probability fractions and
observe how they are influenced by the size of the generative and
discriminative models involved in the process. Based on our results, bigger
generators and larger pretrained discriminators are more appropriate for a
better evaluation of text naturalness. A comprehensive validation procedure
with human participants is required as follow up to check how well this
automatic evaluation scheme correlates with human judgments.