Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP

• Main Authors: Belz, Anya, Thomson, Craig, Reiter, Ehud, Abercrombie, Gavin, Alonso-Moral, Jose M, Arvan, Mohammad, Braggaar, Anouck, Cieliebak, Mark, Clark, Elizabeth, van Deemter, Kees, Dinkar, Tanvi, Dušek, Ondřej, Eger, Steffen, Fang, Qixiang, Gao, Mingqi, Gatt, Albert, Gkatzia, Dimitra, González-Corbelle, Javier, Hovy, Dirk, Hürlimann, Manuela, Ito, Takumi, Kelleher, John D, Klubicka, Filip, Krahmer, Emiel, Lai, Huiyuan, van der Lee, Chris, Li, Yiru, Mahamood, Saad, Mieskes, Margot, van Miltenburg, Emiel, Mosteiro, Pablo, Nissim, Malvina, Parde, Natalie, Plátek, Ondřej, Rieser, Verena, Ruan, Jie, Tetreault, Joel, Toral, Antonio, Wan, Xiaojun, Wanner, Leo, Watson, Lewis, Yang, Diyi
• Format: Journal Article
• Language: English
• Published: 02-05-2023
• Subjects: Computer Science - Computation and Language
• DOI: 10.48550/arxiv.2305.01633

We report our efforts in identifying a set of previous human evaluations in
NLP that would be suitable for a coordinated study examining what makes human
evaluations in NLP more/less reproducible. We present our results and findings,
which include that just 13\% of papers had (i) sufficiently low barriers to
reproduction, and (ii) enough obtainable information, to be considered for
reproduction, and that all but one of the experiments we selected for
reproduction was discovered to have flaws that made the meaningfulness of
conducting a reproduction questionable. As a result, we had to change our
coordinated study design from a reproduce approach to a
standardise-then-reproduce-twice approach. Our overall (negative) finding that
the great majority of human evaluations in NLP is not repeatable and/or not
reproducible and/or too flawed to justify reproduction, paints a dire picture,
but presents an opportunity for a rethink about how to design and report human
evaluations in NLP.