Search Results - "van Miltenburg, Emiel"

Human evaluation of automatically generated text: Current trends and best practice guidelines by van der Lee, Chris, Gatt, Albert, van Miltenburg, Emiel, Krahmer, Emiel

“…•The current paper provides an overview of human evaluation practices in NLG.•The current paper gives an overview of the steps necessary to undertake a human…”
Get full text

Scalar Diversity by Van Tiel, Bob, Van Miltenburg, Emiel, Zevakhina, Natalia, Geurts, Bart

“…Abstract We present experimental evidence showing that there is considerable variation between the rates at which scalar expressions from different lexical…”
Get full text

Image captioning in different languages by van Miltenburg, Emiel

“…This short position paper provides a manually curated list of non-English image captioning datasets (as of May 2024). Through this list, we can observe the…”
Get full text

Evaluating NLG systems: A brief introduction by van Miltenburg, Emiel

“…This year the International Conference on Natural Language Generation (INLG) will feature an award for the paper with the best evaluation. The purpose of this…”
Get full text

On the use of human reference data for evaluating automatic image descriptions by van Miltenburg, Emiel

“…Automatic image description systems are commonly trained and evaluated using crowdsourced, human-generated image descriptions. The best-performing system is…”
Get full text

Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations by Braggaar, Anouck, Liebrecht, Christine, van Miltenburg, Emiel, Krahmer, Emiel

“…This review gives an extensive overview of evaluation methods for task-oriented dialogue systems, paying special attention to practical applications of…”
Get full text

Implicit causality in GPT-2: a case study by Huynh, Hien, Lentz, Tomas O, van Miltenburg, Emiel

“…This case study investigates the extent to which a language model (GPT-2) is able to capture native speakers' intuitions about implicit causality in a sentence…”
Get full text

Stereotyping and Bias in the Flickr30K Dataset by van Miltenburg, Emiel

“…An untested assumption behind the crowdsourced descriptions of the images in the Flickr30K dataset (Young et al., 2014) is that they "focus only on the…”
Get full text

Preregistering NLP Research by van Miltenburg, Emiel, van der Lee, Chris, Krahmer, Emiel

“…Preregistration refers to the practice of specifying what you are going to do, and what you expect to find in your study, before carrying out the study. This…”
Get full text

Room for improvement in automatic image description: an error analysis by van Miltenburg, Emiel, Elliott, Desmond

“…In recent years we have seen rapid and significant progress in automatic image description but what are the open problems in this area? Most work has been…”
Get full text

Detecting and ordering adjectival scalemates by van Miltenburg, Emiel

“…This paper presents a pattern-based method that can be used to infer adjectival scales, such as <lukewarm, warm, hot>, from a corpus. Specifically, the…”
Get full text

Cross-linguistic differences and similarities in image descriptions by van Miltenburg, Emiel, Elliott, Desmond, Vossen, Piek

“…Automatic image description systems are commonly trained and evaluated on large image description datasets. Recently, researchers have started to collect such…”
Get full text

Automatic Construction of Evaluation Suites for Natural Language Generation Datasets by Mille, Simon, Dhole, Kaustubh D, Mahamood, Saad, Perez-Beltrachini, Laura, Gangal, Varun, Kale, Mihir, van Miltenburg, Emiel, Gehrmann, Sebastian

“…Machine learning approaches applied to NLP are often evaluated by summarizing their performance in a single number, for example accuracy. Since most test sets…”
Get full text

Pragmatic factors in image description: the case of negations by van Miltenburg, Emiel, Morante, Roser, Elliott, Desmond

“…We provide a qualitative analysis of the descriptions containing negations (no, not, n't, nobody, etc) in the Flickr30K corpus, and a categorization of…”
Get full text

Underreporting of errors in NLG output, and what to do about it by van Miltenburg, Emiel, Clinciu, Miruna-Adriana, Dušek, Ondřej, Gkatzia, Dimitra, Inglis, Stephanie, Leppänen, Leo, Mahamood, Saad, Manning, Emma, Schoch, Stephanie, Thomson, Craig, Wen, Luou

“…We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an…”
Get full text

Neural data-to-text generation: A comparison between pipeline and end-to-end architectures by Ferreira, Thiago Castro, van der Lee, Chris, van Miltenburg, Emiel, Krahmer, Emiel

“…Traditionally, most data-to-text applications have been designed using a modular pipeline architecture, in which non-linguistic input data is converted into…”
Get full text

Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP by Belz, Anya, Thomson, Craig, Reiter, Ehud, Abercrombie, Gavin, Alonso-Moral, Jose M, Arvan, Mohammad, Braggaar, Anouck, Cieliebak, Mark, Clark, Elizabeth, van Deemter, Kees, Dinkar, Tanvi, Dušek, Ondřej, Eger, Steffen, Fang, Qixiang, Gao, Mingqi, Gatt, Albert, Gkatzia, Dimitra, González-Corbelle, Javier, Hovy, Dirk, Hürlimann, Manuela, Ito, Takumi, Kelleher, John D, Klubicka, Filip, Krahmer, Emiel, Lai, Huiyuan, van der Lee, Chris, Li, Yiru, Mahamood, Saad, Mieskes, Margot, van Miltenburg, Emiel, Mosteiro, Pablo, Nissim, Malvina, Parde, Natalie, Plátek, Ondřej, Rieser, Verena, Ruan, Jie, Tetreault, Joel, Toral, Antonio, Wan, Xiaojun, Wanner, Leo, Watson, Lewis, Yang, Diyi

“…We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human…”
Get full text

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics by Gehrmann, Sebastian, Adewumi, Tosin, Aggarwal, Karmanya, Ammanamanchi, Pawan Sasanka, Anuoluwapo, Aremu, Bosselut, Antoine, Chandu, Khyathi Raghavi, Clinciu, Miruna, Das, Dipanjan, Dhole, Kaustubh D, Du, Wanyu, Durmus, Esin, Dušek, Ondřej, Emezue, Chris, Gangal, Varun, Garbacea, Cristina, Hashimoto, Tatsunori, Hou, Yufang, Jernite, Yacine, Jhamtani, Harsh, Ji, Yangfeng, Jolly, Shailza, Kale, Mihir, Kumar, Dhruv, Ladhak, Faisal, Madaan, Aman, Maddela, Mounica, Mahajan, Khyati, Mahamood, Saad, Majumder, Bodhisattwa Prasad, Martins, Pedro Henrique, McMillan-Major, Angelina, Mille, Simon, van Miltenburg, Emiel, Nadeem, Moin, Narayan, Shashi, Nikolaev, Vitaly, Niyongabo, Rubungo Andre, Osei, Salomey, Parikh, Ankur, Perez-Beltrachini, Laura, Rao, Niranjan Ramesh, Raunak, Vikas, Rodriguez, Juan Diego, Santhanam, Sashank, Sedoc, João, Sellam, Thibault, Shaikh, Samira, Shimorina, Anastasia, Cabezudo, Marco Antonio Sobrevilla, Strobelt, Hendrik, Subramani, Nishant, Xu, Wei, Yang, Diyi, Yerukola, Akhila, Zhou, Jiawei

“…We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly…”
Get full text