A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001–2010

• Published in: Lingua Vol. 134; pp. 219 - 248
• Main Authors: Sprouse, Jon, Schütze, Carson T., Almeida, Diogo
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier B.V 01-09-2013; Elsevier
• Subjects: Acceptability judgments; Descriptive studies and applied theories; Experimental syntax; Grammaticality judgments; Linguistics; Methodology; Syntax; Experimental syntax; Acceptability judgments; Methodology; Grammaticality judgments; English; Grammaticality judgement; Acceptability judgement; Error; Syntax; Linguistic survey; Convergence
• ISSN: 0024-3841; 1872-6135
• DOI: 10.1016/j.lingua.2013.07.002

•We compare the results of informal and formal acceptability judgment methods.•We test 296 sentence types randomly sampled from Linguistic Inquiry.•936 participants, 3 tasks: magnitude estimation, Likert scale, or forced choice.•We find 95% (±5%) convergence between the results of the two methods.•We explore the consequences of this convergence for syntactic methodology.
The goal of the present study is to provide a direct comparison of the results of informal judgment collection methods with the results of formal judgment collection methods, as a first step in understanding the relative merits of each family of methods. Although previous studies have compared small samples of informal and formal results, this article presents the first large-scale comparison based on a random sample of phenomena from a leading theoretical journal (Linguistic Inquiry). We tested 296 data points from the approximately 1743 English data points that were published in Linguistic Inquiry between 2001 and 2010. We tested this sample with 936 naïve participants using three formal judgment tasks (magnitude estimation, 7-point Likert scale, and two-alternative forced-choice) and report five statistical analyses. The results suggest a convergence rate of 95% between informal and formal methods, with a margin of error of 5.3–5.8%. We discuss the implications of this convergence rate for the ongoing conversation about judgment collection methods, and lay out a set of questions for future research into syntactic methodology.