Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA

• Published in: PloS one Vol. 18; no. 3; p. e0279720
• Main Authors: Douglas, Benjamin D, Ewell, Patrick J, Brauer, Markus
• Format: Journal Article
• Language: English
• Published: United States Public Library of Science 14-03-2023; Public Library of Science (PLoS)
• Subjects: Biology and Life Sciences; Comparative analysis; Coronaviruses; COVID-19; Data Accuracy; Data collection; Data entry; Demographics; Humans; Information management; Information services; Medical research; Online health care information services; Online information services; Online services; Pandemics; People and places; Personality; Polls & surveys; Population; Quality management; Research and Analysis Methods; Research Subjects; Researchers; Social Sciences; Surveys; Surveys and Questionnaires; United States; United States > US
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0279720

With the proliferation of online data collection in human-subjects research, concerns have been raised over the presence of inattentive survey participants and non-human respondents (bots). We compared the quality of the data collected through five commonly used platforms. Data quality was indicated by the percentage of participants who meaningfully respond to the researcher's question (high quality) versus those who only contribute noise (low quality). We found that compared to MTurk, Qualtrics, or an undergraduate student sample (i.e., SONA), participants on Prolific and CloudResearch were more likely to pass various attention checks, provide meaningful answers, follow instructions, remember previously presented information, have a unique IP address and geolocation, and work slowly enough to be able to read all the items. We divided the samples into high- and low-quality respondents and computed the cost we paid per high-quality respondent. Prolific ($1.90) and CloudResearch ($2.00) were cheaper than MTurk ($4.36) and Qualtrics ($8.17). SONA cost $0.00, yet took the longest to collect the data.