Standard experimental paradigm designs and data exclusion practices in cognitive psychology can inadvertently introduce systematic “shadow” biases in participant samples

Standard cognitive psychology research practices can introduce inadvertent sampling biases that reduce the reliability and generalizability of the findings. Researchers commonly acknowledge and understand that any given study sample is not perfectly generalizable, especially when implementing typica...

Full description

Saved in:

Bibliographic Details
Published in:	Cognitive research: principles and implications Vol. 8; no. 1; p. 66
Main Authors:	Siritzky, Emma M., Cox, Patrick H., Nadler, Sydni M., Grady, Justin N., Kravitz, Dwight J., Mitroff, Stephen R.
Format:	Journal Article
Language:	English
Published:	Cham Springer International Publishing 21-10-2023 Springer Nature B.V Springer Science + Business Media SpringerOpen
Subjects:	Attention Behavioral Science and Psychology Bias Cognitive ability Cognitive Psychology Color vision Data exclusion Environmental effects Experimental Psychology Humans Individual differences Neurosciences Original Original Article Psychology Reproducibility of Results Research Design Sampling Sampling biases Sampling biases Data exclusion Cognitive psychology Individual differences
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Description
Summary:	Standard cognitive psychology research practices can introduce inadvertent sampling biases that reduce the reliability and generalizability of the findings. Researchers commonly acknowledge and understand that any given study sample is not perfectly generalizable, especially when implementing typical experimental constraints (e.g., limiting recruitment to specific age ranges or to individuals with normal color vision). However, less obvious systematic sampling constraints, referred to here as “shadow” biases, can be unintentionally introduced and can easily go unnoticed. For example, many standard cognitive psychology study designs involve lengthy and tedious experiments with simple, repetitive stimuli. Such testing environments may 1) be aversive to some would-be participants (e.g., those high in certain neurodivergent symptoms) who may self-select not to enroll in such studies, or 2) contribute to participant attrition, both of which reduce the sample’s representativeness. Likewise, standard performance-based data exclusion efforts (e.g., minimum accuracy or response time) or attention checks can systematically remove data from participants from subsets of the population (e.g., those low in conscientiousness). This commentary focuses on the theoretical and practical issues behind these non-obvious and often unacknowledged “shadow” biases, offers a simple illustration with real data as a proof of concept of how applying attention checks can systematically skew latent/hidden variables in the included population, and then discusses the broader implications with suggestions for how to manage and reduce, or at a minimum acknowledge, the problem.
Bibliography:	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23 USDOE
ISSN:	2365-7464 2365-7464
DOI:	10.1186/s41235-023-00520-y