Moderators of the internal consistency of error‐related negativity scores: A meta‐analysis of internal consistency estimates

• Published in: Psychophysiology Vol. 57; no. 8; pp. e13583 - n/a
• Main Author: Clayson, Peter E.
• Format: Journal Article
• Language: English
• Published: United States Blackwell Publishing Ltd 01-08-2020
• Subjects: Clinical trials; error‐related negativity (ERN); Event-related potentials; event‐related potentials (ERPs); internal consistency; Meta-analysis; psychometrics; reliability generalization; Systematic review; internal consistency; meta-analysis; psychometrics; reliability generalization; error-related negativity (ERN); event-related potentials (ERPs)
• ISSN: 0048-5772; 1469-8986; 1540-5958
• DOI: 10.1111/psyp.13583

To ensure adequate reliability (i.e., internal consistency), it is common in studies using event‐related brain potentials (ERPs) to exclude participants for having too few trials. This practice is particularly relevant for error‐related ERPs, such as error‐related negativity (ERN), where the number of recorded ERN trials is not entirely under the researcher's control. Furthermore, there is a widespread practice of inferring reliability based on published psychometric research, which assumes that internal consistency is a universal property of ERN. The present, preregistered reliability generalization study examined whether there is heterogeneity in internal consistency estimates of ERN scores and whether contextual factors moderate reliability. A total of 189 internal consistency estimates from 68 samples nested within 43 studies (n = 4,499 total participants) were analyzed. There was substantial heterogeneity in ERN score internal consistency, which was partially moderated by the type of paradigm (e.g., Stroop, flanker), the clinical status of the sample, the ocular artifact correction procedure, measurement sensors (single versus cluster), and the approach to scoring and estimating reliability, suggesting that contextual factors impact internal consistency at the individual study level. Age, sex, year of publication, artifact rejection procedure, acquisition system, sample type (undergraduate versus community), and length of mean amplitude window did not significantly moderate reliability. Notably, the overall estimated reliability of ERN scores was below established standards. Recommendations for improving ERN score reliability are provided, but the routine failure of most ERN studies to report internal consistency represents a substantial barrier to understanding the factors that impact reliability.
The majority of ERP research infers internal consistency based on published psychometric information. This reliability generalization study calls this practice into question by examining moderators of error‐related negativity (ERN) reliability estimates. Substantial heterogeneity in reliability was observed, and these findings highlight the importance of examining the reliability on a study‐by‐study basis.