Replicating a Meta-Analysis: The Search for the Optimal Word Choice Test Cutoff Continues

• Published in: Assessment (Odessa, Fla.) Vol. 30; no. 8; pp. 2476 - 2490
• Main Authors: Tyson, Bradley T., Shahein, Ayman, Abeare, Christopher A., Baker, Shannon D., Kent, Katrina, Roth, Robert M., Erdodi, Laszlo A.
• Format: Journal Article
• Language: English
• Published: Los Angeles, CA SAGE Publications 01-12-2023
• Subjects: performance validity; Word Choice Test; optimal cutoffs
• ISSN: 1073-1911; 1552-3489
• DOI: 10.1177/10731911221147043

This study was designed to expand on a recent meta-analysis that identified ≤42 as the optimal cutoff on the Word Choice Test (WCT). We examined the base rate of failure and the classification accuracy of various WCT cutoffs in four independent clinical samples (N = 252) against various psychometrically defined criterion groups. WCT ≤ 47 achieved acceptable combinations of specificity (.86–.89) at .49 to .54 sensitivity. Lowering the cutoff to ≤45 improved specificity (.91–.98) at a reasonable cost to sensitivity (.39–.50). Making the cutoff even more conservative (≤42) disproportionately sacrificed sensitivity (.30–.38) for specificity (.98–1.00), while still classifying 26.7% of patients with genuine and severe deficits as non-credible. Critical item (.23–.45 sensitivity at .89–1.00 specificity) and time-to-completion cutoffs (.48–.71 sensitivity at .87–.96 specificity) were effective alternative/complementary detection methods. Although WCT ≤ 45 produced the best overall classification accuracy, scores in the 43 to 47 range provide comparable objective psychometric evidence of non-credible responding. Results question the need for designating a single cutoff as “optimal,” given the heterogeneity of signal detection environments in which individual assessors operate. As meta-analyses often fail to replicate, ongoing research is needed on the classification accuracy of various WCT cutoffs.