A Missing Data Approach to Correct for Direct and Indirect Range Restrictions with a Dichotomous Criterion: A Simulation Study

• Published in: PloS one Vol. 11; no. 3; p. e0152330
• Main Authors: Pfaffel, Andreas, Kollmayer, Marlene, Schober, Barbara, Spiel, Christiane
• Format: Journal Article
• Language: English
• Published: United States Public Library of Science 28-03-2016; Public Library of Science (PLoS)
• Subjects: Analysis; Biology and Life Sciences; Chains; Computer Simulation; Continuity (mathematics); Criteria; Datasets; Entrance examinations; Estimation; Evaluation; Experimental design; Generalized linear models; Higher education; Mathematical models; Methods; Missing data; Models, Statistical; Monte Carlo Method; Monte Carlo methods; Normal distribution; Personnel selection; Physical sciences; Population; Population statistics; Quantitative psychology; Research and analysis methods; Scientific research; Selection bias; Social Sciences; Studies; Success; Test validity; Values; Variables; New York; United States > US; Austria
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0152330

A recurring methodological problem in the evaluation of the predictive validity of selection methods is that the values of the criterion variable are available for selected applicants only. This so-called range restriction problem causes biased population estimates. Correction methods for direct and indirect range restriction scenarios have widely studied for continuous criterion variables but not for dichotomous ones. The few existing approaches are inapplicable because they do not consider the unknown base rate of success. Hence, there is a lack of scientific research on suitable correction methods and the systematic analysis of their accuracies in the cases of a naturally or artificially dichotomous criterion. We aim to overcome this deficiency by viewing the range restriction problem as a missing data mechanism. We used multiple imputation by chained equations to generate complete criterion data before estimating the predictive validity and the base rate of success. Monte Carlo simulations were conducted to investigate the accuracy of the proposed correction in dependence of selection ratio, predictive validity, and base rate of success in an experimental design. In addition, we compared our proposed missing data approach with Thorndike's well-known correction formulas that have only been used in the case of continuous criterion variables so far. The results show that the missing data approach is more accurate in estimating the predictive validity than Thorndike's correction formulas. The accuracy of our proposed correction increases as the selection ratio and the correlation between predictor and criterion increase. Furthermore, the missing data approach provides a valid estimate of the unknown base rate of success. On the basis of our findings, we argue for the use of multiple imputation by chained equations in the evaluation of the predictive validity of selection methods when the criterion is dichotomous.