Analyzing Test-Taking Behavior: Decision Theory Meets Psychometric Theory

• Published in: Psychometrika Vol. 80; no. 4; pp. 1105 - 1122
• Main Authors: Budescu, David V., Bo, Yuanchao
• Format: Journal Article
• Language: English
• Published: New York Springer US 01-12-2015; Springer Nature B.V
• Subjects: Assessment; Behavioral Science and Psychology; Decision Theory; Educational Measurement - methods; Expected values; Feedback (Response); Grading; Guessing (Tests); Humanities; Humans; Item Response Theory; Knowledge; Law; Multiple Choice Tests; Probability; Psychology; Psychometrics; Quantitative psychology; Resistance (Psychology); Risk-Taking; Scoring; Statistical Theory and Methods; Statistics for Social Sciences; Test Taking Skills - psychology; Testing and Evaluation; mis-calibration of probabilities; formula scoring; decision theory; partial information; loss aversion; multiple-choice tests; guessing
• ISSN: 0033-3123; 1860-0980
• DOI: 10.1007/s11336-014-9425-x

We investigate the implications of penalizing incorrect answers to multiple-choice tests, from the perspective of both test-takers and test-makers. To do so, we use a model that combines a well-known item response theory model with prospect theory (Kahneman and Tversky, Prospect theory: An analysis of decision under risk, Econometrica 47:263–91,
1979
). Our results reveal that when test-takers are fully informed of the scoring rule, the use of any penalty has detrimental effects for both test-takers (they are always penalized in excess, particularly those who are risk averse and loss averse) and test-makers (the bias of the estimated scores, as well as the variance and skewness of their distribution, increase as a function of the severity of the penalty).