Rating the ratings: Assessing the psychometric quality of rating data

• Published in: Psychological bulletin Vol. 88; no. 2; pp. 413 - 428
• Main Authors: Saal, Frank E, Downey, Ronald G, Lahey, Mary A
• Format: Journal Article
• Language: English
• Published: Washington, etc American Psychological Association 01-09-1980; American Psychological Association, etc
• Subjects: Evaluation; Human; Literature Review; Psychometrics; Rating; Rating Scales
• ISSN: 0033-2909; 1939-1455
• DOI: 10.1037/0033-2909.88.2.413

Reviews research that is concerned with evaluating the psychometric qualities of data in the form of ratings (rating errors) and that has been plagued with conceptual and operational confusion and inconsistency. Following a brief historical survey, inconsistencies in definitions, quantifications, and methodologies are documented in a review of more than 20 relevant articles published in
Journal of Applied Psychology, Organizational Behavior and Human Performance,
and
Personnel Psychology
(1975-1977). Empirical implications of these inconsistencies are discussed, and a revised typology of rating criteria, combined with a multivariate analytic approach, is suggested. (65 ref)