Discrepancies in Ratings of Behavioral Healthcare Interventions Among Evidence-Based Program Resources Websites

• Published in: Inquiry (Chicago) Vol. 60; p. 469580231186836
• Main Authors: Lee-Easton, Miranda Jade, Magura, Stephen
• Format: Journal Article
• Language: English
• Published: Los Angeles, CA SAGE Publications 01-01-2023; SAGE PUBLICATIONS, INC; SAGE Publishing
• Subjects: Bias; Children & youth; Criteria; Health care; Intervention; Mental health; Military intervention; Original Research; Ratings & rankings; Teenage pregnancy; Websites; evidence-based registers; behavioral healthcare interventions; evidence-based clearinghouses
• ISSN: 0046-9580; 1945-7243
• DOI: 10.1177/00469580231186836

Decision makers in the behavioral health disciplines could benefit from tools to assist them in identifying and implementing evidence-based interventions. One tool is an evidence-based program resources website (EBPR). Prior studies documented that when multiple EBPRs rate an intervention, they may disagree. Prior research concerning the reason for such conflicts is sparse. The present study examines how EBPRs rate interventions and the sources of disagreement between EBPRs when rating the same intervention. This study hypothesizes that EBPRs may disagree about intervention ratings because they either use different rating paradigms or they use different studies as evidence of intervention effectiveness (or both). This study identified 15 EBPRs for inclusion. One author (M.J.L.E.) coded the EBPRs for which “tiers of evidence” each EBPR used to classify behavioral health interventions and which criteria they used when rating interventions. The author then computed one Jaccard index of similarity for the criteria shared between each pair of EBPRs that co-rated interventions, and one for the studies used by EBPR rating pairs when rating the same program. The authors used a combination of chi-square, correlation, and binary logistic regression analyses to analyze the data. There was a statistically significant negative correlation between the number of Cochrane Risk of Bias criteria shared between 2 EBPRs and the likelihood of those 2 EBPRs agreeing on an intervention rating (r = −.12, P ≤ .01). There was no relationship between the number of studies evaluated by 2 EBPRs and the likelihood of those EBPRs agreeing on an intervention rating. The major reason for disagreements between EBPRs when rating the same intervention in this study was due to differences in the rating criteria used by the EBPRs. The studies used by the EBPRs to rate programs does not appear to have an impact.