Analysis of scientific truth status in controlled rehabilitation trials

• Published in: Journal of evaluation in clinical practice Vol. 19; no. 4; pp. 617 - 625
• Main Authors: Kerry, Roger, Madouasse, Aurélien, Arthur, Antony, Mumford, Stephen D.
• Format: Journal Article
• Language: English
• Published: Oxford, UK Blackwell Publishing Ltd 01-08-2013
• Subjects: Bibliometrics; evidence-based practice; Evidence-Based Practice - standards; Humans; philosophy; Physical Therapy Modalities - organization & administration; Physical Therapy Modalities - standards; randomized controlled trials; Randomized Controlled Trials as Topic - standards; Regression Analysis; rehabilitation; Reproducibility of Results; Research Design; systematic reviews; truth; evidence-based practice; rehabilitation; truth; systematic reviews; philosophy; randomized controlled trials
• ISSN: 1356-1294; 1365-2753
• DOI: 10.1111/j.1365-2753.2012.01855.x

Rationale, aims and objectives  Systematic reviews, meta‐analyses and clinical guidelines (reviews) are intended to inform clinical practice, and in this sense can be thought of as scientific truthmakers. High‐quality controlled trials should align to this truth, and method quality markers should predict truth status. We sought to determine in what way controlled trial quality relates to scientific truth, and to determine predictive utility of trial quality and bibliographic markers.
Method  A sample of reviews in rehabilitation medicine was examined. Two scientific truth dimensions were established based on review outcomes. Quality and bibliographic markers were extracted from associated trials for use in a regression analysis of their predictive utility for trial truth status. Probability analysis was undertaken to examine judgments of future trial truth status.
Results  Of the 93 trials included in contemporaneous reviews, overall, n = 45 (48%) were true. Randomization was found more in true trials than false trials in one truth dimension (P = 0.03). Intention‐to‐treat analysis was close to significant in one truth dimension (P = 0.058), being more commonly used in false trials. There were no other significant differences in quality or bibliographic variables between true and false trials. Regression analysis revealed no significant predictors of trial truth status. Probability analysis reported that the reasonable chance of future trials being true was between 2 and 5%, based on a uniform prior.
Conclusions  The findings are at odds with what is considered gold‐standard research methods, but in line with previous reports. Further work should focus on scientific dynamics within healthcare research and evidence‐based practice constructs.