Purpose
Inappropriately reported or conducted studies may decrease the quality of care due
to under- or overestimation of the benefits or harms of interventions. Our aim was
to evaluate how often hand surgical randomized controlled trials (RCTs) use and report
adequate methods to ensure internal validity, and whether inadequate reporting or
methods are associated with the magnitude of treatment effect estimates.
Methods
Data Sources were the Cochrane Central Register of Controlled Trials, MEDLINE, and
Embase databases until November 2020. We included published RCTs investigating the
effects of any surgical intervention in the hand and wrist region. We assessed internal
validity using the Cochrane Risk of Bias (RoB) tool for 6 domains: selection, performance,
detection, attrition, selective reporting, and “other” bias. We extracted the primary
outcome and calculated the effect size for each study. We used mixed-effect meta-regression
to assess whether the RoB modified the magnitude of the effects.
Results
For 207 assessed trials, the RoB was unclear or high for 72% in selection, 93% in
performance, 88% in detection, 25% in attrition, 22% in selective reporting, and 34%
in the “other” bias domain. Trials with a high or unclear risk of selection bias yielded
0.28 standardized mean difference (95% confidence interval, 0.02–0.55) larger effect
sizes compared to studies with a low risk. Risks of bias for other domains did not
modify the intervention effects. The risk for selection bias declined over time: the
odds ratio for a high or unclear RoB was 0.90 (95% confidence interval, 0.85–0.95)
per additional year of publication
Conclusions
The internal validity and credibility of hand surgical RCTs can be improved by using
established methods to achieve true randomization, blinding of the participants and
study personnel, publishing the trial protocol and avoiding selective reporting of
the outcomes, and reporting the trial as recommended in the Consolidated Standards
of Reporting Trials statement.
Clinical relevance
Clinicians should be aware that RCTs that do not use or report proper randomization
and allocation concealment may overestimate the treatment effects.
Key words
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Hand SurgeryAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Systematic reviews in health care: assessing the quality of controlled clinical trials.BMJ. 2001; 323: 42-46
- Avoidable waste in the production and reporting of research evidence.Lancet. 2009; 374: 86-89
- The Empirical Evidence of Bias in Trials Measuring Treatment Differences.Agency for Healthcare Research and Quality (US), 2014
- Chapter 7. Considering bias and conflicts of interest among the included studies.in: Higgins J.P.T. Thomas J. Chandler J. Cumpston M. Li T. Page M.J. Welch V.A. Cochrane Handbook for Systematic Reviews of Interventions. John Wiley & Sons, Hoboken, NJ2019
- The Empirical Evidence of Bias in Trials Measuring Treatment Differences. Methods Research Report. (Prepared by the RTI-UNC Evidence-based Practice Center under Contract No. 290-2007-10056-I.) AHRQ Publication No. 14-EHC050-EF.Agency for Healthcare Research and Quality, Rockville, MD2014
- Trends in the level of evidence in clinical hand surgery research.Hand (N Y). 2016; 11: 211-215
- A systematic review of methodological quality in hand surgical randomized controlled trials.https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=122710Date accessed: January 23, 2019
- The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials.BMJ. 2011; 343: d5928
- RoB 2: a revised tool for assessing risk of bias in randomised trials.BMJ. 2019; 366: l4898
- Cochrane Handbook for Systematic Reviews of Interventions.
- A simple method for converting an odds ratio to effect size for use in meta-analysis.Stat Med. 2000; 19: 3127-3131
- Bridging extema/fixation combined with Kirschner-wire fixation versus volar locked plate fixation for unstable fractures of the distal radius.Zhongguo Zuzhi Gongcheng Yanjiu. 2016; 20: 6590-6598
- Equipoise and the ethics of clinical research.N Engl J Med. 1987; 317: 141-145
- Clinical equipoise: foundational requirement or fundamental error?.in: Steinbock B. The Oxford Handbook of Bioethics. Oxford University Press, 2007: 571-595
- GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.BMJ. 2008; 336: 924-926
- Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials.JAMA. 1995; 273: 408-412
- Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials.Ann Intern Med. 2012; 157: 429-438
- Considerations and methods for placebo controls in surgical trials (ASPIRE guidelines).Lancet. 2020; 395: 828-838
- Influence of blinding on treatment effect size estimate in randomized controlled trials of oral health interventions.BMC Med Res Methodol. 2018; 18: 42
- Impact of selection bias on treatment effect size estimates in randomized trials of oral health interventions: a meta-epidemiological study.J Dent Res. 2018; 97: 5-13
- Blinding in physical therapy trials and its association with treatment effects: a meta-epidemiological study.Am J Phys Med Rehabil. 2017; 96: 34-44
- Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study.BMJ. 2020; 368: l6802
- Subjective and objective outcomes in randomized clinical trials: definitions differed in methods publications and were often absent from trial reports.J Clin Epidemiol. 2014; 67: 1327-1334
- Observer bias in randomized clinical trials with time-to-event outcomes: systematic review of trials with both blinded and non-blinded outcome assessors.Int J Epidemiol. 2014; 43: 937-948
- Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors.CMAJ. 2013; 185: E201-E211
- Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors.BMJ. 2012; 344: e1119
- Empirical evaluation of which trial characteristics are associated with treatment effect estimates.J Clin Epidemiol. 2016; 77: 24-37
- What should be included in meta-analyses? An exploration of methodological issues using the ISPOT meta-analyses.Int J Technol Assess Health Care. 2000; 16: 1109-1119
- Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research.CMAJ. 2004; 171: 735-740
- Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.JAMA. 2004; 291: 2457-2465
- The ClinicalTrials.gov results database--update and key issues.N Engl J Med. 2011; 364: 852-860
- CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials.Ann Intern Med. 2010; 152: 726-732
Article info
Publication history
Published online: March 24, 2022
Accepted:
January 26,
2022
Received:
May 29,
2021
Footnotes
No benefits in any form have been received or will be received related directly or indirectly to the subject of this article.
Identification
Copyright
© 2022 by the American Society for Surgery of the Hand. All rights reserved.
