Inappropriately reported or conducted studies may decrease the quality of care due to under- or overestimation of the benefits or harms of interventions. Our aim was to evaluate how often hand surgical randomized controlled trials (RCTs) use and report adequate methods to ensure internal validity, and whether inadequate reporting or methods are associated with the magnitude of treatment effect estimates.
Data Sources were the Cochrane Central Register of Controlled Trials, MEDLINE, and Embase databases until November 2020. We included published RCTs investigating the effects of any surgical intervention in the hand and wrist region. We assessed internal validity using the Cochrane Risk of Bias (RoB) tool for 6 domains: selection, performance, detection, attrition, selective reporting, and “other” bias. We extracted the primary outcome and calculated the effect size for each study. We used mixed-effect meta-regression to assess whether the RoB modified the magnitude of the effects.
For 207 assessed trials, the RoB was unclear or high for 72% in selection, 93% in performance, 88% in detection, 25% in attrition, 22% in selective reporting, and 34% in the “other” bias domain. Trials with a high or unclear risk of selection bias yielded 0.28 standardized mean difference (95% confidence interval, 0.02–0.55) larger effect sizes compared to studies with a low risk. Risks of bias for other domains did not modify the intervention effects. The risk for selection bias declined over time: the odds ratio for a high or unclear RoB was 0.90 (95% confidence interval, 0.85–0.95) per additional year of publication
The internal validity and credibility of hand surgical RCTs can be improved by using established methods to achieve true randomization, blinding of the participants and study personnel, publishing the trial protocol and avoiding selective reporting of the outcomes, and reporting the trial as recommended in the Consolidated Standards of Reporting Trials statement.
Clinicians should be aware that RCTs that do not use or report proper randomization and allocation concealment may overestimate the treatment effects.
To read this article in full you will need to make a payment
Purchase one-time access:Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
One-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:Subscribe to Journal of Hand Surgery
Already a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
- Systematic reviews in health care: assessing the quality of controlled clinical trials.BMJ. 2001; 323: 42-46
- Avoidable waste in the production and reporting of research evidence.Lancet. 2009; 374: 86-89
- The Empirical Evidence of Bias in Trials Measuring Treatment Differences.Agency for Healthcare Research and Quality (US), 2014
- Chapter 7. Considering bias and conflicts of interest among the included studies.in: Higgins J.P.T. Thomas J. Chandler J. Cumpston M. Li T. Page M.J. Welch V.A. Cochrane Handbook for Systematic Reviews of Interventions. John Wiley & Sons, Hoboken, NJ2019
- The Empirical Evidence of Bias in Trials Measuring Treatment Differences. Methods Research Report. (Prepared by the RTI-UNC Evidence-based Practice Center under Contract No. 290-2007-10056-I.) AHRQ Publication No. 14-EHC050-EF.Agency for Healthcare Research and Quality, Rockville, MD2014
- Trends in the level of evidence in clinical hand surgery research.Hand (N Y). 2016; 11: 211-215
- A systematic review of methodological quality in hand surgical randomized controlled trials.https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=122710Date accessed: January 23, 2019
- The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials.BMJ. 2011; 343: d5928
- RoB 2: a revised tool for assessing risk of bias in randomised trials.BMJ. 2019; 366: l4898
- Cochrane Handbook for Systematic Reviews of Interventions.https://training.cochrane.org/handbookVersion: Version 6.0Date accessed: February 9, 2022
- A simple method for converting an odds ratio to effect size for use in meta-analysis.Stat Med. 2000; 19: 3127-3131
- Bridging extema/fixation combined with Kirschner-wire fixation versus volar locked plate fixation for unstable fractures of the distal radius.Zhongguo Zuzhi Gongcheng Yanjiu. 2016; 20: 6590-6598
- Equipoise and the ethics of clinical research.N Engl J Med. 1987; 317: 141-145
- Clinical equipoise: foundational requirement or fundamental error?.in: Steinbock B. The Oxford Handbook of Bioethics. Oxford University Press, 2007: 571-595
- GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.BMJ. 2008; 336: 924-926
- Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials.JAMA. 1995; 273: 408-412
- Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials.Ann Intern Med. 2012; 157: 429-438
- Considerations and methods for placebo controls in surgical trials (ASPIRE guidelines).Lancet. 2020; 395: 828-838
- Influence of blinding on treatment effect size estimate in randomized controlled trials of oral health interventions.BMC Med Res Methodol. 2018; 18: 42
- Impact of selection bias on treatment effect size estimates in randomized trials of oral health interventions: a meta-epidemiological study.J Dent Res. 2018; 97: 5-13
- Blinding in physical therapy trials and its association with treatment effects: a meta-epidemiological study.Am J Phys Med Rehabil. 2017; 96: 34-44
- Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study.BMJ. 2020; 368: l6802
- Subjective and objective outcomes in randomized clinical trials: definitions differed in methods publications and were often absent from trial reports.J Clin Epidemiol. 2014; 67: 1327-1334
- Observer bias in randomized clinical trials with time-to-event outcomes: systematic review of trials with both blinded and non-blinded outcome assessors.Int J Epidemiol. 2014; 43: 937-948
- Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors.CMAJ. 2013; 185: E201-E211
- Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors.BMJ. 2012; 344: e1119
- Empirical evaluation of which trial characteristics are associated with treatment effect estimates.J Clin Epidemiol. 2016; 77: 24-37
- What should be included in meta-analyses? An exploration of methodological issues using the ISPOT meta-analyses.Int J Technol Assess Health Care. 2000; 16: 1109-1119
- Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research.CMAJ. 2004; 171: 735-740
- Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.JAMA. 2004; 291: 2457-2465
- The ClinicalTrials.gov results database--update and key issues.N Engl J Med. 2011; 364: 852-860
- CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials.Ann Intern Med. 2010; 152: 726-732
Published online: March 24, 2022
Accepted: January 26, 2022
Received: May 29, 2021
No benefits in any form have been received or will be received related directly or indirectly to the subject of this article.
© 2022 by the American Society for Surgery of the Hand. All rights reserved.