If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Corresponding author: Kevin C. Chung, MD, MS, Section of Plastic Surgery, University of Michigan Health System, 2130 Taubman Center, SPC 5340, 1500 E. Medical Center Dr., Ann Arbor, MI 48109.
To evaluate control group selection in nonrandomized studies published in the Journal of Hand Surgery American (JHS).
Methods
We reviewed all papers published in JHS in 2013 to identify studies that used nonrandomized control groups. Data collected included type of study design and control group characteristics. We then appraised studies to determine whether authors discussed confounding and selection bias and how they controlled for confounding.
Results
Thirty-seven nonrandomized studies were published in JHS in 2013. The source of control was either the same institution as the study group, a different institution, a database, or not provided in the manuscript. Twenty-nine (78%) studies statistically compared key characteristics between control and study group. Confounding was controlled with matching, exclusion criteria, or regression analysis. Twenty-two (59%) papers explicitly discussed the threat of confounding and 18 (49%) identified sources of selection bias.
Conclusions
In our review of nonrandomized studies published in JHS, papers had well-defined controls that were similar to the study group, allowing for reasonable comparisons. However, we identified substantial confounding and bias that were not addressed as explicit limitations, which might lead the reader to overestimate the scientific validity of the data.
Clinical relevance
Incorporating a brief discussion of control group selection in scientific manuscripts should help readers interpret the study more appropriately. Authors, reviewers, and editors should strive to address this component of clinical importance.
A control group should be a representative sample of the population from which the study group is derived. Thus, it can be similar to the study group but unexposed to a disease, risk factor, or intervention of interest. In nonrandomized studies, selecting comparable groups is necessary to allow appropriate assessment of associations and effectiveness of an intervention.
Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Available at: www.cochrane-handbook.org. Accessibility verified May 17, 2014.
compared wrist arthrodesis after resection of a giant cell tumor of the distal radius using a vascularized free fibular transfer versus a nonvascularized structural iliac crest transfer. Given the rarity of disease and specialization of treatment, the authors conducted a retrospective cohort study to compare the effectiveness of these interventions.
In nonrandomized studies, the study design determines the selection of controls. In cohort studies, the control group is determined by practice patterns, physicians’ preference, or policy decisions.
the method of wrist arthrodesis was determined by regional preference (Vancouver vs Toronto). In contrast, control selection in case-control studies is at the discretion of a researcher.
Factors that determine whether a participant is placed in the study group or control group of a nonrandomized study may result in comparison groups with unbalanced characteristics.
Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Available at: www.cochrane-handbook.org. Accessibility verified May 17, 2014.
Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Available at: www.cochrane-handbook.org. Accessibility verified May 17, 2014.
performed radial shortening osteotomies in patients with Kienböck disease if they had 2 mm or greater of negative ulnar variance. All other patients were allocated to the control group, who received vascularized bone grafts. This allocation process created comparison groups with slightly differing pathologies. Alternatively, a confounder is an external characteristic that partially or entirely explains an association between an exposure and an outcome of interest.
Because an ideal control group is unattainable in nonrandomized studies, authors should discuss the limitations of their selected controls.
Our aim was to evaluate the control group selection in nonrandomized studies published in the Journal of Hand Surgery American (JHS) in 2013 and to highlight the strengths and weaknesses of various types of controls chosen. We also investigated how often authors adjusted for and discussed the threat of confounding and selection bias. We hypothesized that, in nonrandomized studies in hand surgery, control group selection is appropriate but the discussion of limitations is minimum. Presenting the limitations of a selected control is critical to allow readers to make accurate inferences on the validity of study results. This is an important component of well-written observational research, and peer reviewers and editors share the responsibility of requiring this from their authors.
We performed a literature review of all articles published in 2013 in JHS. We included studies using a nonrandomized control group to make conclusions on their primary research hypothesis. Studies using comparison groups to test secondary hypotheses or outcomes were not included. Lack of sufficient demographic details, source or type of controls to review, and primary outcome not involving control groups led us to exclude those studies. Nonrandomized studies included were retrospective cohorts, prospective cohorts, and case-control studies. The types of studies included were therapeutic, prognostic, and diagnostic.
To assess control group selection, we determined the source, type, and number of controls. The source of controls refers to the population from which the controls were selected. The type of control can be concurrent, historical, or an overlap of both. Concurrent controls are enrolled simultaneously with the intervention group and followed for the same study period.
Conversely, historical controls are participants treated earlier without the intervention of interest but their outcomes are used to compare with the current subjects.
Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Available at: www.cochrane-handbook.org. Accessibility verified May 17, 2014.
We also recorded the number of studies using healthy (ie, normal or nondiseased) controls, controls that received an alternative intervention, and controls in which subjects were self-controlled (eg, when the contralateral hand of a subject was used as the control group).
Statistical comparison of baseline characteristics between study and control groups can identify unbalanced characteristics, thus indicating poor comparability. We recorded the proportion of studies that statistically compared at least 1 baseline characteristic between the study and the control groups. We then determined the number of included articles that controlled for confounding using matching or exclusion criteria at the design stage and standardization, stratification, matched analysis, or regression analyses in the data analysis stage. Lastly, as a surrogate for assessing the authors’ discussion of control group limitations, we assessed whether studies discussed confounding and selection bias. Articles were deemed to have discussed these topics if they provided a possible source of confounding or selection bias, respectively. The authors adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines in the preparation of this manuscript.
Results
A nonrandomized control group was used in 37 of the 236 scientific studies published in 2013 in JHS (Fig. 1). Of the included papers, 19 (51%) were retrospective cohort studies, 5 (14%) were prospective cohort studies, and 13 (35%) were case-control studies. A majority of studies, 20 (54%), investigated the effectiveness of a therapeutic intervention. There were 10 (27%) prognostic studies and 7 (19%) diagnostic studies.
Figure 1Flow diagram for identifying nonrandomized studies in JHS.
The selection of controls was quite variable, with regards to source, type, number, and other defining characteristics (Table 1). The choice of controls appeared to be dependent on study design, study topic, sample size availability, and feasibility. The majority of studies used controls from the same institution (as the study cohort), but in 7 (19%) studies, the authors failed to mention the source. Concurrent controls were used in far greater frequency than historical controls. Historical controls were used when the study topic was rare such as tetraplegia reconstruction
Authors in 29 (78%) studies statistically compared characteristics between their selected cohorts. Of these, 10 articles found at least 1 statistically different characteristic.
Age and sex were the most common comparisons. Duration of disease, disease classification, disease severity, and hand dominance were also frequently compared characteristics.
A majority of studies attempted to control for confounding (Table 2). The most common method was restriction of participant enrollment with rigid exclusion criteria (76%). Age, concomitant injuries, recurrent disease, and comorbidities are examples of exclusion criteria used.
In total, 30 (81%) studies used at minimum 1 method to control for confounding. Although a majority of articles controlled for confounding, approximately half discussed the limitations of their selected control group. Table 3 provides excerpts from articles that discussed control group limitations. Authors explicitly discussed the threat of confounding and identified sources of selection bias in 59% and 49% of articles, respectively.
Table 2How Authors Controlled for Confounders With Study Design and Data Analysis (n = 37 studies)
“That antiplatelet patients were older may have biased us toward finding more ecchymosis and hematoma in the antiplatelet group given age-related changes in the soft tissues.”
“The average age of the repair group was significantly less than the control group, which may have introduced bias, because a younger patient may be more critical of the outcome.”
“Although we do not have epidemiological data on contracture development to explain the difference, it is possible that younger patients were more frequently offered surgical management or that they were less tolerant of functional limitations.”
“Patients operated on through the lateral para-olecranon approach were older, possibly because newer medications have had a major impact on controlling rheumatoid arthritis.”
A majority of the JHS articles reviewed had well-defined controls, but only 59% and 49% elaborated on potential sources of confounding and selection bias, respectively. Previous studies have demonstrated poor disclosure of bias in observational research. Groenwold et al
conducted a systematic review of 174 observational articles and found that the quality of reporting of confounding in articles was poor. The concerns over underreporting bias have led to standardized guidelines for conducting observational research.
Using the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement to assess reporting of observational trials in hand surgery.
identified an improvement in the quality of observational research reporting in hand surgery but not in the discussion of potential bias. Therefore, it is critical that authors discuss their control population. This includes disclosure of the eligibility criteria, selection methods, type, source, and limitations. This allows readers to accurately appraise the control group and assess the validity of conclusions (Table 4).
Table 4Appraisal of Control Group in Observational Studies
Term
How Confounding Is Controlled
Purpose
What is the purpose of the study? What is the purpose of the control group?
Control group selection
What is the source of control? Is the control concurrent or historical? What control group characteristics were provided?
Identification of confounders
Did the study do a literature search to identify known confounders?
Control vs study group
Did the study statistically compare groups to demonstrate lack of comparability?
Controlling for confounding
Did the study control for confounding with their study design? Restriction (exclusion criteria) Matching
Did the study control for confounding with post hoc statistical analysis? Standardization Stratification analysis Matched analysis Regression models
Confounding and bias discussion
Did authors provide an in-depth discussion of confounding? Did authors discuss selection bias?
compared the emotional and behavioral characteristics of children with obstetrical brachial plexus palsy with healthy children presenting for a well-child visit from the same hospital.
The validity of inferences from this study relied on the assumption that the well-child cohort was similar with respect to extraneous factors (eg, psychological stressors) and that their geographic referral patterns were similar to those of the study group.
The temporal relationship between comparison groups is important. As diagnosis, treatment methods, and outcome measures change over time, using historical controls may introduce bias.
found that grip strength was significantly greater in tetraplegic patients treated with single-stage grip-release reconstruction compared with historical controls treated with staged flexion-extension grip-release reconstruction. The authors disclosed that the study group had more mobilization and fewer activity restrictions. In addition, the use of immobilization orthoses was abolished when new publications supported more aggressive rehabilitation regimens. This highlights the limited utility of historical controls with evolution of medical practices. Because of improved treatment and expertise, using historical controls tends to favor more current interventions.
used 3 comparison groups to evaluate outcomes of revision carpal tunnel release. The authors classified patient groups based on presenting symptoms: persistent, recurrent, or new. They found the recurrent group exhibited key clinical differences including a lesser likelihood of presenting with pain. By demonstrating differences among various controls, the authors strengthened their conclusion. However, if multiple controls result in incongruent conclusions, investigators should evaluate reasons for the conflicting results.
A statistical comparison of baseline characteristics between control and study groups allows identification of unbalanced variables. However, it should not be used to demonstrate equivalency because the statistical comparison is often underpowered. Beck et al
compared obese patients with nonobese patients to determine function and complications after reverse total shoulder arthroplasty. Ten variables were compared that revealed a greater proportion of diabetic patients in the obese cohort. The study showed that the obese cohort was at an increased risk for complications including infection and joint instability. Despite identifying a potential confounder, the authors did not explore the impact of this confounder on their conclusion, which may explain the higher incidence of infection in the obese group.
Approaches used by researchers to address confounding are listed in Table 5. At study design level, authors attempt to balance significant variables between comparison groups.
Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Available at: www.cochrane-handbook.org. Accessibility verified May 17, 2014.
Unfortunately, it limits the sample size and decreases the study power. Restriction also decreases external validity, which refers to how well the conclusions can be generalized to everyday practice.
compared volar angulation in patients treated with a single versus 2 rows of distal screws after volar locked plate fixation for distal radius fractures. The case-matched study found no advantage with either method. The matching balanced potential confounders between comparison groups but also highlighted its limitations. By matching 10 variables, the number of participants was reduced greatly, thus reducing statistical power.
In addition, 1-to-1 matching is not always feasible. In this study, age remained statistically different between comparison groups and was, therefore, a residual confounding variable.
Post hoc statistical analysis is another method used to identify associations.
Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Available at: www.cochrane-handbook.org. Accessibility verified May 17, 2014.
Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Available at: www.cochrane-handbook.org. Accessibility verified May 17, 2014.
used multivariable regression analysis to evaluate the influence of demographics on reoperation for flexor pulley reconstruction. They found a higher likelihood of reconstruction among men, after adjusting for age and concomitant nerve and flexor tendon repair. This analysis improved the validity of conclusions by adjusting for exposures that would otherwise be confounders. However, the authors acknowledged that the factors they found to influence reoperation were in fact surrogates for injury severity.
Thus, the influence of injury severity may not be truly represented in this study.
A limitation of our study was that our assessment was dependent on the information provided within the articles. If authors failed to disclose information about control selection or confounding adjustment because of text limit, we assumed these issues were not appropriately addressed.
Reporting guidelines of observational studies emphasize the importance of discussing potential bias or imprecision in studies.
Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group.
Thus, authors of nonrandomized studies are held accountable to select appropriate control groups and provide a discussion of potential selection bias and confounding and of the methods used to address them. Likewise, peer reviewers and editors are also responsible to address the need for such information in published articles. This allows readers to assess the accuracy of results in context of any clinical recommendation.
References
Rochon P.A.
Gurwitz J.H.
Sykora K.
et al.
Reader’s guide to critical appraisal of cohort studies: 1. Role and design.
Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Available at: www.cochrane-handbook.org. Accessibility verified May 17, 2014.
Using the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement to assess reporting of observational trials in hand surgery.
Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group.
Research reported in this publication was supported by awards from the National Institute of Arthritis and Musculoskeletal and Skin Diseases of the National Institutes of Health (2R01 AR047328– 06), the National Institute on Aging and National Institute of Arthritis and Musculoskeletal and Skin Diseases (R01 AR062066), and a Midcareer Investigator Award in Patient-Oriented Research (2K24 AR053120-06) to K.C.C. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
No benefits in any form have been received or will be received related directly or indirectly to the subject of this article.
01315-X&id=gr1.jpg){kind=link}