Advertisement

The Quality of Control Groups in Nonrandomized Studies Published in the Journal of Hand Surgery

Published:October 16, 2014DOI:https://doi.org/10.1016/j.jhsa.2014.09.021

      Purpose

      To evaluate control group selection in nonrandomized studies published in the Journal of Hand Surgery American (JHS).

      Methods

      We reviewed all papers published in JHS in 2013 to identify studies that used nonrandomized control groups. Data collected included type of study design and control group characteristics. We then appraised studies to determine whether authors discussed confounding and selection bias and how they controlled for confounding.

      Results

      Thirty-seven nonrandomized studies were published in JHS in 2013. The source of control was either the same institution as the study group, a different institution, a database, or not provided in the manuscript. Twenty-nine (78%) studies statistically compared key characteristics between control and study group. Confounding was controlled with matching, exclusion criteria, or regression analysis. Twenty-two (59%) papers explicitly discussed the threat of confounding and 18 (49%) identified sources of selection bias.

      Conclusions

      In our review of nonrandomized studies published in JHS, papers had well-defined controls that were similar to the study group, allowing for reasonable comparisons. However, we identified substantial confounding and bias that were not addressed as explicit limitations, which might lead the reader to overestimate the scientific validity of the data.

      Clinical relevance

      Incorporating a brief discussion of control group selection in scientific manuscripts should help readers interpret the study more appropriately. Authors, reviewers, and editors should strive to address this component of clinical importance.

      Key words

      A control group should be a representative sample of the population from which the study group is derived. Thus, it can be similar to the study group but unexposed to a disease, risk factor, or intervention of interest. In nonrandomized studies, selecting comparable groups is necessary to allow appropriate assessment of associations and effectiveness of an intervention.
      • Rochon P.A.
      • Gurwitz J.H.
      • Sykora K.
      • et al.
      Reader’s guide to critical appraisal of cohort studies: 1. Role and design.
      • Peacock J.L.
      • Peacock P.J.
      Oxford Handbook of Medical Statistics.

      Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Available at: www.cochrane-handbook.org. Accessibility verified May 17, 2014.

      • Wacholder S.
      • McLaughlin J.K.
      • Silverman D.T.
      • Mandel J.S.
      Selection of controls in case-control studies: I. Principles.
      Nonrandomized studies are often used in hand surgery research when a randomized controlled trial would be time-consuming or not feasible.
      • Morshed S.
      • Tornetta III, P.
      • Bhandari M.
      Analysis of observational studies: a guide to understanding statistical methods.
      • Busse J.W.
      • Obremskey W.T.
      Principles of designing an orthopaedic case-control study.
      • Bryant D.M.
      • Willits K.
      • Hanson B.P.
      Principles of designing a cohort study in orthopaedics.
      For example, Clarkson et al
      • Clarkson P.W.
      • Sandford K.
      • Phillips A.E.
      • et al.
      Functional results following vascularized versus nonvascularized bone grafts for wrist arthrodesis following excision of giant cell tumors.
      compared wrist arthrodesis after resection of a giant cell tumor of the distal radius using a vascularized free fibular transfer versus a nonvascularized structural iliac crest transfer. Given the rarity of disease and specialization of treatment, the authors conducted a retrospective cohort study to compare the effectiveness of these interventions.
      In nonrandomized studies, the study design determines the selection of controls. In cohort studies, the control group is determined by practice patterns, physicians’ preference, or policy decisions.
      • Mamdani M.
      • Sykora K.
      • Li P.
      • et al.
      Reader’s guide to critical appraisal of cohort studies: 2. Assessing potential for confounding.
      In the study by Clarkson et al,
      • Clarkson P.W.
      • Sandford K.
      • Phillips A.E.
      • et al.
      Functional results following vascularized versus nonvascularized bone grafts for wrist arthrodesis following excision of giant cell tumors.
      the method of wrist arthrodesis was determined by regional preference (Vancouver vs Toronto). In contrast, control selection in case-control studies is at the discretion of a researcher.
      • Rochon P.A.
      • Gurwitz J.H.
      • Sykora K.
      • et al.
      Reader’s guide to critical appraisal of cohort studies: 1. Role and design.
      • Peacock J.L.
      • Peacock P.J.
      Oxford Handbook of Medical Statistics.
      Factors that determine whether a participant is placed in the study group or control group of a nonrandomized study may result in comparison groups with unbalanced characteristics.
      • Rochon P.A.
      • Gurwitz J.H.
      • Sykora K.
      • et al.
      Reader’s guide to critical appraisal of cohort studies: 1. Role and design.
      • Malay S.
      • Chung K.C.
      The choice of controls for providing validity and evidence in clinical research.
      If these characteristics have prognostic importance, then selection bias or confounding may occur, affecting the validity of conclusions.
      • Rochon P.A.
      • Gurwitz J.H.
      • Sykora K.
      • et al.
      Reader’s guide to critical appraisal of cohort studies: 1. Role and design.
      • Peacock J.L.
      • Peacock P.J.
      Oxford Handbook of Medical Statistics.

      Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Available at: www.cochrane-handbook.org. Accessibility verified May 17, 2014.

      • Morshed S.
      • Tornetta III, P.
      • Bhandari M.
      Analysis of observational studies: a guide to understanding statistical methods.
      • Busse J.W.
      • Obremskey W.T.
      Principles of designing an orthopaedic case-control study.
      • Bryant D.M.
      • Willits K.
      • Hanson B.P.
      Principles of designing a cohort study in orthopaedics.
      • Mamdani M.
      • Sykora K.
      • Li P.
      • et al.
      Reader’s guide to critical appraisal of cohort studies: 2. Assessing potential for confounding.
      • Malay S.
      • Chung K.C.
      The choice of controls for providing validity and evidence in clinical research.
      • Paradis C.
      Bias in surgical research.
      • Young J.
      • Solomon M.
      Improving the evidence base in surgery: sources of bias in surgical studies.
      Selection bias refers to systematic differences between baseline characteristics of the comparison groups.

      Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Available at: www.cochrane-handbook.org. Accessibility verified May 17, 2014.

      For example, Afshar et al
      • Afshar A.
      • Eivaziatashbeik K.
      Long-term clinical and radiological outcomes of radial shortening osteotomy and vascularized bone graft in Kienböck disease.
      performed radial shortening osteotomies in patients with Kienböck disease if they had 2 mm or greater of negative ulnar variance. All other patients were allocated to the control group, who received vascularized bone grafts. This allocation process created comparison groups with slightly differing pathologies. Alternatively, a confounder is an external characteristic that partially or entirely explains an association between an exposure and an outcome of interest.
      • Paradis C.
      Bias in surgical research.
      • Vavken P.
      • Culen G.
      • Dorotka R.
      Management of confounding in controlled orthopaedic trials: a cross-sectional study.
      • Müllner M.
      • Matthews H.
      • Altman D.G.
      Reporting on statistical methods to adjust for confounding: a cross-sectional survey.
      If confounders are unequally distributed between comparison groups, they distort the effect of the study intervention.
      • Bryant D.M.
      • Willits K.
      • Hanson B.P.
      Principles of designing a cohort study in orthopaedics.
      Because an ideal control group is unattainable in nonrandomized studies, authors should discuss the limitations of their selected controls.
      Our aim was to evaluate the control group selection in nonrandomized studies published in the Journal of Hand Surgery American (JHS) in 2013 and to highlight the strengths and weaknesses of various types of controls chosen. We also investigated how often authors adjusted for and discussed the threat of confounding and selection bias. We hypothesized that, in nonrandomized studies in hand surgery, control group selection is appropriate but the discussion of limitations is minimum. Presenting the limitations of a selected control is critical to allow readers to make accurate inferences on the validity of study results. This is an important component of well-written observational research, and peer reviewers and editors share the responsibility of requiring this from their authors.
      • Von Elm E.
      • Altman D.G.
      • Egger M.
      • et al.
      The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies.

      Methods

      We performed a literature review of all articles published in 2013 in JHS. We included studies using a nonrandomized control group to make conclusions on their primary research hypothesis. Studies using comparison groups to test secondary hypotheses or outcomes were not included. Lack of sufficient demographic details, source or type of controls to review, and primary outcome not involving control groups led us to exclude those studies. Nonrandomized studies included were retrospective cohorts, prospective cohorts, and case-control studies. The types of studies included were therapeutic, prognostic, and diagnostic.
      To assess control group selection, we determined the source, type, and number of controls. The source of controls refers to the population from which the controls were selected. The type of control can be concurrent, historical, or an overlap of both. Concurrent controls are enrolled simultaneously with the intervention group and followed for the same study period.
      • Malay S.
      • Chung K.C.
      The choice of controls for providing validity and evidence in clinical research.
      Conversely, historical controls are participants treated earlier without the intervention of interest but their outcomes are used to compare with the current subjects.

      Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Available at: www.cochrane-handbook.org. Accessibility verified May 17, 2014.

      We also recorded the number of studies using healthy (ie, normal or nondiseased) controls, controls that received an alternative intervention, and controls in which subjects were self-controlled (eg, when the contralateral hand of a subject was used as the control group).
      Statistical comparison of baseline characteristics between study and control groups can identify unbalanced characteristics, thus indicating poor comparability. We recorded the proportion of studies that statistically compared at least 1 baseline characteristic between the study and the control groups. We then determined the number of included articles that controlled for confounding using matching or exclusion criteria at the design stage and standardization, stratification, matched analysis, or regression analyses in the data analysis stage. Lastly, as a surrogate for assessing the authors’ discussion of control group limitations, we assessed whether studies discussed confounding and selection bias. Articles were deemed to have discussed these topics if they provided a possible source of confounding or selection bias, respectively. The authors adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines in the preparation of this manuscript.

      Results

      A nonrandomized control group was used in 37 of the 236 scientific studies published in 2013 in JHS (Fig. 1). Of the included papers, 19 (51%) were retrospective cohort studies, 5 (14%) were prospective cohort studies, and 13 (35%) were case-control studies. A majority of studies, 20 (54%), investigated the effectiveness of a therapeutic intervention. There were 10 (27%) prognostic studies and 7 (19%) diagnostic studies.
      Figure thumbnail gr1
      Figure 1Flow diagram for identifying nonrandomized studies in JHS.
      The selection of controls was quite variable, with regards to source, type, number, and other defining characteristics (Table 1). The choice of controls appeared to be dependent on study design, study topic, sample size availability, and feasibility. The majority of studies used controls from the same institution (as the study cohort), but in 7 (19%) studies, the authors failed to mention the source. Concurrent controls were used in far greater frequency than historical controls. Historical controls were used when the study topic was rare such as tetraplegia reconstruction
      • Reinholdt C.
      • Friden J.
      Outcomes of single-stage grip-release reconstruction in tetraplegia.
      and split flexor pollicis longus transfer.
      • Rath S.
      Split flexor pollicis longus tendon transfer to A1 pulley for correction of paralytic Z deformity of the thumb.
      Authors in 29 (78%) studies statistically compared characteristics between their selected cohorts. Of these, 10 articles found at least 1 statistically different characteristic.
      • Afshar A.
      • Eivaziatashbeik K.
      Long-term clinical and radiological outcomes of radial shortening osteotomy and vascularized bone graft in Kienböck disease.
      • Bolmers A.
      • Luiten W.E.
      • Doornberg J.N.
      • et al.
      A comparison of the long-term outcome of partial articular (AO Type B) and complete articular (AO Type C) distal radius fractures.
      • Tosti R.
      • Ilyas A.M.
      Prospective evaluation of pronator quadratus repair following volar plate fixation of distal radius fractures.
      • Taylor K.F.
      • Lanzi J.T.
      • Cage J.M.
      • Drake M.L.
      Radial collateral ligament injuries of the thumb metacarpophalangeal joint: Epidemiology in a military population.
      • Beck J.D.
      • Irgit K.S.
      • Andreychik C.M.
      • Maloney P.J.
      • Tang X.
      • Harter G.D.
      Reverse total shoulder arthroplasty in obese patients.
      • Neuhaus V.
      • Badri O.
      • Ferree S.
      • Bot A.G.
      • Ring D.C.
      • Mudgal C.S.
      Radiographic alignment of unstable distal radius fractures fixed with 1 or 2 rows of screws in volar locking plates.
      • Kaszap B.
      • Daecke W.
      • Jung M.
      Outcome comparison of primary trapeziectomy versus secondary trapeziectomy following failed total trapeziometacarpal joint replacement.
      • Bogunovic L.
      • Gelberman R.H.
      • Goldfarb C.A.
      • Boyer M.I.
      • Calfee R.P.
      The impact of antiplatelet medication on hand and wrist surgery.
      • Hartzell T.L.
      • Kuo P.
      • Eberlin K.R.
      • Winograd J.M.
      • Day C.S.
      The overutilization of resources in patients with acute upper extremity trauma and infection.
      • Zieske L.
      • Ebersole G.C.
      • Davidge K.
      • Fox I.
      • Mackinnon S.E.
      Revision carpal tunnel surgery: a 10-year review of intraoperative findings and outcomes.
      The number of characteristics compared per study ranged widely from 1 to 38.
      • Bolmers A.
      • Luiten W.E.
      • Doornberg J.N.
      • et al.
      A comparison of the long-term outcome of partial articular (AO Type B) and complete articular (AO Type C) distal radius fractures.
      • Uehara K.
      • Miura T.
      • Morizaki Y.
      • Miyamoto H.
      • Ohe T.
      • Tanaka S.
      Ultrasonographic evaluation of displaced neurovascular bundle in Dupuytren disease.
      Age and sex were the most common comparisons. Duration of disease, disease classification, disease severity, and hand dominance were also frequently compared characteristics.
      • Tosti R.
      • Ilyas A.M.
      Prospective evaluation of pronator quadratus repair following volar plate fixation of distal radius fractures.
      • Taylor K.F.
      • Lanzi J.T.
      • Cage J.M.
      • Drake M.L.
      Radial collateral ligament injuries of the thumb metacarpophalangeal joint: Epidemiology in a military population.
      • Buckley T.
      • Mitten D.
      • Elfar J.
      The effect of informed consent on results of a standard upper extremity intake questionnaire.
      • Kameyama M.
      • Chen K.R.
      • Mukai K.
      • Shimada A.
      • Atsumi Y.
      • Yanagimoto S.
      Histopathological characteristics of stenosing flexor tenosynovitis in diabetic patients and possible associations with diabetes-related variables.
      Table 1Source and Type of Control Group Selection in JHS in 2013 (n = 37 studies)
      Parametern (%)
      Source of control
      Same institution25 (68)
      Different institution2 (5)
      Multicenter database3 (8)
      Not provided7 (19)
      Type of control
      Concurrent31 (84)
      Historical2 (5)
      Concurrent/historical4 (11)
      Number of controls
      130 (81)
      27 (19)
      Control group characteristics
      Controls receiving a different intervention13 (35)
      Normal/healthy controls8 (22)
      Subjects as self-control2 (5)
      A majority of studies attempted to control for confounding (Table 2). The most common method was restriction of participant enrollment with rigid exclusion criteria (76%). Age, concomitant injuries, recurrent disease, and comorbidities are examples of exclusion criteria used.
      • Beck J.D.
      • Irgit K.S.
      • Andreychik C.M.
      • Maloney P.J.
      • Tang X.
      • Harter G.D.
      Reverse total shoulder arthroplasty in obese patients.
      • Buckley T.
      • Mitten D.
      • Elfar J.
      The effect of informed consent on results of a standard upper extremity intake questionnaire.
      • Schrumpf M.A.
      • Lyman S.
      • Do H.
      • Gay D.M.
      • Marx R.
      • Daluiski A.
      Incidence of postoperative elbow contracture release in New York State.
      • Chen C.
      • Tang P.
      • Zhang X.
      Finger sensory reconstruction with transfer of the proper digital nerve dorsal branch.
      Matching was used in 5 (14%) studies. Age and sex were variables matched in 4 studies.
      • Neuhaus V.
      • Badri O.
      • Ferree S.
      • Bot A.G.
      • Ring D.C.
      • Mudgal C.S.
      Radiographic alignment of unstable distal radius fractures fixed with 1 or 2 rows of screws in volar locking plates.
      • Kaszap B.
      • Daecke W.
      • Jung M.
      Outcome comparison of primary trapeziectomy versus secondary trapeziectomy following failed total trapeziometacarpal joint replacement.
      • Kitay A.
      • Swanstrom M.
      • Schreiver J.J.
      • et al.
      Volar plate position and flexor tendon rupture following distal radius fracture fixation.
      • McKeon K.E.
      • London D.A.
      • Osei D.A.
      • et al.
      Ligamentous hyperlaxity and dorsal wrist ganglions.
      Although more statistically advanced, few studies (14%) used post hoc analysis to adjust for confounding.
      • Reinholdt C.
      • Friden J.
      Outcomes of single-stage grip-release reconstruction in tetraplegia.
      • Bolmers A.
      • Luiten W.E.
      • Doornberg J.N.
      • et al.
      A comparison of the long-term outcome of partial articular (AO Type B) and complete articular (AO Type C) distal radius fractures.
      • Dy C.J.
      • Lyman S.
      • Schreiber J.J.
      • Do H.T.
      • Daluiski A.
      The epidemiology of reoperation after flexor pulley reconstruction.
      • Alyanak B.
      • Kilincasian A.
      • Kutlu L.
      • Bozkurt H.
      • Aydın A.
      Psychological adjustment, maternal distress, and family functioning in children with obstetrical brachial plexus palsy.
      In total, 30 (81%) studies used at minimum 1 method to control for confounding. Although a majority of articles controlled for confounding, approximately half discussed the limitations of their selected control group. Table 3 provides excerpts from articles that discussed control group limitations. Authors explicitly discussed the threat of confounding and identified sources of selection bias in 59% and 49% of articles, respectively.
      Table 2How Authors Controlled for Confounders With Study Design and Data Analysis (n = 37 studies)
      Yes (%)No (%)
      Design
      Exclusion criteria28 (76)9 (24)
      Matching5 (14)32 (86)
      Data analysis
      Standardization0 (0)37 (100)
      Stratification0 (0)37 (100)
      Matched analysis0 (0)37 (100)
      Regression model5 (14)32 (86)
      Table 3Examples of Confounding and Selection Bias Discussion
      AuthorsDiscussionExample
      Alyanak et al
      • Alyanak B.
      • Kilincasian A.
      • Kutlu L.
      • Bozkurt H.
      • Aydın A.
      Psychological adjustment, maternal distress, and family functioning in children with obstetrical brachial plexus palsy.
      Confounding“Self-report measures might often be confounded with social desirability, defensiveness, and other reactive concerns.”
      Lee et al
      • Lee H.J.
      • Gong H.S.
      • Song C.H.
      • Lee J.E.
      • Lee Y.H.
      • Baek G.H.
      Evaluation of vitamin D level and grip strength recovery in women with a distal radius fracture.
      Confounding“Finally, we did not consider psychological factors, which may have influenced grip strength in patients recovering from injury.”
      Nydick et al
      • Nydick J.A.
      • Watt J.F.
      • Garcia M.J.
      • Williams B.D.
      • Hess A.V.
      Clinical outcomes of arthrodesis and arthroplasty for the treatment of posttraumatic wrist arthritis.
      Confounding“However, posttraumatic patients usually have higher functional demands, which may place them at risk for arthroplasty failure.”
      Bogunovic et al
      • Bogunovic L.
      • Gelberman R.H.
      • Goldfarb C.A.
      • Boyer M.I.
      • Calfee R.P.
      The impact of antiplatelet medication on hand and wrist surgery.
      Confounding“That antiplatelet patients were older may have biased us toward finding more ecchymosis and hematoma in the antiplatelet group given age-related changes in the soft tissues.”
      Reinholdt and Friden
      • Reinholdt C.
      • Friden J.
      Outcomes of single-stage grip-release reconstruction in tetraplegia.
      Confounding“Study group received more mobilization and had fewer activity restrictions.”
      Tosti and Hyas
      • Tosti R.
      • Ilyas A.M.
      Prospective evaluation of pronator quadratus repair following volar plate fixation of distal radius fractures.
      Selection bias“The average age of the repair group was significantly less than the control group, which may have introduced bias, because a younger patient may be more critical of the outcome.”
      Schrumpf et al
      • Schrumpf M.A.
      • Lyman S.
      • Do H.
      • Gay D.M.
      • Marx R.
      • Daluiski A.
      Incidence of postoperative elbow contracture release in New York State.
      Selection bias“Although we do not have epidemiological data on contracture development to explain the difference, it is possible that younger patients were more frequently offered surgical management or that they were less tolerant of functional limitations.”
      Studer et al
      • Studer A.
      • Athwal G.S.
      • MacDermid J.C.
      • Faber K.J.
      • King G.J.
      The lateral para-olecranon approach for total elbow arthroplasty.
      Selection bias“Patients operated on through the lateral para-olecranon approach were older, possibly because newer medications have had a major impact on controlling rheumatoid arthritis.”

      Discussion

      A majority of the JHS articles reviewed had well-defined controls, but only 59% and 49% elaborated on potential sources of confounding and selection bias, respectively. Previous studies have demonstrated poor disclosure of bias in observational research. Groenwold et al
      • Groenwold R.H.
      • Van Deursen A.M.
      • Hoes A.W.
      • Hak E.
      Poor quality of reporting bias in observational intervention studies: a systematic review.
      conducted a systematic review of 174 observational articles and found that the quality of reporting of confounding in articles was poor. The concerns over underreporting bias have led to standardized guidelines for conducting observational research.
      • Von Elm E.
      • Altman D.G.
      • Egger M.
      • et al.
      The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies.
      • Groenwold R.H.
      • Van Deursen A.M.
      • Hoes A.W.
      • Hak E.
      Poor quality of reporting bias in observational intervention studies: a systematic review.
      Recently, Sorensen et al
      • Sorensen A.A.
      • Wojahn R.D.
      • Manske M.C.
      • Calfee R.P.
      Using the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement to assess reporting of observational trials in hand surgery.
      identified an improvement in the quality of observational research reporting in hand surgery but not in the discussion of potential bias. Therefore, it is critical that authors discuss their control population. This includes disclosure of the eligibility criteria, selection methods, type, source, and limitations. This allows readers to accurately appraise the control group and assess the validity of conclusions (Table 4).
      Table 4Appraisal of Control Group in Observational Studies
      TermHow Confounding Is Controlled
      PurposeWhat is the purpose of the study?

      What is the purpose of the control group?
      Control group selectionWhat is the source of control?

      Is the control concurrent or historical?

      What control group characteristics were provided?
      Identification of confoundersDid the study do a literature search to identify known confounders?
      Control vs study groupDid the study statistically compare groups to demonstrate lack of comparability?
      Controlling for confoundingDid the study control for confounding with their study design?

      Restriction (exclusion criteria)

      Matching
      Did the study control for confounding with post hoc statistical analysis?

      Standardization

      Stratification analysis

      Matched analysis

      Regression models
      Confounding and bias discussionDid authors provide an in-depth discussion of confounding?

      Did authors discuss selection bias?
      Controls should represent the population from which the study group originates.
      • Peacock J.L.
      • Peacock P.J.
      Oxford Handbook of Medical Statistics.
      Hospital controls are convenient and are similar in demographics to the study group if they come from the same geographic area.
      • Wacholder S.
      • Silverman D.T.
      • McLaughlin J.K.
      • Mandel J.S.
      Selection of controls in case-control studies: II. Types of controls.
      They are most useful when participants do not have diseases related to the exposure being studied.
      • Peacock J.L.
      • Peacock P.J.
      Oxford Handbook of Medical Statistics.
      Alyanak et al
      • Alyanak B.
      • Kilincasian A.
      • Kutlu L.
      • Bozkurt H.
      • Aydın A.
      Psychological adjustment, maternal distress, and family functioning in children with obstetrical brachial plexus palsy.
      compared the emotional and behavioral characteristics of children with obstetrical brachial plexus palsy with healthy children presenting for a well-child visit from the same hospital.
      • Alyanak B.
      • Kilincasian A.
      • Kutlu L.
      • Bozkurt H.
      • Aydın A.
      Psychological adjustment, maternal distress, and family functioning in children with obstetrical brachial plexus palsy.
      The validity of inferences from this study relied on the assumption that the well-child cohort was similar with respect to extraneous factors (eg, psychological stressors) and that their geographic referral patterns were similar to those of the study group.
      The temporal relationship between comparison groups is important. As diagnosis, treatment methods, and outcome measures change over time, using historical controls may introduce bias.
      • Malay S.
      • Chung K.C.
      The choice of controls for providing validity and evidence in clinical research.
      For example, Reinholdt and Friden
      • Reinholdt C.
      • Friden J.
      Outcomes of single-stage grip-release reconstruction in tetraplegia.
      found that grip strength was significantly greater in tetraplegic patients treated with single-stage grip-release reconstruction compared with historical controls treated with staged flexion-extension grip-release reconstruction. The authors disclosed that the study group had more mobilization and fewer activity restrictions. In addition, the use of immobilization orthoses was abolished when new publications supported more aggressive rehabilitation regimens. This highlights the limited utility of historical controls with evolution of medical practices. Because of improved treatment and expertise, using historical controls tends to favor more current interventions.
      • Malay S.
      • Chung K.C.
      The choice of controls for providing validity and evidence in clinical research.
      Multiple control groups increase the power and validity of conclusions when results are concordant across control series.
      • Malay S.
      • Chung K.C.
      The choice of controls for providing validity and evidence in clinical research.
      • Wacholder S.
      • Silverman D.T.
      • McLaughlin J.K.
      • Mandel J.S.
      Selection of controls in case-control studies: III. Design options.
      For example, Zieski et al
      • Zieski L.
      • Ebersole G.C.
      • Davidge K.
      • Mackinnon S.E.
      Revision carpal tunnel surgery: a 10-year review of intraoperative findings and outcomes.
      used 3 comparison groups to evaluate outcomes of revision carpal tunnel release. The authors classified patient groups based on presenting symptoms: persistent, recurrent, or new. They found the recurrent group exhibited key clinical differences including a lesser likelihood of presenting with pain. By demonstrating differences among various controls, the authors strengthened their conclusion. However, if multiple controls result in incongruent conclusions, investigators should evaluate reasons for the conflicting results.
      • Wacholder S.
      • Silverman D.T.
      • McLaughlin J.K.
      • Mandel J.S.
      Selection of controls in case-control studies: III. Design options.
      A statistical comparison of baseline characteristics between control and study groups allows identification of unbalanced variables. However, it should not be used to demonstrate equivalency because the statistical comparison is often underpowered. Beck et al
      • Beck J.D.
      • Irgit K.S.
      • Andreychik C.M.
      • Maloney P.J.
      • Tang X.
      • Harter G.D.
      Reverse total shoulder arthroplasty in obese patients.
      compared obese patients with nonobese patients to determine function and complications after reverse total shoulder arthroplasty. Ten variables were compared that revealed a greater proportion of diabetic patients in the obese cohort. The study showed that the obese cohort was at an increased risk for complications including infection and joint instability. Despite identifying a potential confounder, the authors did not explore the impact of this confounder on their conclusion, which may explain the higher incidence of infection in the obese group.
      Approaches used by researchers to address confounding are listed in Table 5. At study design level, authors attempt to balance significant variables between comparison groups.

      Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Available at: www.cochrane-handbook.org. Accessibility verified May 17, 2014.

      Restriction of participant enrollment with exclusion criteria was used in a majority of articles because it creates similar control and study groups.
      • Rochon P.A.
      • Gurwitz J.H.
      • Sykora K.
      • et al.
      Reader’s guide to critical appraisal of cohort studies: 1. Role and design.
      • Bryant D.M.
      • Willits K.
      • Hanson B.P.
      Principles of designing a cohort study in orthopaedics.
      • Vavken P.
      • Culen G.
      • Dorotka R.
      Management of confounding in controlled orthopaedic trials: a cross-sectional study.
      Unfortunately, it limits the sample size and decreases the study power. Restriction also decreases external validity, which refers to how well the conclusions can be generalized to everyday practice.
      • Bryant D.M.
      • Willits K.
      • Hanson B.P.
      Principles of designing a cohort study in orthopaedics.
      • Paradis C.
      Bias in surgical research.
      Table 5Approaches to Control for Confounding
      Study StageApproachesHow Confounding Is Controlled
      DesignExclusion criteriaCreates a more homogeneous sample and eliminates possible confounders.
      MatchingMatches study and control subjects with respect to key variables (eg, confounder).
      AnalysisStandardizationUses standard population (ie, year 2000 US population) weights to adjust confounder.
      StratificationAnalyze data under confounder’s subgroup; for example, Mantel-Haenszel method.
      Matched analysisAnalyze data by matching subjects on potential confounder and to ensure even distribution among study groups.
      Regression modelUse forward, backward, or stepwise method for model fitting.
      From Bland B. An Introduction to Medical Statistics. 3rd ed. Oxford, UK: Oxford University Press; 2000.
      Matching, another strategy used to manage confounding, creates pairs of study and control subjects who are similar in key prognostic variables.
      • Morshed S.
      • Tornetta III, P.
      • Bhandari M.
      Analysis of observational studies: a guide to understanding statistical methods.
      Matching reduces the chance that any observed differences are a result of the matched variables.
      • Morshed S.
      • Tornetta III, P.
      • Bhandari M.
      Analysis of observational studies: a guide to understanding statistical methods.
      • Bryant D.M.
      • Willits K.
      • Hanson B.P.
      Principles of designing a cohort study in orthopaedics.
      Neuhaus et al
      • Neuhaus V.
      • Badri O.
      • Ferree S.
      • Bot A.G.
      • Ring D.C.
      • Mudgal C.S.
      Radiographic alignment of unstable distal radius fractures fixed with 1 or 2 rows of screws in volar locking plates.
      compared volar angulation in patients treated with a single versus 2 rows of distal screws after volar locked plate fixation for distal radius fractures. The case-matched study found no advantage with either method. The matching balanced potential confounders between comparison groups but also highlighted its limitations. By matching 10 variables, the number of participants was reduced greatly, thus reducing statistical power.
      • Morshed S.
      • Tornetta III, P.
      • Bhandari M.
      Analysis of observational studies: a guide to understanding statistical methods.
      In addition, 1-to-1 matching is not always feasible. In this study, age remained statistically different between comparison groups and was, therefore, a residual confounding variable.
      Post hoc statistical analysis is another method used to identify associations.

      Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Available at: www.cochrane-handbook.org. Accessibility verified May 17, 2014.

      • Normand S.L.T.
      • Sykora K.
      • Li P.
      • Rochon P.A.
      • Anderson G.M.
      Readers guide to critical appraisal of cohort studies: 3. Analytical strategies to reduce confounding.
      Regression models can estimate intervention effect adjusted for imbalances in observed prognostic factors.

      Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Available at: www.cochrane-handbook.org. Accessibility verified May 17, 2014.

      Bland M. An Introduction to Medical Statistics. 3rd ed. Oxford, UK: Oxford University Press; 2000.

      Dy et al
      • Dy C.J.
      • Lyman S.
      • Schreiber J.J.
      • Do H.T.
      • Daluiski A.
      The epidemiology of reoperation after flexor pulley reconstruction.
      used multivariable regression analysis to evaluate the influence of demographics on reoperation for flexor pulley reconstruction. They found a higher likelihood of reconstruction among men, after adjusting for age and concomitant nerve and flexor tendon repair. This analysis improved the validity of conclusions by adjusting for exposures that would otherwise be confounders. However, the authors acknowledged that the factors they found to influence reoperation were in fact surrogates for injury severity.
      • Dy C.J.
      • Lyman S.
      • Schreiber J.J.
      • Do H.T.
      • Daluiski A.
      The epidemiology of reoperation after flexor pulley reconstruction.
      Thus, the influence of injury severity may not be truly represented in this study.
      A limitation of our study was that our assessment was dependent on the information provided within the articles. If authors failed to disclose information about control selection or confounding adjustment because of text limit, we assumed these issues were not appropriately addressed.
      Reporting guidelines of observational studies emphasize the importance of discussing potential bias or imprecision in studies.
      • Von Elm E.
      • Altman D.G.
      • Egger M.
      • et al.
      The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies.
      • Stroup D.F.
      • Berlin J.A.
      • Morton S.C.
      • et al.
      Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group.
      Poor, unbalanced reporting prevents accurate assessment of strengths and weaknesses of the study and clouds the applicability of results.
      • Manchikanti L.
      • Singh V.
      • Smith H.S.
      • Hirsch J.A.
      Evidence-based medicine, systematic reviews, and guidelines in interventional pain management: part 4: observational studies.
      Thus, authors of nonrandomized studies are held accountable to select appropriate control groups and provide a discussion of potential selection bias and confounding and of the methods used to address them. Likewise, peer reviewers and editors are also responsible to address the need for such information in published articles. This allows readers to assess the accuracy of results in context of any clinical recommendation.

      References

        • Rochon P.A.
        • Gurwitz J.H.
        • Sykora K.
        • et al.
        Reader’s guide to critical appraisal of cohort studies: 1. Role and design.
        BMJ. 2005; 33: 895-897
        • Peacock J.L.
        • Peacock P.J.
        Oxford Handbook of Medical Statistics.
        Oxford University Press, New York2011: 1-73
      1. Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Available at: www.cochrane-handbook.org. Accessibility verified May 17, 2014.

        • Wacholder S.
        • McLaughlin J.K.
        • Silverman D.T.
        • Mandel J.S.
        Selection of controls in case-control studies: I. Principles.
        Am J Epidemiol. 1992; 135: 1019-1028
        • Morshed S.
        • Tornetta III, P.
        • Bhandari M.
        Analysis of observational studies: a guide to understanding statistical methods.
        J Bone Joint Surg Am. 2009; 91: 50-60
        • Busse J.W.
        • Obremskey W.T.
        Principles of designing an orthopaedic case-control study.
        J Bone Joint Surg Am. 2009; 91: 15-20
        • Bryant D.M.
        • Willits K.
        • Hanson B.P.
        Principles of designing a cohort study in orthopaedics.
        J Bone Joint Surg Am. 2009; 91: 10-14
        • Clarkson P.W.
        • Sandford K.
        • Phillips A.E.
        • et al.
        Functional results following vascularized versus nonvascularized bone grafts for wrist arthrodesis following excision of giant cell tumors.
        J Hand Surg Am. 2013; 38: 935-940.e1
        • Mamdani M.
        • Sykora K.
        • Li P.
        • et al.
        Reader’s guide to critical appraisal of cohort studies: 2. Assessing potential for confounding.
        BMJ. 2005; 330: 960-962
        • Malay S.
        • Chung K.C.
        The choice of controls for providing validity and evidence in clinical research.
        Plast Reconstr Surg. 2012; 130: 959-965
        • Paradis C.
        Bias in surgical research.
        Ann Surg. 2008; 248: 180-188
        • Young J.
        • Solomon M.
        Improving the evidence base in surgery: sources of bias in surgical studies.
        ANZ J Surg. 2003; 73: 504-506
        • Afshar A.
        • Eivaziatashbeik K.
        Long-term clinical and radiological outcomes of radial shortening osteotomy and vascularized bone graft in Kienböck disease.
        J Hand Surg Am. 2013; 38: 289-296
        • Vavken P.
        • Culen G.
        • Dorotka R.
        Management of confounding in controlled orthopaedic trials: a cross-sectional study.
        Clin Orthop Relat Res. 2008; 466: 985-989
        • Müllner M.
        • Matthews H.
        • Altman D.G.
        Reporting on statistical methods to adjust for confounding: a cross-sectional survey.
        Ann Intern Med. 2002; 136: 122-126
        • Von Elm E.
        • Altman D.G.
        • Egger M.
        • et al.
        The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies.
        J Clin Epidemiol. 2008; 61: 344-349
        • Reinholdt C.
        • Friden J.
        Outcomes of single-stage grip-release reconstruction in tetraplegia.
        J Hand Surg Am. 2013; 38: 1137-1144
        • Rath S.
        Split flexor pollicis longus tendon transfer to A1 pulley for correction of paralytic Z deformity of the thumb.
        J Hand Surg Am. 2013; 38: 1172-1180
        • Bolmers A.
        • Luiten W.E.
        • Doornberg J.N.
        • et al.
        A comparison of the long-term outcome of partial articular (AO Type B) and complete articular (AO Type C) distal radius fractures.
        J Hand Surg Am. 2013; 38: 753-759
        • Tosti R.
        • Ilyas A.M.
        Prospective evaluation of pronator quadratus repair following volar plate fixation of distal radius fractures.
        J Hand Surg Am. 2013; 38: 1678-1684
        • Taylor K.F.
        • Lanzi J.T.
        • Cage J.M.
        • Drake M.L.
        Radial collateral ligament injuries of the thumb metacarpophalangeal joint: Epidemiology in a military population.
        J Hand Surg Am. 2013; 38: 532-536
        • Beck J.D.
        • Irgit K.S.
        • Andreychik C.M.
        • Maloney P.J.
        • Tang X.
        • Harter G.D.
        Reverse total shoulder arthroplasty in obese patients.
        J Hand Surg Am. 2013; 38: 965-970
        • Neuhaus V.
        • Badri O.
        • Ferree S.
        • Bot A.G.
        • Ring D.C.
        • Mudgal C.S.
        Radiographic alignment of unstable distal radius fractures fixed with 1 or 2 rows of screws in volar locking plates.
        J Hand Surg Am. 2013; 38: 297-301
        • Kaszap B.
        • Daecke W.
        • Jung M.
        Outcome comparison of primary trapeziectomy versus secondary trapeziectomy following failed total trapeziometacarpal joint replacement.
        J Hand Surg Am. 2013; 38: 863-871
        • Bogunovic L.
        • Gelberman R.H.
        • Goldfarb C.A.
        • Boyer M.I.
        • Calfee R.P.
        The impact of antiplatelet medication on hand and wrist surgery.
        J Hand Surg Am. 2013; 38: 1063-1070
        • Hartzell T.L.
        • Kuo P.
        • Eberlin K.R.
        • Winograd J.M.
        • Day C.S.
        The overutilization of resources in patients with acute upper extremity trauma and infection.
        J Hand Surg Am. 2013; 38: 766-773
        • Zieske L.
        • Ebersole G.C.
        • Davidge K.
        • Fox I.
        • Mackinnon S.E.
        Revision carpal tunnel surgery: a 10-year review of intraoperative findings and outcomes.
        J Hand Surg Am. 2013; 38: 1530-1539
        • Uehara K.
        • Miura T.
        • Morizaki Y.
        • Miyamoto H.
        • Ohe T.
        • Tanaka S.
        Ultrasonographic evaluation of displaced neurovascular bundle in Dupuytren disease.
        J Hand Surg Am. 2013; 38: 23-28
        • Buckley T.
        • Mitten D.
        • Elfar J.
        The effect of informed consent on results of a standard upper extremity intake questionnaire.
        J Hand Surg Am. 2013; 38: 366-371
        • Kameyama M.
        • Chen K.R.
        • Mukai K.
        • Shimada A.
        • Atsumi Y.
        • Yanagimoto S.
        Histopathological characteristics of stenosing flexor tenosynovitis in diabetic patients and possible associations with diabetes-related variables.
        J Hand Surg Am. 2013; 38: 1331-1339
        • Schrumpf M.A.
        • Lyman S.
        • Do H.
        • Gay D.M.
        • Marx R.
        • Daluiski A.
        Incidence of postoperative elbow contracture release in New York State.
        J Hand Surg Am. 2013; 38: 1746-1752.e1–3
        • Chen C.
        • Tang P.
        • Zhang X.
        Finger sensory reconstruction with transfer of the proper digital nerve dorsal branch.
        J Hand Surg Am. 2013; 38A: 82-89
        • Kitay A.
        • Swanstrom M.
        • Schreiver J.J.
        • et al.
        Volar plate position and flexor tendon rupture following distal radius fracture fixation.
        J Hand Surg Am. 2013; 38: 1091-1096
        • McKeon K.E.
        • London D.A.
        • Osei D.A.
        • et al.
        Ligamentous hyperlaxity and dorsal wrist ganglions.
        J Hand Surg Am. 2013; 38: 2138-2143
        • Dy C.J.
        • Lyman S.
        • Schreiber J.J.
        • Do H.T.
        • Daluiski A.
        The epidemiology of reoperation after flexor pulley reconstruction.
        J Hand Surg Am. 2013; 38: 1705-1711
        • Alyanak B.
        • Kilincasian A.
        • Kutlu L.
        • Bozkurt H.
        • Aydın A.
        Psychological adjustment, maternal distress, and family functioning in children with obstetrical brachial plexus palsy.
        J Hand Surg Am. 2013; 38: 137-142
        • Groenwold R.H.
        • Van Deursen A.M.
        • Hoes A.W.
        • Hak E.
        Poor quality of reporting bias in observational intervention studies: a systematic review.
        Ann Epidemiol. 2008; 18: 746-751
        • Sorensen A.A.
        • Wojahn R.D.
        • Manske M.C.
        • Calfee R.P.
        Using the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement to assess reporting of observational trials in hand surgery.
        J Hand Surg Am. 2013; 38: 1584-1589
        • Wacholder S.
        • Silverman D.T.
        • McLaughlin J.K.
        • Mandel J.S.
        Selection of controls in case-control studies: II. Types of controls.
        Am J Epidemiol. 1992; 135: 1029-1041
        • Wacholder S.
        • Silverman D.T.
        • McLaughlin J.K.
        • Mandel J.S.
        Selection of controls in case-control studies: III. Design options.
        Am J Epidemiol. 1992; 135: 1042-1050
        • Zieski L.
        • Ebersole G.C.
        • Davidge K.
        • Mackinnon S.E.
        Revision carpal tunnel surgery: a 10-year review of intraoperative findings and outcomes.
        J Hand Surg Am. 2013; 38: 1530-1539
        • Normand S.L.T.
        • Sykora K.
        • Li P.
        • Rochon P.A.
        • Anderson G.M.
        Readers guide to critical appraisal of cohort studies: 3. Analytical strategies to reduce confounding.
        BMJ. 2005; 330: 1021-1023
      2. Bland M. An Introduction to Medical Statistics. 3rd ed. Oxford, UK: Oxford University Press; 2000.

        • Stroup D.F.
        • Berlin J.A.
        • Morton S.C.
        • et al.
        Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group.
        JAMA. 2000; 283: 2008-2012
        • Manchikanti L.
        • Singh V.
        • Smith H.S.
        • Hirsch J.A.
        Evidence-based medicine, systematic reviews, and guidelines in interventional pain management: part 4: observational studies.
        Pain Physician. 2009; 12: 73-108
        • Lee H.J.
        • Gong H.S.
        • Song C.H.
        • Lee J.E.
        • Lee Y.H.
        • Baek G.H.
        Evaluation of vitamin D level and grip strength recovery in women with a distal radius fracture.
        J Hand Surg Am. 2013; 38: 519-525
        • Nydick J.A.
        • Watt J.F.
        • Garcia M.J.
        • Williams B.D.
        • Hess A.V.
        Clinical outcomes of arthrodesis and arthroplasty for the treatment of posttraumatic wrist arthritis.
        J Hand Surg Am. 2013; 38: 899-903
        • Studer A.
        • Athwal G.S.
        • MacDermid J.C.
        • Faber K.J.
        • King G.J.
        The lateral para-olecranon approach for total elbow arthroplasty.
        J Hand Surg Am. 2013; 38: 2219-2226