Advertisement

A Machine Learning Algorithm to Estimate the Probability of a True Scaphoid Fracture After Wrist Trauma

      Purpose

      To identify predictors of a true scaphoid fracture among patients with radial wrist pain following acute trauma, train 5 machine learning (ML) algorithms in predicting scaphoid fracture probability, and design a decision rule to initiate advanced imaging in high-risk patients.

      Methods

      Two prospective cohorts including 422 patients with radial wrist pain following wrist trauma were combined. There were 117 scaphoid fractures (28%) confirmed on computed tomography, magnetic resonance imaging, or radiographs. Eighteen fractures (15%) were occult. Predictors of a scaphoid fracture were identified among demographics, mechanism of injury and examination maneuvers. Five ML-algorithms were trained in calculating scaphoid fracture probability. ML-algorithms were assessed on ability to discriminate between patients with and without a fracture (area under the receiver operating characteristic curve), agreement between observed and predicted probabilities (calibration), and overall performance (Brier score). The best performing ML-algorithm was incorporated into a probability calculator. A decision rule was proposed to initiate advanced imaging among patients with negative radiographs.

      Results

      Pain over the scaphoid on ulnar deviation, sex, age, and mechanism of injury were most strongly associated with a true scaphoid fracture. The best performing ML-algorithm yielded an area under the receiver operating characteristic curve, calibration slope, intercept, and Brier score of 0.77, 0.84, −0.01 and 0.159, respectively. The ML-derived decision rule proposes to initiate advanced imaging in patients with radial-sided wrist pain, negative radiographs, and a fracture probability of ≥10%. When applied to our cohort, this would yield 100% sensitivity, 38% specificity, and would have reduced the number of patients undergoing advanced imaging by 36% without missing a fracture.

      Conclusions

      The ML-algorithm accurately calculated scaphoid fracture probability based on scaphoid pain on ulnar deviation, sex, age, and mechanism of injury. The ML-decision rule may reduce the number of patients undergoing advanced imaging by a third with a small risk of missing a fracture. External validation is required before implementation.

      Type of study/level of evidence

      Diagnostic II.

      Key words

      JHS Podcast

      August 2, 2022

      JHS Podcast Episode 77

      Dr. Graham interviews Dr. Anne Eva Bulstra about her paper “A Machine Learning Algorithm to Estimate the Probability of a True Scaphoid Fracture After Wrist Trauma”, which is the lead article in the August 2022 issue of the Journal of Hand Surgery.

      Loading ...
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Hand Surgery
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Mallee W.
        • Doornberg J.N.
        • Ring D.
        • van Dijk C.N.
        • Maas M.
        • Goslings J.C.
        Comparison of CT and MRI for diagnosis of suspected scaphoid fractures.
        J Bone Joint Surg Am. 2011; 93: 20-28
        • Bergh T.H.
        • Lindau T.
        • Soldal L.A.
        • et al.
        Clinical scaphoid score (CSS) to identify scaphoid fracture with MRI in patients with normal x-ray after a wrist trauma.
        Emerg Med J. 2014; 31: 659-664
        • Daniels A.M.
        • Bevers M.S.A.M.
        • Sassen S.
        • et al.
        Improved detection of scaphoid fractures with high-resolution peripheral quantitative CT Compared with conventional CT.
        J Bone Joint Surg Am. 2020; 102: 2138-2145
        • Rua T.
        • Gidwani S.
        • Malhotra B.
        • et al.
        Cost-effectiveness of immediate magnetic resonance imaging in the management of patients with suspected scaphoid fracture: results from a randomized clinical trial.
        Value Health. 2020; 23: 1444-1452
        • Suh N.
        • Grewal R.
        Controversies and best practices for acute scaphoid fracture management.
        J Hand Surg Eur Vol. 2018; 43: 4-12
        • Jenkins P.J.
        • Slade K.
        • Huntley J.S.
        • Robinson C.M.
        A comparative analysis of the accuracy, diagnostic uncertainty and cost of imaging modalities in suspected scaphoid fractures.
        Injury. 2008; 39: 768-774
        • Karl J.W.
        • Swart E.
        • Strauch R.J.
        Diagnosis of occult scaphoid fractures: a cost-effectiveness analysis.
        J Bone Joint Surg Am. 2015; 97: 1860-1868
        • Mallee W.H.
        • Wang J.
        • Poolman R.W.
        • et al.
        Computed tomography versus magnetic resonance imaging versus bone scintigraphy for clinically suspected scaphoid fractures in patients with negative plain radiographs.
        Cochrane Database Syst Rev. 2015; : CD010023
        • Ring D.
        • Lozano-Calderon S.
        Imaging for suspected scaphoid fracture.
        J Hand Surg Am. 2008; 33: 954-957
        • Pillai A.
        • Jain M.
        Management of clinical fractures of the scaphoid: results of an audit and literature review.
        Eur J Emerg Med. 2005; 12: 47-51
        • DaCruz D.J.
        • Bodiwala G.G.
        • Finlay D.B.
        The suspected fracture of the scaphoid: a rational approach to diagnosis.
        Injury. 1988; 19: 149-152
        • Duckworth A.D.
        • Buijze G.A.
        • Moran M.
        • et al.
        Predictors of fracture following suspected injury to the scaphoid.
        J Bone Joint Surg Br. 2012; 94: 961-968
        • Mallee W.H.
        • Walenkamp M.M.J.
        • Mulders M.A.M.
        • Goslings J.C.
        • Schep N.W.L.
        Detecting scaphoid fractures in wrist injury: a clinical decision rule.
        Arch Orthop Trauma Surg. 2020; 140: 575-581
        • Bertsimas D.
        • Dunn J.
        • Velmahos G.C.
        • Kaafarani H.M.A.
        Surgical risk is not linear: derivation and validation of a novel, user-friendly, and machine-learning-based Predictive OpTimal Trees in Emergency Surgery Risk (POTTER) calculator.
        Ann Surg. 2018; 268: 574-583
        • Staartjes V.E.
        • de Wispelaere M.P.
        • Vandertop W.P.
        • Schroder M.L.
        Deep learning-based preoperative predictive analytics for patient-reported outcomes following lumbar diskectomy: feasibility of center-specific modeling.
        Spine J. 2018;
        • Christodoulou E.
        • Ma J.
        • Collins G.S.
        • Steyerberg E.W.
        • Verbakel J.Y.
        • Van Calster B.
        A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models.
        J Clin Epidemiol. 2019; 110: 12-22
        • Kim D.H.
        • MacKinnon T.
        Artificial intelligence in fracture detection: transfer learning from deep convolutional neural networks.
        Clin Radiol. 2018; 73: 439-445
        • Langerhuizen D.W.G.
        • Bulstra A.E.J.
        • Janssen S.J.
        • et al.
        Is Deep learning on par with human observers for detection of radiographically visible and occult fractures of the scaphoid?.
        Clin Orthop Relat Res. 2020; 478: 2653-2659
        • Langerhuizen D.W.G.
        • Janssen S.J.
        • Mallee W.H.
        • et al.
        What are the applications and limitations of artificial intelligence for fracture detection and classification in orthopaedic trauma imaging? A systematic review.
        Clin Orthop Relat Res. 2019; 477: 2482-2491
        • Goto T.
        • Camargo Jr., C.A.
        • Faridi M.K.
        • Freishtat R.J.
        • Hasegawa K.
        Machine learning-based prediction of clinical outcomes for children during emergency department triage.
        JAMA Netw Open. 2019; 2e186937
        • Oosterhoff J.H.F.
        • Doornberg J.N.
        Machine Learning Consortium. Artificial intelligence in orthopaedics: false hope or not? A narrative review along the line of Gartner's hype cycle.
        EFORT Open Rev. 2020; 5: 593-603
        • Karhade A.V.
        • Ogink P.
        • Thio Q.
        • et al.
        Development of machine learning algorithms for prediction of discharge disposition after elective inpatient surgery for lumbar degenerative disc disorders.
        Neurosurg Focus. 2018; 45: E6
        • Karhade A.V.
        • Thio Q.C.B.S.
        • Ogink P.T.
        • et al.
        Development of machine learning algorithms for prediction of 30-day mortality after surgery for spinal metastasis.
        Neurosurgery. 2019; 85: E83-E91
        • Levin S.
        • Toerper M.
        • Hamrock E.
        • et al.
        Machine-learning-based electronic triage more accurately differentiates patients with respect to clinical outcomes compared with the emergency severity index.
        Ann Emerg Med. 2018; 71: 565-574 e562
        • Thio Q.C.B.S.
        • Karhade A.V.
        • Ogink P.T.
        • et al.
        Can machine-learning techniques be used for 5-year survival prediction of patients with chondrosarcoma?.
        Clin Orthop Relat Res. 2018; 476: 2040-2048
        • Machine Learning Consortium on behalf of the SPRINT and FLOW Investigators
        A machine learning algorithm to identify patients with tibial shaft fractures at risk for infection after operative treatment.
        J Bone Joint Surg Am. 2021; 103: 532-540
        • Bayliss L.
        • Jones L.D.
        The role of artificial intelligence and machine learning in predicting orthopaedic outcomes.
        Bone Joint J. 2019; 101-B: 1476-1478
        • Luo W.
        • Phung D.
        • Tran T.
        • et al.
        Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view.
        J Med Internet Res. 2016; 18: e323
        • Moons K.G.
        • Altman D.G.
        • Reitsma J.B.
        • et al.
        Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration.
        Ann Intern Med. 2015; 162: W1-W73
        • Stekhoven D.J.
        • Buhlmann P.
        MissForest--non-parametric missing value imputation for mixed-type data.
        Bioinformatics. 2012; 28: 112-118
        • Kursa M.B.
        • Rudnkicki W.R.
        Feature selection with the Boruta Package.
        J Stat Softw. 2010; 36
        • Hendrickx L.A.M.
        • Sobol G.L.
        • Langerhuizen D.W.G.
        • et al.
        A machine learning algorithm to predict the probability of (occult) posterior malleolar fractures associated with tibial shaft fractures to guide “malleolus first” fixation.
        J Orthop Trauma. 2020; 34: 131-138
        • Steyerberg E.W.
        Validation in prediction research: the waste by data splitting.
        J Clin Epidemiol. 2018; 103: 131-133
        • Esteva A.
        • Kuprel B.
        • Novoa R.A.
        • et al.
        Dermatologist-level classification of skin cancer with deep neural networks.
        Nature. 2017; 542: 115-118
        • Maroco J.
        • Silva D.
        • Rodrigues A.
        • Guerreiro M.
        • Santana I.
        • de Mendonca A.
        Data mining methods in the prediction of dementia: a real-data comparison of the accuracy, sensitivity and specificity of linear discriminant analysis, logistic regression, neural networks, support vector machines, classification trees and random forests.
        BMC Res Notes. 2011; 4: 299
        • Steyerberg E.W.
        • Vickers A.J.
        • Cook N.R.
        • et al.
        Assessing the performance of prediction models: a framework for traditional and novel measures.
        Epidemiology. 2010; 21: 128-138
        • Steyerberg E.W.
        • Vergouwe Y.
        Towards better clinical prediction models: seven steps for development and an ABCD for validation.
        Eur Heart J. 2014; 35: 1925-1931
        • Mallee W.H.
        • Henny E.P.
        • van Dijk C.N.
        • Kamminga S.P.
        • van Enst W.A.
        • Kloen P.
        Clinical diagnostic evaluation for scaphoid fractures: a systematic review and meta-analysis.
        J Hand Surg Am. 2014; 39: 1683-1691 e1682
        • Walker-Bone K.
        • Byng P.
        • Linaker C.
        • et al.
        Reliability of the Southampton examination schedule for the diagnosis of upper limb disorders in the general population.
        Ann Rheum Dis. 2002; 61: 1103-1106
      1. Scaphoid Fractures - Artificial Intelligence Prediction Tool - Artificial Intelligence.

      Linked Article

      • Erratum
        Journal of Hand SurgeryVol. 47Issue 12
        • Preview
          In the article by Bulstra and Machine Learning Consortium in the August 2022 issue of The Journal of Hand Surgery (“A Machine Learning Algorithm to Estimate the Probability of a True Scaphoid Fracture After Wrist Trauma”, Vol. 47, No. 8, p. 709-718), one of the Machine Learning Consortium collaborators was listed incorrectly. “Carel (J.C.) Goslings, MD, PhD” should be “J. Carel Goslings, MD, PhD.” The online version of the article has been updated. The authors regret this error.
        • Full-Text
        • PDF