Advertisement

Readability of Online Hand Surgery Patient Educational Materials: Evaluating the Trend Since 2008

      Purpose

      Online patient educational materials have historically been written at a higher-than-recommended sixth grade reading level. The objectives of this study were to assess the readability of online hand surgery patient educational materials from the official online patient resource website of the American Society for Surgery of the Hand (ASSH) and to compare changes in the readability of the current ASSH online patient educational materials with those in 2008 and 2015.

      Methods

      An internet-based study of all 88 English language patient educational materials on HandCare.org, the official online patient resource website of the ASSH, was performed. The readability of each article was assessed using the Flesch reading ease formula, Flesch-Kincaid grade level, Coleman-Liau index, Gunning-Fog index, and Simple Measure of Gobbledygook grade level. To evaluate the trend in the readability of ASSH online hand surgery patient educational materials, the Flesch-Kincaid grade levels of articles published in 2020 were compared with those of data published in 2008 and 2015.

      Results

      The average Flesch reading ease score of the patient educational materials was 57.6, which is at the high-school reading level. The average reading grade level of patient educational materials ranged from 9.0 to 12.3 depending on the readability metric used. The average Flesch-Kincaid grade level of all the ASSH patient educational materials was 9.8 in 2020, which is significantly better than 10.4 in 2008 but significantly worse than 8.5 in 2015.

      Conclusions

      Online hand surgery patient educational materials continue to be written for the general public at a higher-than-recommended reading grade level. There has been no substantial improvement in the readability of online hand surgery patient educational materials since 2008.

      Clinical relevance

      Improvements are needed in the readability of online patient educational materials to ensure that patients with all health literacy levels are able to comprehend and benefit from health information.

      Key words

      Patients undergoing hand surgery commonly use the internet as a resource for health information. Nearly 50% of patients undergoing hand surgery research their condition prior to their initial consultation.
      • Rao A.J.
      • Dy C.J.
      • Goldfarb C.A.
      • Cohen M.S.
      • Wysocki R.W.
      Patient preferences and utilization of online resources for patients treated in hand surgery practices.
      For some, the internet is a resource for self-diagnosis, whereas for others, online information may frame the understanding of their hand pathology and affect subsequent discussions and decision-making in the surgeon’s office.
      • Rao A.J.
      • Dy C.J.
      • Goldfarb C.A.
      • Cohen M.S.
      • Wysocki R.W.
      Patient preferences and utilization of online resources for patients treated in hand surgery practices.
      ,
      • Hageman M.G.
      • Anderson J.
      • Blok R.
      • Bossen J.K.
      • Ring D.
      Internet self-diagnosis in hand surgery.
      Therefore, it is important that patients of all backgrounds have access to high-quality hand surgery online educational materials that are appropriately written for their health literacy level.
      The National Institutes of Health, Centers for Disease Control and Prevention, and American Medical Association have recommended that health information be written at no higher than a sixth grade reading level
      U.S. Department for Health and Human Services. National Institutes of Health
      Clear Communication.
      U.S. Department for Health and Human Services. Centers for Disease Control and Prevention
      Simply Put: A Guide for Creating Easy-To-Understand Materials.
      • Weiss B.D.
      Health Literacy: A Manual for Clinicians.
      ; however, prior evidence has demonstrated that online patient educational materials for orthopedic surgery are consistently written at a higher-than-recommended reading level.
      • Roberts H.
      • Zhang D.
      • Dyer G.S.
      The readability of AAOS patient education materials: evaluating the progress since 2008.
      Similarly, in hand surgery, patient educational materials from the American Academy of Orthopaedic Surgeons, American Society for Surgery of the Hand (ASSH), and American Association for Hand Surgery have been shown to have inadequate readability.
      • Beutel B.G.
      • Danna N.R.
      • Melamed E.
      • Capo J.T.
      Comparative readability of shoulder and elbow patient education materials within orthopaedic websites.
      • Wang S.W.
      • Capo J.T.
      • Orillaza N.
      Readability and comprehensibility of patient education material in hand-related web sites.
      • Hadden K.
      • Prince L.Y.
      • Schnaekel A.
      • Couch C.G.
      • Stephenson J.M.
      • Wyrick T.O.
      Readability of patient education naterials in hand surgery and health literacy best practices for improvement.
      • Yi P.H.
      • Chang M.M.
      • Haughom B.D.
      • Jawa A.
      Readability of patient education materials from the AAHS.
      Although the analysis of ASSH online patient educational materials in 2008, and subsequently in 2015, showed small improvements in readability over time, the materials were largely written at a higher-than-recommended reading level.
      • Wang S.W.
      • Capo J.T.
      • Orillaza N.
      Readability and comprehensibility of patient education material in hand-related web sites.
      ,
      • Hadden K.
      • Prince L.Y.
      • Schnaekel A.
      • Couch C.G.
      • Stephenson J.M.
      • Wyrick T.O.
      Readability of patient education naterials in hand surgery and health literacy best practices for improvement.
      We aimed to perform a replication study of earlier works on the readability of online hand surgery patient educational materials from HandCare.org, the official online patient resource website of the ASSH, in order to determine the trend in the readability of ASSH online patient educational materials since 2008.
      • Wang S.W.
      • Capo J.T.
      • Orillaza N.
      Readability and comprehensibility of patient education material in hand-related web sites.
      ,
      • Hadden K.
      • Prince L.Y.
      • Schnaekel A.
      • Couch C.G.
      • Stephenson J.M.
      • Wyrick T.O.
      Readability of patient education naterials in hand surgery and health literacy best practices for improvement.
      Our null hypothesis was that there has been no change in the readability of ASSH online patient educational materials from 2008 to 2020.

      Materials and Methods

      An internet-based readability study was performed without clinical subjects, and institutional review board approval was not required and not obtained. In April 2020, online hand surgery patient educational materials were collected from HandCare.org, the official online patient resource website of the ASSH. All articles written in the English language were included. The articles were copied in plain-text format and analyzed using an open-source software for the calculation of readability scores (ReadabilityFormulas.com, My Byline Media). Where applicable, the articles were categorized based on relevant anatomic location: hand and digit, wrist and forearm, elbow and arm, or shoulder and brachial plexus. Eight articles were classified by HandCare.org as topics related to hand safety; these topics included firework safety, gardening safety, lawn mower safety, pumpkin carving safety, snow blower safety, table saw safety, turkey carving safety, and how to remove a ring. A total of 88 unique hand surgery patient educational articles in the English language were included in this study.
      The readability of each of the 88 articles was assessed using 5 validated and widely accepted readability metrics by one of the authors (D.Z.): Flesch reading ease formula,
      • Flesch R.
      A new readability yardstick.
      Flesch-Kincaid grade level,
      • Kincaid P.J.
      • Fishburne Jr., R.P.
      • Rogers R.L.
      • Chissom B.S.
      Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel.
      Coleman-Liau index,
      • Coleman M.
      • Liau T.L.
      A computer readability formula designed for machine scoring.
      Gunning-Fog index,
      • Gunning R.
      The Technique of Clear Writing.
      and Simple Measure of Gobbledygook (SMOG) grade level (Table 1).
      • McLaughlin H.G.
      SMOG grading—a new readability formula.
      The Flesch reading ease score ranges from 0 to 100, where higher scores represent more readable materials. A Flesch reading ease score of 90–100 is suitable for elementary school students, a score of 60–70 is suitable for middle school students, and a score of 0–30 is suitable for university students. The remaining 4 readability metrics (the Flesch-Kincaid grade level, Coleman-Liau index, Gunning-Fog index, and SMOG grade level) generate scores that correspond directly to the academic grade level required to comprehend the material. Therefore, lower scores on these readability metrics represent more readable materials. Twenty of the 88 articles were selected by a random number generator. Assessments of the readability metrics, by copying the plain-text format articles into an open-source readability score software, were performed by a second independent investigator (E.K.). The intraclass correlation coefficient among the scores of these 20 articles was used to assess the interobserver reliability of the task of calculating readability scores from the plain-text articles using the open-source readability score software.
      Table 1Algorithms for Readability Metrics
      Readability MetricAlgorithm
      Complex words are defined as words with 3 or more syllables.
      Flesch reading ease206.8351.015(wordssentences)84.6(syllableswords)
      Flesch-Kincaid grade level0.39(wordssentences)+11.8(syllableswords)15.59
      Coleman-Liau index0.0588(letters100words)0.296(sentences100words)15.8
      Gunning-Fog index0.4[(wordssentences)+100(complexwordswords)]
      SMOG grade level1.0430complexwords×30sentences+3.1291
      Complex words are defined as words with 3 or more syllables.
      To evaluate the trend in the readability of ASSH online hand surgery patient educational materials, a readability scale was used. The Flesch-Kincaid grade level was chosen because it is the most commonly used readability metric in the field of orthopedic surgery and has been previously used for temporal comparisons.
      • Roberts H.
      • Zhang D.
      • Dyer G.S.
      The readability of AAOS patient education materials: evaluating the progress since 2008.
      The Flesch-Kincaid grade levels of the 88 articles included in this study were compared with those of 49 articles measured by Wang et al
      • Wang S.W.
      • Capo J.T.
      • Orillaza N.
      Readability and comprehensibility of patient education material in hand-related web sites.
      in 2008 and 77 articles measured by Hadden et al
      • Hadden K.
      • Prince L.Y.
      • Schnaekel A.
      • Couch C.G.
      • Stephenson J.M.
      • Wyrick T.O.
      Readability of patient education naterials in hand surgery and health literacy best practices for improvement.
      in 2015.
      Descriptive statistics for the readability scores were calculated. Histograms were generated to graphically depict the distribution of readability scores obtained using each readability metric. Concordance between the readability metrics was calculated using the Pearson correlation coefficient and depicted graphically to show pairwise relationships. Multiple linear regression was used to test for differences in the readability of the articles based on anatomic location. The t test was used to test for differences in the readability of articles classified as hand safety-related educational materials because we hypothesized that these articles are written at lower reading grade levels. The t test was used to analyze differences in the readability of ASSH patient educational materials in 2020 compared with that in 2008 and 2015. A standard significance criteria of α of 0.05 was employed.

      Results

      Readability scores were calculated for all 88 online hand surgery patient educational articles (Table 2). The average Flesch reading ease score of the patient educational materials was 57.6, which is at the high-school reading level (Fig. 1A). The average reading grade level of the patient educational materials ranged from 9.0 to 12.3 depending on the readability metric used (Fig. 1B–E). No article was written below the sixth grade reading level, as determined by any of the readability metrics. The intraclass correlation coefficient was 99.2%, indicating excellent interobserver reliability among the study investigators.
      Table 2Readability scores of Online Hand Surgery Patient Educational Materials
      Readability MetricScore (Mean ± SD)Range (Minimum–Maximum)
      Flesch reading ease57.6 ± 8.235.7–75.6
      Flesch-Kincaid grade level9.8 ± 1.76.5–16.6
      Coleman-Liau index10.1 ± 1.47.0–14.0
      Gunning-Fog index12.3 ± 1.89.2–19.7
      SMOG grade level9.0 ± 1.46.3–13.9
      Figure thumbnail gr1
      Figure 1Histograms demonstrating the readability of online hand surgery patient educational materials assessed using the A Flesch reading ease formula, B Flesch-Kincaid grade level, C Coleman-Liau index, D Gunning-Fog index, and E SMOG grade level.
      The readability scores differed based on algorithm. The Coleman-Liau index and Gunning-Fog index consistently resulted in higher reading grade levels than the Flesch-Kincaid grade level test and SMOG grade level test. Concordance was observed between the algorithms for reading grade level used in this study (Fig. 2). Strong positive correlations were seen between the Flesch-Kincaid grade level and Gunning-Fog index, Flesch-Kincaid grade level, and SMOG grade level, between the Gunning-Fog index and SMOG grade level, and between the Coleman-Liau index and SMOG grade level. Moderate positive correlations were seen between the Flesch-Kincaid grade level and Coleman-Liau index and between the Coleman-Liau index and Gunning-Fog index.
      Figure thumbnail gr2
      Figure 2Scatter plots with regression lines demonstrating correlations between the A Flesch-Kincaid grade level and Coleman-Liau index, B Flesch-Kincaid grade level and Gunning-Fog index, C Flesch-Kincaid grade level and SMOG grade level, D Gunning-Fog index and Coleman-Liau index, E Gunning-Fog index and SMOG formula, and F Coleman-Liau index and SMOG grade level.
      Sixty-seven of the 88 articles were categorized based on anatomic location, including 41 articles on hand and digit conditions, 11 articles on wrist and forearm conditions, 9 articles on elbow and arm conditions, and 6 articles on shoulder and brachial plexus conditions. The average (± SD) Flesch-Kincaid grade level of the articles pertaining to these anatomic locations was 9.5 ± 1.7, 9.6 ± 1.2, 9.7 ± 1.5, and 10.7 ± 1.2, respectively. The multiple linear regression showed no significant difference in the Flesch-Kincaid grade level based on anatomic location (P = .3).
      The average (± SD) Flesch-Kincaid grade level of the 8 articles on hand safety was 8.2 ± 1.4. The average Flesch-Kincaid grade level of the remaining patient educational articles was 10.0 ± 1.6. This difference was statistically significant (P < .05).
      The average (± SD) Flesch-Kincaid grade level of all ASSH patient educational materials was 9.8 ± 1.7 in 2020. This was significantly better than a reading grade level of 10.4 ± 1.6 in 2008 (P < .05)
      • Wang S.W.
      • Capo J.T.
      • Orillaza N.
      Readability and comprehensibility of patient education material in hand-related web sites.
      ; however, this was significantly worse than a reading grade level of 8.5 ± 1.5 in in 2015 (P < .05, Fig. 3).
      • Hadden K.
      • Prince L.Y.
      • Schnaekel A.
      • Couch C.G.
      • Stephenson J.M.
      • Wyrick T.O.
      Readability of patient education naterials in hand surgery and health literacy best practices for improvement.
      From 2015 to 2020, 15 new articles were added to HandCare.org, and 4 existing articles were removed. Among the 73 articles on identical topics, paired t test showed a significant difference in the Flesch-Kincaid grade level (P < .05), and only 4 articles had the same Flesch-Kincaid grade level, suggesting that the content had been revised over time. For the 73 topics carried over from 2015 to 2020, the reading grade level was significantly higher in 2020 (9.6 ± 1.5) compared with that in 2015 (8.5 ± 1.5, P < .05). The reading grade level of the newly added educational materials was 10.7 ± 2.2, which was significantly higher than 9.6 ± 1.5 for the revised existing materials (P < .05).
      Figure thumbnail gr3
      Figure 3Violin plot demonstrating the reading grade level of online hand surgery patient educational materials in 2008 (n = 49), 2015 (n = 77), and 2020 (n = 88). The width of each violin represents the density of articles written at a particular reading grade level. The red line represents the NIH-recommended sixth grade reading level for health information. NIH, National Institutes of Health.

      Discussion

      The internet has become an important source of health information for patients; however, patients with varying levels of health literacy approach online resources. Since the average adult reading grade level in the United States is at the eighth grade level, the National Institutes of Health and numerous other agencies have recommended that health information be written at the sixth grade or lower reading level to ensure appropriate readability.
      U.S. Department for Health and Human Services. National Institutes of Health
      Clear Communication.
      U.S. Department for Health and Human Services. Centers for Disease Control and Prevention
      Simply Put: A Guide for Creating Easy-To-Understand Materials.
      • Weiss B.D.
      Health Literacy: A Manual for Clinicians.
      ,
      • Hadden K.
      • Prince L.Y.
      • Schnaekel A.
      • Couch C.G.
      • Stephenson J.M.
      • Wyrick T.O.
      Readability of patient education naterials in hand surgery and health literacy best practices for improvement.
      Patients undergoing hand surgery may choose to obtain online information from simple search engine results,
      • Akinleye S.D.
      • Garofolo-Gonzalez G.
      • Montuori M.
      • Culbertson M.D.
      • Hashem J.
      • Edelstein D.M.
      Readability of the most commonly accessed online patient education materials pertaining to pathology of the hand.
      institutional and hospital websites,
      • Cook J.A.
      • Sasor S.E.
      • Tholpady S.S.
      • Momeni A.
      • Chu M.W.
      Hand surgery resources exceed American health literacy.
      or national organizations,
      • Beutel B.G.
      • Danna N.R.
      • Melamed E.
      • Capo J.T.
      Comparative readability of shoulder and elbow patient education materials within orthopaedic websites.
      • Wang S.W.
      • Capo J.T.
      • Orillaza N.
      Readability and comprehensibility of patient education material in hand-related web sites.
      • Hadden K.
      • Prince L.Y.
      • Schnaekel A.
      • Couch C.G.
      • Stephenson J.M.
      • Wyrick T.O.
      Readability of patient education naterials in hand surgery and health literacy best practices for improvement.
      • Yi P.H.
      • Chang M.M.
      • Haughom B.D.
      • Jawa A.
      Readability of patient education materials from the AAHS.
      but no option has been shown to consistently provide readable materials on hand surgery conditions. The problem of poor readability of patient educational materials is pervasive and comparable with other surgical subspecialties, such as spine surgery,
      • Eltorai A.E.
      • Cheatham M.
      • Naqvi S.S.
      • et al.
      Is the readability of spine-related patient education material improving? An assessment of subspecialty websites.
      joint arthroplasty,
      • Doinn T.Ó.
      • Broderick J.M.
      • Abdelhalim M.M.
      • Quinlan J.F.
      Readability of patient educational materials in hip and knee arthroplasty: has a decade nade a difference?.
      vascular surgery,
      • Scott B.B.
      • Johnson A.R.
      • Doval A.F.
      • Tran B.N.
      • Lee B.T.
      Readability and understandability analysis of online materials related to abdominal aortic aneurysm repair.
      bariatric surgery,
      • Meleo-Erwin Z.
      • Basch C.
      • Fera J.
      • Ethan D.
      • Garcia P.
      Readability of online patient-based information on bariatric surgery.
      and pediatric surgery,
      • Prieto J.M.
      • West-Santos C.
      • Montgomery A.S.
      • et al.
      Patient-oriented online resources in pediatric surgery: are we failing the readability test?.
      as well as medical topics, such as diabetes mellitus,
      • Lipari M.
      • Berlie H.
      • Saleh Y.
      • Hang P.
      • Moser L.
      Understandability, actionability, and readability of online patient education materials about diabetes mellitus.
      heart murmurs,
      • Arslan D.
      • Tutar M.S.
      • Kozanhan B.
      • Bagci Z.
      The quality, understandability, readability, and popularity of online educational materials for heart murmur.
      and cancer.
      • Kim C.
      • Prabhu A.V.
      • Hansberry D.R.
      • Agarwal N.
      • Heron D.E.
      • Beriwal S.
      Digital era of mobile communications and smartphones: a novel analysis of patient comprehension of cancer-related information available through mobile applications.
      In this study of ASSH online hand surgery patient educational materials, we showed that information is written at a ninth to 12th grade reading level on an average. Of the 88 articles on HandCare.org, no article was written at the recommended sixth grade reading level, as assessed by any of the 5 readability metrics used.
      Although there has been a significant improvement in the readability of online hand surgery patient educational materials since 2008, the progress has been modest and liable to fluctuation. Wang et al
      • Wang S.W.
      • Capo J.T.
      • Orillaza N.
      Readability and comprehensibility of patient education material in hand-related web sites.
      demonstrated that ASSH online hand surgery patient educational materials had an average Flesch-Kincaid grade level of 10.4 in 2008. In a follow-up study, Hadden et al
      • Hadden K.
      • Prince L.Y.
      • Schnaekel A.
      • Couch C.G.
      • Stephenson J.M.
      • Wyrick T.O.
      Readability of patient education naterials in hand surgery and health literacy best practices for improvement.
      demonstrated an improvement in the average reading grade level to 8.5 in 2015. In our study conducted in 2020, we demonstrated an average reading grade level of 9.8. Although this represents a modest improvement from the level in 2008, it is worse than that in 2015. Because the volume of materials available to patients has increased, it is possible that newer materials are being produced at higher-than-recommended reading levels. It is unclear whether these minor fluctuations in readability have an important effect on the comprehension of online patient educational materials by the general public. We did not find a substantial improvement in online patient educational materials in this replication study, and the materials continue to be written at a higher-than-suggested reading grade level.
      A number of strategies exist for improving the accessibility of online patient health information. One strategy is the establishment of a committee to oversee online content and ensure appropriate readability. Using shorter words and sentences, avoiding unnecessary jargon, and minimizing compound sentences can help improve readability. Information on even the most complicated medical diagnoses can be effectively conveyed at the recommended reading grade level (Table 3). Another strategy is the incorporation of multimedia supplements, such as pictures or videos, which can aid in the understanding of hand conditions, regardless of reading proficiency. The use of interactive web interfaces can leverage motion, animation, and user participation to provide the end user with a customized educational experience, which may translate to better content comprehension compared with that by reading text alone.
      • Roberts H.J.
      • Zhang D.
      • Earp B.E.
      • Blazar P.
      • Dyer G.S.
      Patient self-reported utility of hand surgery online patient education materials.
      Finally, an increasing number of patients undergoing hand surgery now use social-networking platforms.
      • Klietz M.L.
      • Kaiser H.W.
      • Machens H.G.
      • Aitzetmüller M.M.
      Social media marketing: what do prospective patients want to see?.
      ,
      • Rozental T.D.
      • George T.M.
      • Chacko A.T.
      Social networking among upper extremity patients.
      The content of social-networking websites has been demonstrated to be generally more readable,
      • Zhang D.
      • Schumacher C.
      • Harris M.B.
      • Bono C.M.
      The quality and readability of information available on the internet regarding lumbar fusion.
      and these platforms may be leveraged by providers, institutions, and organizations to deliver high-quality and understandable health information.
      Table 3Excerpt of Patient Educational Material Written at a Higher-Than-Recommended Reading Grade Level
      Brachial Plexus Injury
      HandCare: The Upper Extremity Expert.
      and at the Appropriate Reading Grade Level
      Poor ReadabilityAppropriate Readability
      Brachial plexus injuryBrachial plexus injury
      The brachial plexus is a group of nerves that come from the spinal cord in the neck and travel down the arm. These nerves control the muscles of the shoulder, elbow, wrist and hand, as well as provide feeling in the arm. Some brachial plexus injuries are minor and will completely recover in several weeks. Other injuries are severe enough and could cause some permanent disability in the arm.The brachial plexus is a group of nerves that travel from the neck down the arm. These nerves allow the shoulder, elbow, wrist, and hand to move. They provide a sense of feeling in the hand and arm. When injuries happen, some are minor and get better in weeks. Others are severe and can cause permanent disability of the arm.
      These nerves can be damaged by stretching, pressure or cutting. Stretching can occur when the head and neck are forced away from the shoulder, such as during a motorcycle fall or car accident. If severe enough, the nerves can tear out of the spinal cord in the neck. Pressure could occur from the crushing of the brachial plexus between the collarbone and first rib, which can happen during a fracture or dislocation. Swelling in this area from excessive bleeding or injured soft tissues can also cause an injury.These nerves can be hurt by a stretch, a squeeze, or a cut. Bad stretches can occur when the head and neck are pulled away from the shoulder. This can happen in motorcycle accidents, car accidents, or falls. If bad enough, the nerves can tear away from the neck. These nerves can also be hurt by a squeeze, such as from a broken collar bone or first rib. Bleeding or swelling in this area can also causes these nerves to get squeezed.
      Nerve injuries can stop signals to and from the brain, preventing the muscles of the arm and hand from working properly, and causing loss of feeling in the area.Nerve injuries can stop signals to and from the brain. They can cause the muscles of the arm and hand to not work. They can also cause loss of feeling in the area.
      Many adult injuries will not recover on their own, and early evaluation by physicians who have experience treating these problems is essential. Some injuries can recover with time and therapy. The time for recovery can be weeks or months. When an injury is unlikely to improve, several surgical techniques can be used to improve the recovery.Many adult injuries do not get better on their own. It is important to be seen early on by a doctor who deals with these problems. Some injuries can get better with time and exercise. This can take weeks or months. There are surgeries that can help treat bad injuries not likely to recover on their own.
      To help decide which injuries are likely to recover, your physician will rely upon multiple examinations of the arm and hand to check the strength of muscles and presence of feeling in different areas. Additional testing, such as an MRI scan or CT scan/myelography, may be used. A Nerve Conduction Study/Electromyogram (NCS/EMG), a test that measures the electrical activity transmitted by nerves and muscles, may also be performed.To judge which injuries are likely to get better, your doctor will examine muscle strength and feeling in different areas of the arm time and again. Additional tests, such as special MRI scans, CT scans, and nerve tests, that measure the electrical activity of injured nerves, may also be used.
      Flesch-Kincaid grade level: 10.4Flesch-Kincaid grade level: 5.9
      CT, computed tomography; EMG, electromyography; MRI, magnetic resonance imaging; NCS, nerve conduction study.
      There are several limitations to this study. First, we assessed readability using 5 validated and widely used readability algorithms. Although comprehensive, these algorithms are not exhaustive. The other metrics of readability include the Fry readability graph and New Dale-Chall score.
      • Hadden K.
      • Prince L.Y.
      • Schnaekel A.
      • Couch C.G.
      • Stephenson J.M.
      • Wyrick T.O.
      Readability of patient education naterials in hand surgery and health literacy best practices for improvement.
      Second, because we analyzed the plain-text version of online patient educational materials, we were unable to account for multimedia illustrations, which may improve comprehension. To the authors’ knowledge, no tool exists that measures the overall comprehensibility of a website that accounts for illustrations and videos, although there may be a role for further studies of the patient-perceived utility of educational materials.
      • Roberts H.J.
      • Zhang D.
      • Earp B.E.
      • Blazar P.
      • Dyer G.S.
      Patient self-reported utility of hand surgery online patient education materials.
      Third, we assessed readability at 1 time point, and we were unable to comment on whether or how frequently online materials changed from 2008 to 2020. Finally, any metric of readability is algorithmic, and scores are generally calculated based on the number of letters, syllables, words, complex words, or sentences, but these metrics are unable to account for the nuances of medical jargon. For example, the words “gout” and “pseudogout” describe similar conditions, with comparable medical sophistication; however, many readability algorithms would consider “gout” a simple word but “pseudogout” a complex word because it has 3 or more syllables. Although these algorithms are validated measures of reading grade level, their correlation with patients’ perception of the utility of educational materials has not been demonstrated.
      • Roberts H.J.
      • Zhang D.
      • Earp B.E.
      • Blazar P.
      • Dyer G.S.
      Patient self-reported utility of hand surgery online patient education materials.
      Our findings suggest that online hand surgery patient educational materials, even those published by our national society, are written for the general public at too high a reading grade level. No substantial improvements in the readability of online hand surgery patient educational materials have been made since 2015. Our results support a need for continual readability assessments of patient educational materials from official organizations to ensure that patients with all levels of health literacy levels are able to comprehend and benefit from online information. Future research is needed to study the effects of multimedia content, interactive modules, and social-networking features on the effective delivery of health information.

      Acknowledgments

      The authors acknowledge Dr Jia Zhu for her help in creating the violin plot.

      References

        • Rao A.J.
        • Dy C.J.
        • Goldfarb C.A.
        • Cohen M.S.
        • Wysocki R.W.
        Patient preferences and utilization of online resources for patients treated in hand surgery practices.
        Hand. 2019; 14: 277-283
        • Hageman M.G.
        • Anderson J.
        • Blok R.
        • Bossen J.K.
        • Ring D.
        Internet self-diagnosis in hand surgery.
        Hand. 2015; 10: 565-569
        • U.S. Department for Health and Human Services. National Institutes of Health
        Clear Communication.
        (Accessed April 26, 2020.)
        • U.S. Department for Health and Human Services. Centers for Disease Control and Prevention
        Simply Put: A Guide for Creating Easy-To-Understand Materials.
        (July 2010. Accessed April 26)
        • Weiss B.D.
        Health Literacy: A Manual for Clinicians.
        American Medical Association Foundation, 2003
        • Roberts H.
        • Zhang D.
        • Dyer G.S.
        The readability of AAOS patient education materials: evaluating the progress since 2008.
        J Bone Joint Surg Am. 2016; 98: e70
        • Beutel B.G.
        • Danna N.R.
        • Melamed E.
        • Capo J.T.
        Comparative readability of shoulder and elbow patient education materials within orthopaedic websites.
        Bull Hosp Jt Dis. 2015; 73: 249-256
        • Wang S.W.
        • Capo J.T.
        • Orillaza N.
        Readability and comprehensibility of patient education material in hand-related web sites.
        J Hand Surg Am. 2009; 34: 1308-1315
        • Hadden K.
        • Prince L.Y.
        • Schnaekel A.
        • Couch C.G.
        • Stephenson J.M.
        • Wyrick T.O.
        Readability of patient education naterials in hand surgery and health literacy best practices for improvement.
        J Hand Surg Am. 2016; 41: 825-832
        • Yi P.H.
        • Chang M.M.
        • Haughom B.D.
        • Jawa A.
        Readability of patient education materials from the AAHS.
        Hand. 2014; 9: 393-394
        • Flesch R.
        A new readability yardstick.
        J Appl Psychol. 1948; 32: 221-233
        • Kincaid P.J.
        • Fishburne Jr., R.P.
        • Rogers R.L.
        • Chissom B.S.
        Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel.
        Institute for Simulation and Training, University of Central Florida, 1975
        • Coleman M.
        • Liau T.L.
        A computer readability formula designed for machine scoring.
        J Appl Psychol. 1975; 60: 283-284
        • Gunning R.
        The Technique of Clear Writing.
        McGraw-Hill, 1968
        • McLaughlin H.G.
        SMOG grading—a new readability formula.
        J Read. 1969; 12: 639-646
        • Akinleye S.D.
        • Garofolo-Gonzalez G.
        • Montuori M.
        • Culbertson M.D.
        • Hashem J.
        • Edelstein D.M.
        Readability of the most commonly accessed online patient education materials pertaining to pathology of the hand.
        Hand. 2018; 13: 705-714
        • Cook J.A.
        • Sasor S.E.
        • Tholpady S.S.
        • Momeni A.
        • Chu M.W.
        Hand surgery resources exceed American health literacy.
        Hand. 2018; 13: 547-551
        • Eltorai A.E.
        • Cheatham M.
        • Naqvi S.S.
        • et al.
        Is the readability of spine-related patient education material improving? An assessment of subspecialty websites.
        Spine. 2016; 41: 1041-1048
        • Doinn T.Ó.
        • Broderick J.M.
        • Abdelhalim M.M.
        • Quinlan J.F.
        Readability of patient educational materials in hip and knee arthroplasty: has a decade nade a difference?.
        J Arthroplasty. 2020; 35: 3076-3083
        • Scott B.B.
        • Johnson A.R.
        • Doval A.F.
        • Tran B.N.
        • Lee B.T.
        Readability and understandability analysis of online materials related to abdominal aortic aneurysm repair.
        Vasc Endovascular Surg. 2020; 54: 111-117
        • Meleo-Erwin Z.
        • Basch C.
        • Fera J.
        • Ethan D.
        • Garcia P.
        Readability of online patient-based information on bariatric surgery.
        Health Promot Perspect. 2019; 9: 156-160
        • Prieto J.M.
        • West-Santos C.
        • Montgomery A.S.
        • et al.
        Patient-oriented online resources in pediatric surgery: are we failing the readability test?.
        J Pediatr Surg. 2020; 55: 2048-2051
        • Lipari M.
        • Berlie H.
        • Saleh Y.
        • Hang P.
        • Moser L.
        Understandability, actionability, and readability of online patient education materials about diabetes mellitus.
        Am J Health Syst Pharm. 2019; 76: 182-186
        • Arslan D.
        • Tutar M.S.
        • Kozanhan B.
        • Bagci Z.
        The quality, understandability, readability, and popularity of online educational materials for heart murmur.
        Cardiol Young. 2020; 30: 328-336
        • Kim C.
        • Prabhu A.V.
        • Hansberry D.R.
        • Agarwal N.
        • Heron D.E.
        • Beriwal S.
        Digital era of mobile communications and smartphones: a novel analysis of patient comprehension of cancer-related information available through mobile applications.
        Cancer Invest. 2019; 37: 127-133
        • Brachial Plexus Injury
        HandCare: The Upper Extremity Expert.
        (Accessed August 10, 2020)
        • Roberts H.J.
        • Zhang D.
        • Earp B.E.
        • Blazar P.
        • Dyer G.S.
        Patient self-reported utility of hand surgery online patient education materials.
        Musculoskeletal Care. 2018; 16: 458-462
        • Klietz M.L.
        • Kaiser H.W.
        • Machens H.G.
        • Aitzetmüller M.M.
        Social media marketing: what do prospective patients want to see?.
        Aesthet Surg J. 2020; 40: 577-583
        • Rozental T.D.
        • George T.M.
        • Chacko A.T.
        Social networking among upper extremity patients.
        J Hand Surg Am. 2010; 35: 819-823.e1
        • Zhang D.
        • Schumacher C.
        • Harris M.B.
        • Bono C.M.
        The quality and readability of information available on the internet regarding lumbar fusion.
        Global Spine J. 2016; 6: 133-138