Advertisement

Factors Associated With Quality of Online Information on Trapeziometacarpal Arthritis

      Purpose

      People increasingly search the Internet for information about common medical problems such as trapeziometacarpal (TMC) joint arthritis. But this information can be biased, inaccurate, and misleading. Medical professionals should be aware of what patients may be reading about their condition because concepts and beliefs can affect symptoms, limitations, and decision making. This study sought factors associated with the quality of design and content of health information Web sites about TMC arthritis.

      Methods

      Using 3 search engines we entered “thumb arthritis” and measured the quality of design and content of 67 Web sites using the DISCERN and LIDA tools, dominant tones using the IBM Watson Tone Analyzer, and readability, and we recorded Web site characteristics. All but 1 Web site exceeded the recommended sixth-grade reading level. We created 2 backward stepwise regression models to identify independent factors associated with Web site design and content quality.

      Results

      In multivariable analysis, the Web site not having a clear preference for treatment was independently associated with greater design and content quality measured by DISCERN. Health On the Net (HON) code certification—a code of conduct for medical Web sites—and nonprofit Web sites had higher LIDA scores.

      Conclusions

      Online information on TMC arthrosis is difficult to read, often biased in favor of a particular treatment and influenced by profit and HONcode.

      Clinical relevance

      Hand surgeons should prepare to gently correct misconceptions established or reinforced, in part, by material found on the Internet.

      Key words

      People increasingly search the Internet for information about medical problems (Appendices A-C, available on the Journal's Web site at www.jhandsurg.org). Seventy-four percent of adults in the United States have searched online for medical information.

      Taylor H. The Growing Influence and Use Of Health Care Information Obtained Online. Available at: http://www.harrisinteractive.com/ vault/HI-Harris-Poll-Cyberchondriacs-2011-09-15.pdf. The Harris Poll #98. September 15, 2011. Accessed September 21, 2017.

      There is a great deal of information available online for common illnesses such as trapeziometacarpal (TMC) arthritis.
      • Becker S.J.
      • Briet J.P.
      • Hageman M.G.
      • Ring D.
      Death, taxes, and trapeziometacarpal arthrosis hand.
      • Becker S.J.
      • Makarawung D.J.
      • Spit S.A.
      • King J.D.
      • Ring D.
      Disability in patients with trapeziometacarpal joint arthrosis: incidental versus presenting diagnosis.
      • Ochtman A.E.A.
      • Guitton T.G.
      • Buijze G.A.
      • et al.
      Trapeziometacarpal arthrosis: predictors of a second visit and surgery.
      The American Medical Association and the National Institutes of Health recommend that patient information be written at a sixth-grade reading level

      Clear & Simple: Achieving Qualtiy and Effectiveness in Health Communication. Bethesda, MD: National Institutes of Health, Office of Communications and Public Liaison; 2016. Available at: https://www.nih.gov/institutes-nih/nih-office-director/office-communications-public-liaison/clear-communication/clear-simple. Accessed October 30, 2017.

      Weiss BD. Health Literacy and Patient Safety: Help Patients Understand. Chicago: American Medical Association Foundation and American Medical Association; 2007. Available at: https://med.fsu.edu/userFiles/file/ahec_health_clinicians_manual.pdf. Accessed October 30, 2017.

      ; however, a recent article found that TMC arthritis online information uniformly exceeds this recommended reading level.
      • Kamal R.N.
      • Paci G.M.
      • Daniels A.H.
      • Gosselin M.
      • Rainbow M.J.
      • Weiss A.P.
      Quality of internet health information on thumb carpometacarpal joint arthritis.
      The quality of online information varies.
      • Kamal R.N.
      • Paci G.M.
      • Daniels A.H.
      • Gosselin M.
      • Rainbow M.J.
      • Weiss A.P.
      Quality of internet health information on thumb carpometacarpal joint arthritis.
      • John E.S.
      • John A.M.
      • Hansberry D.R.
      • et al.
      Colorectal cancer screening patient education materials—how effective is online health information?.
      • McKearney T.C.
      • McKearney R.M.
      The quality and accuracy of internet information on the subject of ear tubes.
      • Grewal P.
      • Alagaratnam S.
      The quality and readability of colorectal cancer information on the internet.
      The information patients find can be biased, inaccurate, or misleading.
      • McKearney T.C.
      • McKearney R.M.
      The quality and accuracy of internet information on the subject of ear tubes.
      Although there is a risk of misinformation and misunderstanding, many people are not aware of the accuracy or trustworthiness of the information they obtain. One study found that 9 out of 10 people describe the information they find as reliable.

      Taylor H. The Growing Influence and Use Of Health Care Information Obtained Online. Available at: http://www.harrisinteractive.com/ vault/HI-Harris-Poll-Cyberchondriacs-2011-09-15.pdf. The Harris Poll #98. September 15, 2011. Accessed September 21, 2017.

      It can be helpful for medical professionals to know the quality, tone, and readability of online information most commonly read by their patients. There may be greater risk of incorrectly identifying patient preferences (going with preferences based on misconceptions rather than preferences based on values) if people have preconceptions and misconceptions that are influenced by online information.
      • Mulley A.G.
      • Trimble C.
      • Elwyn G.
      Stop the silent misdiagnosis: patients' preferences matter.
      This study addressed factors independently associated with TMC osteoarthritis online information design and content quality measured using the (1) DISCERN and (2) LIDA instruments.

      Materials and Methods

      Data source

      Because our study involves no participants, it was exempt from institutional review board approval. On July 31, 2017, we searched for “thumb arthritis” in 3 of the most popular Web search engines (Google, Yahoo!, and Bing).

      Top 15 Most Popular Search Engines. eBizMBA Rank. July 2017. Available at: http://www.ebizmba.com/articles/search-engines. Accessed October 30, 2017.

      Because results can be influenced by search history, we searched with the browser in privacy mode after deleting our search history and cookies. We recorded the first 50 hits of each search engine. After excluding duplicates (n = 57) and irrelevant Web sites (n = 26, eg, blogs, web-shops, research journals, videos), 67 Web sites were included.

      Measurements

      We recorded what treatment options were discussed (operative, nonoperative, both, or none), if there was a clear preference for 1 treatment, and if the Web site was profit or nonprofit. For U.S. Web sites we used Nonprofit Explorer, a Web site that registers tax-exempt organizations, to look for nonprofit organizations.

      Tigas M, Wei S, Glassford A. Nonprofit Explorer: Research Tax-Exempt Organizations. ProPublica. 2017. Available at: https://projects.propublica.org/nonprofits/. Accessed November 2, 2017.

      In addition, we registered if websites had a Health On the Net (HON) code. The Health On the Net Foundation is an internationally recognized organization that provides certification of conduct, a HONcode, for medical Web sites. A HON seal is accredited when a Web site meets their 8 principles: authoritative, complementary, privacy, attribution, justifiability, transparency, financial disclosure, and advertising policy.

      Health On the Net Foundation. 2017. Available at: https://www.hon.ch/index.html. Accessed October 30, 2017.

      The HONcode is an accepted quality measure for online health sites.

      Health On the Net Foundation. 2017. Available at: https://www.hon.ch/index.html. Accessed October 30, 2017.

      • Alamoudi U.
      • Hong P.
      Readability and quality assessment of websites related to microtia and aural atresia.
      • Fast A.M.
      • Deibert C.M.
      • Hruby G.W.
      • Glassberg K.I.
      Evaluating the quality of Internet health resources in pediatric urology.
      Readability is defined as the ease with which a reader can understand a written text.
      • Davis K.S.
      • McCormick A.A.
      • Jabbour N.
      What might parents read: sorting webs of online information on vascular anomalies.
      We used 3 of the most reliable scores for readability: the Flesch Reading Ease Score (FRES), the Flesch-Kincaid Grade Level (FKGL), and the Gunning Fog Index (GFI). We used the spelling and grammar tool in Microsoft Word (Microsoft Corp., Redmond, WA) to determine the FRES and FKGL. We used an online calculator to assess the GFI.
      The FRES rates text on a 100-point rating scale with higher scores representing easier readability. The FKGL reflects the required U.S. grade level to comprehend text, for example, a score of 7.2 indicates that text is expected to be understandable for an average student in seventh grade. It ranges from 3 to 12, with greater score indicating more difficult readability.

      Kincaid JP, Fishburne RP Jr, Rogers RL, Chissom BS. Derivation of New Readability Formulas (Automated Readability Index, Fog Count, and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Millington, TN: Chief of Naval Technical Training: Naval Air Station Memphis, Research Branch Report; 1975:8–75.

      The GFI is a measure of text readability based on the use of difficult words and the length of sentences.
      • Gunning R.
      The Technique of Clear Writing.
      Scores range from 0 to 15 and represent the estimated years of school education that is required to understand text on first reading.
      The IBM Watson Tone Analyzer measures tones that are present in written text in 3 categories: emotion (tones of anger, disgust, fear, joy, or sadness), language style (language that is analytical, confident, or tentative), and social tendencies (language reflecting openness, conscientiousness, extraversion, agreeableness, or emotional range) (Table 1). Scores of each tone range from 0 to 1.0 with scores greater than 0.5 indicating likely present, and scores greater than 0.75 indicating very likely present tones.
      IBM Watson Developer Cloud
      Tone Analyzer.
      We copied all Web site texts in the analyzer and used the document level to get a sense of the dominant tones of the website.
      Table 1Definition of Dominant Tones
      Definition
      Emotional Tones
       AngerIs evoked due to injustice, conflict, humiliation, negligence, or betrayal. If anger is active, a person attacks the target, verbally or physically. If passive, a person silently sulks and feels tension and hostility.
       DisgustA feeling of revulsion or strong disapproval aroused by something unpleasant or offensive.
       FearFear is a response to impending danger. It is a survival mechanism that is triggered as a reaction to some negative stimulus. Fear can be a mild caution or an extreme phobia.
       JoyJoy or happiness has shades of enjoyment, satisfaction, and pleasure. There is a sense of well-being, inner peace, love, safety, and contentment.
       SadnessIndicates a feeling of loss and disadvantage. When a person can be observed to be quiet, less energetic, and withdrawn, it may be inferred that sadness exists.
      Language Styles
       AnalyticalA person's reasoning and analytical attitude about things.
       ConfidentA person's degree of certainty.
       TentativeA person's degree of inhibition.
      Social Tendencies
       OpennessThe extent to which a person is open to experiencing a variety of activities.
       ConscientiousnessA person's tendency to act in an organized or thoughtful way.
       ExtraversionA person's tendency to seek stimulation in the company of others.
       AgreeablenessA person's tendency to be compassionate and cooperative toward others.
       Emotional rangeThe extent to which a person's emotions are sensitive to the individual's environment.
      From IBM Cloud Docs. Personality Insights. Available at: https://console.bluemix.net/docs/services/personality-insights/models.html#models. Accessed June 2, 2018.
      The DISCERN and LIDA tools gauge health information quality. The DISCERN contains 16 questions, 8 on reliability, 7 on treatment information and choices, and 1 question on overall quality. Each question is rated on a 5-point Likert scale with answers from 1 no to 5 yes with a maximum score of 80. A higher score indicates greater overall quality.
      • Charnock D.
      The DISCERN Handbook: Quality Criteria for Consumer Health Information on Treatment Choices.
      The LIDA measures 3 domains: accessibility, usability, and reliability. It consists of 27 questions scored on a 4-point Likert scale, with scores ranging from 0 never to 3 always. Total scores range between 0 and 100 with a higher score indicating greater quality.

      Minervation. The LIDA Instrument: Minervation Validation Instrument for Health Care Web Sites; 2007. http://www.minervation.com/wp-content/uploads/2011/04/Minervation-LIDA-instrument-v1-2.pdf. Accessed October 30, 2017.

      We rated usability and reliability because the online calculation for accessibility is no longer available.

      Reliability

      To assess intraobserver reliability, DISCERN and LIDA were scored twice by a single reviewer (J.S.E.O.) with 2 weeks in between evaluations. The DISCERN and LIDA scores are the mean of both ratings. The intraobserver reliability, measured by intraclass correlation coefficient, of DISCERN was 0.91 (95% confidence interval [95% CI], 0.85–0.94) and LIDA was 0.85 (95% CI, 0.77–0.90) (Table 2). Bland-Altman plots showed a small difference in agreement between the 2 ratings and minimal systematic differences (Appendix C).
      Table 2Reliability Scores
      Variables ScoredICC
      Two-way mixed-effects model for ICC with 95% CI.
      (95% CI)
      P Value
      Intraobserver reliability
       DISCERN0.91 (0.85–0.94)< .001
       LIDA0.85 (0.77–0.90)< .001
      Interobserver reliability
       DISCERN0.95 (0.91–0.98)< .001
       LIDA0.88 (0.78– 0.94)< .001
      Interobserver agreement
      Interobserver agreement as kappa with SE.
      Kappa ± SE
       Treatment options discussed1.00 (0.17)< .001
       Clear preference for treatment1.00 (0.13)< .001
       Nonprofit Web site1.00 (0.17)< .001
      ICC, intraclass correlation coefficient.
      Bold values indicate statistical significance (P < .05).
      Two-way mixed-effects model for ICC with 95% CI.
      Interobserver agreement as kappa with SE.
      To assess interobserver reliability, a second reviewer (J.T.P.K.) rated 30 Web sites on DISCERN, LIDA, profit versus nonprofit, clear preference for 1 treatment or not, and what treatment options were discussed. The interobserver reliability, measured by intraclass correlation coefficient, for DISCERN was 0.95 (95% CI, 0.91–0.98) and for LIDA was 0.88 (95% CI, 0.78–0.94) (Table 2). The interobserver reliability, measured by kappa, for nonprofit was 1.00 (standard error [SE], 0.17), for discussed treatment options 1.00 (SE, 0.17), and for clear preference for treatment 1.00 (SE, 0.13) (Table 2).
      The IBM Watson Tone Analyzer was developed to analyze tones in customer service conversations. It is based on 96,000 customer service Twitter conversations, rated by 5 trained annotators for tone. IBM trained a machine-learning model based on this dataset. IBM states the model demonstrated high accuracy compared with a benchmark dataset, but they do not mention actual numbers to gauge the reliability of the machine-learning program.

      IBM Cloud Docs. The Science Behind the Service; 2017. Available at: https://console.bluemix.net/docs/services/tone-analyzer/science.html#the-science-behind-the-service. Accessed December 18, 2017.

      Website characteristics

      Of the 67 Web sites, 19 (28%) were nonprofit and 14 (21%) HONcode-accredited (Table 3). Mean readability scores for FRES were 53 ± 9.7 (indicating “difficult to read”); for FKGL, 10 ± 1.7 (corresponds with 10th-grade reading level); and for GFI, 13 ± 2.0 (indicating 13 years of education required to understand text). Only 1 Web site was written below seventh-grade level (FKGL 4.7), as recommended by The National Institutes of Health and The American Medical Association.

      Weiss BD. Health Literacy and Patient Safety: Help Patients Understand. Chicago: American Medical Association Foundation and American Medical Association; 2007. Available at: https://med.fsu.edu/userFiles/file/ahec_health_clinicians_manual.pdf. Accessed October 30, 2017.

      • Kamal R.N.
      • Paci G.M.
      • Daniels A.H.
      • Gosselin M.
      • Rainbow M.J.
      • Weiss A.P.
      Quality of internet health information on thumb carpometacarpal joint arthritis.
      The most likely present tones in Web sites texts were openness (0.85 ± 0.09) and tentative (0.72 ± 0.19). The mean LIDA score was 61 ± 13; the mean DISCERN was 45 ± 12.
      Table 3Web Site Characteristics
      Variables
      Continuous variables as mean (± SD); discrete variables as number (percentage).
      Values
      Web sites67
      HONcode, n (%)14 (21)
      Nonprofit, n (%)19 (28)
      Both treatment options discussed, n (%)53 (79)
      Clear preference for treatment, n (%)
       None44 (66)
       Nonoperative treatment15 (22)
       Operative treatment8 (12)
      Readability scores
       FRES53 ± 9.7
       FKGL10 ± 1.7
       GFI13 ± 2.0
      Web site tone
       Anger0.07 ± 0.05
       Disgust0.08 ± 0.08
       Fear0.26 ± 0.13
       Joy0.35 ± 0.18
       Sadness0.61 ± 0.04
       Analytical0.63 ± 0.14
       Confident0.00 ± 0.00
       Tentative0.72 ± 0.19
       Openness0.85 ± 0.09
       Conscientiousness0.29 ± 0.14
       Extraversion0.09 ± 0.07
       Agreeableness0.13 ± 0.10
       Emotional range0.24 ± 0.12
      Quality of information
       DISCERN45 ± 12
       LIDA61 ± 13
      Continuous variables as mean (± SD); discrete variables as number (percentage).

      Statistical analysis

      Continuous variables are reported as mean and standard deviation (SD) and discrete variables as number and percentage. We used Student t test to compare continuous and dichotomous variables, and Pearson correlation for 2 continuous variables.
      We created 2 backward stepwise regression models to identify independent predictors of Web sites’ quality of information measured with DISCERN and LIDA. We included all factors with P less than .10 on bivariate analysis (Table 4) in the final multivariable models (Table 5). Adjusted R2 indicates the proportion of variability in the outcome variable (either DISCERN or LIDA score) that is accounted for by the model. Semipartial R2 expresses the specific variability of a given independent variable in the model.
      Table 4Bivariate Analyses
      Pearson correlation indicated by r; bold indicates statistically significant difference; continuous variables as mean (± SD); discrete variables as number (percentage).
      VariablesDISCERNP ValueLIDAP Value
      HONcode
       Yes51 ± 10.03271 ± 11.0017
       No43 ± 1259 ± 13
      Nonprofit
       Yes50 ± 11.03369 ± 12.0020
       No43 ± 1258 ± 13
      Both treatment options discussed
       Yes47 ± 11.00262 ± 14.17
       No36 ± 1257 ± 10
      Clear preference for treatment
       None49 ± 10< .00164 ± 13.019
       Nonoperative treatment36 ± 1255 ± 13
       Operative treatment39 ± 1255 ± 9.5
      Readability scores (r)
       FRES–0.063.61–0.2.10
       FKGL–0.1.420.086.49
       GFI0.16.210.046.71
      Web site tone (r)
       Anger0.18.140.14.27
       Disgust0.17.180.13.31
       Fear0.29.020.18.14
       Joy–0.04.75–0.16.18
       Sadness0.18.14–0.12.32
       Analytical0.11.39–0.13.32
       Confident0.00.000.00.00
       Tentative0.23.0590.27.03
       Openness–0.042.74–0.14.26
       Conscientiousness0.030.81–0.17.16
       Extraversion0.23.057–0.11.38
       Agreeableness0.018.890.07.58
       Emotional range0.21.084–0.18.14
      Bold indicates statistical significance (P < .05).
      Pearson correlation indicated by r; bold indicates statistically significant difference; continuous variables as mean (± SD); discrete variables as number (percentage).
      Table 5Stepwise Regression for Predictors of Quality of Information
      Variables inserted in the stepwise regression for DISCERN model: HONcode, nonprofit, treatment options, treatment preference, Web site tones: fear, tentative, extraversion, emotional range. For LIDA model: HONcode, nonprofit, treatment preference, Web site tone: tentative.
      Dependent

      Variables
      PredictorsRegression Coefficient (95% CI)Standard ErrorP ValueSemipartial R2Adjusted R2
      DISCERNClear preference for treatment
       NoneReference value.19
       Nonoperative treatment–12 (–19 to –6.0)3.2< .0010.18
       Operative treatment–9.7 (–18 to –1.4)4.1.0220.068
      LIDAHONcode11 (4.3 to 18)3.5.0020.12.23
      Nonprofit10 (3.7 to 16)3.2.0020.12
      Bold indicates statistical significance (P < .05).
      Variables inserted in the stepwise regression for DISCERN model: HONcode, nonprofit, treatment options, treatment preference, Web site tones: fear, tentative, extraversion, emotional range. For LIDA model: HONcode, nonprofit, treatment preference, Web site tone: tentative.
      An a priori sample size estimate indicated that a sample of 65 Web sites would provide 80% statistical power, with alpha set at 0.05, for a regression with 5 independent variables if a single variable would account for 10% or more of the variability in quality, and our complete model would account for 20% of the overall variability in quality.

      Results

      DISCERN

      Accounting for potential interaction of variables using multivariable analysis, greater Web site quality measured by DISCERN was independently associated with not having a clear preference for treatment compared with having a clear preference (preference for either nonoperative treatment: β regression coefficient [β] = –12; 95% CI, –19 to –6.0; semipartial R2 = 0.18; P < .05; or preference for operative treatment: β = –9.7; 95% CI = –18 to –1.4; semipartial R2 = 0.068; P < .05) (adjusted R2 = 0.19) (Table 5). Dominant Web site tone of fear (r = 0.29; P < .05), being HONcode-certified (P = .032), and nonprofit Web sites (P = 0.033) were significant in bivariate analysis but were not retained in the final model (Table 4).

      LIDA

      Greater Website quality, measured by LIDA, was independently associated with being HONcode-certified (β = 11; 95% CI = 4.3 to 18; semipartial R2 = 0.12; P < .05) and being nonprofit (β = 10; 95% CI = 3.7 to 16; semipartial R2 = 0.12; P < .05) (adjusted R2 = 0.23) (Table 5). Dominant tentative tone (r = 0.27; P = .030) and clear preference for treatment (P = .019) were significant in bivariate analysis but were not retained in the final model (Table 4).

      Discussion

      Information that patients find online about common health conditions like TMC arthritis can be difficult to read, biased, and misleading.

      Taylor H. The Growing Influence and Use Of Health Care Information Obtained Online. Available at: http://www.harrisinteractive.com/ vault/HI-Harris-Poll-Cyberchondriacs-2011-09-15.pdf. The Harris Poll #98. September 15, 2011. Accessed September 21, 2017.

      • Kamal R.N.
      • Paci G.M.
      • Daniels A.H.
      • Gosselin M.
      • Rainbow M.J.
      • Weiss A.P.
      Quality of internet health information on thumb carpometacarpal joint arthritis.
      • John E.S.
      • John A.M.
      • Hansberry D.R.
      • et al.
      Colorectal cancer screening patient education materials—how effective is online health information?.
      • McKearney T.C.
      • McKearney R.M.
      The quality and accuracy of internet information on the subject of ear tubes.
      • Grewal P.
      • Alagaratnam S.
      The quality and readability of colorectal cancer information on the internet.
      This study addressed factors associated with the design and content quality of TMC arthritis online information measured by DISCERN and LIDA.
      We acknowledge some study limitations. First, this study is limited to Web sites that were written in English and available on the date of search, July 31, 2017. The Internet is changing quickly, and its dynamic character makes it impossible to predict when and how online information will be edited or updated. Our study might not reflect the most recent available online information regarding TMC arthritis. Second, we used the search term “thumb arthritis” and no other medical terms such as “carpometacarpal” or “trapeziometacarpal osteoarthritis.” Different terms would have produced a wider range of Web sites; however, the term “thumb arthritis” may be more commonly used by patients than sophisticated medical terms. Third, the quality of videos, images, and other multimedia factors were not assessed, although these materials might contribute to overall comprehension and better understanding of health information. Fourth, the DISCERN and LIDA quality assessment tools rely on subjective input, despite strict criteria for assessment of each rating. This may lead to bias of the observers, although our study showed good intra- and interobserver reliability scores. Fifth, we addressed the quality and not the accuracy of the content based on current best evidence. It is possible to present quality content that is not evidence-based. Sixth, the reliability of the Watson tone analyzer for medical Web sites is unknown. In addition, language assessment tools only give an estimation of the likelihood of dominant tones and emotions in written text. It does not address the purposes of the author nor the existence of possible bias. Although little is studied about the correlation between language assessment tools and patient information Web sites, tone analyzers are increasingly used by communication experts in the field of marketing, customer service, and education.
      The finding that lower Web site quality measured by DISCERN was independently associated with a clear preference for treatment, compared with no clear preference, can be explained by the fact that the DISCERN tool focuses specifically on the quality of written information about treatment choices.
      • Charnock D.
      The DISCERN Handbook: Quality Criteria for Consumer Health Information on Treatment Choices.
      • Charnock D.
      • Shepperd S.
      • Needham G.
      • Gann R.
      DISCERN: an instrument for judging the quality of written consumer health information on treatment choices.
      The DISCERN instrument can be used to assess the quality of a Web site that discusses 1 particular treatment choice, as long as it is clear that other treatment choices are available (Questions 6 and 14) and that only 1 treatment option is discussed by the authors (Question 1). Websites that only focus on 1 treatment option and are not clear about other available treatments score lower on the DISCERN measure. A previous study investigated TMC arthritis information Web sites for their readability and quality.
      • Kamal R.N.
      • Paci G.M.
      • Daniels A.H.
      • Gosselin M.
      • Rainbow M.J.
      • Weiss A.P.
      Quality of internet health information on thumb carpometacarpal joint arthritis.
      To assess quality, they used more limited and less often used measures with fewer answer options, such as the HONcode Site Evaluation Form, instead of DISCERN and LIDA tools.

      Minervation. The LIDA Instrument: Minervation Validation Instrument for Health Care Web Sites; 2007. http://www.minervation.com/wp-content/uploads/2011/04/Minervation-LIDA-instrument-v1-2.pdf. Accessed October 30, 2017.

      • Charnock D.
      • Shepperd S.
      • Needham G.
      • Gann R.
      DISCERN: an instrument for judging the quality of written consumer health information on treatment choices.

      Dobrogowska-Schlebusch E, Niedźwiedzka B. Assessment of the Quality of Online Health Resources in Order to Identify the Examples of Best Practices in Creating Portals for Patients. Part One of the Study: Development and Validation of the Assessment Tool; 2011. Available at: http://www.iss.it/binary/eahi/cont/95_Ewa_Dobrogowska_Schlebusch_Full_text.pdf. Accessed December 24, 2017.

      In addition, they did not analyze Web site tones or perform multivariable analysis to assess factors associated with the quality of information. They found that health information on TMC arthritis is of generally poor quality, is predominantly posted by physician authors, and is too hard to read. Of the 60 Web sites analyzed, only 3 contained a HONcode.
      • Kamal R.N.
      • Paci G.M.
      • Daniels A.H.
      • Gosselin M.
      • Rainbow M.J.
      • Weiss A.P.
      Quality of internet health information on thumb carpometacarpal joint arthritis.
      The finding that greater Website quality measured by LIDA, was associated with being HONcode-certified, is in line with previous findings. Several studies on the quality of online information about colorectal cancer information, chronic pain, asthma, and scoliosis indicate HON certification as a factor associated with greater website quality.
      • Grewal P.
      • Alagaratnam S.
      The quality and readability of colorectal cancer information on the internet.
      • O’Neill S.C.
      • Baker J.F.
      • Fitzgerald C.
      • et al.
      Cauda equina syndrome: assessing the readability and quality of patient information on the internet.
      • Kaicker J.
      • Debono V.B.
      • Dang W.
      • Buckley N.
      • Thabane L.
      Assessment of the quality and variability of health information on chronic pain websites using the DISCERN instrument.
      • Nason G.J.
      • Baker J.F.
      • Byrne D.P.
      • Noel J.
      • Moore D.
      • Kiely P.J.
      Scoliosis-specific information on the internet: has the “information highway” led to better information provision?.
      • Banasiak N.C.
      • Meadows-Oliver M.
      Evaluating asthma websites using the Brief DISCERN instrument.
      Conversely, other studies on TMC arthritis, vascular anomalies, and vertebroplasty, including HON certification as an independent variable, did not find this association,
      • Kamal R.N.
      • Paci G.M.
      • Daniels A.H.
      • Gosselin M.
      • Rainbow M.J.
      • Weiss A.P.
      Quality of internet health information on thumb carpometacarpal joint arthritis.
      • Davis K.S.
      • McCormick A.A.
      • Jabbour N.
      What might parents read: sorting webs of online information on vascular anomalies.
      • Sullivan T.B.
      • Anderson J.T.
      • Ahn U.M.
      • Ahn N.U.
      Can internet information on vertebroplasty be a reliable means of patient self-education?.
      possibly owing to the low number of HON-certified Web sites in the latter studies’ samples. Nonprofit Web sites showed better quality, probably because they have no commercial aims to sell products or services. Two studies on the quality and readability of online colorectal cancer information
      • Grewal P.
      • Alagaratnam S.
      The quality and readability of colorectal cancer information on the internet.
      and of online cauda equina syndrome information
      • Nason G.J.
      • Baker J.F.
      • Byrne D.P.
      • Noel J.
      • Moore D.
      • Kiely P.J.
      Scoliosis-specific information on the internet: has the “information highway” led to better information provision?.
      indicate a similar relationship between greater Web site quality and government-authored websites. Whereas DISCERN assesses the presentation of treatment options, the LIDA tool focuses more on usability and reliability. Specific tones were significant in bivariate analysis, suggesting that they might be clues to lower quality, but were not independent of the profit status and HONcode. Readability was not associated with quality scores, perhaps because the Web sites on average were written for relatively highly educated people.
      • Kamal R.N.
      • Paci G.M.
      • Daniels A.H.
      • Gosselin M.
      • Rainbow M.J.
      • Weiss A.P.
      Quality of internet health information on thumb carpometacarpal joint arthritis.
      • McKearney T.C.
      • McKearney R.M.
      The quality and accuracy of internet information on the subject of ear tubes.
      • O’Neill S.C.
      • Baker J.F.
      • Fitzgerald C.
      • et al.
      Cauda equina syndrome: assessing the readability and quality of patient information on the internet.
      • Eberlin K.R.
      • Vargas C.R.
      • Chuang D.J.
      • Lee B.T.
      Patient education for carpal tunnel syndrome: analysis of readability.
      • Moody E.
      • Clemens K.
      • Storsley L.
      • Waterman A.
      • Parikh C.
      • Garg A.
      Improving on-line information for potential living kidney donors.
      • Chi E.
      • Jabbour N.
      • Aaronson N.L.
      Quality and readability of websites for patient information on tonsillectomy and sleep apnea.
      Another explanation is that the LIDA and DISCERN tools do not sufficiently assess and capture readability levels of Web sites. The DISCERN instrument does not contain any question regarding readability, the LIDA tool only contains 1 (2.1.2.: Is the level of detail appropriate to their level of knowledge?). This lack of readability items should be kept in mind when using these outcome measures to rate Web site quality.
      This study found that Web-based information on TMC arthritis is difficult to read, often biased in favor of a particular treatment, and influenced by profit and HONcode. Improved Web sites and decision aids (tools providing unbiased information to patients and that help clarify patients’ preferences) could help ensure that patient decisions about TMC arthritis and other common health conditions are based on their values and not on misconceptions. Measures of design, content, tone, and readability can assist with the development of improved patient information.

      Acknowledgments

      T.T. receives payments or benefits from AO Trauma, Stryker, DePuy Synthes, PATIENT+, and VCC. D.R. receives royalties from Skeletal Dynamics and Wright Medical; receives a stipend from Clinical Orthopaedics and Related Research; receives honoraria for talks from universities and hospitals; and receives payment form lawyers for expert review.

      Appendix

      Appendix AIncluded and Excluded Web Sites
      Included Web Sites
      EngineWeb SiteURL
      YahooAAHShttp://handsurgery.org/multimedia/files/public/thumbarthritis.pdf
      GoogleAAOShttp://orthoinfo.aaos.org/topic.cfm?topic=A00210
      YahooAAOShttp://orthoinfo.aaos.org/topic.cfm?topic=A00224
      GoogleAgilityOrthopedicshttp://www.agilitydoctor.com/learning/agility-magazine/200-thumb-arthritis-what-can-be-done-about-it
      GoogleArthritis Foundationhttp://www.arthritis.org/living-with-arthritis/treatments/joint-surgery/types/other/new-thumb-surgery-options.php
      YahooArthritis Foundationhttp://www.arthritis.org/living-with-arthritis/treatments/medication/drug-types/other/oa-thumb-treatment.php
      GoogleArthritisHealhhttps://www.arthritis-health.com/types/rheumatoid/hand-rheumatoid-arthritis-signs-and-symptoms
      BingArthritisRelievedhttp://arthritisrelieved.com/arthritis-thumb-pain-thumb-arthritis/
      YahooASSHhttp://www.assh.org/LinkClick.aspx?fileticket=AsKorJBUZsw%3d&portalid=1
      GoogleBelMarrahttp://www.belmarrahealth.com/thumb-arthritis-pain-causes-treatments/
      GoogleBioProhttp://bioproimplants.com/portfolio-view/thumb-arthritis
      GoogleBoneTalkshttp://www.bonetalks.com/thumbarthritis/
      GoogleBSSHhttp://www.bssh.ac.uk/patients/conditions/24/basal_thumb_arthritis
      GoogleBurlingtonFreePresshttp://www.burlingtonfreepress.com/story/life/2017/03/13/thumb-arthritis-treatment-symptoms-mikolyzk-manitowoc-hfm/99124612/
      GoogleC.NoelHenlyhttp://noelhenley.com/arthritis-base-of-the-thumb/
      GoogleChicagoTribunehttp://www.chicagotribune.com/suburbs/highland-park/community/chi-ugc-article-5-main-causes-you-must-know-about-thumb-joint-2014-05-15-story.html
      GoogleClevelandClinichttps://my.clevelandclinic.org/health/articles/arthritis-thumb-base
      YahooConsumerHealthDigesthttps://www.consumerhealthdigest.com/joint-pain/thumb-arthritis.html
      GoogleDailyMailhttp://www.dailymail.co.uk/health/article-3952498/Blast-arthritis-Exocet-thumb-implant-Using-latest-technology-joint-replacement-restore-mobility.html
      GoogleDallasNewshttps://www.dallasnews.com/life/healthy-living/2013/11/04/is-that-pain-in-your-thumb-arthritis
      YahooDicksonDiveleyhttps://www.dd-clinic.com/thumb-cmc-arthritis/
      GoogleDr.Sinnerhttp://sinnerchiropractic.com/z-thumb-arthritis/
      GoogleDrugs.comhttps://www.drugs.com/mcd/thumb-arthritis
      GoogleEatonHandhttp://www.eatonhand.com/hw/hw003.htm
      GoogleEverydayHealthhttps://www.everydayhealth.com/arthritis/0404/thumbs-down-thumb-arthritis-on-the-rise.aspx
      YahooFitzmauricehttp://www.fitzhand.com/thumb-arthritis/
      GoogleGibaudhttps://www.gibaud.com/EN/pathology/hand-wrist/basal-thumb-arthritis.htm
      BingHandAndWristClinichttp://www.handandwristclinic.com/article.asp?article=107
      BingHandToShoulderCenterhttps://handtoshoulderwisconsin.com/our-specialties/hand/thumb-arthritis/
      GoogleHealthCentralhttps://www.healthcentral.com/article/oh-my-aching-hands-the-thumb
      GoogleHealthlinehttp://www.healthline.com/health/osteoarthritis/thumb#overview1
      GoogleHealthlinehttp://www.healthline.com/health/basal-joint-arthritis
      GoogleHoustonMethodisthttp://www.houstonmethodist.org/orthopedics/where-does-it-hurt/hand/arthritis-of-the-thumb/
      GoogleHSShttps://www.hss.edu/conditions_basal-joint-arthritis-overview.asp
      GoogleIrishTimeshttp://www.irishtimes.com/business/innovation/new-implant-could-improve-treatment-of-thumb-arthritis-1.2991175
      GoogleJohnMuirhttps://www.johnmuirhealth.com/health-education/conditions-treatments/bones-joints/thumb-arthritis.html
      YahooJointPainSolutionshttp://www.joint-pain-solutions.com/thumb-joint-pain.html
      YahooLawrenceLihttps://www.orthopedicshoulder.com/services/hand/thumb-arthritis/
      YahooLifescripthttp://www.lifescript.com/health/a-z/mayo/t/thumb_arthritis.aspx
      GoogleLondonOrthopaedicClinichttp://www.londonorthopaedic.com/thumb-arthritis/
      GoogleMayoClinichttp://www.mayoclinic.org/diseases-conditions/thumb-arthritis/basics/treatment/con-20027798
      YahooMHShttp://www.midwesthand.com/specialty-treatment/thumb-arthritis/
      GoogleMichiganMedicinehttps://medicine.umich.edu/dept/orthopaedic-surgery/patient-care-services-hand-upper-extremity/basilar-thumb-arthritis
      GoogleMyHandhttp://www.myhand.com.au/handouts/anatomy/joint/basal-thumb-arthritis-joint
      GoogleMyHealthAlbertahttps://myhealth.alberta.ca/health/pages/conditions.aspx?Hwid=hw125723
      YahooNebraskahttp://www.carpaltunnelrelief.net/Basilar%20Thumb%20Arthritis
      BingNHShttp://www.nhs.uk/conditions/osteoarthritis/Pages/Introduction.aspx
      YahooNorthwellHealthhttps://www.northwell.edu/find-care/conditions-we-treat/arthritis-thumb
      GoogleOhMyArthritishttp://www.blog.ohmyarthritis.com/my-thumb-hurts-do-i-have-thumb-arthritis/
      YahooOhMyArthritishttp://www.ohmyarthritis.com/Learn/About-Health-Conditions/CMC-thumb-arthritis.html
      GoogleOrthoBulletshttp://www.orthobullets.com/hand/6054/basilar-thumb-arthritis
      YahooOrthopodhttp://eorthopod.com/arthritis-of-the-thumb/
      BingPHShttp://phsurgery.com/thumb-arthritis/
      GoogleProlotherapyhttp://www.prolotherapy.org/thumb-arthritis/
      GoogleRegenexxhttps://www.regenexx.com/hand-basal-joint-cmc-arthritis-treatment/
      GoogleTaluneUniversityhttps://medicine.tulane.edu/find-doctor/orthopaedics-clinics/problems-conditions/thumb-arthritisbasal-thumb-arthritis
      GoogleTheHandandWristInstitutehttp://www.handandwristinstitute.com/basal-thumb-arthritis/
      GoogleTheRheumatologisthttp://www.the-rheumatologist.org/article/nonsurgical-treatments-can-relieve-pain-improve-hand-function-in-thumb-carpometacarpal-joint-osteoarthritis/
      GoogleUWMedicinehttp://www.orthop.washington.edu/?q=patient-care/hand/thumb-arthritis.html
      GoogleUWMedicinehttp://www.uwmedicine.org/health-library/Pages/thumb-arthritis.aspx
      YahooVeryWellhttps://www.verywell.com/thumb-arthritis-2549457
      YahooVeryWellhttps://www.verywell.com/thumb-osteoarthritis-what-you-need-to-know-2552320
      GoogleViveHealthhttps://www.vivehealth.com/blogs/resources/thumb-brace-for-arthritis
      GoogleWellingtonhttp://www.wellington-hand-physiotherapy.co.nz/thumb-pain-and-arthritis.html
      YahooWikipediahttps://en.wikipedia.org/wiki/Arthritis
      YahooYgoyhttp://arthritis.ygoy.com/remedies-for-thumb-arthritis/
      BingYourHealthyJointshttp://yourhealthyjoints.com/thumb-arthritis/
      Excluded websites:
      EngineWebsiteURLReason for Exclusion
      YahooAmazonhttps://www.amazon.com/arthritis-thumb-splint/s?ie=UTF8&page=1&rh=i%3Aaps%2Ck%3Aarthritis%20thumb%20splintWebshop
      YahooEbayhttps://www.ebay.com/sch/i.html?_nkw=arthritis+thumb+splintWebshop
      BingPinteresthttps://nl.pinterest.com/studiomaya2/thumb-arthritis/Pictures only
      BingPinteresthttps://nl.pinterest.com/loriakm/thumb-arthritis/Pictures only
      GooglePubMedhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC2599975/Research journal
      YahooPubMedhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC2684204/Research journal
      BingWalmarthttps://www.walmart.com/c/ep/thumb-bracesWebshop
      YahooYoutubehttps://www.youtube.com/watch?v=3V3YEUYRvg0Video
      BingYoutubehttps://www.youtube.com/watch?v=XZ_Gy35DGPkVideo
      BingYoutubehttps://www.youtube.com/watch?v=SfOAOq5QPZkVideo
      GoogleBoulderCentrehttps://bouldercentre.com/thumb-arthritis/Video
      Google3PointProductshttp://www.3pointproducts.com/design-line-thumb-arthritis-splintWebshop
      GoogleMayoClinichttp://www.mayoclinic.org/diseases-conditions/thumb-arthritis/basics/definition/con-20027798Overlapping
      GoogleMayoClinichttp://www.mayoclinic.org/diseases-conditions/thumb-arthritis/basics/symptoms/con-20027798Overlapping
      GoogleASSHhttp://www.assh.org/handcare/hand-arm-conditions/thumb-arthritisOverlapping
      BingClickBankhttp://hiddensurvivalmuscle.com/?hop=indy2559Not relevant
      YahooDJOGlobalhttp://www.djoglobal.eu/en_UK/index.htmlNot relevant
      YahooHealthlinehttp://www.healthline.com/health/osteoarthritis/thumbOverlapping
      YahooHeraldTimesReporterhttp://www.htrnews.com/story/life/2017/06/08/ask-doctor-thumb-arthritis-there-such-thing/381549001/Overlapping
      YahooHSShttps://www.hss.edu/conditions_basal-joint-arthritis-therapy.aspOverlapping
      GoogleNYHandandWristCenterhttp://www.handsurgeonsnewyork.com/arthritis-of-the-thumb/Overlapping
      YahooOhMyArthritishttp://www.blog.ohmyarthritis.com/my-thumb-hurts-whats-wrong/Overlapping
      BingPureNaturalHealinghttp://www.purenaturalhealing.com/go/Not relevant
      YahooUconnHealthhttp://uconnsportsmed.uchc.edu/injury/handwrist/arthritis_thumb.htmlOverlapping
      YahooWebMDhttp://www.webmd.com/rheumatoid-arthritis/guide/trigger-finger#1Not relevant
      GoogleOrthoCarehttps://www.ortho-care.eu/en/hand-and-wrist/disorders/basal-thumb-arthritisNot operational anymore
      AAHS, American Association for Hand Surgery; AAOS, American Academy of Orthopaedic Surgeons; ASSH, American Society for Surgery of the Hand; BSSH, British Society for Surgery of the Hand; HSS, Hospital for Special Surgery; MHS, Midwest Hand Service; NHS, National Health Service; PHS, Public Health Service; UWMedicine, University of Washington Medicine; DJOGlobal, DJ Orthopedics Global; NYHandandWristCenter: New York Hand and Wrist Center.
      Appendix BDISCERN and LIDA Questionnaires
      DISCERN questions
       1. Are the aims clear?
       2. Does it achieve its aims?
       3. Is it relevant?
       4. Is it clear what sources of information were used to compile the publication (other than the author or producer)?
       5. Is it clear when the information used or reported in the publication was produced?
       6. Is it balanced and unbiased?
       7. Does it provide details of additional sources of support and information?
       8. Does it refer to areas of uncertainty?
       9. Does it describe how each treatment works?
       10. Does it describe the benefits of each treatment?
       11. Does it describe the risks of each treatment?
       12. Does it describe what would happen if no treatment is used?
       13. Does it describe how the treatment choices affect overall quality of life?
       14. Is it clear that there may be more than one possible treatment choice?
       15. Does it provide support for shared decision-making?
       16. Based on the answers to all the above questions, rate the overall quality of the information about treatment choices.
      LIDA questions
       Usability
      1.1. Is there a clear statement of who this Web site is for?
      1.2. Is the level of detail appropriate to their level of knowledge?
      1.3. Is the layout of the main block of information clear and readable?
      1.4. Is the navigation clear and well structured?
      1.5. Can you always tell your current location in the site?
      1.6. Is the color scheme appropriate and engaging?
      2.1. Is the same page layout used throughout the site?
      2.2. Do navigational links have a consistent function?
      2.3. Is the site structure (categories or organization of pages) applied consistently?
      3.1. Does the site provide an effective search facility?
      3.2. Does the site provide effective browsing facilities?
      3.3. Does the design minimize the cognitive overhead of using the site?
      3.4. Does the site support the normal browser navigational tools?
      3.5. Can you use the site without third party plug-ins?
      4.1. Can the user make an effective judgment of whether the site applies to them?
      4.2. Is the Web site interactive?
      4.3. Can the user personalize their experience of using the site?
      4.4. Does the Web site integrate nontextual media?
       Reliability
      1.1. Does the site respond to recent events?
      1.2. Can users submit comments on specific content?
      1.3. Is site content updated at an appropriate interval?
      2.1. Is it clear who runs the site?
      2.2. Is it clear who pays for the site?
      2.3. Is there a declaration of the objectives of the people who run the site?
      3.1. Does the site report a clear content production method?
      3.2. Is this a robust method?
      3.3. Can the information be checked from original sources?
      Figure thumbnail fx1
      Appendix CBland-Altman Plots for Inter- and Intra-observer Reliability

      References

      1. Taylor H. The Growing Influence and Use Of Health Care Information Obtained Online. Available at: http://www.harrisinteractive.com/ vault/HI-Harris-Poll-Cyberchondriacs-2011-09-15.pdf. The Harris Poll #98. September 15, 2011. Accessed September 21, 2017.

        • Becker S.J.
        • Briet J.P.
        • Hageman M.G.
        • Ring D.
        Death, taxes, and trapeziometacarpal arthrosis hand.
        Clin Orthop Relat Res. 2013; 47: 3738-3744
        • Becker S.J.
        • Makarawung D.J.
        • Spit S.A.
        • King J.D.
        • Ring D.
        Disability in patients with trapeziometacarpal joint arthrosis: incidental versus presenting diagnosis.
        J Hand Surg Am. 2014; 39: 2009-2015
        • Ochtman A.E.A.
        • Guitton T.G.
        • Buijze G.A.
        • et al.
        Trapeziometacarpal arthrosis: predictors of a second visit and surgery.
        J Hand Microsurg. 2013; 5: 9-13
      2. Clear & Simple: Achieving Qualtiy and Effectiveness in Health Communication. Bethesda, MD: National Institutes of Health, Office of Communications and Public Liaison; 2016. Available at: https://www.nih.gov/institutes-nih/nih-office-director/office-communications-public-liaison/clear-communication/clear-simple. Accessed October 30, 2017.

      3. Weiss BD. Health Literacy and Patient Safety: Help Patients Understand. Chicago: American Medical Association Foundation and American Medical Association; 2007. Available at: https://med.fsu.edu/userFiles/file/ahec_health_clinicians_manual.pdf. Accessed October 30, 2017.

        • Kamal R.N.
        • Paci G.M.
        • Daniels A.H.
        • Gosselin M.
        • Rainbow M.J.
        • Weiss A.P.
        Quality of internet health information on thumb carpometacarpal joint arthritis.
        R I Med J (2013). 2014; 97: 31-35
        • John E.S.
        • John A.M.
        • Hansberry D.R.
        • et al.
        Colorectal cancer screening patient education materials—how effective is online health information?.
        Int J Colorectal Dis. 2016; 31: 1817-1824
        • McKearney T.C.
        • McKearney R.M.
        The quality and accuracy of internet information on the subject of ear tubes.
        Int J Pediatr Otorhinolaryngol. 2013; 77: 894-897
        • Grewal P.
        • Alagaratnam S.
        The quality and readability of colorectal cancer information on the internet.
        Int J Surg. 2013; 11: 410-413
        • Mulley A.G.
        • Trimble C.
        • Elwyn G.
        Stop the silent misdiagnosis: patients' preferences matter.
        BMJ. 2012; 345: e6572
      4. Top 15 Most Popular Search Engines. eBizMBA Rank. July 2017. Available at: http://www.ebizmba.com/articles/search-engines. Accessed October 30, 2017.

      5. Tigas M, Wei S, Glassford A. Nonprofit Explorer: Research Tax-Exempt Organizations. ProPublica. 2017. Available at: https://projects.propublica.org/nonprofits/. Accessed November 2, 2017.

      6. Health On the Net Foundation. 2017. Available at: https://www.hon.ch/index.html. Accessed October 30, 2017.

        • Alamoudi U.
        • Hong P.
        Readability and quality assessment of websites related to microtia and aural atresia.
        Int J Pediatr Otorhinolaryngol. 2015; 79: 151-156
        • Fast A.M.
        • Deibert C.M.
        • Hruby G.W.
        • Glassberg K.I.
        Evaluating the quality of Internet health resources in pediatric urology.
        J Pediatr Urol. 2013; 9: 151-156
        • Davis K.S.
        • McCormick A.A.
        • Jabbour N.
        What might parents read: sorting webs of online information on vascular anomalies.
        Int J Pediatr Otorhinolaryngol. 2017; 93: 63-67
        • Bond S.
        Gunning Fog Index.
        (Available at:) (Accessed December 20, 2017)
      7. Kincaid JP, Fishburne RP Jr, Rogers RL, Chissom BS. Derivation of New Readability Formulas (Automated Readability Index, Fog Count, and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Millington, TN: Chief of Naval Technical Training: Naval Air Station Memphis, Research Branch Report; 1975:8–75.

        • Gunning R.
        The Technique of Clear Writing.
        McGraw-Hill International Book Co., New York1952
        • IBM Watson Developer Cloud
        Tone Analyzer.
        (Available at:) (Accessed October 30, 2017)
        • Charnock D.
        The DISCERN Handbook: Quality Criteria for Consumer Health Information on Treatment Choices.
        Radcliffe Medical Press, Abingdon, UK1998
      8. Minervation. The LIDA Instrument: Minervation Validation Instrument for Health Care Web Sites; 2007. http://www.minervation.com/wp-content/uploads/2011/04/Minervation-LIDA-instrument-v1-2.pdf. Accessed October 30, 2017.

      9. IBM Cloud Docs. The Science Behind the Service; 2017. Available at: https://console.bluemix.net/docs/services/tone-analyzer/science.html#the-science-behind-the-service. Accessed December 18, 2017.

        • Charnock D.
        • Shepperd S.
        • Needham G.
        • Gann R.
        DISCERN: an instrument for judging the quality of written consumer health information on treatment choices.
        J Epidemiol Community Health. 1999; 53: 105-111
        • HONcode Site Evaluation Form
        Health On the Net Foundation; 2017.
        (Available at:) (Accessed December 24, 2017)
      10. Dobrogowska-Schlebusch E, Niedźwiedzka B. Assessment of the Quality of Online Health Resources in Order to Identify the Examples of Best Practices in Creating Portals for Patients. Part One of the Study: Development and Validation of the Assessment Tool; 2011. Available at: http://www.iss.it/binary/eahi/cont/95_Ewa_Dobrogowska_Schlebusch_Full_text.pdf. Accessed December 24, 2017.

        • O’Neill S.C.
        • Baker J.F.
        • Fitzgerald C.
        • et al.
        Cauda equina syndrome: assessing the readability and quality of patient information on the internet.
        Spine (Phila Pa 1976). 2014; 39: 645-649
        • Kaicker J.
        • Debono V.B.
        • Dang W.
        • Buckley N.
        • Thabane L.
        Assessment of the quality and variability of health information on chronic pain websites using the DISCERN instrument.
        BMC Med. 2010; 8: 59
        • Nason G.J.
        • Baker J.F.
        • Byrne D.P.
        • Noel J.
        • Moore D.
        • Kiely P.J.
        Scoliosis-specific information on the internet: has the “information highway” led to better information provision?.
        Spine (Phila Pa 1976). 2012; 37: E1364-E1369
        • Banasiak N.C.
        • Meadows-Oliver M.
        Evaluating asthma websites using the Brief DISCERN instrument.
        J Asthma Allergy. 2017; 10: 191-196
        • Sullivan T.B.
        • Anderson J.T.
        • Ahn U.M.
        • Ahn N.U.
        Can internet information on vertebroplasty be a reliable means of patient self-education?.
        Clin Orthop Relat Res. 2014; 472: 1597-1604
        • Eberlin K.R.
        • Vargas C.R.
        • Chuang D.J.
        • Lee B.T.
        Patient education for carpal tunnel syndrome: analysis of readability.
        Hand (N Y). 2015; 10: 374-380
        • Moody E.
        • Clemens K.
        • Storsley L.
        • Waterman A.
        • Parikh C.
        • Garg A.
        Improving on-line information for potential living kidney donors.
        Kidney Int. 2007; 71: 1062-1070
        • Chi E.
        • Jabbour N.
        • Aaronson N.L.
        Quality and readability of websites for patient information on tonsillectomy and sleep apnea.
        Int J Pediatr Otorhinolaryngol. 2017; 98: 1-3