Purpose
People increasingly search the Internet for information about common medical problems such as trapeziometacarpal (TMC) joint arthritis. But this information can be biased, inaccurate, and misleading. Medical professionals should be aware of what patients may be reading about their condition because concepts and beliefs can affect symptoms, limitations, and decision making. This study sought factors associated with the quality of design and content of health information Web sites about TMC arthritis.
Methods
Using 3 search engines we entered “thumb arthritis” and measured the quality of design and content of 67 Web sites using the DISCERN and LIDA tools, dominant tones using the IBM Watson Tone Analyzer, and readability, and we recorded Web site characteristics. All but 1 Web site exceeded the recommended sixth-grade reading level. We created 2 backward stepwise regression models to identify independent factors associated with Web site design and content quality.
Results
In multivariable analysis, the Web site not having a clear preference for treatment was independently associated with greater design and content quality measured by DISCERN. Health On the Net (HON) code certification—a code of conduct for medical Web sites—and nonprofit Web sites had higher LIDA scores.
Conclusions
Online information on TMC arthrosis is difficult to read, often biased in favor of a particular treatment and influenced by profit and HONcode.
Clinical relevance
Hand surgeons should prepare to gently correct misconceptions established or reinforced, in part, by material found on the Internet.
Key words
People increasingly search the Internet for information about medical problems (Appendices A-C, available on the Journal's Web site at www.jhandsurg.org). Seventy-four percent of adults in the United States have searched online for medical information.
1
There is a great deal of information available online for common illnesses such as trapeziometacarpal (TMC) arthritis.Taylor H. The Growing Influence and Use Of Health Care Information Obtained Online. Available at: http://www.harrisinteractive.com/ vault/HI-Harris-Poll-Cyberchondriacs-2011-09-15.pdf. The Harris Poll #98. September 15, 2011. Accessed September 21, 2017.
2
, 3
, 4
The American Medical Association and the National Institutes of Health recommend that patient information be written at a sixth-grade reading level5
, Clear & Simple: Achieving Qualtiy and Effectiveness in Health Communication. Bethesda, MD: National Institutes of Health, Office of Communications and Public Liaison; 2016. Available at: https://www.nih.gov/institutes-nih/nih-office-director/office-communications-public-liaison/clear-communication/clear-simple. Accessed October 30, 2017.
6
; however, a recent article found that TMC arthritis online information uniformly exceeds this recommended reading level.Weiss BD. Health Literacy and Patient Safety: Help Patients Understand. Chicago: American Medical Association Foundation and American Medical Association; 2007. Available at: https://med.fsu.edu/userFiles/file/ahec_health_clinicians_manual.pdf. Accessed October 30, 2017.
7
The quality of online information varies.
7
, 8
, 9
, 10
The information patients find can be biased, inaccurate, or misleading.9
Although there is a risk of misinformation and misunderstanding, many people are not aware of the accuracy or trustworthiness of the information they obtain. One study found that 9 out of 10 people describe the information they find as reliable.1
Taylor H. The Growing Influence and Use Of Health Care Information Obtained Online. Available at: http://www.harrisinteractive.com/ vault/HI-Harris-Poll-Cyberchondriacs-2011-09-15.pdf. The Harris Poll #98. September 15, 2011. Accessed September 21, 2017.
It can be helpful for medical professionals to know the quality, tone, and readability of online information most commonly read by their patients. There may be greater risk of incorrectly identifying patient preferences (going with preferences based on misconceptions rather than preferences based on values) if people have preconceptions and misconceptions that are influenced by online information.
11
This study addressed factors independently associated with TMC osteoarthritis online information design and content quality measured using the (1) DISCERN and (2) LIDA instruments.Materials and Methods
Data source
Because our study involves no participants, it was exempt from institutional review board approval. On July 31, 2017, we searched for “thumb arthritis” in 3 of the most popular Web search engines (Google, Yahoo!, and Bing).
12
Because results can be influenced by search history, we searched with the browser in privacy mode after deleting our search history and cookies. We recorded the first 50 hits of each search engine. After excluding duplicates (n = 57) and irrelevant Web sites (n = 26, eg, blogs, web-shops, research journals, videos), 67 Web sites were included.Top 15 Most Popular Search Engines. eBizMBA Rank. July 2017. Available at: http://www.ebizmba.com/articles/search-engines. Accessed October 30, 2017.
Measurements
We recorded what treatment options were discussed (operative, nonoperative, both, or none), if there was a clear preference for 1 treatment, and if the Web site was profit or nonprofit. For U.S. Web sites we used Nonprofit Explorer, a Web site that registers tax-exempt organizations, to look for nonprofit organizations.
13
In addition, we registered if websites had a Health On the Net (HON) code. The Health On the Net Foundation is an internationally recognized organization that provides certification of conduct, a HONcode, for medical Web sites. A HON seal is accredited when a Web site meets their 8 principles: authoritative, complementary, privacy, attribution, justifiability, transparency, financial disclosure, and advertising policy.Tigas M, Wei S, Glassford A. Nonprofit Explorer: Research Tax-Exempt Organizations. ProPublica. 2017. Available at: https://projects.propublica.org/nonprofits/. Accessed November 2, 2017.
14
The HONcode is an accepted quality measure for online health sites.Health On the Net Foundation. 2017. Available at: https://www.hon.ch/index.html. Accessed October 30, 2017.
14
, Health On the Net Foundation. 2017. Available at: https://www.hon.ch/index.html. Accessed October 30, 2017.
15
, 16
Readability is defined as the ease with which a reader can understand a written text.
17
We used 3 of the most reliable scores for readability: the Flesch Reading Ease Score (FRES), the Flesch-Kincaid Grade Level (FKGL), and the Gunning Fog Index (GFI). We used the spelling and grammar tool in Microsoft Word (Microsoft Corp., Redmond, WA) to determine the FRES and FKGL. We used an online calculator to assess the GFI.The FRES rates text on a 100-point rating scale with higher scores representing easier readability. The FKGL reflects the required U.S. grade level to comprehend text, for example, a score of 7.2 indicates that text is expected to be understandable for an average student in seventh grade. It ranges from 3 to 12, with greater score indicating more difficult readability.
19
The GFI is a measure of text readability based on the use of difficult words and the length of sentences.Kincaid JP, Fishburne RP Jr, Rogers RL, Chissom BS. Derivation of New Readability Formulas (Automated Readability Index, Fog Count, and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Millington, TN: Chief of Naval Technical Training: Naval Air Station Memphis, Research Branch Report; 1975:8–75.
20
Scores range from 0 to 15 and represent the estimated years of school education that is required to understand text on first reading.The IBM Watson Tone Analyzer measures tones that are present in written text in 3 categories: emotion (tones of anger, disgust, fear, joy, or sadness), language style (language that is analytical, confident, or tentative), and social tendencies (language reflecting openness, conscientiousness, extraversion, agreeableness, or emotional range) (Table 1). Scores of each tone range from 0 to 1.0 with scores greater than 0.5 indicating likely present, and scores greater than 0.75 indicating very likely present tones. We copied all Web site texts in the analyzer and used the document level to get a sense of the dominant tones of the website.
Table 1Definition of Dominant Tones
| Definition | |
|---|---|
| Emotional Tones | |
| Anger | Is evoked due to injustice, conflict, humiliation, negligence, or betrayal. If anger is active, a person attacks the target, verbally or physically. If passive, a person silently sulks and feels tension and hostility. |
| Disgust | A feeling of revulsion or strong disapproval aroused by something unpleasant or offensive. |
| Fear | Fear is a response to impending danger. It is a survival mechanism that is triggered as a reaction to some negative stimulus. Fear can be a mild caution or an extreme phobia. |
| Joy | Joy or happiness has shades of enjoyment, satisfaction, and pleasure. There is a sense of well-being, inner peace, love, safety, and contentment. |
| Sadness | Indicates a feeling of loss and disadvantage. When a person can be observed to be quiet, less energetic, and withdrawn, it may be inferred that sadness exists. |
| Language Styles | |
| Analytical | A person's reasoning and analytical attitude about things. |
| Confident | A person's degree of certainty. |
| Tentative | A person's degree of inhibition. |
| Social Tendencies | |
| Openness | The extent to which a person is open to experiencing a variety of activities. |
| Conscientiousness | A person's tendency to act in an organized or thoughtful way. |
| Extraversion | A person's tendency to seek stimulation in the company of others. |
| Agreeableness | A person's tendency to be compassionate and cooperative toward others. |
| Emotional range | The extent to which a person's emotions are sensitive to the individual's environment. |
From IBM Cloud Docs. Personality Insights. Available at: https://console.bluemix.net/docs/services/personality-insights/models.html#models. Accessed June 2, 2018.
The DISCERN and LIDA tools gauge health information quality. The DISCERN contains 16 questions, 8 on reliability, 7 on treatment information and choices, and 1 question on overall quality. Each question is rated on a 5-point Likert scale with answers from 1 no to 5 yes with a maximum score of 80. A higher score indicates greater overall quality.
22
The LIDA measures 3 domains: accessibility, usability, and reliability. It consists of 27 questions scored on a 4-point Likert scale, with scores ranging from 0 never to 3 always. Total scores range between 0 and 100 with a higher score indicating greater quality.23
We rated usability and reliability because the online calculation for accessibility is no longer available.Minervation. The LIDA Instrument: Minervation Validation Instrument for Health Care Web Sites; 2007. http://www.minervation.com/wp-content/uploads/2011/04/Minervation-LIDA-instrument-v1-2.pdf. Accessed October 30, 2017.
Reliability
To assess intraobserver reliability, DISCERN and LIDA were scored twice by a single reviewer (J.S.E.O.) with 2 weeks in between evaluations. The DISCERN and LIDA scores are the mean of both ratings. The intraobserver reliability, measured by intraclass correlation coefficient, of DISCERN was 0.91 (95% confidence interval [95% CI], 0.85–0.94) and LIDA was 0.85 (95% CI, 0.77–0.90) (Table 2). Bland-Altman plots showed a small difference in agreement between the 2 ratings and minimal systematic differences (Appendix C).
Table 2Reliability Scores
| Variables Scored | ICC (95% CI) | P Value |
|---|---|---|
| Intraobserver reliability | ||
| DISCERN | 0.91 (0.85–0.94) | < .001 |
| LIDA | 0.85 (0.77–0.90) | < .001 |
| Interobserver reliability | ||
| DISCERN | 0.95 (0.91–0.98) | < .001 |
| LIDA | 0.88 (0.78– 0.94) | < .001 |
| Interobserver agreement | Kappa ± SE | |
| Treatment options discussed | 1.00 (0.17) | < .001 |
| Clear preference for treatment | 1.00 (0.13) | < .001 |
| Nonprofit Web site | 1.00 (0.17) | < .001 |
ICC, intraclass correlation coefficient.
Bold values indicate statistical significance (P < .05).
∗ Two-way mixed-effects model for ICC with 95% CI.
† Interobserver agreement as kappa with SE.
To assess interobserver reliability, a second reviewer (J.T.P.K.) rated 30 Web sites on DISCERN, LIDA, profit versus nonprofit, clear preference for 1 treatment or not, and what treatment options were discussed. The interobserver reliability, measured by intraclass correlation coefficient, for DISCERN was 0.95 (95% CI, 0.91–0.98) and for LIDA was 0.88 (95% CI, 0.78–0.94) (Table 2). The interobserver reliability, measured by kappa, for nonprofit was 1.00 (standard error [SE], 0.17), for discussed treatment options 1.00 (SE, 0.17), and for clear preference for treatment 1.00 (SE, 0.13) (Table 2).
The IBM Watson Tone Analyzer was developed to analyze tones in customer service conversations. It is based on 96,000 customer service Twitter conversations, rated by 5 trained annotators for tone. IBM trained a machine-learning model based on this dataset. IBM states the model demonstrated high accuracy compared with a benchmark dataset, but they do not mention actual numbers to gauge the reliability of the machine-learning program.
24
IBM Cloud Docs. The Science Behind the Service; 2017. Available at: https://console.bluemix.net/docs/services/tone-analyzer/science.html#the-science-behind-the-service. Accessed December 18, 2017.
Website characteristics
Of the 67 Web sites, 19 (28%) were nonprofit and 14 (21%) HONcode-accredited (Table 3). Mean readability scores for FRES were 53 ± 9.7 (indicating “difficult to read”); for FKGL, 10 ± 1.7 (corresponds with 10th-grade reading level); and for GFI, 13 ± 2.0 (indicating 13 years of education required to understand text). Only 1 Web site was written below seventh-grade level (FKGL 4.7), as recommended by The National Institutes of Health and The American Medical Association.
6
, Weiss BD. Health Literacy and Patient Safety: Help Patients Understand. Chicago: American Medical Association Foundation and American Medical Association; 2007. Available at: https://med.fsu.edu/userFiles/file/ahec_health_clinicians_manual.pdf. Accessed October 30, 2017.
7
The most likely present tones in Web sites texts were openness (0.85 ± 0.09) and tentative (0.72 ± 0.19). The mean LIDA score was 61 ± 13; the mean DISCERN was 45 ± 12.Table 3Web Site Characteristics
| Variables | Values |
|---|---|
| Web sites | 67 |
| HONcode, n (%) | 14 (21) |
| Nonprofit, n (%) | 19 (28) |
| Both treatment options discussed, n (%) | 53 (79) |
| Clear preference for treatment, n (%) | |
| None | 44 (66) |
| Nonoperative treatment | 15 (22) |
| Operative treatment | 8 (12) |
| Readability scores | |
| FRES | 53 ± 9.7 |
| FKGL | 10 ± 1.7 |
| GFI | 13 ± 2.0 |
| Web site tone | |
| Anger | 0.07 ± 0.05 |
| Disgust | 0.08 ± 0.08 |
| Fear | 0.26 ± 0.13 |
| Joy | 0.35 ± 0.18 |
| Sadness | 0.61 ± 0.04 |
| Analytical | 0.63 ± 0.14 |
| Confident | 0.00 ± 0.00 |
| Tentative | 0.72 ± 0.19 |
| Openness | 0.85 ± 0.09 |
| Conscientiousness | 0.29 ± 0.14 |
| Extraversion | 0.09 ± 0.07 |
| Agreeableness | 0.13 ± 0.10 |
| Emotional range | 0.24 ± 0.12 |
| Quality of information | |
| DISCERN | 45 ± 12 |
| LIDA | 61 ± 13 |
∗ Continuous variables as mean (± SD); discrete variables as number (percentage).
Statistical analysis
Continuous variables are reported as mean and standard deviation (SD) and discrete variables as number and percentage. We used Student t test to compare continuous and dichotomous variables, and Pearson correlation for 2 continuous variables.
We created 2 backward stepwise regression models to identify independent predictors of Web sites’ quality of information measured with DISCERN and LIDA. We included all factors with P less than .10 on bivariate analysis (Table 4) in the final multivariable models (Table 5). Adjusted R2 indicates the proportion of variability in the outcome variable (either DISCERN or LIDA score) that is accounted for by the model. Semipartial R2 expresses the specific variability of a given independent variable in the model.
Table 4Bivariate Analyses
| Variables | DISCERN | P Value | LIDA | P Value |
|---|---|---|---|---|
| HONcode | ||||
| Yes | 51 ± 10 | .032 | 71 ± 11 | .0017 |
| No | 43 ± 12 | 59 ± 13 | ||
| Nonprofit | ||||
| Yes | 50 ± 11 | .033 | 69 ± 12 | .0020 |
| No | 43 ± 12 | 58 ± 13 | ||
| Both treatment options discussed | ||||
| Yes | 47 ± 11 | .002 | 62 ± 14 | .17 |
| No | 36 ± 12 | 57 ± 10 | ||
| Clear preference for treatment | ||||
| None | 49 ± 10 | < .001 | 64 ± 13 | .019 |
| Nonoperative treatment | 36 ± 12 | 55 ± 13 | ||
| Operative treatment | 39 ± 12 | 55 ± 9.5 | ||
| Readability scores (r) | ||||
| FRES | –0.063 | .61 | –0.2 | .10 |
| FKGL | –0.1 | .42 | 0.086 | .49 |
| GFI | 0.16 | .21 | 0.046 | .71 |
| Web site tone (r) | ||||
| Anger | 0.18 | .14 | 0.14 | .27 |
| Disgust | 0.17 | .18 | 0.13 | .31 |
| Fear | 0.29 | .02 | 0.18 | .14 |
| Joy | –0.04 | .75 | –0.16 | .18 |
| Sadness | 0.18 | .14 | –0.12 | .32 |
| Analytical | 0.11 | .39 | –0.13 | .32 |
| Confident | 0.00 | .00 | 0.00 | .00 |
| Tentative | 0.23 | .059 | 0.27 | .03 |
| Openness | –0.042 | .74 | –0.14 | .26 |
| Conscientiousness | 0.030 | .81 | –0.17 | .16 |
| Extraversion | 0.23 | .057 | –0.11 | .38 |
| Agreeableness | 0.018 | .89 | 0.07 | .58 |
| Emotional range | 0.21 | .084 | –0.18 | .14 |
Bold indicates statistical significance (P < .05).
∗ Pearson correlation indicated by r; bold indicates statistically significant difference; continuous variables as mean (± SD); discrete variables as number (percentage).
Table 5Stepwise Regression for Predictors of Quality of Information
| Dependent Variables | Predictors | Regression Coefficient (95% CI) | Standard Error | P Value | Semipartial R2 | Adjusted R2 |
|---|---|---|---|---|---|---|
| DISCERN | Clear preference for treatment | |||||
| None | Reference value | .19 | ||||
| Nonoperative treatment | –12 (–19 to –6.0) | 3.2 | < .001 | 0.18 | ||
| Operative treatment | –9.7 (–18 to –1.4) | 4.1 | .022 | 0.068 | ||
| LIDA | HONcode | 11 (4.3 to 18) | 3.5 | .002 | 0.12 | .23 |
| Nonprofit | 10 (3.7 to 16) | 3.2 | .002 | 0.12 | ||
Bold indicates statistical significance (P < .05).
∗ Variables inserted in the stepwise regression for DISCERN model: HONcode, nonprofit, treatment options, treatment preference, Web site tones: fear, tentative, extraversion, emotional range. For LIDA model: HONcode, nonprofit, treatment preference, Web site tone: tentative.
An a priori sample size estimate indicated that a sample of 65 Web sites would provide 80% statistical power, with alpha set at 0.05, for a regression with 5 independent variables if a single variable would account for 10% or more of the variability in quality, and our complete model would account for 20% of the overall variability in quality.
Results
DISCERN
Accounting for potential interaction of variables using multivariable analysis, greater Web site quality measured by DISCERN was independently associated with not having a clear preference for treatment compared with having a clear preference (preference for either nonoperative treatment: β regression coefficient [β] = –12; 95% CI, –19 to –6.0; semipartial R2 = 0.18; P < .05; or preference for operative treatment: β = –9.7; 95% CI = –18 to –1.4; semipartial R2 = 0.068; P < .05) (adjusted R2 = 0.19) (Table 5). Dominant Web site tone of fear (r = 0.29; P < .05), being HONcode-certified (P = .032), and nonprofit Web sites (P = 0.033) were significant in bivariate analysis but were not retained in the final model (Table 4).
LIDA
Greater Website quality, measured by LIDA, was independently associated with being HONcode-certified (β = 11; 95% CI = 4.3 to 18; semipartial R2 = 0.12; P < .05) and being nonprofit (β = 10; 95% CI = 3.7 to 16; semipartial R2 = 0.12; P < .05) (adjusted R2 = 0.23) (Table 5). Dominant tentative tone (r = 0.27; P = .030) and clear preference for treatment (P = .019) were significant in bivariate analysis but were not retained in the final model (Table 4).
Discussion
Information that patients find online about common health conditions like TMC arthritis can be difficult to read, biased, and misleading.
1
, Taylor H. The Growing Influence and Use Of Health Care Information Obtained Online. Available at: http://www.harrisinteractive.com/ vault/HI-Harris-Poll-Cyberchondriacs-2011-09-15.pdf. The Harris Poll #98. September 15, 2011. Accessed September 21, 2017.
7
, 8
, 9
, 10
This study addressed factors associated with the design and content quality of TMC arthritis online information measured by DISCERN and LIDA.We acknowledge some study limitations. First, this study is limited to Web sites that were written in English and available on the date of search, July 31, 2017. The Internet is changing quickly, and its dynamic character makes it impossible to predict when and how online information will be edited or updated. Our study might not reflect the most recent available online information regarding TMC arthritis. Second, we used the search term “thumb arthritis” and no other medical terms such as “carpometacarpal” or “trapeziometacarpal osteoarthritis.” Different terms would have produced a wider range of Web sites; however, the term “thumb arthritis” may be more commonly used by patients than sophisticated medical terms. Third, the quality of videos, images, and other multimedia factors were not assessed, although these materials might contribute to overall comprehension and better understanding of health information. Fourth, the DISCERN and LIDA quality assessment tools rely on subjective input, despite strict criteria for assessment of each rating. This may lead to bias of the observers, although our study showed good intra- and interobserver reliability scores. Fifth, we addressed the quality and not the accuracy of the content based on current best evidence. It is possible to present quality content that is not evidence-based. Sixth, the reliability of the Watson tone analyzer for medical Web sites is unknown. In addition, language assessment tools only give an estimation of the likelihood of dominant tones and emotions in written text. It does not address the purposes of the author nor the existence of possible bias. Although little is studied about the correlation between language assessment tools and patient information Web sites, tone analyzers are increasingly used by communication experts in the field of marketing, customer service, and education.
The finding that lower Web site quality measured by DISCERN was independently associated with a clear preference for treatment, compared with no clear preference, can be explained by the fact that the DISCERN tool focuses specifically on the quality of written information about treatment choices.
22
, 25
The DISCERN instrument can be used to assess the quality of a Web site that discusses 1 particular treatment choice, as long as it is clear that other treatment choices are available (Questions 6 and 14) and that only 1 treatment option is discussed by the authors (Question 1). Websites that only focus on 1 treatment option and are not clear about other available treatments score lower on the DISCERN measure. A previous study investigated TMC arthritis information Web sites for their readability and quality.7
To assess quality, they used more limited and less often used measures with fewer answer options, such as the HONcode Site Evaluation Form,26
instead of DISCERN and LIDA tools.23
, Minervation. The LIDA Instrument: Minervation Validation Instrument for Health Care Web Sites; 2007. http://www.minervation.com/wp-content/uploads/2011/04/Minervation-LIDA-instrument-v1-2.pdf. Accessed October 30, 2017.
25
, 27
In addition, they did not analyze Web site tones or perform multivariable analysis to assess factors associated with the quality of information. They found that health information on TMC arthritis is of generally poor quality, is predominantly posted by physician authors, and is too hard to read. Of the 60 Web sites analyzed, only 3 contained a HONcode.Dobrogowska-Schlebusch E, Niedźwiedzka B. Assessment of the Quality of Online Health Resources in Order to Identify the Examples of Best Practices in Creating Portals for Patients. Part One of the Study: Development and Validation of the Assessment Tool; 2011. Available at: http://www.iss.it/binary/eahi/cont/95_Ewa_Dobrogowska_Schlebusch_Full_text.pdf. Accessed December 24, 2017.
7
The finding that greater Website quality measured by LIDA, was associated with being HONcode-certified, is in line with previous findings. Several studies on the quality of online information about colorectal cancer information, chronic pain, asthma, and scoliosis indicate HON certification as a factor associated with greater website quality.
10
, 28
, 29
, 30
, 31
Conversely, other studies on TMC arthritis, vascular anomalies, and vertebroplasty, including HON certification as an independent variable, did not find this association,7
, 17
, 32
possibly owing to the low number of HON-certified Web sites in the latter studies’ samples. Nonprofit Web sites showed better quality, probably because they have no commercial aims to sell products or services. Two studies on the quality and readability of online colorectal cancer information10
and of online cauda equina syndrome information30
indicate a similar relationship between greater Web site quality and government-authored websites. Whereas DISCERN assesses the presentation of treatment options, the LIDA tool focuses more on usability and reliability. Specific tones were significant in bivariate analysis, suggesting that they might be clues to lower quality, but were not independent of the profit status and HONcode. Readability was not associated with quality scores, perhaps because the Web sites on average were written for relatively highly educated people.7
, 9
, 28
, 33
, 34
, 35
Another explanation is that the LIDA and DISCERN tools do not sufficiently assess and capture readability levels of Web sites. The DISCERN instrument does not contain any question regarding readability, the LIDA tool only contains 1 (2.1.2.: Is the level of detail appropriate to their level of knowledge?). This lack of readability items should be kept in mind when using these outcome measures to rate Web site quality.This study found that Web-based information on TMC arthritis is difficult to read, often biased in favor of a particular treatment, and influenced by profit and HONcode. Improved Web sites and decision aids (tools providing unbiased information to patients and that help clarify patients’ preferences) could help ensure that patient decisions about TMC arthritis and other common health conditions are based on their values and not on misconceptions. Measures of design, content, tone, and readability can assist with the development of improved patient information.
Acknowledgments
T.T. receives payments or benefits from AO Trauma, Stryker, DePuy Synthes, PATIENT+, and VCC. D.R. receives royalties from Skeletal Dynamics and Wright Medical; receives a stipend from Clinical Orthopaedics and Related Research; receives honoraria for talks from universities and hospitals; and receives payment form lawyers for expert review.
Appendix
Appendix AIncluded and Excluded Web Sites
AAHS, American Association for Hand Surgery; AAOS, American Academy of Orthopaedic Surgeons; ASSH, American Society for Surgery of the Hand; BSSH, British Society for Surgery of the Hand; HSS, Hospital for Special Surgery; MHS, Midwest Hand Service; NHS, National Health Service; PHS, Public Health Service; UWMedicine, University of Washington Medicine; DJOGlobal, DJ Orthopedics Global; NYHandandWristCenter: New York Hand and Wrist Center.
Appendix BDISCERN and LIDA Questionnaires
| DISCERN questions |
| 1. Are the aims clear? |
| 2. Does it achieve its aims? |
| 3. Is it relevant? |
| 4. Is it clear what sources of information were used to compile the publication (other than the author or producer)? |
| 5. Is it clear when the information used or reported in the publication was produced? |
| 6. Is it balanced and unbiased? |
| 7. Does it provide details of additional sources of support and information? |
| 8. Does it refer to areas of uncertainty? |
| 9. Does it describe how each treatment works? |
| 10. Does it describe the benefits of each treatment? |
| 11. Does it describe the risks of each treatment? |
| 12. Does it describe what would happen if no treatment is used? |
| 13. Does it describe how the treatment choices affect overall quality of life? |
| 14. Is it clear that there may be more than one possible treatment choice? |
| 15. Does it provide support for shared decision-making? |
| 16. Based on the answers to all the above questions, rate the overall quality of the information about treatment choices. |
| LIDA questions |
| Usability |
| 1.1. Is there a clear statement of who this Web site is for? |
| 1.2. Is the level of detail appropriate to their level of knowledge? |
| 1.3. Is the layout of the main block of information clear and readable? |
| 1.4. Is the navigation clear and well structured? |
| 1.5. Can you always tell your current location in the site? |
| 1.6. Is the color scheme appropriate and engaging? |
| 2.1. Is the same page layout used throughout the site? |
| 2.2. Do navigational links have a consistent function? |
| 2.3. Is the site structure (categories or organization of pages) applied consistently? |
| 3.1. Does the site provide an effective search facility? |
| 3.2. Does the site provide effective browsing facilities? |
| 3.3. Does the design minimize the cognitive overhead of using the site? |
| 3.4. Does the site support the normal browser navigational tools? |
| 3.5. Can you use the site without third party plug-ins? |
| 4.1. Can the user make an effective judgment of whether the site applies to them? |
| 4.2. Is the Web site interactive? |
| 4.3. Can the user personalize their experience of using the site? |
| 4.4. Does the Web site integrate nontextual media? |
| Reliability |
| 1.1. Does the site respond to recent events? |
| 1.2. Can users submit comments on specific content? |
| 1.3. Is site content updated at an appropriate interval? |
| 2.1. Is it clear who runs the site? |
| 2.2. Is it clear who pays for the site? |
| 2.3. Is there a declaration of the objectives of the people who run the site? |
| 3.1. Does the site report a clear content production method? |
| 3.2. Is this a robust method? |
| 3.3. Can the information be checked from original sources? |
Appendix CBland-Altman Plots for Inter- and Intra-observer Reliability
References
Taylor H. The Growing Influence and Use Of Health Care Information Obtained Online. Available at: http://www.harrisinteractive.com/ vault/HI-Harris-Poll-Cyberchondriacs-2011-09-15.pdf. The Harris Poll #98. September 15, 2011. Accessed September 21, 2017.
- Death, taxes, and trapeziometacarpal arthrosis hand.Clin Orthop Relat Res. 2013; 47: 3738-3744
- Disability in patients with trapeziometacarpal joint arthrosis: incidental versus presenting diagnosis.J Hand Surg Am. 2014; 39: 2009-2015
- Trapeziometacarpal arthrosis: predictors of a second visit and surgery.J Hand Microsurg. 2013; 5: 9-13
Clear & Simple: Achieving Qualtiy and Effectiveness in Health Communication. Bethesda, MD: National Institutes of Health, Office of Communications and Public Liaison; 2016. Available at: https://www.nih.gov/institutes-nih/nih-office-director/office-communications-public-liaison/clear-communication/clear-simple. Accessed October 30, 2017.
Weiss BD. Health Literacy and Patient Safety: Help Patients Understand. Chicago: American Medical Association Foundation and American Medical Association; 2007. Available at: https://med.fsu.edu/userFiles/file/ahec_health_clinicians_manual.pdf. Accessed October 30, 2017.
- Quality of internet health information on thumb carpometacarpal joint arthritis.R I Med J (2013). 2014; 97: 31-35
- Colorectal cancer screening patient education materials—how effective is online health information?.Int J Colorectal Dis. 2016; 31: 1817-1824
- The quality and accuracy of internet information on the subject of ear tubes.Int J Pediatr Otorhinolaryngol. 2013; 77: 894-897
- The quality and readability of colorectal cancer information on the internet.Int J Surg. 2013; 11: 410-413
- Stop the silent misdiagnosis: patients' preferences matter.BMJ. 2012; 345: e6572
Top 15 Most Popular Search Engines. eBizMBA Rank. July 2017. Available at: http://www.ebizmba.com/articles/search-engines. Accessed October 30, 2017.
Tigas M, Wei S, Glassford A. Nonprofit Explorer: Research Tax-Exempt Organizations. ProPublica. 2017. Available at: https://projects.propublica.org/nonprofits/. Accessed November 2, 2017.
Health On the Net Foundation. 2017. Available at: https://www.hon.ch/index.html. Accessed October 30, 2017.
- Readability and quality assessment of websites related to microtia and aural atresia.Int J Pediatr Otorhinolaryngol. 2015; 79: 151-156
- Evaluating the quality of Internet health resources in pediatric urology.J Pediatr Urol. 2013; 9: 151-156
- What might parents read: sorting webs of online information on vascular anomalies.Int J Pediatr Otorhinolaryngol. 2017; 93: 63-67
- Gunning Fog Index.(Available at:) (Accessed December 20, 2017)
Kincaid JP, Fishburne RP Jr, Rogers RL, Chissom BS. Derivation of New Readability Formulas (Automated Readability Index, Fog Count, and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Millington, TN: Chief of Naval Technical Training: Naval Air Station Memphis, Research Branch Report; 1975:8–75.
- The Technique of Clear Writing.McGraw-Hill International Book Co., New York1952
- Tone Analyzer.(Available at:) (Accessed October 30, 2017)
- The DISCERN Handbook: Quality Criteria for Consumer Health Information on Treatment Choices.Radcliffe Medical Press, Abingdon, UK1998
Minervation. The LIDA Instrument: Minervation Validation Instrument for Health Care Web Sites; 2007. http://www.minervation.com/wp-content/uploads/2011/04/Minervation-LIDA-instrument-v1-2.pdf. Accessed October 30, 2017.
IBM Cloud Docs. The Science Behind the Service; 2017. Available at: https://console.bluemix.net/docs/services/tone-analyzer/science.html#the-science-behind-the-service. Accessed December 18, 2017.
- DISCERN: an instrument for judging the quality of written consumer health information on treatment choices.J Epidemiol Community Health. 1999; 53: 105-111
- Health On the Net Foundation; 2017.(Available at:) (Accessed December 24, 2017)
Dobrogowska-Schlebusch E, Niedźwiedzka B. Assessment of the Quality of Online Health Resources in Order to Identify the Examples of Best Practices in Creating Portals for Patients. Part One of the Study: Development and Validation of the Assessment Tool; 2011. Available at: http://www.iss.it/binary/eahi/cont/95_Ewa_Dobrogowska_Schlebusch_Full_text.pdf. Accessed December 24, 2017.
- Cauda equina syndrome: assessing the readability and quality of patient information on the internet.Spine (Phila Pa 1976). 2014; 39: 645-649
- Assessment of the quality and variability of health information on chronic pain websites using the DISCERN instrument.BMC Med. 2010; 8: 59
- Scoliosis-specific information on the internet: has the “information highway” led to better information provision?.Spine (Phila Pa 1976). 2012; 37: E1364-E1369
- Evaluating asthma websites using the Brief DISCERN instrument.J Asthma Allergy. 2017; 10: 191-196
- Can internet information on vertebroplasty be a reliable means of patient self-education?.Clin Orthop Relat Res. 2014; 472: 1597-1604
- Patient education for carpal tunnel syndrome: analysis of readability.Hand (N Y). 2015; 10: 374-380
- Improving on-line information for potential living kidney donors.Kidney Int. 2007; 71: 1062-1070
- Quality and readability of websites for patient information on tonsillectomy and sleep apnea.Int J Pediatr Otorhinolaryngol. 2017; 98: 1-3
Article info
Publication history
Accepted:
August 1,
2018
Received:
January 12,
2018
Footnotes
No benefits in any form have been received or will be received related directly or indirectly to the subject of this article.
Identification
Copyright
© 2018 by the American Society for Surgery of the Hand. All rights reserved.
30031-5&id=fx1.jpg){kind=link}