Evaluation of internet websites about retinopathy of prematurity patient education
http://www.100md.com
《英国眼科学杂志》
1 Department of Ophthalmology, Federal University of S?o Paulo, S?o Paulo, Brazil
2 Department of Ophthalmology, University of California-Davis, Sacramento, CA, USA
Correspondence to:
Lawrence S Morse
MD, PhD, Department of Ophthalmology, University of California-Davis, 4860 Y Street, Suite 2400, Sacramento, CA 95817, USA; lsmorse@ucdavis.edu
Accepted for publication 6 October 2004
ABSTRACT
Background/aims: The success of the treatment in patients with retinopathy of prematurity (ROP) is mainly associated with timely diagnosis and appropriate management. Information dissemination is crucial for the outcome of ROP. This study aimed to evaluate the quality of the information about ROP available for patients on the internet.
Methods: Cross sectional study. In March 2004 the ROP information available on the internet was evaluated using two search engines (MetaCrawler and MSN) and four key terms ("retinopathy of prematurity," "premature eye," "premature retina," and "ROP"). The quality of each website was evaluated using a score system. The sites were classified as academic, organisational, or commercial. Readability, general quality of the website (based on ownership, purpose, authorship, author qualification, attribution, interactivity, and currency), and quality of the content specific to ROP (definition, causes, epidemiology, risk factors, diagnosis, classification, treatment, and prognosis) were analysed.
Results: Of 114 websites evaluated, 40 were included. 10 sites (25.0%) were academic, eight (20.0%) organisational, and 22 (55.0%) commercial. In the majority of the sites (62.5%) the ROP information was fair or poor.
Conclusions: A large amount of information about ROP is available on the internet. However, most websites were considered incomplete.
Abbreviations: ROP, retinopathy of prematurity
Keywords: internet; retinopathy of prematurity
Because of its accessibility and speed, the internet has become a popular mean of obtaining information. The use of understandable language and visual effects is frequent, making websites very attractive to patients and, therefore, an option for the improvement of medical information accessibility.
However, the large volume of information available on the world wide web makes it difficult for patients to judge the accuracy of the data provided. In an attempt to evaluate the appropriateness of health information provided on the internet, studies have been conducted.1,2,3,4,5,6,7,8,9,10,11,12,13
Retinopathy of prematurity (ROP) is a disease that affects vasculature development in the eyes of premature infants and presents five progressive clinical stages, being a major cause of poor vision and blindness in premature infants. Its prognosis is strongly related to timely diagnosis and appropriate treatment.
The purpose of this study was to evaluate the quality of ROP information presented on the internet.
MATERIALS AND METHODS
Site selection
In March 2004 we used two search engines, MSN (www.msn.com) and MetaCrawler (www.metacrawler.com) to select the sites, as previously reported,2 using the keywords "retinopathy of prematurity," "ROP," "premature eye," and "premature retina." MetaCrawler integrates the results for a query from different well known search engines including Google, Alta Vista, Ask Jeeves, About, Looksmart, Teoma (DirectHit), Overture, Find What, and Yahoo, providing an accurate approximation to a list of the most popular ROP sites that would be encountered by a typical user. The keywords were arbitrarily chosen to represent likely possibilities used by patients during an internet search.
The studied sites were chosen by screening the first 30 sites retrieved under each keyword. Sites charging service fees or in a language other than English, mailing lists, bulletin boards, discussion groups, and sites explicitly stating healthcare professionals or students as their target audience were not included.
Site evaluation
Site evaluation was divided into readability, quality, and technical.
Readability
To determine the site readability, we copied the ROP material from each site, pasted it into a Microsoft Word document (Microsoft Office, 2000), and obtained the Flesch-Kincaid grade level score. This score rates text on US school grade levels or years (range third to 12th grade) and it is based on the average of sentence length and number of syllables per word.14 A score of 8.0 or less is the recommended level for standard documents.
Quality
A score system based on those previously determined for medical site evaluation and on the Health On The Net foundation code principles was used.1,7,8,15,16 Necessary modifications were made in applying the previously tested criteria for the purpose of this specific disease.
The sites were scored on ownership (statement of provider, or any form of support), purpose (education, profits), authorship (author’s name), author qualification, attribution (references and sources citation), interactivity, and currency (first posting date and subsequent revision dates). Each of these characteristics was further divided into other subheadings (table 1), which then received a score of 0, 1, or 2, depending on the amount of information provided, for a total score of 13.
Table 1 Quality component scoring system
Technical
We rated each site as to its provision of background information about ROP: definition, causes, epidemiology, risk factors, diagnosis, classification, treatment, and prognosis and a score of 0, 1, or 2 was assigned, for a total score of 16 (table 2).
Table 2 Technical component score system
Sites were classified into three groups8: (1) academic, defined as those providing information with or without an educational institution affiliation and not for purposes of profit; (2) organisational, sites created by an association to provide information and not for purposes of profit; and (3) commercial, sites providing information for profit either through direct sales or indirectly through commercial sponsorship. The same investigator (ENM) performed all evaluations. Website classification was performed only after quality, technical, and readability data were acquired.
Statistical analysis
Scores for each site were converted to a percentage value of the maximum range. Based on the overall percentage score, a label was assigned to each site as described previously8: excellent (80%), very good (70% to 79%), good (60% to 69%), fair (50% to 59%), and poor (49%).
One way ANOVA was used to compare percentage scores. Spearman correlation was used to analyse readability, quality, and technical scores. A p value of <0.05 was considered significant.
RESULTS
For each heading search, a range of 73 to 151 101 sites was listed. The first 30 sites were reviewed (240 sites), but because the searches sometimes discovered the same sites, only 114 unique sites were retrieved. A total of 74 (64.9%) sites were excluded because the information contained was not related to ROP (43 websites), the information was aimed at healthcare professionals (24 sites), the site provided only a short abstract of scientific papers about ROP (three sites), the site only gave links to other sites (two sites), or to have access to the page a fee or registration was required (two sites). Of the 40 sites evaluated, 10 (25.0%) were classified as academic, eight (20.0%) as organisational, and 22 (55.0%) as commercial.
The mean overall percentage score was 59.1% (SD 17.4%). The mean quality percentage score was 54.8% (20.2%) and the mean technical percentage score was 62.6% (21.3%). The quality, technical, and overall percentage scores did not differ among the website groups (one way ANOVA; p = 0.664, p = 0.129, p = 0.185, respectively). The proportion of each grading per criteria is presented in table 3.
Table 3 Quality and technical scores (2 is the best score) for ROP information on 40 websites
According to the overall percentage score, eight sites (20.0%) were considered excellent, three (7.5%) very good, four (10.0%) good, 10 (25.0%) fair, and 15 (37.5%) were poor. The rankings for sites in each of the three categories are provided in table 4.
Table 4 Relative ranking of the three categories of ROP websites
Clarity of the text was very variable between sites and most sites (95.0%) required a high "reading level." The average Flesch-Kincaid grade level was 10.83 (or 10th grade, eighth month) with a standard deviation of plus or minus 1.33; 17 sites (42.5%) scored at the maximum readability, 12.0.
The quality score was correlated with the technical score (r = 0.4, p = 0.01). There was no correlation between the quality score and the readability level (r = 0.192, p = 0.236). There was a significant correlation between the technical score and the Flesch-Kincaid grade level (r = 0.372, p = 0.018), demonstrating that the sites providing better information related to ROP presented this using a more complex language.
DISCUSSION
We assessed the general quality, technical content, and readability of ROP information available on the internet. Although website evaluations were conducted by the same investigator, we consider that subjectivity was minimised by using well defined quantitative standard scores.
Overall, the majority of ROP sites were of poor or fair quality (62.5%). Similar results have been reported for health information websites addressing other diseases.1–3,5,6,11,17 Sites produced by organisations were better designed, with 62.5% of them presenting excellent, very good, or good quality information.
We observed a great variation in scores among the 40 analysed sites. In the general quality evaluation, a common flaw was the inadequate evidence of currency (95.0%), attribution (80.0%), and authorship (62.5%). These are important principles to ensure quality and reliability of site content.6,18
In the technical component of our evaluation, the most commonly omitted aspect was disease classification (50.0%). We consider that mentioning the classification helps parents to better understand the possible progression of the retinopathy to more complicated stages, often signalling poorer prognosis, highlighting the importance of the prompt treatment and adequate follow up. Another troubling finding is that 75.0% of the sites did not mention or adequately discuss the epidemiology of ROP or how the disease is detected (70.0%). These topics, in our opinion, should also receive special attention from website authors, as they can be important not only for the parents but also for the general physicians involved with premature babies.
The appropriateness of the language used in the website is also important. It is estimated that 20% of the adult population in the United States have low literacy skills or read below or at the fifth grade level.19 To ensure that the information is at an appropriate level for the general public the Flesch-Kincaid grade level ideally should be at the eighth level or below. In our study, however, 95% of the sites required a high reading level, with the sites presenting better ROP information using a more complex language.
Although an abundance of ROP information is available on the internet for patients and families, this study highlighted the poor quality of this information. Additionally, most patients who use this source of information do not have the background for evaluating the quality of the provided material. Therefore, it is important that ophthalmologists help to develop good quality websites and to ensure that their patients are directed to sites that provide accurate information.
Preparing and providing a list of "approved" sites for patients, or a list of "tips" on how to evaluate the general quality of a site (for example, currency, authorship), or even asking the patient about information they found on the internet and possible questions about it are means to guarantee that your patient will be benefit from the internet resources.
REFERENCES
Sandvik H. Health information and interaction on the internet: a survey of female urinary incontinence. BMJ 1999;319:29–32.
Griffiths K, Christensen H. Quality of web based information on treatment of depression: cross sectional survey. BMJ 2000;321:1511–15.
Li L, Irvin E, Guzman J, et al. Surfing for back pain patients. Spine 2001;26:545–57.
Rigby M, Forsstrom J, Roberts R, et al. Verifying quality and safety in health informatics services. BMJ 2001;323:552–6.
Croft D, Peterson M. An evaluation of the quality and contents of asthma education on the world wide web. Chest 2002;121:1301–7.
Reed M, Anderson C. Evaluation of patient information Internet websites about menopause and hormone replacement therapy. Maturitas 2002;43:135–54.
Butler L, Foster N. A cross-sectional survey of the quality of web-based information on low back pain. Spine 2003;28:395–401.
Bohacek L, Gomez M, Fish J. An evaluation of internet sites for burn scar management. J Burn Care Rehabil 2003;24:246–51.
Madan A, Frantzides C, Pesce C. The quality of information about laparoscopic bariatric surgery on the internet. Surg Endosc 2003;17:685–7.
Curro V, Buonuomo P, De Rose P, et al. The evolution of web-based medical information on sore throat: a longitudinal study. J Med Internet Res 2003;5:e10.
Kisely S, Ong G, Takyar A. A survey of the quality of web based information on the treatment of schizophrenia and attention deficit hyperactivity disorder. Aust N Z J Psychiatry 2003;37:85–91.
Oermann M, Lowery N, Thornley J. Evaluation of websites on management of pain in children. Pain Manag Nurs 2003;4:99–105.
Murphy R, Frost S, Webster P, et al. An evaluation of web-based information. Int Eat Disord 2004;35:145–54.
Ohio State University. Revising and editing with Word: spelling, grammar, and style. Available at ccl.english.ohio-state.edu/handouts/miscellaneous/word_revision_spelling_grammar_pc. htm, accessed 12 April 2004.
Silberg W, Lundberg G, Musacchio R. Assessing, controlling, and assuring the quality of medical information on the internet. JAMA 1997;277:1244–5.
Health on the net foundation code of conduct for medical and health web sites. Available at www.hon.ch/HONcode/Conduct.html, accessed 12 April 2004.
McClung H, Murray R, Heitlinger L. The internet as a source for current patient information. Pediatrics 1998;101:E2.
Kim P, Eng T, Deering M, et al. Published criteria for evaluating health related web sites: review. BMJ 1999;318:647–9.
Doak C, Doak L, Friedell G, et al. Improving comprehension for cancer patients with low literacy skills: strategies for clinicians. CA Cancer J Clin 1998;48:151–62.(E N Martins1,2 and L S Mo)
2 Department of Ophthalmology, University of California-Davis, Sacramento, CA, USA
Correspondence to:
Lawrence S Morse
MD, PhD, Department of Ophthalmology, University of California-Davis, 4860 Y Street, Suite 2400, Sacramento, CA 95817, USA; lsmorse@ucdavis.edu
Accepted for publication 6 October 2004
ABSTRACT
Background/aims: The success of the treatment in patients with retinopathy of prematurity (ROP) is mainly associated with timely diagnosis and appropriate management. Information dissemination is crucial for the outcome of ROP. This study aimed to evaluate the quality of the information about ROP available for patients on the internet.
Methods: Cross sectional study. In March 2004 the ROP information available on the internet was evaluated using two search engines (MetaCrawler and MSN) and four key terms ("retinopathy of prematurity," "premature eye," "premature retina," and "ROP"). The quality of each website was evaluated using a score system. The sites were classified as academic, organisational, or commercial. Readability, general quality of the website (based on ownership, purpose, authorship, author qualification, attribution, interactivity, and currency), and quality of the content specific to ROP (definition, causes, epidemiology, risk factors, diagnosis, classification, treatment, and prognosis) were analysed.
Results: Of 114 websites evaluated, 40 were included. 10 sites (25.0%) were academic, eight (20.0%) organisational, and 22 (55.0%) commercial. In the majority of the sites (62.5%) the ROP information was fair or poor.
Conclusions: A large amount of information about ROP is available on the internet. However, most websites were considered incomplete.
Abbreviations: ROP, retinopathy of prematurity
Keywords: internet; retinopathy of prematurity
Because of its accessibility and speed, the internet has become a popular mean of obtaining information. The use of understandable language and visual effects is frequent, making websites very attractive to patients and, therefore, an option for the improvement of medical information accessibility.
However, the large volume of information available on the world wide web makes it difficult for patients to judge the accuracy of the data provided. In an attempt to evaluate the appropriateness of health information provided on the internet, studies have been conducted.1,2,3,4,5,6,7,8,9,10,11,12,13
Retinopathy of prematurity (ROP) is a disease that affects vasculature development in the eyes of premature infants and presents five progressive clinical stages, being a major cause of poor vision and blindness in premature infants. Its prognosis is strongly related to timely diagnosis and appropriate treatment.
The purpose of this study was to evaluate the quality of ROP information presented on the internet.
MATERIALS AND METHODS
Site selection
In March 2004 we used two search engines, MSN (www.msn.com) and MetaCrawler (www.metacrawler.com) to select the sites, as previously reported,2 using the keywords "retinopathy of prematurity," "ROP," "premature eye," and "premature retina." MetaCrawler integrates the results for a query from different well known search engines including Google, Alta Vista, Ask Jeeves, About, Looksmart, Teoma (DirectHit), Overture, Find What, and Yahoo, providing an accurate approximation to a list of the most popular ROP sites that would be encountered by a typical user. The keywords were arbitrarily chosen to represent likely possibilities used by patients during an internet search.
The studied sites were chosen by screening the first 30 sites retrieved under each keyword. Sites charging service fees or in a language other than English, mailing lists, bulletin boards, discussion groups, and sites explicitly stating healthcare professionals or students as their target audience were not included.
Site evaluation
Site evaluation was divided into readability, quality, and technical.
Readability
To determine the site readability, we copied the ROP material from each site, pasted it into a Microsoft Word document (Microsoft Office, 2000), and obtained the Flesch-Kincaid grade level score. This score rates text on US school grade levels or years (range third to 12th grade) and it is based on the average of sentence length and number of syllables per word.14 A score of 8.0 or less is the recommended level for standard documents.
Quality
A score system based on those previously determined for medical site evaluation and on the Health On The Net foundation code principles was used.1,7,8,15,16 Necessary modifications were made in applying the previously tested criteria for the purpose of this specific disease.
The sites were scored on ownership (statement of provider, or any form of support), purpose (education, profits), authorship (author’s name), author qualification, attribution (references and sources citation), interactivity, and currency (first posting date and subsequent revision dates). Each of these characteristics was further divided into other subheadings (table 1), which then received a score of 0, 1, or 2, depending on the amount of information provided, for a total score of 13.
Table 1 Quality component scoring system
Technical
We rated each site as to its provision of background information about ROP: definition, causes, epidemiology, risk factors, diagnosis, classification, treatment, and prognosis and a score of 0, 1, or 2 was assigned, for a total score of 16 (table 2).
Table 2 Technical component score system
Sites were classified into three groups8: (1) academic, defined as those providing information with or without an educational institution affiliation and not for purposes of profit; (2) organisational, sites created by an association to provide information and not for purposes of profit; and (3) commercial, sites providing information for profit either through direct sales or indirectly through commercial sponsorship. The same investigator (ENM) performed all evaluations. Website classification was performed only after quality, technical, and readability data were acquired.
Statistical analysis
Scores for each site were converted to a percentage value of the maximum range. Based on the overall percentage score, a label was assigned to each site as described previously8: excellent (80%), very good (70% to 79%), good (60% to 69%), fair (50% to 59%), and poor (49%).
One way ANOVA was used to compare percentage scores. Spearman correlation was used to analyse readability, quality, and technical scores. A p value of <0.05 was considered significant.
RESULTS
For each heading search, a range of 73 to 151 101 sites was listed. The first 30 sites were reviewed (240 sites), but because the searches sometimes discovered the same sites, only 114 unique sites were retrieved. A total of 74 (64.9%) sites were excluded because the information contained was not related to ROP (43 websites), the information was aimed at healthcare professionals (24 sites), the site provided only a short abstract of scientific papers about ROP (three sites), the site only gave links to other sites (two sites), or to have access to the page a fee or registration was required (two sites). Of the 40 sites evaluated, 10 (25.0%) were classified as academic, eight (20.0%) as organisational, and 22 (55.0%) as commercial.
The mean overall percentage score was 59.1% (SD 17.4%). The mean quality percentage score was 54.8% (20.2%) and the mean technical percentage score was 62.6% (21.3%). The quality, technical, and overall percentage scores did not differ among the website groups (one way ANOVA; p = 0.664, p = 0.129, p = 0.185, respectively). The proportion of each grading per criteria is presented in table 3.
Table 3 Quality and technical scores (2 is the best score) for ROP information on 40 websites
According to the overall percentage score, eight sites (20.0%) were considered excellent, three (7.5%) very good, four (10.0%) good, 10 (25.0%) fair, and 15 (37.5%) were poor. The rankings for sites in each of the three categories are provided in table 4.
Table 4 Relative ranking of the three categories of ROP websites
Clarity of the text was very variable between sites and most sites (95.0%) required a high "reading level." The average Flesch-Kincaid grade level was 10.83 (or 10th grade, eighth month) with a standard deviation of plus or minus 1.33; 17 sites (42.5%) scored at the maximum readability, 12.0.
The quality score was correlated with the technical score (r = 0.4, p = 0.01). There was no correlation between the quality score and the readability level (r = 0.192, p = 0.236). There was a significant correlation between the technical score and the Flesch-Kincaid grade level (r = 0.372, p = 0.018), demonstrating that the sites providing better information related to ROP presented this using a more complex language.
DISCUSSION
We assessed the general quality, technical content, and readability of ROP information available on the internet. Although website evaluations were conducted by the same investigator, we consider that subjectivity was minimised by using well defined quantitative standard scores.
Overall, the majority of ROP sites were of poor or fair quality (62.5%). Similar results have been reported for health information websites addressing other diseases.1–3,5,6,11,17 Sites produced by organisations were better designed, with 62.5% of them presenting excellent, very good, or good quality information.
We observed a great variation in scores among the 40 analysed sites. In the general quality evaluation, a common flaw was the inadequate evidence of currency (95.0%), attribution (80.0%), and authorship (62.5%). These are important principles to ensure quality and reliability of site content.6,18
In the technical component of our evaluation, the most commonly omitted aspect was disease classification (50.0%). We consider that mentioning the classification helps parents to better understand the possible progression of the retinopathy to more complicated stages, often signalling poorer prognosis, highlighting the importance of the prompt treatment and adequate follow up. Another troubling finding is that 75.0% of the sites did not mention or adequately discuss the epidemiology of ROP or how the disease is detected (70.0%). These topics, in our opinion, should also receive special attention from website authors, as they can be important not only for the parents but also for the general physicians involved with premature babies.
The appropriateness of the language used in the website is also important. It is estimated that 20% of the adult population in the United States have low literacy skills or read below or at the fifth grade level.19 To ensure that the information is at an appropriate level for the general public the Flesch-Kincaid grade level ideally should be at the eighth level or below. In our study, however, 95% of the sites required a high reading level, with the sites presenting better ROP information using a more complex language.
Although an abundance of ROP information is available on the internet for patients and families, this study highlighted the poor quality of this information. Additionally, most patients who use this source of information do not have the background for evaluating the quality of the provided material. Therefore, it is important that ophthalmologists help to develop good quality websites and to ensure that their patients are directed to sites that provide accurate information.
Preparing and providing a list of "approved" sites for patients, or a list of "tips" on how to evaluate the general quality of a site (for example, currency, authorship), or even asking the patient about information they found on the internet and possible questions about it are means to guarantee that your patient will be benefit from the internet resources.
REFERENCES
Sandvik H. Health information and interaction on the internet: a survey of female urinary incontinence. BMJ 1999;319:29–32.
Griffiths K, Christensen H. Quality of web based information on treatment of depression: cross sectional survey. BMJ 2000;321:1511–15.
Li L, Irvin E, Guzman J, et al. Surfing for back pain patients. Spine 2001;26:545–57.
Rigby M, Forsstrom J, Roberts R, et al. Verifying quality and safety in health informatics services. BMJ 2001;323:552–6.
Croft D, Peterson M. An evaluation of the quality and contents of asthma education on the world wide web. Chest 2002;121:1301–7.
Reed M, Anderson C. Evaluation of patient information Internet websites about menopause and hormone replacement therapy. Maturitas 2002;43:135–54.
Butler L, Foster N. A cross-sectional survey of the quality of web-based information on low back pain. Spine 2003;28:395–401.
Bohacek L, Gomez M, Fish J. An evaluation of internet sites for burn scar management. J Burn Care Rehabil 2003;24:246–51.
Madan A, Frantzides C, Pesce C. The quality of information about laparoscopic bariatric surgery on the internet. Surg Endosc 2003;17:685–7.
Curro V, Buonuomo P, De Rose P, et al. The evolution of web-based medical information on sore throat: a longitudinal study. J Med Internet Res 2003;5:e10.
Kisely S, Ong G, Takyar A. A survey of the quality of web based information on the treatment of schizophrenia and attention deficit hyperactivity disorder. Aust N Z J Psychiatry 2003;37:85–91.
Oermann M, Lowery N, Thornley J. Evaluation of websites on management of pain in children. Pain Manag Nurs 2003;4:99–105.
Murphy R, Frost S, Webster P, et al. An evaluation of web-based information. Int Eat Disord 2004;35:145–54.
Ohio State University. Revising and editing with Word: spelling, grammar, and style. Available at ccl.english.ohio-state.edu/handouts/miscellaneous/word_revision_spelling_grammar_pc. htm, accessed 12 April 2004.
Silberg W, Lundberg G, Musacchio R. Assessing, controlling, and assuring the quality of medical information on the internet. JAMA 1997;277:1244–5.
Health on the net foundation code of conduct for medical and health web sites. Available at www.hon.ch/HONcode/Conduct.html, accessed 12 April 2004.
McClung H, Murray R, Heitlinger L. The internet as a source for current patient information. Pediatrics 1998;101:E2.
Kim P, Eng T, Deering M, et al. Published criteria for evaluating health related web sites: review. BMJ 1999;318:647–9.
Doak C, Doak L, Friedell G, et al. Improving comprehension for cancer patients with low literacy skills: strategies for clinicians. CA Cancer J Clin 1998;48:151–62.(E N Martins1,2 and L S Mo)