Beating Asthma: A Community-Based Asthma Education Initiative
http://www.100md.com
《美国整骨期刊》
the Department of Preventive Medicine at Edward Via Virginia College of Osteopathic Medicine in Blacksburg (Scariati); Transactional Records Access Clearinghouse at Syracuse University in NY (Roberge); and the Department of Community and Preventive Medicine at University of Rochester in Minn (Dye).
Beating Asthma is a community-based educational intervention designed to empower people with asthma by providing them with information about their pulmonary disease. The project consists of a series of three lectures delivered in a single 2-hour evening program by a pulmonologist, a health educator, and a licensed clinical psychologist. Surveys were distributed to participants before and after the program to assess general knowledge of asthma, the disease's pathophysiology, and asthma-management skills. Seventy-eight (77%) of the 101 families participating in the event completed both questionnaires. Regression model analysis of survey results showed that participants with the lowest scores before intervention achieved the greatest gains after intervention (P<.001). Analysis of the three topic areas revealed that only the subscore for disease management differed significantly from zero (P<.001). The authors assess the quality and usefulness of the survey instrument and the lecture content for future use.
Asthma is a chronic respiratory disease characterized by episodes of inflammation and narrowing of small airways in response to asthma triggers. Asthma attacks can vary from mild to life threatening and are characterized by shortness of breath, coughing, wheezing, chest pain or tightness, or a combination of these symptoms. Many factors can trigger an asthma attack, including allergens, infections, exercise, abrupt changes in the weather, or exposure to airway irritants.
The burden of asthma in the United States has increased over the past 2 decades.1 Asthma currently afflicts more than 20 million people, including an estimated 6.2 million children aged under 18 years, and causes approximately 4000 deaths annually.1–7 Estimates for the direct annual cost of asthma in the United States range from $6 billion to $11.5 billion annually.7–9 Indirect costs, which include lost productivity and prescription medication, increase this amount by $4.6 billion.7
Because of asthma-related illness, children miss more than 12.8 million school days annually,7 while adults miss 24.5 million workdays every year.7–11 In the United States, the indirect cost of lost productivity as a result of asthma has been estimated at more than $700 million.8–10 In fact, lost productivity as a result of death (recently estimated at $1.7 billion) was the largest single indirect cost shown.11
Previous researchers9,10 have documented a rapid increase in the national cost for asthma-related care over the past 2 decades. The public health burden of asthma is clear, and affecting change remains a challenge.
In 2002, the National Asthma Education and Prevention Program of the National Heart, Lung, and Blood Institute (NHLBI) in Bethesda, Md, updated selected topics of its Guidelines for the Diagnosis and Management of Asthma.12,13 As in the first two editions of these guidelines, the 2002 expert panel placed an emphasis on patient education and prevention—a reflection of the association between asthma outcomes and the commitment a patient brings to appropriate, day-to-day self-care. The Beating Asthma program, which was designed to raise awareness about asthma in an interactive, support group–focused educational forum, can be used as part of a patient-centered response to the NHLBI recommendations.
Methods
The Beating Asthma project, which took place in Syracuse, NY, in 1997, received approval from the institutional review board at Edward Via Virginia College of Osteopathic Medicine in Blacksburg.
Syracuse is located in the geographic center of New York State. With a population of approximately 460,8000, it is the largest metropolitan area in Onondaga County. Although asthma rates are not routinely monitored in Onondaga County, 102 children, aged up to 4 years, were hospitalized with asthma in 2000 (3.41 per 1000 population).
Beating Asthma was an educational evening program focused on empowering people with asthma with practical information about the mechanics and the management of their disease. This community-based event was held from 6:30 PM to 8:30 PM on a weekday evening in the meeting hall of a centrally located shopping mall. It was sponsored by the Syracuse Quality Initiative Group (SQIG), an alliance of health maintenance organizations, hospitals, and other interested parties working together to streamline processes common to healthcare organizations operating in a single market (eg, credentialing, preventive medical services).
A multifaceted approach was used to recruit participants for the Beating Asthma program. Six weeks before the event, a mailing was sent to select primary care physicians and relevant specialists (ie, allergists; ear, nose, and throat specialists; obstetric physicians/gynecologists; and pulmonologists) announcing the program and requesting that they inform their patients diagnosed with asthma of the event. Four to five weeks later, a second mailing was sent to primary care physicians only. Each managed care organization participating in the SQIG also sent a personalized letter of invitation to its patients diagnosed with asthma (identified using agreed-upon ICD-9 [International Classification of Diseases, Ninth Revision] codes). Investigators also displayed posters in popular community locations, sent a press release to the local media, and contracted with several local radio stations to air a commercial announcement of the event. The cost to participate in the program was nominal—$5.00 per family for advance registrants, or $7.00 per family if they registered at the event.
Each family group was checked in on arrival and asked to complete an anonymous, voluntary one-page questionnaire before the start of the program. One side of the questionnaire consisted of 16 true/false questions on asthma drawn from educational materials created by NHLBI and the now-defunct PhyLine Practice Management, LLC, as well as speakers' self-determined educational objectives (Figure, part 1). The reverse side of the questionnaire requested patients' demographic, diagnostic, and healthcare-related information (Figure, part 2). Because all questionnaires were completed anonymously, pretest and posttest data were linked by a series of matching numbers. Each family was assigned a sequential identification number when they registered for the event (eg, 1, 2, 3, 4). Families who completed the pretest were invited to enter a raffle to win tickets to a popular local hockey event.
The educational intervention consisted of three lectures provided during the 2-hour program. A pulmonologist, health educator, and licensed clinical psychologist presented material to program attendees. The pulmonologist and health educator lectured regarding general knowledge of the disease, pathophysiology of asthma, and disease-management skills. The licensed clinical psychologist delivered a lecture on the psychosocial aspects of asthma. An asthmatic athlete from nearby Syracuse University also spoke about the management of her disease. An open question-and-answer session between the experts and the audience took place at the end of the program.
At the end of the lecture series, each family was asked to complete a posttest questionnaire composed of the same 16 true/false questions found on the pretest. In return for completing the posttest, the family received an extensive packet of information, a peak flow meter, a stress ball, and the correct answers to the true/false questions supplemented with explanations (Figure, part 1).
Information was entered and preliminary analyses were performed in a public domain epidemiologic analysis program (Epi Info, Version 6.01; Centers for Disease Control and Prevention, Atlanta, Ga). The data were exported to SAS statistical software (version 6.12 for Windows; SAS Institute Inc, Cary, NC) for further analysis.
Results
Eighty-six families preregistered for the event and 40 families registered at the door on the night of the event. Of the 86 preregistered families, 61 attended the event, for a total of 101 participating families. Among these families, 97 (96%) completed a pretest and 78 (77%) completed the posttest.
Program participants were separated into two subgroups: those who completed the pretest only (Group A, n=97) and those who completed the pretest and the posttest (Group B, n=78).
The two study groups were then compared on the basis of demographic data, date of diagnosis, illness severity (as measured by hospitalizations and emergency room visits within the previous 2 years), and pretest scores and subscores. There were no statistically significant differences between the two study groups in these comparisons.
Answers from Group B pretests were linked to their posttest results, allowing for more thorough data analysis. Total test differences were analyzed, as were question subgroups and individual questions. A "difference score" was computed for families in this group by subtracting pretest scores from posttest scores. Similarly, three subscores were computed for areas of general knowledge (questions 3, 5, 10, 11, 13), pathophysiology (questions 1, 12, 14, 16), and disease management (questions 2, 4, 7, 8, 9, 15). Only one question was posed to participants regarding the psychosocial aspects of asthma (question 6), so this area was not analyzed as a separate subgroup.
The average pretest score for Group B participants was 69.96%, and the average posttest score was 75.10%. Neither pretest nor posttest scores were normally distributed, so meaningful confidence intervals to detect true differences in means could not be constructed. Instead, as noted, difference scores were calculated for families in Group B by subtracting pretest results from posttest results. Difference scores were normally distributed with a mean of 5.14 (95% confidence interval, 2.31±7.97, P<.001). Multiple regression analysis was used to test for the effects of age, sex, date of diagnosis, and illness severity on the posttest score. The model was estimated using a transformation of the posttest score to its square to correct for model nonlinearity. None of the variables were significant. Simple regression, again with a transformation of the response variable to remove nonlinearity, was used to test the effect of pretest score on posttest score. This model was significant (P<.001) with the pretest score explaining less than 20% of the variation in the square of the posttest score (adjusted R2=0.1988). This model suggests that participants with the lowest pretest scores achieved the greatest gains on the posttest. Our analysis of subsection scores by topic area (ie, pathophysiology, general knowledge, disease management) revealed that only the subscore for disease-management questions differed significantly from zero (P<.001).
Individual questions were categorized into one of the four following subgroups based on respondents' answers on the pretest and posttest:
answered correctly on both the pretest and posttest;
answered incorrectly on the pretest, but correctly on the posttest;
answered correctly on the pretest, but incorrectly on the posttest; or
answered incorrectly on both the pretest and posttest.
For each answer provided by respondents that placed them in the first subgroup (ie, item answered correctly on both tests), the answer seemed to be based on information participants had known previously and on which the educational intervention had no impact. Furthermore, for each answer provided by respondents that placed them in the fourth subgroup (ie, item answered incorrectly on both tests), the information covered in that question seemed to focus on material that was either not covered or was poorly covered during intervention and for which our training had no impact.
It is tempting to assume that questions categorized in the second group (ie, item answered incorrectly on the pretest, correctly on the posttest) indicate effective presentation of course materials, but the number of people who got a question right on the pretest but wrong on the posttest must also be considered. If it is assumed that misinformation was not disseminated during the presentations, then this inconsistency might be explainable by participants guessing the correct answer during the pretest. To correct for guessing in the pretest, a correction factor (right::wrong - wrong::right) was applied to our calculations.14–16
Among the 16 questions, seven were answered correctly by more than 75% of respondents on both pretest and posttest. Of the questions following this response pattern, two questions tested general knowledge (questions 3 and 5), three tested knowledge of pathophysiology (questions 1, 12, 14), and two tested knowledge of disease management (questions 2 and 15).
Conversely, three questions were notable because twice as many respondents answered them incorrectly on both the pretest and the posttest. Two were general knowledge questions (questions 10 and 13) and one was a disease-management question (question 8).
After applying the correction factor for guessing, four disease-management questions (questions 2, 4, 7, 9) are notable for having the greatest number of respondents making posttest gains. We were pleasantly surprised that 80% of respondents knew the correct answer to question 2 before the Beating Asthma educational intervention. We were disappointed, however, that respondents did not exhibit better understanding of the material presented as addressed in question 4; slightly more than 23% answered question 4 incorrectly in both tests.
The results for six questions were remarkable in that they had negative scores after application of the correction factor. Among these six questions, two (questions 8 and 13) were also among the three questions that received the most incorrect responses on both tests. Another two (questions 14 and 12) were grouped with those questions that had the greatest number of respondents answering correctly on both tests. Of the six questions with negative scores after correction, three were related to pathophysiology (questions 12, 14, 16); two, to general knowledge (questions 11 and 13); and one, to disease management (question 8).
Our application of a correction factor for guessing allowed us to remove guessing as an explanation for respondents selecting the correct answer. Negative scores on these questions (ie, after the correction factor was applied) implied that the correct answer was obtained by chance, not because respondents had developed new knowledge as a result of our educational intervention. These results require that we consider whether respondents truly gained knowledge in those particular topic areas.
Conclusions
We found a statistically significant improvement between pretest and posttest asthma knowledge scores for families participating in the one-night Beating Asthma program and completing both tests. Participants with the lowest pretest scores achieved the greatest gains on the posttest.
In addition, our survey instrument allowed us to assess the quality of the lecture content. Of the three main topic areas included in this educational intervention (ie, pathophysiology, general knowledge, disease management), only the subscore for disease management demonstrated a significant improvement. Anecdotally, this result was consistent with the better-defined objectives of the presenter on disease management when compared with the other speakers at the event. We believe this factor translated into more clearly defined questions on the survey instrument.
We were able to use the survey instrument to critique itself. By looking at the pattern of pretest/posttest answers for each question, we were able to identify questions that were too basic (ie, those in the general knowledge subgroup), those that were poorly covered or not covered during the one-night educational intervention, and those that appeared to have value in measuring the impact of the intervention. This self-evaluation has allowed investigators to redesign the survey instrument for use in future asthma-intervention programs.
The survey instrument was a powerful educational tool, capable of sensitizing event participants to the issues that would be addressed in the lecture series. When completing the posttest, a number of participants made notes on the questionnaire about why a given statement was true or false based on information presented within the evening's lectures.
Finally, the survey instrument offered added value by allowing participating organizations to document their efforts in meeting criteria for reporting preventive care and health promotion activities (ie, number of patient-members participating in a health promotion/education programming) as established by the National Committee for Quality Assurance in Washington, DC.17
The lessons learned from the Beating Asthma project provide insights to researchers contemplating the creation of educational interventions for patients with asthma. We determined, for example, that, if we were to plan a second Beating Asthma event, the effectiveness of the program would be greatly increased by having speaker objectives and the survey instrument drafted by event organizers a priori to ensure continuity and quality in the intervention's content. In addition, we propose the use of an alternate system of personal identifiers on the survey instruments that would allow researchers to link the gathered data to patient medical records, thereby enabling researchers to conduct longitudinal follow-up studies.
Surveillance of the Beating Asthma educational program provided valuable information for assessing the efficacy of the educational intervention offered to community patients diagnosed with asthma. Although program coordination for Beating Asthma was somewhat cumbersome and time consuming for event planners, the program content was uniform; cost, resources, and efforts were not duplicated; surveillance was comparable; and the chance of working with a small sample size was minimized. We found great benefit in obtaining program sponsorship through the SQIG, an amalgamation of healthcare organizations.
Many insurance providers hesitate to provide education-based preventive services because they cannot demonstrate cost-benefit, especially in an health insurance climate in which patients—through their employers—change carriers on a regular basis (ie, approximately every 5 years),18 thereby creating the perception in carriers that any money committed to prevention efforts is "lost." If all the health insurance carriers in one geographic area are providing the same preventive services, however, the value of preventive care is maintained even if the patient changes carriers.
Beating Asthma was a successful collaborative effort between multiple health entities focused on improving asthma awareness and education at a community level. The lessons learned from this project will inform the development of future endeavors.
References
8. Weiss KB, Gergen PJ, Hodgson TA. An economic evaluation of asthma in the United States. N Engl J Med.1992; 326:862 -866.
9. Weiss KB, Sullivan SD, Lyttle CS. Trends in the cost of illness for asthma in the United States, 1985–1994. J Allergy Clin Immunol. 2000;106:493 -499.
10. Redd SC. Asthma in the United States: burden and current theories [review]. Environ Health Perspect.2002; 110(Suppl 4):557 -560.
12. Highlights of the Expert Panel Report 2: Guidelines for the diagnosis and management of asthma. Bethesda, Md: National Asthma Education and Prevention Program, National Heart, Lung, and Blood Institute; 1997. Publication 97-4051A.
13. National Asthma Education and Prevention Program. Expert Panel Report: Guidelines for the Diagnosis and Management of Asthma: Update on Selected Topics–2002 [published correction appears in J Allergy Clin Immunol. 2003;111:466]. J Allergy Clin Immunol.2002; 110(5 Suppl):S141 -S219.
14. Bereby-Meyer Y, Meyer J, Flascher OM. Prospect theory analysis of guessing in multiple choice tests. J Behav Decision Making. 2002;15:313 -327.
15. Vanhuele M, Drèze X. Measuring the price knowledge shoppers bring to the store. J Marketing. October2002; 66:72 -85.
16. Chan N, Kennedy PE. Are multiple-choice exams easier for economics students A comparison of multiple-choice and "equivalent" constructed-response exam questions. South Econ J.2002; 68:957 -972.
18. Survey Shows Private Health Insurance Premiums Rose 11.2% in 2004 [press release]. Washington DC: Henry J Kaiser Foundation/Health Research and Educational Trust; September 9, 2004. Available at: http://www.kff.org/insurance/chcm090904nr.cfm. Accessed December 28, 2005.(Paula D. Scariati, DO, MP)
Beating Asthma is a community-based educational intervention designed to empower people with asthma by providing them with information about their pulmonary disease. The project consists of a series of three lectures delivered in a single 2-hour evening program by a pulmonologist, a health educator, and a licensed clinical psychologist. Surveys were distributed to participants before and after the program to assess general knowledge of asthma, the disease's pathophysiology, and asthma-management skills. Seventy-eight (77%) of the 101 families participating in the event completed both questionnaires. Regression model analysis of survey results showed that participants with the lowest scores before intervention achieved the greatest gains after intervention (P<.001). Analysis of the three topic areas revealed that only the subscore for disease management differed significantly from zero (P<.001). The authors assess the quality and usefulness of the survey instrument and the lecture content for future use.
Asthma is a chronic respiratory disease characterized by episodes of inflammation and narrowing of small airways in response to asthma triggers. Asthma attacks can vary from mild to life threatening and are characterized by shortness of breath, coughing, wheezing, chest pain or tightness, or a combination of these symptoms. Many factors can trigger an asthma attack, including allergens, infections, exercise, abrupt changes in the weather, or exposure to airway irritants.
The burden of asthma in the United States has increased over the past 2 decades.1 Asthma currently afflicts more than 20 million people, including an estimated 6.2 million children aged under 18 years, and causes approximately 4000 deaths annually.1–7 Estimates for the direct annual cost of asthma in the United States range from $6 billion to $11.5 billion annually.7–9 Indirect costs, which include lost productivity and prescription medication, increase this amount by $4.6 billion.7
Because of asthma-related illness, children miss more than 12.8 million school days annually,7 while adults miss 24.5 million workdays every year.7–11 In the United States, the indirect cost of lost productivity as a result of asthma has been estimated at more than $700 million.8–10 In fact, lost productivity as a result of death (recently estimated at $1.7 billion) was the largest single indirect cost shown.11
Previous researchers9,10 have documented a rapid increase in the national cost for asthma-related care over the past 2 decades. The public health burden of asthma is clear, and affecting change remains a challenge.
In 2002, the National Asthma Education and Prevention Program of the National Heart, Lung, and Blood Institute (NHLBI) in Bethesda, Md, updated selected topics of its Guidelines for the Diagnosis and Management of Asthma.12,13 As in the first two editions of these guidelines, the 2002 expert panel placed an emphasis on patient education and prevention—a reflection of the association between asthma outcomes and the commitment a patient brings to appropriate, day-to-day self-care. The Beating Asthma program, which was designed to raise awareness about asthma in an interactive, support group–focused educational forum, can be used as part of a patient-centered response to the NHLBI recommendations.
Methods
The Beating Asthma project, which took place in Syracuse, NY, in 1997, received approval from the institutional review board at Edward Via Virginia College of Osteopathic Medicine in Blacksburg.
Syracuse is located in the geographic center of New York State. With a population of approximately 460,8000, it is the largest metropolitan area in Onondaga County. Although asthma rates are not routinely monitored in Onondaga County, 102 children, aged up to 4 years, were hospitalized with asthma in 2000 (3.41 per 1000 population).
Beating Asthma was an educational evening program focused on empowering people with asthma with practical information about the mechanics and the management of their disease. This community-based event was held from 6:30 PM to 8:30 PM on a weekday evening in the meeting hall of a centrally located shopping mall. It was sponsored by the Syracuse Quality Initiative Group (SQIG), an alliance of health maintenance organizations, hospitals, and other interested parties working together to streamline processes common to healthcare organizations operating in a single market (eg, credentialing, preventive medical services).
A multifaceted approach was used to recruit participants for the Beating Asthma program. Six weeks before the event, a mailing was sent to select primary care physicians and relevant specialists (ie, allergists; ear, nose, and throat specialists; obstetric physicians/gynecologists; and pulmonologists) announcing the program and requesting that they inform their patients diagnosed with asthma of the event. Four to five weeks later, a second mailing was sent to primary care physicians only. Each managed care organization participating in the SQIG also sent a personalized letter of invitation to its patients diagnosed with asthma (identified using agreed-upon ICD-9 [International Classification of Diseases, Ninth Revision] codes). Investigators also displayed posters in popular community locations, sent a press release to the local media, and contracted with several local radio stations to air a commercial announcement of the event. The cost to participate in the program was nominal—$5.00 per family for advance registrants, or $7.00 per family if they registered at the event.
Each family group was checked in on arrival and asked to complete an anonymous, voluntary one-page questionnaire before the start of the program. One side of the questionnaire consisted of 16 true/false questions on asthma drawn from educational materials created by NHLBI and the now-defunct PhyLine Practice Management, LLC, as well as speakers' self-determined educational objectives (Figure, part 1). The reverse side of the questionnaire requested patients' demographic, diagnostic, and healthcare-related information (Figure, part 2). Because all questionnaires were completed anonymously, pretest and posttest data were linked by a series of matching numbers. Each family was assigned a sequential identification number when they registered for the event (eg, 1, 2, 3, 4). Families who completed the pretest were invited to enter a raffle to win tickets to a popular local hockey event.
The educational intervention consisted of three lectures provided during the 2-hour program. A pulmonologist, health educator, and licensed clinical psychologist presented material to program attendees. The pulmonologist and health educator lectured regarding general knowledge of the disease, pathophysiology of asthma, and disease-management skills. The licensed clinical psychologist delivered a lecture on the psychosocial aspects of asthma. An asthmatic athlete from nearby Syracuse University also spoke about the management of her disease. An open question-and-answer session between the experts and the audience took place at the end of the program.
At the end of the lecture series, each family was asked to complete a posttest questionnaire composed of the same 16 true/false questions found on the pretest. In return for completing the posttest, the family received an extensive packet of information, a peak flow meter, a stress ball, and the correct answers to the true/false questions supplemented with explanations (Figure, part 1).
Information was entered and preliminary analyses were performed in a public domain epidemiologic analysis program (Epi Info, Version 6.01; Centers for Disease Control and Prevention, Atlanta, Ga). The data were exported to SAS statistical software (version 6.12 for Windows; SAS Institute Inc, Cary, NC) for further analysis.
Results
Eighty-six families preregistered for the event and 40 families registered at the door on the night of the event. Of the 86 preregistered families, 61 attended the event, for a total of 101 participating families. Among these families, 97 (96%) completed a pretest and 78 (77%) completed the posttest.
Program participants were separated into two subgroups: those who completed the pretest only (Group A, n=97) and those who completed the pretest and the posttest (Group B, n=78).
The two study groups were then compared on the basis of demographic data, date of diagnosis, illness severity (as measured by hospitalizations and emergency room visits within the previous 2 years), and pretest scores and subscores. There were no statistically significant differences between the two study groups in these comparisons.
Answers from Group B pretests were linked to their posttest results, allowing for more thorough data analysis. Total test differences were analyzed, as were question subgroups and individual questions. A "difference score" was computed for families in this group by subtracting pretest scores from posttest scores. Similarly, three subscores were computed for areas of general knowledge (questions 3, 5, 10, 11, 13), pathophysiology (questions 1, 12, 14, 16), and disease management (questions 2, 4, 7, 8, 9, 15). Only one question was posed to participants regarding the psychosocial aspects of asthma (question 6), so this area was not analyzed as a separate subgroup.
The average pretest score for Group B participants was 69.96%, and the average posttest score was 75.10%. Neither pretest nor posttest scores were normally distributed, so meaningful confidence intervals to detect true differences in means could not be constructed. Instead, as noted, difference scores were calculated for families in Group B by subtracting pretest results from posttest results. Difference scores were normally distributed with a mean of 5.14 (95% confidence interval, 2.31±7.97, P<.001). Multiple regression analysis was used to test for the effects of age, sex, date of diagnosis, and illness severity on the posttest score. The model was estimated using a transformation of the posttest score to its square to correct for model nonlinearity. None of the variables were significant. Simple regression, again with a transformation of the response variable to remove nonlinearity, was used to test the effect of pretest score on posttest score. This model was significant (P<.001) with the pretest score explaining less than 20% of the variation in the square of the posttest score (adjusted R2=0.1988). This model suggests that participants with the lowest pretest scores achieved the greatest gains on the posttest. Our analysis of subsection scores by topic area (ie, pathophysiology, general knowledge, disease management) revealed that only the subscore for disease-management questions differed significantly from zero (P<.001).
Individual questions were categorized into one of the four following subgroups based on respondents' answers on the pretest and posttest:
answered correctly on both the pretest and posttest;
answered incorrectly on the pretest, but correctly on the posttest;
answered correctly on the pretest, but incorrectly on the posttest; or
answered incorrectly on both the pretest and posttest.
For each answer provided by respondents that placed them in the first subgroup (ie, item answered correctly on both tests), the answer seemed to be based on information participants had known previously and on which the educational intervention had no impact. Furthermore, for each answer provided by respondents that placed them in the fourth subgroup (ie, item answered incorrectly on both tests), the information covered in that question seemed to focus on material that was either not covered or was poorly covered during intervention and for which our training had no impact.
It is tempting to assume that questions categorized in the second group (ie, item answered incorrectly on the pretest, correctly on the posttest) indicate effective presentation of course materials, but the number of people who got a question right on the pretest but wrong on the posttest must also be considered. If it is assumed that misinformation was not disseminated during the presentations, then this inconsistency might be explainable by participants guessing the correct answer during the pretest. To correct for guessing in the pretest, a correction factor (right::wrong - wrong::right) was applied to our calculations.14–16
Among the 16 questions, seven were answered correctly by more than 75% of respondents on both pretest and posttest. Of the questions following this response pattern, two questions tested general knowledge (questions 3 and 5), three tested knowledge of pathophysiology (questions 1, 12, 14), and two tested knowledge of disease management (questions 2 and 15).
Conversely, three questions were notable because twice as many respondents answered them incorrectly on both the pretest and the posttest. Two were general knowledge questions (questions 10 and 13) and one was a disease-management question (question 8).
After applying the correction factor for guessing, four disease-management questions (questions 2, 4, 7, 9) are notable for having the greatest number of respondents making posttest gains. We were pleasantly surprised that 80% of respondents knew the correct answer to question 2 before the Beating Asthma educational intervention. We were disappointed, however, that respondents did not exhibit better understanding of the material presented as addressed in question 4; slightly more than 23% answered question 4 incorrectly in both tests.
The results for six questions were remarkable in that they had negative scores after application of the correction factor. Among these six questions, two (questions 8 and 13) were also among the three questions that received the most incorrect responses on both tests. Another two (questions 14 and 12) were grouped with those questions that had the greatest number of respondents answering correctly on both tests. Of the six questions with negative scores after correction, three were related to pathophysiology (questions 12, 14, 16); two, to general knowledge (questions 11 and 13); and one, to disease management (question 8).
Our application of a correction factor for guessing allowed us to remove guessing as an explanation for respondents selecting the correct answer. Negative scores on these questions (ie, after the correction factor was applied) implied that the correct answer was obtained by chance, not because respondents had developed new knowledge as a result of our educational intervention. These results require that we consider whether respondents truly gained knowledge in those particular topic areas.
Conclusions
We found a statistically significant improvement between pretest and posttest asthma knowledge scores for families participating in the one-night Beating Asthma program and completing both tests. Participants with the lowest pretest scores achieved the greatest gains on the posttest.
In addition, our survey instrument allowed us to assess the quality of the lecture content. Of the three main topic areas included in this educational intervention (ie, pathophysiology, general knowledge, disease management), only the subscore for disease management demonstrated a significant improvement. Anecdotally, this result was consistent with the better-defined objectives of the presenter on disease management when compared with the other speakers at the event. We believe this factor translated into more clearly defined questions on the survey instrument.
We were able to use the survey instrument to critique itself. By looking at the pattern of pretest/posttest answers for each question, we were able to identify questions that were too basic (ie, those in the general knowledge subgroup), those that were poorly covered or not covered during the one-night educational intervention, and those that appeared to have value in measuring the impact of the intervention. This self-evaluation has allowed investigators to redesign the survey instrument for use in future asthma-intervention programs.
The survey instrument was a powerful educational tool, capable of sensitizing event participants to the issues that would be addressed in the lecture series. When completing the posttest, a number of participants made notes on the questionnaire about why a given statement was true or false based on information presented within the evening's lectures.
Finally, the survey instrument offered added value by allowing participating organizations to document their efforts in meeting criteria for reporting preventive care and health promotion activities (ie, number of patient-members participating in a health promotion/education programming) as established by the National Committee for Quality Assurance in Washington, DC.17
The lessons learned from the Beating Asthma project provide insights to researchers contemplating the creation of educational interventions for patients with asthma. We determined, for example, that, if we were to plan a second Beating Asthma event, the effectiveness of the program would be greatly increased by having speaker objectives and the survey instrument drafted by event organizers a priori to ensure continuity and quality in the intervention's content. In addition, we propose the use of an alternate system of personal identifiers on the survey instruments that would allow researchers to link the gathered data to patient medical records, thereby enabling researchers to conduct longitudinal follow-up studies.
Surveillance of the Beating Asthma educational program provided valuable information for assessing the efficacy of the educational intervention offered to community patients diagnosed with asthma. Although program coordination for Beating Asthma was somewhat cumbersome and time consuming for event planners, the program content was uniform; cost, resources, and efforts were not duplicated; surveillance was comparable; and the chance of working with a small sample size was minimized. We found great benefit in obtaining program sponsorship through the SQIG, an amalgamation of healthcare organizations.
Many insurance providers hesitate to provide education-based preventive services because they cannot demonstrate cost-benefit, especially in an health insurance climate in which patients—through their employers—change carriers on a regular basis (ie, approximately every 5 years),18 thereby creating the perception in carriers that any money committed to prevention efforts is "lost." If all the health insurance carriers in one geographic area are providing the same preventive services, however, the value of preventive care is maintained even if the patient changes carriers.
Beating Asthma was a successful collaborative effort between multiple health entities focused on improving asthma awareness and education at a community level. The lessons learned from this project will inform the development of future endeavors.
References
8. Weiss KB, Gergen PJ, Hodgson TA. An economic evaluation of asthma in the United States. N Engl J Med.1992; 326:862 -866.
9. Weiss KB, Sullivan SD, Lyttle CS. Trends in the cost of illness for asthma in the United States, 1985–1994. J Allergy Clin Immunol. 2000;106:493 -499.
10. Redd SC. Asthma in the United States: burden and current theories [review]. Environ Health Perspect.2002; 110(Suppl 4):557 -560.
12. Highlights of the Expert Panel Report 2: Guidelines for the diagnosis and management of asthma. Bethesda, Md: National Asthma Education and Prevention Program, National Heart, Lung, and Blood Institute; 1997. Publication 97-4051A.
13. National Asthma Education and Prevention Program. Expert Panel Report: Guidelines for the Diagnosis and Management of Asthma: Update on Selected Topics–2002 [published correction appears in J Allergy Clin Immunol. 2003;111:466]. J Allergy Clin Immunol.2002; 110(5 Suppl):S141 -S219.
14. Bereby-Meyer Y, Meyer J, Flascher OM. Prospect theory analysis of guessing in multiple choice tests. J Behav Decision Making. 2002;15:313 -327.
15. Vanhuele M, Drèze X. Measuring the price knowledge shoppers bring to the store. J Marketing. October2002; 66:72 -85.
16. Chan N, Kennedy PE. Are multiple-choice exams easier for economics students A comparison of multiple-choice and "equivalent" constructed-response exam questions. South Econ J.2002; 68:957 -972.
18. Survey Shows Private Health Insurance Premiums Rose 11.2% in 2004 [press release]. Washington DC: Henry J Kaiser Foundation/Health Research and Educational Trust; September 9, 2004. Available at: http://www.kff.org/insurance/chcm090904nr.cfm. Accessed December 28, 2005.(Paula D. Scariati, DO, MP)