A randomised control study of a fully automated internet based smoking cessation programme
http://www.100md.com
《烟草控制杂志》
Oregon Center for Applied Science, Eugene, Oregon, USA
ABSTRACT
Objective: The objective of this project was to test the short term (90 days) efficacy of an automated behavioural intervention for smoking cessation, the "1-2-3 Smokefree" programme, delivered via an internet website.
Design: Randomised control trial. Subjects surveyed at baseline, immediately post-intervention, and 90 days later.
Settings: The study and the intervention occurred entirely via the internet site. Subjects were recruited primarily via worksites, which referred potential subjects to the website.
Subjects: The 351 qualifying subjects were notified of the study via their worksite and required to have internet access. Additionally, subjects were required to be over 18 years of age, smoke cigarettes, and be interested in quitting smoking in the next 30 days. Eligible subjects were randomly assigned individually to treatment or control condition by computer algorithm.
Intervention: The intervention consisted of a video based internet site that presented current strategies for smoking cessation and motivational materials tailored to the user’s race/ethnicity, sex, and age. Control subjects received nothing for 90 days and were then allowed access to the programme.
Main outcome measures: The primary outcome measure was abstinence from smoking at 90 day follow up.
Results: At follow up, the cessation rate at 90 days was 24.1% (n = 21) for the treatment group and 8.2% (n = 9) for the control group (p = 0.002). Using an intent-to-treat model, 12.3% (n = 21) of the treatment group were abstinent, compared to 5.0% (n = 9) in the control group (p = 0.015).
Conclusions: These evaluation results suggest that a smoking cessation programme, with at least short term efficacy, can be successfully delivered via the internet.
Keywords: smoking cessation; intervention; internet; computer; website
This article describes a test of a stand alone behavioural intervention for smoking cessation that is delivered entirely via an internet website, makes extensive use of audio and video content, and is highly individualised. While there have been numerous articles written about the potential of the internet for health education and behaviour change,1–4 most discuss only the promise of the internet; relatively few have reported data from actual interventions.5 Furthermore, of the studies that have reported data from behavioural interventions described as internet based, few were conducted in a rigorous manner.6 Thus, there remains a substantial lack of data concerning the efficacy of internet based interventions.
It must be noted that there are many permutations of "internet based" programmes. Some smoking cessation interventions have used questionnaires administered via internet websites to collect data and then, using computerised algorithms, have generated personalised or tailored materials unique to each user.7–9 In some cases tailored, computer generated materials have been printed and sent via postal mail,7,8,10,11 while in other cases email has been used to deliver the materials.9 While it appears that most of the generated materials presented to smokers in these studies has been text based, computers clearly have the potential to deliver graphics, audio, and even video to the end user via the internet. It has long been noted that the impact of messages delivered via video (for example, television) and even audio (radio) can be more effective than print delivered messages such as newspapers and magazines.12 Thus, this website delivers smoking cessation assistance through the extensive use of video segments, as well as audio and some text. Of course, due to the lack of universal standards and rapidly changing technologies, there are many technical challenges to delivering graphics, audio, and video via the internet. However, the potential for greater effectiveness, and the rapid evolution of multimedia and internet technology, have made it important to develop and test such enhanced interventions.2
METHODS
Subjects and recruitment
Subjects were recruited primarily through large worksites. Promotional materials (for example, posters and brochures) with smoking cessation messages and the website address (www.Quitcigs.org) were displayed in the worksites. Some organisations also placed a link to the Quitcigs website on their intranet websites or sent broadcast emails or electronic newsletters to employees promoting the research study. Worksites were used simply because they are convenient locations where it is relatively easy to advertise a smoking cessation programme to fairly large numbers of smokers at low cost. Subjects could then access the website from their computers at their worksite or home.
Actual enrolment took place entirely via the website. The requirements for participation were being 18 years or older, currently smoking cigarettes on a daily basis, considering quitting smoking in the next 30 days, and being able to access the website. (Access to the website was given to anyone interested in quitting in the next six months so that they could explore the content as part of preparation for quitting. However, only those people who said they were willing to make a quit attempt within the next 30 days were included in this study. The programme’s quit plan calendar permitted choosing a quit day only within a 30 day time frame.) Access required having an internet connected computer with compatible browser software (for example, Internet Explorer or Netscape), and QuickTime (multimedia software). Potential subjects were required to complete an online informed consent process before being enrolled.
The study was a randomised control clinical trial. Eligible subjects were randomly assigned by computer algorithm with equal probability to the treatment condition, which received the intervention, or to a wait list control group that received access to the programme after a waiting period of 90 days. Guidelines for ethical conduct were followed throughout the study and all subject activities were monitored by the Oregon Center for Applied Science Institutional Review Board in Eugene, Oregon. At enrolment, all subjects were informed that they would receive either immediate access or delayed access to the website. Only if they consented to this stipulation were they permitted to enrol. Although project staff were not blinded to subjects’ group assignment, staff had no interaction with subjects and no control over what the subjects saw, as the intervention was completely automated and delivered entirely by computer.
A total of 351 subjects—52% female (n = 182), 48% male (n = 169)—were enrolled and randomly assigned to either the treatment or control group.
Seven per cent (n = 26) of the subjects were 18–25 years old, 38% (n = 134) were 26–39 years old, 48% (n = 170) were 40–55 years old, and 6% (n = 21) were over 55. Most subjects (83.5%; n = 288) self identified as white, 6.7% (n = 23) as African American, 4.3% (n = 15) as Hispanic, 2.0% (n = 7) as Native American/Indian, and 3.5% (n = 12) as "Other". All subjects were assessed at enrolment (baseline) and after 90 days. Subjects in the intervention condition were also assessed immediately after using the intervention programme. All surveys were administered via the internet.
Objective
The objective of this evaluation study was to test the efficacy of an automated behavioural intervention for smoking cessation, the "1-2-3 Smokefree" programme, delivered entirely via an internet website. Our hypothesis was that users of this highly structured intervention with personalised presentations would quit at rates greater than those who attempted to quit on their own.
Description of intervention
The intervention consisted entirely of a website programme (fig 1). The overall programme consisted of 13 separate versions or strands, including 12 demographically targeted versions, and one multicultural version, each with the same basic structure and content. The targeted versions were based on user sex, age (< 40 or 40 years old), and race/ethnicity (white, African American, or Hispanic). Using demographic data from the baseline survey, users were assigned to either one of the 12 basic content versions, or to the multicultural version if they did not fit one of the primary race/ethnicity categories. The content for each version was developed using data obtained in a very extensive set of focus groups conducted with each of the 12 targeted demographic groups (Deprey TM, Noell J. Clinic-based multimedia smoking cessation program. Presented at the American Public Health Association Conference, 9–13 November 1997, Indianapolis, Indiana; Deprey TM, Noell J. Results of clinic-based multimedia smoking cessation program. Presented at the American Public Health Association Conference, 15–19 November 1998, Washington DC). Demographic status was selected for the main set of targeting variables based on the importance of social modelling,12,13 as this programme makes extensive use of video and it was considered important that the models viewed by the users appeared to be "people like them". There also is reason to believe that sex and race/ethnicity are important variables in individual choice of cessation strategies,14,15 and these choices can be influenced by message modelling.
Overall, the website was designed to be an automated approximation of the experience a smoker would receive when working with a live smoking cessation counsellor, using principles such as those described by Abrams and colleagues16 and those contained in the Clinical Practice Guideline by Fiore and colleagues.17 Five major content modules occurred across all versions: benefits of stopping smoking, overcoming common barriers to cessation, strategies for avoiding situations that prompt cravings, strategies for dealing with cravings, and setting a quit date (which included creating a personalised quit plan calendar with individualised tips and enlisting social support). There also were modules with descriptions of Zyban (bupropion) and then-current nicotine replacement therapy (NRT) options (nicotine gum, patch, inhaler, and nasal spray). Programme modules and their content were chosen because they have been empirically proven in previous smoking cessation programme use16–19 or because they were identified as something that smokers in the developmental focus groups said they wanted (or expected) to see.
The experience of each participant was personalised (that is, tailored) both by programmatic branching (using baseline survey data) and by multiple choices offered at many points in the programme. With the exception of brief introductions to some modules, nearly all modules (each with multiple choices and levels of content) contained optional sections that could be skipped if the user chose to do so. The largest number of alternative elements for optional viewing was within the modules on barriers to cessation, dealing with situations that prompt cravings, and dealing with cravings. The culmination of the programme was a module that combined setting an actual quit date and developing a "Quit Calendar" with the chosen date circled and individualised tips selected by the user. The quit calendar permitted only dates within the next 30 days to be chosen.
Extensive use was made of video segments to present highly personalised content. One artefact of the choice to extensively individualise the presentation of the intervention material was that the amount of video required to do so was quite large. In fact, the programme contained approximately 20 hours of video material, although individual subjects saw only a fraction of that amount. The video segments presented three types of characters: a physician who presented a brief message on the health importance of stopping smoking and information regarding pharmacological aids; an ex-smoker "guide" matched to the user in sex and race/ethnicity; and numerous testimonials from ex-smokers. Most of the testimonials were delivered by actors, following focus group work demonstrating that actors were perceived as "real" more often than actual ex-smokers, when videos of each were compared side by side. Numerous audio segments were also used, especially in combination with animated graphics. QuickTime was used as the format for the video and audio segments, due to numerous technical factors applicable at the time this intervention was developed. Although repeat use was encouraged, the programme was designed so that a user could complete it in one extended session if so desired.
The entire intervention was provided by the website server program (using HTML, JavaScript, PERL CGIs, and an SQL database). Excepting technical support for users having trouble connecting to the website and automated email reminders when subjects did not complete the follow up surveys on time, there was no personal, email, or telephone interaction with subjects. Subjects interacted only with the website program.
Measures
At baseline, we assessed: demographics (age, sex, race/ethnicity); tobacco use (number of cigarettes smoked per day and use of other forms of tobacco); time from waking to first cigarette; stage of change (readiness to quit); number of previous quit attempts; techniques used in previous quit attempts (including pharmacological aids, counselling, and self help guides); presence of others in household who smoke; socialisation with other smokers; reasons for quitting; self efficacy (that is, confidence in ability to quit); perceived difficulty of not smoking under various conditions (that is, when drinking, stressed, angry, talking on phone, or if gaining weight); and perceived benefits of quitting (that is, feel better, avoid health problems, have more money, smell better, feel more in control). Measures administered immediately post-intervention included: stage of change; strength of desire to quit; self efficacy; perceived difficulty of not smoking under various conditions; perceived benefits of quitting; how helpful they found the programme; and whether or not they would recommend it to others attempting to stop smoking.
At 90 day follow up, the same measures were administered as at baseline, excepting demographic items. The 90 day follow up period was chosen for the following reasons: (1) the project had a very short timeline (due to the limitations of the primary funding mechanism); (2) 90 day follow up periods are commonly used (cf Cobb N, Graham A, Bock B. 3-month smoking outcomes on QuitNet.Com. Presented at Annual Meeting of Society for Nicotine and Tobacco Research, February 19–23, 2003, New Orleans), and (3) 90 day rates are predictive of abstinence over longer periods.20,21 Current smoking status was determined with two items, a stage of change measure (planning to quit in next 30 days, planning to quit in next six months, not planning to quit, have quit in last six months, and quit more than six months ago) and an item measuring the average number of cigarettes smoked over the previous seven days; both items had to indicate abstinence. Biochemical verification was considered, but we concluded it was not necessary as this was a case where it "...is not required and may not be desirable" due to the "limited face-to-face contact" and "data collection...through the...internet", as included in recent recommendations from a Society for Research on Nicotine and Tobacco subcommittee on the use of biochemical verification in clinical trials.22
Self efficacy for quitting smoking was measured with a single item using a five point Likert scale. Abstinence from smoking was defined as only those subjects who answered, at the follow up survey, that the average number of cigarettes smoked in the previous seven days was "zero" and indicated on the stage of change measure that they did not smoke. All assessments were conducted via the website. Automated reminders that a participant was due for the 90 day assessment were emailed to the participant.
Statistical methods
For all analyses logistic regression methods were employed to examine quit status across six predictor variables: experimental condition, sex, age, race/ethnicity, pre-test self efficacy, and number of cigarettes smoked per day at baseline. In addition, differential treatment effects were examined by entering into the regression equation the five interaction terms between experimental condition and each of the other predictor variables (for example, condition by sex, condition by age, etc). In the event that none of the main or interaction effects (other than the experimental condition main effect) were significant, the model was simplified and a simple 2 test for experimental condition was carried out.
RESULTS
A total of 351 participants were assigned to either the treatment condition (n = 171) or the control condition (n = 180). Participants were recruited through worksites from May 2000 through September 2001. Three worksites accounted for 44.2% of the total sample (American Airlines, 18.7%; Delta Airlines, 18.4%; and Hallmark Corporation, 7.2%). The remaining participants came from a variety of companies (none providing more than 5%) and through word of mouth. There were no adverse events or side effects reported during the recruitment and intervention period.
At baseline, there were no variables on which treatment and control subjects differed significantly (table 1). At 90 day follow up, a total of 6.1% (197) of all subjects returned to complete the assessment. While a somewhat smaller proportion of treatment subjects (50.9%; n = 87) returned than control condition subjects (61.1%; n = 110), the difference in attrition rates (2 = 3.73, p = 0.053) did not quite reach the 0.05 level. Thus, the analysis utilising those subjects who completed the follow up survey is internally valid—that is, the clinical trial was not compromised by differential attrition. Nonetheless, because the difference is so close to being significant, analyses are also presented for the more conservative intent-to-treat model, in which all non-respondents are presumed to still be smoking and thus categorised as smokers.
Due to the nature of the web based intervention, there was no programmatic control over when people left the website after initially accessing it. Although we attempted to assess subjects just before they left the website following their initial visit, only a minority (38.5%; n = 135) of all subjects (56.7% of treatment subjects and 21.1% of control subjects) completed the immediate session survey. Therefore, we have used only the baseline and 90 day follow up data in our analyses.
Programme use
A substantial majority of users viewed at least one optional section within modules (70.2%; n = 120). Fifty six per cent of users (n = 96) viewed the quit plan module and set an actual quit date; the same percentage of users viewed the descriptions of pharmacological aids. However, only a minority of subjects viewed optional sections within each of the five major content modules: overcoming barriers to cessation (viewed by 48.5% (n = 83) of users); avoiding situations that prompt cravings (viewed by 42.1% (n = 72) of users); dealing with cravings (viewed by 42.1% (n = 72) of users); and benefits of quitting smoking (viewed by 34.5% (n = 59)). None of the demographic or cigarette use variables predicted use of specific programme sections.
Follow up sample outcomes
For the sample of 197 subjects who returned to complete the 90 day follow up survey, the cessation rate among treatment group subjects (n = 87) was 24.1% (n = 21). The cessation rate for control condition subjects (n = 110) was 8.2% (n = 9). Logistic regression analysis was carried out to determine if there were differential condition effects across age, sex, race/ethnicity, self efficacy, and number of cigarettes smoked per day at baseline (that is, interactions with condition). There were no significant interactions between condition and the other main effects (that is, age, sex, race/ethnicity, self efficacy, and number of cigarettes smoked per day at baseline). Thus, these terms were dropped from the model and a simple 2 test was carried out. The 2 test indicated that there was a significant difference across condition (2 = 9.58, 1df, p = 0.002; odds ratio 3.57, 95% confidence interval 1.54 to 8.27). No specific aspect of programme use (for example, number of optional screens viewed) predicted abstinence at follow up.
Bivariate comparisons of baseline data for those lost to follow up at 90 days and those who were retained did not result in any significant differences that could be used to explain the observed attrition.
Intent-to-treat outcomes
The cessation rate among all treatment group subjects (n = 171) was 12.3% (n = 21) and among control condition subjects (n = 180) was 5.0% (n = 9). As with the follow up sample, logistic regression analysis indicated that there were no significant interactions between condition and the other main effects (that is, age, sex, race/ethnicity, self efficacy, and number of cigarettes smoked per day at baseline). Thus, these terms were dropped from the model and a simple 2 test was carried out. The 2 test indicated that there was a significant difference across condition (2 = 5.95, 1df, p = 0.015; odds ratio 2.66, 95% confidence interval 1.18 to 5.99). No specific aspect of programme use (for example, number of optional screens viewed) predicted abstinence at follow up.
DISCUSSION
This evaluation of the fully automated internet based smoking cessation intervention described herein has yielded promising results. The fully randomised control design lends confidence to the finding that the website intervention actually helped people stop smoking. The intent-to-treat analysis may have been excessively conservative as it is questionable to assume that all missing subjects are, in fact, smokers; however, even this conservative analysis showed significant intervention effects.
It is important to note several limitations to this study. First, all data are from self reports, and smoking status was not physiologically confirmed. Second, the attrition rate was quite high. Third, two findings are puzzling and inconsistent with other smoking cessation studies. In this study neither self efficacy nor number of cigarettes smoked per day was predictive of outcome. Despite repeated analyses using different approaches to data reduction and analysis, our data do not show an association between cessation outcome and measures of self efficacy or amount smoked per day. Fourth, longer follow up periods are needed to determine the long term impact of programmes such as this.
What this paper adds
Although numerous articles have been written about the promise of the internet for health education and behaviour change, most discuss only the potentials of the internet; relatively few have reported data from actual interventions and fewer have used fully controlled designs. The internet, which reaches a large and rapidly increasing number of people, has great potential as a means of providing behaviour change interventions to many people at low incremental cost. Thus, there is a need for studies that evaluate the actual impact of web based interventions.
The results from this randomised control trial of an entirely automated smoking cessation intervention, delivered via the internet, suggest that this type of intervention, with no direct personal contact of any kind, can be efficacious in helping people change their behaviour. Using a randomised control design, this study found more smokers were able to quit smoking when they used the website. Therefore, this study provides additional evidence that internet interventions can have a positive impact on health behaviours.
Because the attrition rate did not differ significantly across treatment and control groups, and the intent-to-treat analysis still showed significant effects, there is reason to have some confidence about these findings. However, improved retention rates would lend greater confidence in the generalisability of the findings. Interestingly, very high attrition rates have been found in other web based smoking cessation programme evaluations23 (R Munoz, personal communication, email 2 November 2002). The reasons for these high attrition rates could be related to: (1) the innate anonymity of internet contact, where users may feel no personal connection to intervention staff, thus perhaps lowering the motivation to quit; and (2) users’ access to the internet or email addresses may change. Clearly, this is an area of importance for future studies.
One last concern is that although the treatment condition had significantly higher abstinence rates, both treatment and control condition quit rates were lower than might be expected in a group that is mostly in the contemplation or preparation stages of change.24 It is not clear why these rates are low. Perhaps people are more willing to declare themselves ready to quit in the next 30 days when they are in a completely anonymous situation and endorsing that position to a computer, not a person. Perhaps the emphasis on planning to quit (given concerns about the number of participants who would continue to return for assistance) rather than relapse prevention was misplaced. Further studies are needed to clarify the reasons for these findings if replicated.
Although there are limitations, the internet remains an attractive intervention tool because the incremental cost per user is negligible and the potential cost effectiveness is great. When viewed from the RE-AIM perspective (Reach, Efficacy, Adoption, Implementation fidelity, and Maintenance) as described by Glasgow et al,25 this intervention may yield significant public health effects. The reach (that is, to anyone with a personal computer and a connection to the internet) is very large. The efficacy is reasonable for a single session, relatively "light touch" intervention, and is something that probably can be improved significantly. The adoptability is high as anyone (that is, worksites) with a website can include a link to the intervention site, and individual users can directly access the intervention via a computer with an internet connection. Perfect implementation fidelity is provided by the website itself. And lastly, maintenance costs are minimal for a programme such as this, once it has been developed. Thus, the overall potential public health value (R x E x A x I x M) is large. However, it must be noted that there often is a very large gap between potential reach and actual reach. The cost to achieve a given reach (and the time required to do so) may well be the impediment in implementing cost effective, efficacious internet interventions. In this study, despite recruitment at very large worksites, the number of enrolees per worksite was low and recruitment was slow, as has been observed in similar studies by others (R Munoz, personal communication, email 2 November 2002). How this affects the representativeness of the sample, and its generalisability, is unclear. Enrolment rates and retention are critical issues that must be addressed in future internet intervention studies.
In sum, the use of a completely automated intervention delivered via the internet resulted in elevated abstinence from cigarettes. This stands in contrast to other self administered interventions, such as self help manuals, which have minimal effectiveness when used alone.26,27 Although there are other strategies that might be employed for increasing the quit rates, such as using an online support group with a real moderator, this study indicates that it is possible to provide effective support with an automated system.
ACKNOWLEDGEMENTS
The authors would like to acknowledge the National Cancer Institute for support of this research (grant R44 CA64028).
FOOTNOTES
Competing interests: none declared
Work performed at the Oregon Center for Applied Science, Eugene, Oregon, USA
REFERENCES
Cassell MM, Jackson C, Cheuvront B. Health communication on the internet: an effective channel for health behavior change J Health Commun 1998;3:71–9.
Etter JF. Using new information technology to treat tobacco dependence. Respiration 2002;69:7–11.
Frotheringham MJ, Owies D, Leslie E, et al. Interactive health communication in preventive medicine: Internet-based strategies in teaching and research. Am J Prev Med 2000;19:113–20.
PriceWaterhouseCoopers. HealthCast 2010: Healthcare on the internet. 2 November 1999. Retrieved 17 April 2003 from http// www.pwchealth.com/healthcast2010/html.
Robinson TN, Patrick K, Eng TR, et al. An evidence-based approach to interactive health communication. JAMA 1998;280:1264–9.
Bessell TL, McDonald S, Silagy CA, et al. Do internet interventions for consumers cause more harm than good A systematic review. Health Expect 2002;5:28–37.
Etter JF, Perneger TV. Effectiveness of a computer-tailored smoking cessation program: a randomized trial. Arch Intern Med 2001;161:2596–601.
Shiffman S, Paty JA, Rohay JM, et al. The efficacy of computer-tailored smoking cessation material as a supplement to nicotine patch therapy. Drug Alcohol Depend 2001;64:35–46.
Takahashi Y, Satomura K, Miyagishima K, et al. A new smoking cessation programme using the internet. Tobacco Control 1999;8:109–10.
Strecher VJ. Computer-tailored smoking cessation materials: a review and discussion. Patient Educ Couns 1999;36:107–17.
Velicer WF, Prochaska JO, Fava JL, et al. Interactive versus noninteractive interventions and dose-response relationships for stage-matched smoking cessation programs in a managed care setting. Health Psychol 1999;18:21–8.
Bandura A. Principles of behavior modification. New York: Holt, Rinehart, and Winston, 1969.
Bandura A. Self-efficacy: The exercise of control. New York: Freeman, 1997.
Piper ME, Fox BJ, Welsch SK, et al. Gender and racial/ethnic differences in tobacco-dependence treatment: a commentary and research recommendations. Nicotine Tob Res 2001;3:291–7.
Wetter DW, Fiore MC, Young TB, et al. Gender differences in response to nicotine replacement therapy: objective and subjective indexes of tobacco withdrawal. Exp Clin Psychopharmacol 1999;7:135–44.
Abrams DB, Niaura R, Brown RB, et al. The tobacco dependence treatment handbook: a guide to best practices. New York: The Guilford Press, 2003.
Fiore MC, Bailey WC, Cohen SJ, et al. Treating tobacco use and dependence. Clinical Practice Guideline. Rockville, Maryland: US Department of Health and Human Services, Public Health Service, June 2000.
Hollis JF, Lichtenstein E, Vogt TM, et al. Nurse-assisted counseling for smokers in primary care. Ann Intern Med 1993;118:521–5.
Lichtenstein E, Hollis JF, Severson HH, et al. Tobacco cessation interventions in health care settings: rationale, model, outcomes. Addict Behav 1996;21:709–20.
Schneider NG, Olmstead R, Mody FV, et al. Efficacy of a nicotine nasal spray in smoking cessation: a placebo-controlled, double-blind trial. Addiction 1995;90:1671–82.
Zhu SH, Anderson CM, Tedeschi GJ, et al. Evidence of real-world effectiveness of a telephone quitline for smokers. N Engl J Med 2002;347:1087–93.
Society for Nicotine and Tobacco Research, Subcommittee on Biochemical Verification. Biochemical verification of tobacco use and cessation. Nicotine Tob Res 2002;4:149–59.
Feil EG, Noell J, Lichtenstein E, et al. Evaluation of an internet-based smoking cessation program: lessons learned from a pilot study. Nicotine Tob Res 2003;5:1–6.
Prochaska JO, Velicer WF, Fava JL. Evaluating a population-based recruitment approach and a stage-based expert system intervention for smoking cessation. Addict Behav 2001;26:583–602.
Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999;89:322–7.
Curry SJ, Ludman EJ, McClure J. Self-administered treatment for smoking cessation. J Clin Psychol 2003;59:305–19.
Lancaster T, Stead LF. Self-help interventions for smoking cessation (Cochrane Review). In: The Cochrane Library, issue 4, 2004. Chichester, UK: John Wiley & Sons, Ltd,.(L H G Swartz, J W Noell, S W Schroeder a)
ABSTRACT
Objective: The objective of this project was to test the short term (90 days) efficacy of an automated behavioural intervention for smoking cessation, the "1-2-3 Smokefree" programme, delivered via an internet website.
Design: Randomised control trial. Subjects surveyed at baseline, immediately post-intervention, and 90 days later.
Settings: The study and the intervention occurred entirely via the internet site. Subjects were recruited primarily via worksites, which referred potential subjects to the website.
Subjects: The 351 qualifying subjects were notified of the study via their worksite and required to have internet access. Additionally, subjects were required to be over 18 years of age, smoke cigarettes, and be interested in quitting smoking in the next 30 days. Eligible subjects were randomly assigned individually to treatment or control condition by computer algorithm.
Intervention: The intervention consisted of a video based internet site that presented current strategies for smoking cessation and motivational materials tailored to the user’s race/ethnicity, sex, and age. Control subjects received nothing for 90 days and were then allowed access to the programme.
Main outcome measures: The primary outcome measure was abstinence from smoking at 90 day follow up.
Results: At follow up, the cessation rate at 90 days was 24.1% (n = 21) for the treatment group and 8.2% (n = 9) for the control group (p = 0.002). Using an intent-to-treat model, 12.3% (n = 21) of the treatment group were abstinent, compared to 5.0% (n = 9) in the control group (p = 0.015).
Conclusions: These evaluation results suggest that a smoking cessation programme, with at least short term efficacy, can be successfully delivered via the internet.
Keywords: smoking cessation; intervention; internet; computer; website
This article describes a test of a stand alone behavioural intervention for smoking cessation that is delivered entirely via an internet website, makes extensive use of audio and video content, and is highly individualised. While there have been numerous articles written about the potential of the internet for health education and behaviour change,1–4 most discuss only the promise of the internet; relatively few have reported data from actual interventions.5 Furthermore, of the studies that have reported data from behavioural interventions described as internet based, few were conducted in a rigorous manner.6 Thus, there remains a substantial lack of data concerning the efficacy of internet based interventions.
It must be noted that there are many permutations of "internet based" programmes. Some smoking cessation interventions have used questionnaires administered via internet websites to collect data and then, using computerised algorithms, have generated personalised or tailored materials unique to each user.7–9 In some cases tailored, computer generated materials have been printed and sent via postal mail,7,8,10,11 while in other cases email has been used to deliver the materials.9 While it appears that most of the generated materials presented to smokers in these studies has been text based, computers clearly have the potential to deliver graphics, audio, and even video to the end user via the internet. It has long been noted that the impact of messages delivered via video (for example, television) and even audio (radio) can be more effective than print delivered messages such as newspapers and magazines.12 Thus, this website delivers smoking cessation assistance through the extensive use of video segments, as well as audio and some text. Of course, due to the lack of universal standards and rapidly changing technologies, there are many technical challenges to delivering graphics, audio, and video via the internet. However, the potential for greater effectiveness, and the rapid evolution of multimedia and internet technology, have made it important to develop and test such enhanced interventions.2
METHODS
Subjects and recruitment
Subjects were recruited primarily through large worksites. Promotional materials (for example, posters and brochures) with smoking cessation messages and the website address (www.Quitcigs.org) were displayed in the worksites. Some organisations also placed a link to the Quitcigs website on their intranet websites or sent broadcast emails or electronic newsletters to employees promoting the research study. Worksites were used simply because they are convenient locations where it is relatively easy to advertise a smoking cessation programme to fairly large numbers of smokers at low cost. Subjects could then access the website from their computers at their worksite or home.
Actual enrolment took place entirely via the website. The requirements for participation were being 18 years or older, currently smoking cigarettes on a daily basis, considering quitting smoking in the next 30 days, and being able to access the website. (Access to the website was given to anyone interested in quitting in the next six months so that they could explore the content as part of preparation for quitting. However, only those people who said they were willing to make a quit attempt within the next 30 days were included in this study. The programme’s quit plan calendar permitted choosing a quit day only within a 30 day time frame.) Access required having an internet connected computer with compatible browser software (for example, Internet Explorer or Netscape), and QuickTime (multimedia software). Potential subjects were required to complete an online informed consent process before being enrolled.
The study was a randomised control clinical trial. Eligible subjects were randomly assigned by computer algorithm with equal probability to the treatment condition, which received the intervention, or to a wait list control group that received access to the programme after a waiting period of 90 days. Guidelines for ethical conduct were followed throughout the study and all subject activities were monitored by the Oregon Center for Applied Science Institutional Review Board in Eugene, Oregon. At enrolment, all subjects were informed that they would receive either immediate access or delayed access to the website. Only if they consented to this stipulation were they permitted to enrol. Although project staff were not blinded to subjects’ group assignment, staff had no interaction with subjects and no control over what the subjects saw, as the intervention was completely automated and delivered entirely by computer.
A total of 351 subjects—52% female (n = 182), 48% male (n = 169)—were enrolled and randomly assigned to either the treatment or control group.
Seven per cent (n = 26) of the subjects were 18–25 years old, 38% (n = 134) were 26–39 years old, 48% (n = 170) were 40–55 years old, and 6% (n = 21) were over 55. Most subjects (83.5%; n = 288) self identified as white, 6.7% (n = 23) as African American, 4.3% (n = 15) as Hispanic, 2.0% (n = 7) as Native American/Indian, and 3.5% (n = 12) as "Other". All subjects were assessed at enrolment (baseline) and after 90 days. Subjects in the intervention condition were also assessed immediately after using the intervention programme. All surveys were administered via the internet.
Objective
The objective of this evaluation study was to test the efficacy of an automated behavioural intervention for smoking cessation, the "1-2-3 Smokefree" programme, delivered entirely via an internet website. Our hypothesis was that users of this highly structured intervention with personalised presentations would quit at rates greater than those who attempted to quit on their own.
Description of intervention
The intervention consisted entirely of a website programme (fig 1). The overall programme consisted of 13 separate versions or strands, including 12 demographically targeted versions, and one multicultural version, each with the same basic structure and content. The targeted versions were based on user sex, age (< 40 or 40 years old), and race/ethnicity (white, African American, or Hispanic). Using demographic data from the baseline survey, users were assigned to either one of the 12 basic content versions, or to the multicultural version if they did not fit one of the primary race/ethnicity categories. The content for each version was developed using data obtained in a very extensive set of focus groups conducted with each of the 12 targeted demographic groups (Deprey TM, Noell J. Clinic-based multimedia smoking cessation program. Presented at the American Public Health Association Conference, 9–13 November 1997, Indianapolis, Indiana; Deprey TM, Noell J. Results of clinic-based multimedia smoking cessation program. Presented at the American Public Health Association Conference, 15–19 November 1998, Washington DC). Demographic status was selected for the main set of targeting variables based on the importance of social modelling,12,13 as this programme makes extensive use of video and it was considered important that the models viewed by the users appeared to be "people like them". There also is reason to believe that sex and race/ethnicity are important variables in individual choice of cessation strategies,14,15 and these choices can be influenced by message modelling.
Overall, the website was designed to be an automated approximation of the experience a smoker would receive when working with a live smoking cessation counsellor, using principles such as those described by Abrams and colleagues16 and those contained in the Clinical Practice Guideline by Fiore and colleagues.17 Five major content modules occurred across all versions: benefits of stopping smoking, overcoming common barriers to cessation, strategies for avoiding situations that prompt cravings, strategies for dealing with cravings, and setting a quit date (which included creating a personalised quit plan calendar with individualised tips and enlisting social support). There also were modules with descriptions of Zyban (bupropion) and then-current nicotine replacement therapy (NRT) options (nicotine gum, patch, inhaler, and nasal spray). Programme modules and their content were chosen because they have been empirically proven in previous smoking cessation programme use16–19 or because they were identified as something that smokers in the developmental focus groups said they wanted (or expected) to see.
The experience of each participant was personalised (that is, tailored) both by programmatic branching (using baseline survey data) and by multiple choices offered at many points in the programme. With the exception of brief introductions to some modules, nearly all modules (each with multiple choices and levels of content) contained optional sections that could be skipped if the user chose to do so. The largest number of alternative elements for optional viewing was within the modules on barriers to cessation, dealing with situations that prompt cravings, and dealing with cravings. The culmination of the programme was a module that combined setting an actual quit date and developing a "Quit Calendar" with the chosen date circled and individualised tips selected by the user. The quit calendar permitted only dates within the next 30 days to be chosen.
Extensive use was made of video segments to present highly personalised content. One artefact of the choice to extensively individualise the presentation of the intervention material was that the amount of video required to do so was quite large. In fact, the programme contained approximately 20 hours of video material, although individual subjects saw only a fraction of that amount. The video segments presented three types of characters: a physician who presented a brief message on the health importance of stopping smoking and information regarding pharmacological aids; an ex-smoker "guide" matched to the user in sex and race/ethnicity; and numerous testimonials from ex-smokers. Most of the testimonials were delivered by actors, following focus group work demonstrating that actors were perceived as "real" more often than actual ex-smokers, when videos of each were compared side by side. Numerous audio segments were also used, especially in combination with animated graphics. QuickTime was used as the format for the video and audio segments, due to numerous technical factors applicable at the time this intervention was developed. Although repeat use was encouraged, the programme was designed so that a user could complete it in one extended session if so desired.
The entire intervention was provided by the website server program (using HTML, JavaScript, PERL CGIs, and an SQL database). Excepting technical support for users having trouble connecting to the website and automated email reminders when subjects did not complete the follow up surveys on time, there was no personal, email, or telephone interaction with subjects. Subjects interacted only with the website program.
Measures
At baseline, we assessed: demographics (age, sex, race/ethnicity); tobacco use (number of cigarettes smoked per day and use of other forms of tobacco); time from waking to first cigarette; stage of change (readiness to quit); number of previous quit attempts; techniques used in previous quit attempts (including pharmacological aids, counselling, and self help guides); presence of others in household who smoke; socialisation with other smokers; reasons for quitting; self efficacy (that is, confidence in ability to quit); perceived difficulty of not smoking under various conditions (that is, when drinking, stressed, angry, talking on phone, or if gaining weight); and perceived benefits of quitting (that is, feel better, avoid health problems, have more money, smell better, feel more in control). Measures administered immediately post-intervention included: stage of change; strength of desire to quit; self efficacy; perceived difficulty of not smoking under various conditions; perceived benefits of quitting; how helpful they found the programme; and whether or not they would recommend it to others attempting to stop smoking.
At 90 day follow up, the same measures were administered as at baseline, excepting demographic items. The 90 day follow up period was chosen for the following reasons: (1) the project had a very short timeline (due to the limitations of the primary funding mechanism); (2) 90 day follow up periods are commonly used (cf Cobb N, Graham A, Bock B. 3-month smoking outcomes on QuitNet.Com. Presented at Annual Meeting of Society for Nicotine and Tobacco Research, February 19–23, 2003, New Orleans), and (3) 90 day rates are predictive of abstinence over longer periods.20,21 Current smoking status was determined with two items, a stage of change measure (planning to quit in next 30 days, planning to quit in next six months, not planning to quit, have quit in last six months, and quit more than six months ago) and an item measuring the average number of cigarettes smoked over the previous seven days; both items had to indicate abstinence. Biochemical verification was considered, but we concluded it was not necessary as this was a case where it "...is not required and may not be desirable" due to the "limited face-to-face contact" and "data collection...through the...internet", as included in recent recommendations from a Society for Research on Nicotine and Tobacco subcommittee on the use of biochemical verification in clinical trials.22
Self efficacy for quitting smoking was measured with a single item using a five point Likert scale. Abstinence from smoking was defined as only those subjects who answered, at the follow up survey, that the average number of cigarettes smoked in the previous seven days was "zero" and indicated on the stage of change measure that they did not smoke. All assessments were conducted via the website. Automated reminders that a participant was due for the 90 day assessment were emailed to the participant.
Statistical methods
For all analyses logistic regression methods were employed to examine quit status across six predictor variables: experimental condition, sex, age, race/ethnicity, pre-test self efficacy, and number of cigarettes smoked per day at baseline. In addition, differential treatment effects were examined by entering into the regression equation the five interaction terms between experimental condition and each of the other predictor variables (for example, condition by sex, condition by age, etc). In the event that none of the main or interaction effects (other than the experimental condition main effect) were significant, the model was simplified and a simple 2 test for experimental condition was carried out.
RESULTS
A total of 351 participants were assigned to either the treatment condition (n = 171) or the control condition (n = 180). Participants were recruited through worksites from May 2000 through September 2001. Three worksites accounted for 44.2% of the total sample (American Airlines, 18.7%; Delta Airlines, 18.4%; and Hallmark Corporation, 7.2%). The remaining participants came from a variety of companies (none providing more than 5%) and through word of mouth. There were no adverse events or side effects reported during the recruitment and intervention period.
At baseline, there were no variables on which treatment and control subjects differed significantly (table 1). At 90 day follow up, a total of 6.1% (197) of all subjects returned to complete the assessment. While a somewhat smaller proportion of treatment subjects (50.9%; n = 87) returned than control condition subjects (61.1%; n = 110), the difference in attrition rates (2 = 3.73, p = 0.053) did not quite reach the 0.05 level. Thus, the analysis utilising those subjects who completed the follow up survey is internally valid—that is, the clinical trial was not compromised by differential attrition. Nonetheless, because the difference is so close to being significant, analyses are also presented for the more conservative intent-to-treat model, in which all non-respondents are presumed to still be smoking and thus categorised as smokers.
Due to the nature of the web based intervention, there was no programmatic control over when people left the website after initially accessing it. Although we attempted to assess subjects just before they left the website following their initial visit, only a minority (38.5%; n = 135) of all subjects (56.7% of treatment subjects and 21.1% of control subjects) completed the immediate session survey. Therefore, we have used only the baseline and 90 day follow up data in our analyses.
Programme use
A substantial majority of users viewed at least one optional section within modules (70.2%; n = 120). Fifty six per cent of users (n = 96) viewed the quit plan module and set an actual quit date; the same percentage of users viewed the descriptions of pharmacological aids. However, only a minority of subjects viewed optional sections within each of the five major content modules: overcoming barriers to cessation (viewed by 48.5% (n = 83) of users); avoiding situations that prompt cravings (viewed by 42.1% (n = 72) of users); dealing with cravings (viewed by 42.1% (n = 72) of users); and benefits of quitting smoking (viewed by 34.5% (n = 59)). None of the demographic or cigarette use variables predicted use of specific programme sections.
Follow up sample outcomes
For the sample of 197 subjects who returned to complete the 90 day follow up survey, the cessation rate among treatment group subjects (n = 87) was 24.1% (n = 21). The cessation rate for control condition subjects (n = 110) was 8.2% (n = 9). Logistic regression analysis was carried out to determine if there were differential condition effects across age, sex, race/ethnicity, self efficacy, and number of cigarettes smoked per day at baseline (that is, interactions with condition). There were no significant interactions between condition and the other main effects (that is, age, sex, race/ethnicity, self efficacy, and number of cigarettes smoked per day at baseline). Thus, these terms were dropped from the model and a simple 2 test was carried out. The 2 test indicated that there was a significant difference across condition (2 = 9.58, 1df, p = 0.002; odds ratio 3.57, 95% confidence interval 1.54 to 8.27). No specific aspect of programme use (for example, number of optional screens viewed) predicted abstinence at follow up.
Bivariate comparisons of baseline data for those lost to follow up at 90 days and those who were retained did not result in any significant differences that could be used to explain the observed attrition.
Intent-to-treat outcomes
The cessation rate among all treatment group subjects (n = 171) was 12.3% (n = 21) and among control condition subjects (n = 180) was 5.0% (n = 9). As with the follow up sample, logistic regression analysis indicated that there were no significant interactions between condition and the other main effects (that is, age, sex, race/ethnicity, self efficacy, and number of cigarettes smoked per day at baseline). Thus, these terms were dropped from the model and a simple 2 test was carried out. The 2 test indicated that there was a significant difference across condition (2 = 5.95, 1df, p = 0.015; odds ratio 2.66, 95% confidence interval 1.18 to 5.99). No specific aspect of programme use (for example, number of optional screens viewed) predicted abstinence at follow up.
DISCUSSION
This evaluation of the fully automated internet based smoking cessation intervention described herein has yielded promising results. The fully randomised control design lends confidence to the finding that the website intervention actually helped people stop smoking. The intent-to-treat analysis may have been excessively conservative as it is questionable to assume that all missing subjects are, in fact, smokers; however, even this conservative analysis showed significant intervention effects.
It is important to note several limitations to this study. First, all data are from self reports, and smoking status was not physiologically confirmed. Second, the attrition rate was quite high. Third, two findings are puzzling and inconsistent with other smoking cessation studies. In this study neither self efficacy nor number of cigarettes smoked per day was predictive of outcome. Despite repeated analyses using different approaches to data reduction and analysis, our data do not show an association between cessation outcome and measures of self efficacy or amount smoked per day. Fourth, longer follow up periods are needed to determine the long term impact of programmes such as this.
What this paper adds
Although numerous articles have been written about the promise of the internet for health education and behaviour change, most discuss only the potentials of the internet; relatively few have reported data from actual interventions and fewer have used fully controlled designs. The internet, which reaches a large and rapidly increasing number of people, has great potential as a means of providing behaviour change interventions to many people at low incremental cost. Thus, there is a need for studies that evaluate the actual impact of web based interventions.
The results from this randomised control trial of an entirely automated smoking cessation intervention, delivered via the internet, suggest that this type of intervention, with no direct personal contact of any kind, can be efficacious in helping people change their behaviour. Using a randomised control design, this study found more smokers were able to quit smoking when they used the website. Therefore, this study provides additional evidence that internet interventions can have a positive impact on health behaviours.
Because the attrition rate did not differ significantly across treatment and control groups, and the intent-to-treat analysis still showed significant effects, there is reason to have some confidence about these findings. However, improved retention rates would lend greater confidence in the generalisability of the findings. Interestingly, very high attrition rates have been found in other web based smoking cessation programme evaluations23 (R Munoz, personal communication, email 2 November 2002). The reasons for these high attrition rates could be related to: (1) the innate anonymity of internet contact, where users may feel no personal connection to intervention staff, thus perhaps lowering the motivation to quit; and (2) users’ access to the internet or email addresses may change. Clearly, this is an area of importance for future studies.
One last concern is that although the treatment condition had significantly higher abstinence rates, both treatment and control condition quit rates were lower than might be expected in a group that is mostly in the contemplation or preparation stages of change.24 It is not clear why these rates are low. Perhaps people are more willing to declare themselves ready to quit in the next 30 days when they are in a completely anonymous situation and endorsing that position to a computer, not a person. Perhaps the emphasis on planning to quit (given concerns about the number of participants who would continue to return for assistance) rather than relapse prevention was misplaced. Further studies are needed to clarify the reasons for these findings if replicated.
Although there are limitations, the internet remains an attractive intervention tool because the incremental cost per user is negligible and the potential cost effectiveness is great. When viewed from the RE-AIM perspective (Reach, Efficacy, Adoption, Implementation fidelity, and Maintenance) as described by Glasgow et al,25 this intervention may yield significant public health effects. The reach (that is, to anyone with a personal computer and a connection to the internet) is very large. The efficacy is reasonable for a single session, relatively "light touch" intervention, and is something that probably can be improved significantly. The adoptability is high as anyone (that is, worksites) with a website can include a link to the intervention site, and individual users can directly access the intervention via a computer with an internet connection. Perfect implementation fidelity is provided by the website itself. And lastly, maintenance costs are minimal for a programme such as this, once it has been developed. Thus, the overall potential public health value (R x E x A x I x M) is large. However, it must be noted that there often is a very large gap between potential reach and actual reach. The cost to achieve a given reach (and the time required to do so) may well be the impediment in implementing cost effective, efficacious internet interventions. In this study, despite recruitment at very large worksites, the number of enrolees per worksite was low and recruitment was slow, as has been observed in similar studies by others (R Munoz, personal communication, email 2 November 2002). How this affects the representativeness of the sample, and its generalisability, is unclear. Enrolment rates and retention are critical issues that must be addressed in future internet intervention studies.
In sum, the use of a completely automated intervention delivered via the internet resulted in elevated abstinence from cigarettes. This stands in contrast to other self administered interventions, such as self help manuals, which have minimal effectiveness when used alone.26,27 Although there are other strategies that might be employed for increasing the quit rates, such as using an online support group with a real moderator, this study indicates that it is possible to provide effective support with an automated system.
ACKNOWLEDGEMENTS
The authors would like to acknowledge the National Cancer Institute for support of this research (grant R44 CA64028).
FOOTNOTES
Competing interests: none declared
Work performed at the Oregon Center for Applied Science, Eugene, Oregon, USA
REFERENCES
Cassell MM, Jackson C, Cheuvront B. Health communication on the internet: an effective channel for health behavior change J Health Commun 1998;3:71–9.
Etter JF. Using new information technology to treat tobacco dependence. Respiration 2002;69:7–11.
Frotheringham MJ, Owies D, Leslie E, et al. Interactive health communication in preventive medicine: Internet-based strategies in teaching and research. Am J Prev Med 2000;19:113–20.
PriceWaterhouseCoopers. HealthCast 2010: Healthcare on the internet. 2 November 1999. Retrieved 17 April 2003 from http// www.pwchealth.com/healthcast2010/html.
Robinson TN, Patrick K, Eng TR, et al. An evidence-based approach to interactive health communication. JAMA 1998;280:1264–9.
Bessell TL, McDonald S, Silagy CA, et al. Do internet interventions for consumers cause more harm than good A systematic review. Health Expect 2002;5:28–37.
Etter JF, Perneger TV. Effectiveness of a computer-tailored smoking cessation program: a randomized trial. Arch Intern Med 2001;161:2596–601.
Shiffman S, Paty JA, Rohay JM, et al. The efficacy of computer-tailored smoking cessation material as a supplement to nicotine patch therapy. Drug Alcohol Depend 2001;64:35–46.
Takahashi Y, Satomura K, Miyagishima K, et al. A new smoking cessation programme using the internet. Tobacco Control 1999;8:109–10.
Strecher VJ. Computer-tailored smoking cessation materials: a review and discussion. Patient Educ Couns 1999;36:107–17.
Velicer WF, Prochaska JO, Fava JL, et al. Interactive versus noninteractive interventions and dose-response relationships for stage-matched smoking cessation programs in a managed care setting. Health Psychol 1999;18:21–8.
Bandura A. Principles of behavior modification. New York: Holt, Rinehart, and Winston, 1969.
Bandura A. Self-efficacy: The exercise of control. New York: Freeman, 1997.
Piper ME, Fox BJ, Welsch SK, et al. Gender and racial/ethnic differences in tobacco-dependence treatment: a commentary and research recommendations. Nicotine Tob Res 2001;3:291–7.
Wetter DW, Fiore MC, Young TB, et al. Gender differences in response to nicotine replacement therapy: objective and subjective indexes of tobacco withdrawal. Exp Clin Psychopharmacol 1999;7:135–44.
Abrams DB, Niaura R, Brown RB, et al. The tobacco dependence treatment handbook: a guide to best practices. New York: The Guilford Press, 2003.
Fiore MC, Bailey WC, Cohen SJ, et al. Treating tobacco use and dependence. Clinical Practice Guideline. Rockville, Maryland: US Department of Health and Human Services, Public Health Service, June 2000.
Hollis JF, Lichtenstein E, Vogt TM, et al. Nurse-assisted counseling for smokers in primary care. Ann Intern Med 1993;118:521–5.
Lichtenstein E, Hollis JF, Severson HH, et al. Tobacco cessation interventions in health care settings: rationale, model, outcomes. Addict Behav 1996;21:709–20.
Schneider NG, Olmstead R, Mody FV, et al. Efficacy of a nicotine nasal spray in smoking cessation: a placebo-controlled, double-blind trial. Addiction 1995;90:1671–82.
Zhu SH, Anderson CM, Tedeschi GJ, et al. Evidence of real-world effectiveness of a telephone quitline for smokers. N Engl J Med 2002;347:1087–93.
Society for Nicotine and Tobacco Research, Subcommittee on Biochemical Verification. Biochemical verification of tobacco use and cessation. Nicotine Tob Res 2002;4:149–59.
Feil EG, Noell J, Lichtenstein E, et al. Evaluation of an internet-based smoking cessation program: lessons learned from a pilot study. Nicotine Tob Res 2003;5:1–6.
Prochaska JO, Velicer WF, Fava JL. Evaluating a population-based recruitment approach and a stage-based expert system intervention for smoking cessation. Addict Behav 2001;26:583–602.
Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999;89:322–7.
Curry SJ, Ludman EJ, McClure J. Self-administered treatment for smoking cessation. J Clin Psychol 2003;59:305–19.
Lancaster T, Stead LF. Self-help interventions for smoking cessation (Cochrane Review). In: The Cochrane Library, issue 4, 2004. Chichester, UK: John Wiley & Sons, Ltd,.(L H G Swartz, J W Noell, S W Schroeder a)