Communicating Quality of Life Information to Cancer Patients: A Study of Six Presentation Formats
http://www.100md.com
《临床肿瘤学》
the Radiation Oncology Research Unit, Queen's University, Kingston
University of Toronto
Princess Margaret Hospital, Toronto, Ontario
Department of Community Health & Epidemiology, University of Saskatchewan, Saskatoon, Saskatchewan
Faculty of Nursing, University of Manitoba, Winnipeg, Manitoba, Canada
the National Cancer Institute of Canada Clinical Trials Group
ABSTRACT
PURPOSE: To determine which formats for presenting health-related quality of life (HRQL) data are interpreted most accurately and are most preferred by cancer patients. Patients often want a great deal of information about cancer treatments, including information relevant to HRQL. Clinical trials provide methodologically sound HRQL data that may be useful to patients.
PATIENTS AND METHODS: In a multicenter study, 198 patients with previously treated cancer participated in a structured interview. Participants judged HRQL information presented in one textual and five graphical formats. Outcome measures included the accuracy of patients' interpretations and ease-of-use and helpfulness ratings for each format.
RESULTS: Correct interpretations ranged from 85% to 98% across formats (F = 10.3, P < .0001) with line graphs of mean HRQL scores over time being interpreted correctly most often. Older patients and less-educated patients were less likely to interpret graphs accurately (F = 7.3, P = .008; and F = 10.6, P = .001, respectively), but all groups were most accurate on simple line graphs. Multivariate analysis revealed that format type, participant age and education were independent predictors of accuracy rates. Patients' ratings also varied across formats both for ease of understanding scores (F = 12.1, P < .0001) and for helpfulness scores (F = 13.2, P < .0001), with line graphs being rated highest on both outcomes.
CONCLUSION: Patients generally prefer a simple linear representation of group mean HRQL scores, and can accurately interpret data presented in this format more than 98% of the time irrespective of their age group and educational level. The findings have important implications for the communication of clinical trial HRQL results.
INTRODUCTION
Research evidence demonstrates that cancer patients often want a great deal of information about cancer treatments, including information relevant to their health-related quality of life (HRQL).1-6 Patients often desire information about treatment risks and benefits,2,4 and HRQL may be useful, therefore, for helping patients clarify their treatment preferences.1,6 HRQL information may also be useful to patients for other reasons.6,7 For example, to improve a patient’s ability to live with his disease8-11 or to recover from his treatment.12 Regardless of the reason(s) patients desire HRQL information, more effective ways of communicating it to patients are part of progressing towards optimal care.
The measurement of HRQL in clinical trials, in general, has been increasing in frequency.13-15 In particular, the inclusion of HRQL outcome measures in trials of the National Cancer Institute of Canada Clinical Trials Group has been a focus of the group in both national and international trials.16 Although there exists some degree of consensus and some ongoing debate regarding the most appropriate way to measure and report HRQL in clinical trials,17 there is general agreement that HRQL is subjective and multidimensional,18-20 and that instruments with robust psychometric properties are required for its proper measurement and analysis.21-23 What has not been well evaluated, however, is an appropriate method of feeding the HRQL results of clinical trials to patients—or other consumers of this information—for the purposes alluded to in the preceding paragraph. The focus of this study, therefore, is on patients' perceptions of how HRQL data might be best communicated to them.
Curran24 has hypothesized that one way to reduce the cognitive effort required to understand quantitative information is to present the data in a graphical display, especially when the data are intended to represent change over time. Various theories of graphical comprehension have been hypothesized.25-27 Shaw's cognitive model for understanding information displayed graphically includes three phases: a search for visual qualitative information; a search for quantitative relationships; and a subsequent integration of both that allows the reader to interpret the graph.28 A graph allows the reader to process quantitative information in a format that is easier to understand and retain than a textual presentation.29 Evidence also suggests that the most effective graphical display depends on the type of task for which the information is used.30
Although many studies have addressed the different instruments for collecting HRQL data and their usefulness, the best approach to describing and presenting HRQL information to patients has not been thoroughly investigated. In one HRQL study, Machin and Weeden31 found that the best means to provide HRQL information is by presenting a detailed graphical description of the major HRQL items, such as the domains and their scores; however, the most appropriate type of graph to provide such information had not been studied directly. General recommendations are available as to how best to display quantitative information graphically, although these suggestions tend not to be empirically based.32 Research of data presentation formats in the context of patients making medical treatment decisions suggests that general principles do require empiric investigation.33 Moreover, although some research has explored the cognitive processing principles that underlie the interpretation of different graphical presentations,34 it remains unclear as to which type of presentation would best convey complex data such as HRQL information. Inherent in determining which formats are best to convey the data is the need to consider the level of complexity that patients want from the data, for example, trends for a group over time, individual response rates, variability between patients, or other descriptions.
In a previous qualitative study35 involving 14 men and 19 women with a variety of cancer diagnoses, we found that simple formats for presenting HRQL information (simple graphs or written text) were generally preferred over more complex graphical information, regardless of participants' educational level. Individual patients, however, varied as to which of the visual formats they most preferred. The present study builds on our previous qualitative focus group studies by exploring these issues in a more systematic and quantitative way with a large, heterogeneous sample of cancer patients.
The purpose of this study was to determine, using a multicenter standardized protocol, how accurately cancer patients interpret HRQL data from clinical trials, and, which data presentation formats patients most prefer when presented HRQL data from a clinical trial within a treatment choice scenario. Our purpose was, furthermore, to determine the most accurate and most preferred HRQL data presentation formats to use in clinical applications, such as decision aids or educational aids requiring communication of clinical trials results to patients. In order to avoid interfering with treatment decision making in real time, we employed a study design asking previously treated cancer patients to evaluate HRQL information in the context of having been recently diagnosed with a new cancer requiring treatment. We chose not to address the scope of HRQL information presented, nor the choice of instrument, but rather to focus on how the selected domains of HRQL assessed by a particular instrument (the European Organisation for Research and Treatment of Cancer [EORTC] Quality of Life Questionnaire C30 [QLQ-C30]) might be best communicated.
PATIENTS AND METHODS
Participants
Participants were recruited in three cancer centers: Princess Margaret Hospital, University of Saskatchewan, and the Kingston Regional Cancer Centre. Potentially eligible patients were identified from upcoming outpatient appointment lists (Kingston and Toronto) or from a patient registry (Saskatchewan). A randomly selected cohort of patients identified in this way was contacted through a letter of invitation from the attending physician. Patient eligibility was confirmed by a preliminary telephone interview. Inclusion criteria specified that all participants: be able to read and write English; have a previous diagnosis of cancer (with either stable or no active disease); and had not received chemotherapy or radiotherapy in the preceding 6 months. The study was approved by the three local research ethics boards and all participants gave written consent.
Study Design
Structured one-on-one interviews were conducted with each participant. The interview design is illustrated in Figure 1.
In order to establish a hypothetical context for their responses, patients were given a short scenario that asked them to imagine that they had just been diagnosed with a new lung cancer and that they had the option of one of two treatments. The participant then completed 18 judgements (trials), each of which compared one pair of hypothetical treatments. On each trial the interviewer assessed the accuracy of the patient's interpretation of HQOL information by asking the participant to identify which of the two treatments provided the better HRQL. In order to minimize the role of chance correct responses, the interviewer also asked the patient to explain the rationale for his/her response. Responses were scored as correct if the explanation substantiated the judgment. The six formats were identified in our previous qualitative study as those most preferred by patients,35 including five graphical formats and one textual description (format 3) of treatment outcomes. The graphical formats included a line graph (format 1), a line graph with ranges (format 2), side-by-side change (response) bars (format 4), stacked change (response) bars (format 5), and stacked raw data (format 6). Each graphical format was supplemented by a brief section of text explaining the presentation. A typical presentation is shown in Figure 2, and examples of remaining formats (without the accompanying text supplements) are shown in Figure 3.
The 18 accuracy trials consisted of each of six formats being presented in three magnitude conditions: a small difference between the two treatments, a medium difference, and a mixed condition (in which one treatment showed higher HRQL scores at one point in time, and the other treatment showed higher scores at other time points). Trials were stratified by magnitude type (six trials in each of three blocks). To reduce the potential effects of presentation order on the results, a modified Latin square design was used, employing six unique orders for format presentation, and three unique orders for magnitude type. In addition, to help participants expect a change in HRQL scores being presented from one magnitude block to the next, a different HRQL domain (global, physical functioning, and emotional functioning) was used to represent each magnitude type.
After the completion of the 18 accuracy judgements, each participant rated their preferences for each presentation format using three different measures. Patients rated formats on two 10-point Likert scales assessing how easy the format was to understand, and how helpful each format was in deciding which treatment had the better HRQL. Finally, each participant ranked each format on how they would prefer to receive HRQL information. We used both Likert ratings and relative rankings to increase our understanding of patients' format preferences.
Data Analyses
Our first objective of determining how accurately patients interpreted HRQL graphical information was assessed using descriptive statistics of correct response rates. The null hypothesis that all formats were equally accurately interpreted was tested using repeated-measures analysis of variance. A related hypothesis that accuracy rates varied according to patient characteristics (age, education, or sex) and/or other factors (institution, magnitude group, format order, HRQL domain) was tested using repeated-measures analysis of variance (for univariate testing) and generalized linear modeling (for multivariate testing). The latter method was selected given the multiple-level within patient repeated nature of the accuracy outcome measure. Initial sample size estimates of 60 patients per institution were based on considerations of feasibility and on estimated power calculations for logistic regression.
Our second objective of exploring patients' preference was addressed using descriptive summaries of preference ratings. The null hypothesis that patient preferences did not differ between formats was tested using both parametric (analysis of variance) and nonparametric (Wilcoxan rank-sum) methods; since the two methods yielded similar findings we report only the parametric statistics.
RESULTS
Overall, 198 patients with a variety of cancer diagnoses participated. One patient could not complete the interview because of not understanding the tasks; this individual is included in the denominator for the accuracy calculations. Table 1 lists the characteristics of the participants. About two thirds of the sample comprised patients with either breast or prostate cancer, as these patients were most available to recruitment from well follow-up clinics. None had been treated for metastatic disease. Just less than one half of participants had received post–secondary school education.
Figure 4 shows the accuracy rates for each of the six formats: Patients judged formats accurately most of the time, with correct response rates ranging from 85% to 98%, but accuracy was affected by format. The line graphs illustrating mean HRQL scores (format 1) were interpreted correctly most often, and analysis of variance revealed that the different accuracy rates between formats were unlikely to have occurred by chance (F = 10.3, P < .0001). The figure also shows that textual summaries of the data were interpreted more accurately (95%) than some of the more complex graphical representations.
The accuracy with which patients interpreted each format varied with participants' age and educational level. Patients with postsecondary education interpreted formats more accurately overall than did patients with high school or less than high school education (F = 10.5, P = .001). An interaction between education and accuracy was further seen (Fig 5) in which accuracy of the less educated group compared with those more educated dropped for the more complex formats (F = 4.0, P = .001). Likewise, older patients (those at or older than the median age of 65 years) were less likely overall to interpret graphs accurately compared to those younger than 65 years (F = 7.3, P = .008), and an interaction between age and format type (F = 2.4, P = .03) similar to that for education and format type was observed.
Multivariate analysis revealed that format type, participant age and education were independent predictors of accuracy rates. Accuracy rates were weakly associated with the magnitude of the HRQL data, in that mixed magnitudes were less accurately interpreted in the change-score and raw-score formats compared with the other four formats (F = 2.7, P = .01). Study institution, format order, and magnitude order were not significantly associated with accuracy rates.
Patients' preference ratings for the six formats are summarized in Figure 6. Formats differed in how patients rated both ease of understanding scores (F = 12.1, P < .0001) and helpfulness scores (F = 13.2, P < .0001). A line graph representation of average scores (format 1) was rated highest on both outcomes. Interestingly, participants rated the text format as the least helpful (although we note that its accuracy rate was high). Participants ranked format 1 (line graphs of mean scores) as first (or tied for first) most often (65.8%); line graphs with ranges (format 2) was ranked as first or tied for first by 42.3% of patients, and each of the remaining formats were ranked first or tied for first by less than 8% of patients.
DISCUSSION
We have demonstrated that patients can accurately interpret graphical representations of HRQL data that are similar to the results typically generated from clinical trials of cancer therapies. Graphs representing mean HRQL scores over time (format 1) were interpreted accurately most often overall (98% mean accuracy rate). Although accuracy rates were seen to vary significantly across formats according to participants' ages and educational groups, the line graphs of mean scores were interpreted accurately in at least 97% of trials in all subgroups (ie, irrespective of patients' age, educational status or participating center). Moreover, patients rated the line graph format highest both in terms of perceived ease of interpretation and in perceived helpfulness to them, and it was ranked as most preferred by the largest proportion of participants. Adding a graphical representation of variation in the data (error bars depicting standard deviation of responses, format 2) did not improve preference outcomes for the line graph format. Consistent with the literature, the five graphical formats were at least as preferred as, and generally more preferred than, text-only summaries of HRQL data.
Our findings are also consistent with previous qualitative evaluation of patients' attitudes toward the presentation of HRQL data, indicating a consistent message from patient participants that graphical representation of trial results should be included in the information provided to patients. In both our qualitative and quantitative studies, simpler formats for data presentation were preferred, and line graphs illustrating group mean scores were the most favored. These findings, along with the finding that accuracy rates were consistent across the three participating institutions in the present study, suggest that patients' general preferences are consistent and robust. Simply put, both the qualitative35 and the present quantitative findings support the "keep it simple" axiom.
The graphical representation of clinical trial HQRL data for consumers of those data has important potential. More than thirty years ago, Skeel36 commented that "there is uncertainty about how to translate QOL (quality of life) information from the research setting to meaningful practice decisions," and the need to "market" HRQL information is well described.14,37 More recently, physicians have been found to generally support the relevance and importance of HRQL information.7 Our findings are relevant to how HRQL information might be integrated into specific interventions such as decision aids to inform patients' decisions about their treatment, and/or may be integrated into educational materials for uses beyond determining treatment preferences, such as patients' understanding of their situation or planning for the future. The findings are also relevant to informing nonpatient consumers of clinical trial HRQL data such as physicians and policy makers.14,37
Other studies addressing how individuals interpret the presentation of quantitative information have demonstrated that the interpretation of data is sensitive to the format used for its presentation. As noted earlier, Cleveland developed a hierarchy of perceptual skills that provides a theoretical basis for the accurate interpretation of graphical data.34 Feldman-Stewart33 compared participants' accuracy and speed of processing six different presentation formats and noted that pie charts and random ovals caused slower and less accurate performance compared with other formats for the task of choosing between two quantities. Elting et al38 demonstrated that experienced physicians interpreting identical data might draw different conclusions depending on the presentation format used. Differences in the interpretation of data as a function of the graphical format have also been described in other contexts.39,40
Although we were successful in demonstrating patients' consistent accuracy in format interpretation, and were rigorous in our approach to the analyses, our conclusions must be considered subject to the limitations inherent in this study. For example, we did not explore more textured descriptions of HRQL data (eg, the presentation of results in several HRQL domains simultaneously) and we have not tested the interpretation of data in patients considering a real decision (when additional stresses reduce cognitive resources). Nonetheless, our findings suggest strongly that a simple linear representation of mean scores over time represents an appropriate initial strategy for communicating HRQL data. This simple strategy may be at odds with more complex methods for HRQL data analysis, and in particular, complex strategies for dealing with data not missing at random41 or other methodologic challenges in data interpretation.42 It remains to be seen how complex analyses might be distilled down into the simpler display formats that patients seem to prefer, or alternatively, how new formats can accommodate complexities in data analysis without compromising the interpretation accuracy and the usefulness of the graphical displays. It may also be possible, in the context of Web-based information provision, to make available alternative data displays and in doing so accommodate consumers' preferences for information display. Such strategies, however, should not accommodate preference at the expense of the accuracy in how they are interpreted.
A further limitation of our method concerns the limited scope of the scenario under consideration, namely, that of adjuvant therapy following resection of a lung cancer. This scenario, which we selected based on our previous method development in focus groups, depicts changes in HRQL over time. In the setting of adjuvant treatment, some HRQL changes result from the adverse effects of treatment rather than from the improvements in HRQL conferred by treatment. It is possible that in a setting of symptom palliation, response change scores (format 4), for example, illustrating the proportion of improvement of HRQL attributed to treatment, may be valued differently. These issues require exploration with further research.
We conclude that cancer patients generally prefer a simple linear representation of group mean HRQL scores over time, and can accurately interpret data presented in this format more than 98% of the time irrespective of their age group and educational level. In general, graphic formats were deemed preferable to textual descriptions of the data. Research designed to further explore more complex issues of HRQL data communication in a variety of clinical scenarios is required.
Appendix
Description of clinical scenario used in the study: Card with case scenario.
Imagine the following: You went to your doctor 6 weeks ago and had a routine chest x-ray, which showed a cancer in your right lung. You were referred to a surgeon who admitted you to the hospital and performed an operation, which removed all visible cancer. You were then referred to a cancer specialist who explained to you that two new treatments were available to patients in your situation.
The new treatments are each intended to increase your chances of being cured. In other words, the surgery is not always successful—it is known that for every 10 patients that have the operation, four will be cured of the cancer. The new treatments increase the number of patients who are cured by treating any cancer that may have spread to other parts of the body.
The two new treatments are very similar, and require the same number of treatment visits. The treatments are equally successful at increasing the chances of the cancer being cured. The only difference that has been noted is the number of side effects with each treatment, which can impact how people feel about their quality of life.
The participant is then shown the questions (items) that were used in the clinical trial to collect data on quality of life.
For example, for global quality of life, participants would be told that patients in clinical studies responded to the following two questions:
How would you rate your overall health during the past week?
1 2 3 4 5 6 7
Very Poor to Excellent
How would you rate your overall quality of life during the past week?
1 2 3 4 5 6 7
Very Poor to Excellent
The global scores are described as follows: When a patient responded with the highest possible responses ("7" for each question), their quality of life was scored as "100"; when a patient responded with the lowest possible responses ("1" for each question), their quality of life was scored as a "0." Other combinations are scored accordingly (for instance, a "4" to each question would be scored as "50").
Authors' Disclosures of Potential Conflicts of Interest
The authors indicated no potential conflicts of interest.
NOTES
Supported by an operating grant from the National Cancer Institute of Canada. The funding agreement ensured the authors' independence in designing the study, interpreting the data, writing and publishing the report.
Presented at the 39th annual meeting of the American Society of Clinical Oncology, Chicago, IL, May 31-June 3, 2003.
Authors' disclosures of potential conflicts of interest are found at the end of this article.
REFERENCES
Davison J, Degner LF, Morgan TR: Information and decision-making preferences of men with prostate cancer. Oncol Nurs Forum 22:1401-1408, 1995
Sutherland HJ, Llewellyn-Thomas HA, Lockwood GA, et al: Cancer patients: Their desire for information and participation in treatment decisions. J R Soc Med 82:260-263, 1989
Wiggers JH, Donovan KO, Redman S, et al: Cancer patient satisfaction with care. Cancer 66:610-616, 1990
Cassileth BR, Zupkis RV, Sutton-Smith K, et al: Information and participation preferences among cancer patients. Ann Intern Med 92:832-836, 1980
Chaitchik S, Kreitler S, Shaked S, et al: Doctor-patient communication in a cancer ward. J Cancer Educ 7:41-54, 1992
Feldman-Stewart D, Brundage MD, Hayter C, et al: What questions do patients with curable prostate cancer want answered? Med Decis Making 20:7-19, 2000
Bezjak A, Taylor KM, Ng P, et al: Quality-of-life information and clinical practice: The oncologist's perspective. Cancer Prev Control 2:230-235, 1998
Greenfield S, Kaplan S, Ware JE: Expanding patient involvement in care. Ann Intern Med 102:520-528, 1985
Jacobs C, Ross RD, Walker IM, et al: Behavior of cancer patients: A randomized study of the effects of education and peer support groups. Am J Clin Oncol (CCT) 6:347-350, 1983
Meyerowitz BE, Watkins IK, Sparks FC: Psychosocial implications of adjuvant chemotherapy. Cancer 52:1541-1545, 1983
Molleman E, Krabbendam PJ, Annyas AA, et al: The significance of the doctor-patient relationship in coping with cancer. Soc Sci Med 18:475-480, 1984
Wallace LM: Informed consent to elective surgery: The ‘therapeutic’ value? Soc Sci Med 22:29-33, 1986
Anderson RT, Aaronson NK, Wilkin D: Critical review of the international assessments of health-related quality of life. Qual Life Res 2:369-395, 1993
Till JE, Osoba D, Pater JL, et al: Research on health-related quality of life: Dissemination into practical applications. Qual Life Res 3:279-283, 1994
Kiebert G, Wait S, Bernhard J, et al: Practice and policy of measuring quality of life and health economics in cancer clinical trials: A survey among co-operative trial groups. Qual Life Res 9:1073-1080, 2000
Osoba D, Dancey J, Zee B, et al: Health-related quality-of-life studies of the National Cancer Institute of Canada Clinical Trials Group. J Natl Cancer Inst Monographs:107-111, 1996
Sprangers MA, Moinpour CM, Moynihan TJ, et al: Assessing meaningful change in quality of life over time: A users' guide for clinicians. Mayo Clin Proc 77:561-571, 2002
Aaronson NK, Ahmedzai S, Bergman B: The European Organization for Research and Treatment of Cancer QLQ-C30: A quality-of-life instrument for use in international clinical trials in oncology. J Natl Cancer Inst 85:365-376, 1993
de Haes JC, van Knippenberg FC: Quality of life instruments for cancer patients: "Babel's Tower revisited." J Clin Epidemiol 42:1239-1241, 1989
Cohen SR, Mount BM, Bruera E, et al: Validity of the McGill Quality of Life Questionnaire in the palliative care setting: A multi-centre Canadian study demonstrating the importance of the existential domain. Palliative Medicine 11:3-20, 1997
Gill TM, Feinstein AR: A critical appraisal of the quality of quality-of-life measurements. JAMA 272:619-626, 1994
Guyatt GH, Feeny DH, Patrick DL: Measuring health-related quality of life. Ann Intern Med 118:622-629, 1993
Neymark N, Kiebert W, Torfs K, et al: Methodological and statistical issues of quality of life (QoL) and economic evaluation in cancer clinical trials: Report of a workshop. Eur J Cancer 34:1317-1333, 1998
Curran CR: Data display techniques. Appl Nurs Res 12:153-158, 1999
Berton J: Semiology of Graphs. Madison, WI, University of Wisconsin Press, 1983
Carpenter PA, Shaw PA: A model of the perceptual and conceptual processes in graph comprehension. J of Exp Psych 4:75-100, 1998
Cleveland WS: The elements of graphing data. Belmont, CA, Wadsworth, 1994
Shaw PA, Mayer RE, Hegarty M: Graphs as aids to knowledge construction: Signaling techniques for guiding the process of graph comprehension. J Ed Psych 91:690-702,1999
Standing L, Conezio J, Haber R: Perception and memory for pictures: Single-trial learning of 2500 visual stimuli. Psychonomic Sci 19:73-74, 1970
Jarvenpaa SL, Dickson GW: Graphics and managerial decision making: Research based guidelines. Comm ACM 31:764-774, 1988
Machin D, Weeden S: Suggestions for the presentation of quality of life data from clinical trials. Stat Med 17:711-724, 1998
Tufte ER: The visual display of quantitative information. Cheshire, CT, Graphics Press, 1983
Feldman-Stewart D, Kocovski N, McConnell B, et al: Perception of quantitative information for treatment decisions. Med Decis Making 20:228-238, 2000
Cleveland WS, McGill R: Graphical perception: Theory, experimentation, and application to the development of graphical methods. J Am Stat Assoc 79:531-554, 1984
Brundage MD, Leis A, Bezjak A, et al: Cancer patients' preferences for communicating clinical trial quality of life information: A qualitative study. Qual Life Res 12:395-404, 2002
Skeel RT: Quality of life dimensions that are most important to cancer patients. Oncology 7:55-61, 1969
Osoba D, Till JE, Pater JL, et al: Health-related quality of life: Measurement and clinical application: A workshop report. Can J Oncol 5:338-343, 1995
Elting LS, Martin CG, Cantor SB, et al: Influence of data display formats on physician investigators' decisions to stop clinical trials: Prospective trial with repeated measures. BMJ 318:1527-1531, 1999
Ottenbacher KJ: When is a picture worth a thousand p values? A comparison of visual and quantitative methods to analyze single subject data. J Spec Ed 23:436-449, 1990
Carswell CM: The art and psychological science of graphical communication. Psych Sci Agenda 1995;8-9
Fairclough DL, Gagnon DD, Zagari MJ, et al: Evaluation of quality of life in a clinical trial with non-random dropout: The effect of epoetin alfa in anemic cancer patients. Qual Life Res 12:1013-1027, 2003
Sprangers MA: Quality-of-life assessment in oncology: Acheivements and challenges. Acta Oncol 41:229-237, 2002(M. Brundage, D. Feldman-S)
University of Toronto
Princess Margaret Hospital, Toronto, Ontario
Department of Community Health & Epidemiology, University of Saskatchewan, Saskatoon, Saskatchewan
Faculty of Nursing, University of Manitoba, Winnipeg, Manitoba, Canada
the National Cancer Institute of Canada Clinical Trials Group
ABSTRACT
PURPOSE: To determine which formats for presenting health-related quality of life (HRQL) data are interpreted most accurately and are most preferred by cancer patients. Patients often want a great deal of information about cancer treatments, including information relevant to HRQL. Clinical trials provide methodologically sound HRQL data that may be useful to patients.
PATIENTS AND METHODS: In a multicenter study, 198 patients with previously treated cancer participated in a structured interview. Participants judged HRQL information presented in one textual and five graphical formats. Outcome measures included the accuracy of patients' interpretations and ease-of-use and helpfulness ratings for each format.
RESULTS: Correct interpretations ranged from 85% to 98% across formats (F = 10.3, P < .0001) with line graphs of mean HRQL scores over time being interpreted correctly most often. Older patients and less-educated patients were less likely to interpret graphs accurately (F = 7.3, P = .008; and F = 10.6, P = .001, respectively), but all groups were most accurate on simple line graphs. Multivariate analysis revealed that format type, participant age and education were independent predictors of accuracy rates. Patients' ratings also varied across formats both for ease of understanding scores (F = 12.1, P < .0001) and for helpfulness scores (F = 13.2, P < .0001), with line graphs being rated highest on both outcomes.
CONCLUSION: Patients generally prefer a simple linear representation of group mean HRQL scores, and can accurately interpret data presented in this format more than 98% of the time irrespective of their age group and educational level. The findings have important implications for the communication of clinical trial HRQL results.
INTRODUCTION
Research evidence demonstrates that cancer patients often want a great deal of information about cancer treatments, including information relevant to their health-related quality of life (HRQL).1-6 Patients often desire information about treatment risks and benefits,2,4 and HRQL may be useful, therefore, for helping patients clarify their treatment preferences.1,6 HRQL information may also be useful to patients for other reasons.6,7 For example, to improve a patient’s ability to live with his disease8-11 or to recover from his treatment.12 Regardless of the reason(s) patients desire HRQL information, more effective ways of communicating it to patients are part of progressing towards optimal care.
The measurement of HRQL in clinical trials, in general, has been increasing in frequency.13-15 In particular, the inclusion of HRQL outcome measures in trials of the National Cancer Institute of Canada Clinical Trials Group has been a focus of the group in both national and international trials.16 Although there exists some degree of consensus and some ongoing debate regarding the most appropriate way to measure and report HRQL in clinical trials,17 there is general agreement that HRQL is subjective and multidimensional,18-20 and that instruments with robust psychometric properties are required for its proper measurement and analysis.21-23 What has not been well evaluated, however, is an appropriate method of feeding the HRQL results of clinical trials to patients—or other consumers of this information—for the purposes alluded to in the preceding paragraph. The focus of this study, therefore, is on patients' perceptions of how HRQL data might be best communicated to them.
Curran24 has hypothesized that one way to reduce the cognitive effort required to understand quantitative information is to present the data in a graphical display, especially when the data are intended to represent change over time. Various theories of graphical comprehension have been hypothesized.25-27 Shaw's cognitive model for understanding information displayed graphically includes three phases: a search for visual qualitative information; a search for quantitative relationships; and a subsequent integration of both that allows the reader to interpret the graph.28 A graph allows the reader to process quantitative information in a format that is easier to understand and retain than a textual presentation.29 Evidence also suggests that the most effective graphical display depends on the type of task for which the information is used.30
Although many studies have addressed the different instruments for collecting HRQL data and their usefulness, the best approach to describing and presenting HRQL information to patients has not been thoroughly investigated. In one HRQL study, Machin and Weeden31 found that the best means to provide HRQL information is by presenting a detailed graphical description of the major HRQL items, such as the domains and their scores; however, the most appropriate type of graph to provide such information had not been studied directly. General recommendations are available as to how best to display quantitative information graphically, although these suggestions tend not to be empirically based.32 Research of data presentation formats in the context of patients making medical treatment decisions suggests that general principles do require empiric investigation.33 Moreover, although some research has explored the cognitive processing principles that underlie the interpretation of different graphical presentations,34 it remains unclear as to which type of presentation would best convey complex data such as HRQL information. Inherent in determining which formats are best to convey the data is the need to consider the level of complexity that patients want from the data, for example, trends for a group over time, individual response rates, variability between patients, or other descriptions.
In a previous qualitative study35 involving 14 men and 19 women with a variety of cancer diagnoses, we found that simple formats for presenting HRQL information (simple graphs or written text) were generally preferred over more complex graphical information, regardless of participants' educational level. Individual patients, however, varied as to which of the visual formats they most preferred. The present study builds on our previous qualitative focus group studies by exploring these issues in a more systematic and quantitative way with a large, heterogeneous sample of cancer patients.
The purpose of this study was to determine, using a multicenter standardized protocol, how accurately cancer patients interpret HRQL data from clinical trials, and, which data presentation formats patients most prefer when presented HRQL data from a clinical trial within a treatment choice scenario. Our purpose was, furthermore, to determine the most accurate and most preferred HRQL data presentation formats to use in clinical applications, such as decision aids or educational aids requiring communication of clinical trials results to patients. In order to avoid interfering with treatment decision making in real time, we employed a study design asking previously treated cancer patients to evaluate HRQL information in the context of having been recently diagnosed with a new cancer requiring treatment. We chose not to address the scope of HRQL information presented, nor the choice of instrument, but rather to focus on how the selected domains of HRQL assessed by a particular instrument (the European Organisation for Research and Treatment of Cancer [EORTC] Quality of Life Questionnaire C30 [QLQ-C30]) might be best communicated.
PATIENTS AND METHODS
Participants
Participants were recruited in three cancer centers: Princess Margaret Hospital, University of Saskatchewan, and the Kingston Regional Cancer Centre. Potentially eligible patients were identified from upcoming outpatient appointment lists (Kingston and Toronto) or from a patient registry (Saskatchewan). A randomly selected cohort of patients identified in this way was contacted through a letter of invitation from the attending physician. Patient eligibility was confirmed by a preliminary telephone interview. Inclusion criteria specified that all participants: be able to read and write English; have a previous diagnosis of cancer (with either stable or no active disease); and had not received chemotherapy or radiotherapy in the preceding 6 months. The study was approved by the three local research ethics boards and all participants gave written consent.
Study Design
Structured one-on-one interviews were conducted with each participant. The interview design is illustrated in Figure 1.
In order to establish a hypothetical context for their responses, patients were given a short scenario that asked them to imagine that they had just been diagnosed with a new lung cancer and that they had the option of one of two treatments. The participant then completed 18 judgements (trials), each of which compared one pair of hypothetical treatments. On each trial the interviewer assessed the accuracy of the patient's interpretation of HQOL information by asking the participant to identify which of the two treatments provided the better HRQL. In order to minimize the role of chance correct responses, the interviewer also asked the patient to explain the rationale for his/her response. Responses were scored as correct if the explanation substantiated the judgment. The six formats were identified in our previous qualitative study as those most preferred by patients,35 including five graphical formats and one textual description (format 3) of treatment outcomes. The graphical formats included a line graph (format 1), a line graph with ranges (format 2), side-by-side change (response) bars (format 4), stacked change (response) bars (format 5), and stacked raw data (format 6). Each graphical format was supplemented by a brief section of text explaining the presentation. A typical presentation is shown in Figure 2, and examples of remaining formats (without the accompanying text supplements) are shown in Figure 3.
The 18 accuracy trials consisted of each of six formats being presented in three magnitude conditions: a small difference between the two treatments, a medium difference, and a mixed condition (in which one treatment showed higher HRQL scores at one point in time, and the other treatment showed higher scores at other time points). Trials were stratified by magnitude type (six trials in each of three blocks). To reduce the potential effects of presentation order on the results, a modified Latin square design was used, employing six unique orders for format presentation, and three unique orders for magnitude type. In addition, to help participants expect a change in HRQL scores being presented from one magnitude block to the next, a different HRQL domain (global, physical functioning, and emotional functioning) was used to represent each magnitude type.
After the completion of the 18 accuracy judgements, each participant rated their preferences for each presentation format using three different measures. Patients rated formats on two 10-point Likert scales assessing how easy the format was to understand, and how helpful each format was in deciding which treatment had the better HRQL. Finally, each participant ranked each format on how they would prefer to receive HRQL information. We used both Likert ratings and relative rankings to increase our understanding of patients' format preferences.
Data Analyses
Our first objective of determining how accurately patients interpreted HRQL graphical information was assessed using descriptive statistics of correct response rates. The null hypothesis that all formats were equally accurately interpreted was tested using repeated-measures analysis of variance. A related hypothesis that accuracy rates varied according to patient characteristics (age, education, or sex) and/or other factors (institution, magnitude group, format order, HRQL domain) was tested using repeated-measures analysis of variance (for univariate testing) and generalized linear modeling (for multivariate testing). The latter method was selected given the multiple-level within patient repeated nature of the accuracy outcome measure. Initial sample size estimates of 60 patients per institution were based on considerations of feasibility and on estimated power calculations for logistic regression.
Our second objective of exploring patients' preference was addressed using descriptive summaries of preference ratings. The null hypothesis that patient preferences did not differ between formats was tested using both parametric (analysis of variance) and nonparametric (Wilcoxan rank-sum) methods; since the two methods yielded similar findings we report only the parametric statistics.
RESULTS
Overall, 198 patients with a variety of cancer diagnoses participated. One patient could not complete the interview because of not understanding the tasks; this individual is included in the denominator for the accuracy calculations. Table 1 lists the characteristics of the participants. About two thirds of the sample comprised patients with either breast or prostate cancer, as these patients were most available to recruitment from well follow-up clinics. None had been treated for metastatic disease. Just less than one half of participants had received post–secondary school education.
Figure 4 shows the accuracy rates for each of the six formats: Patients judged formats accurately most of the time, with correct response rates ranging from 85% to 98%, but accuracy was affected by format. The line graphs illustrating mean HRQL scores (format 1) were interpreted correctly most often, and analysis of variance revealed that the different accuracy rates between formats were unlikely to have occurred by chance (F = 10.3, P < .0001). The figure also shows that textual summaries of the data were interpreted more accurately (95%) than some of the more complex graphical representations.
The accuracy with which patients interpreted each format varied with participants' age and educational level. Patients with postsecondary education interpreted formats more accurately overall than did patients with high school or less than high school education (F = 10.5, P = .001). An interaction between education and accuracy was further seen (Fig 5) in which accuracy of the less educated group compared with those more educated dropped for the more complex formats (F = 4.0, P = .001). Likewise, older patients (those at or older than the median age of 65 years) were less likely overall to interpret graphs accurately compared to those younger than 65 years (F = 7.3, P = .008), and an interaction between age and format type (F = 2.4, P = .03) similar to that for education and format type was observed.
Multivariate analysis revealed that format type, participant age and education were independent predictors of accuracy rates. Accuracy rates were weakly associated with the magnitude of the HRQL data, in that mixed magnitudes were less accurately interpreted in the change-score and raw-score formats compared with the other four formats (F = 2.7, P = .01). Study institution, format order, and magnitude order were not significantly associated with accuracy rates.
Patients' preference ratings for the six formats are summarized in Figure 6. Formats differed in how patients rated both ease of understanding scores (F = 12.1, P < .0001) and helpfulness scores (F = 13.2, P < .0001). A line graph representation of average scores (format 1) was rated highest on both outcomes. Interestingly, participants rated the text format as the least helpful (although we note that its accuracy rate was high). Participants ranked format 1 (line graphs of mean scores) as first (or tied for first) most often (65.8%); line graphs with ranges (format 2) was ranked as first or tied for first by 42.3% of patients, and each of the remaining formats were ranked first or tied for first by less than 8% of patients.
DISCUSSION
We have demonstrated that patients can accurately interpret graphical representations of HRQL data that are similar to the results typically generated from clinical trials of cancer therapies. Graphs representing mean HRQL scores over time (format 1) were interpreted accurately most often overall (98% mean accuracy rate). Although accuracy rates were seen to vary significantly across formats according to participants' ages and educational groups, the line graphs of mean scores were interpreted accurately in at least 97% of trials in all subgroups (ie, irrespective of patients' age, educational status or participating center). Moreover, patients rated the line graph format highest both in terms of perceived ease of interpretation and in perceived helpfulness to them, and it was ranked as most preferred by the largest proportion of participants. Adding a graphical representation of variation in the data (error bars depicting standard deviation of responses, format 2) did not improve preference outcomes for the line graph format. Consistent with the literature, the five graphical formats were at least as preferred as, and generally more preferred than, text-only summaries of HRQL data.
Our findings are also consistent with previous qualitative evaluation of patients' attitudes toward the presentation of HRQL data, indicating a consistent message from patient participants that graphical representation of trial results should be included in the information provided to patients. In both our qualitative and quantitative studies, simpler formats for data presentation were preferred, and line graphs illustrating group mean scores were the most favored. These findings, along with the finding that accuracy rates were consistent across the three participating institutions in the present study, suggest that patients' general preferences are consistent and robust. Simply put, both the qualitative35 and the present quantitative findings support the "keep it simple" axiom.
The graphical representation of clinical trial HQRL data for consumers of those data has important potential. More than thirty years ago, Skeel36 commented that "there is uncertainty about how to translate QOL (quality of life) information from the research setting to meaningful practice decisions," and the need to "market" HRQL information is well described.14,37 More recently, physicians have been found to generally support the relevance and importance of HRQL information.7 Our findings are relevant to how HRQL information might be integrated into specific interventions such as decision aids to inform patients' decisions about their treatment, and/or may be integrated into educational materials for uses beyond determining treatment preferences, such as patients' understanding of their situation or planning for the future. The findings are also relevant to informing nonpatient consumers of clinical trial HRQL data such as physicians and policy makers.14,37
Other studies addressing how individuals interpret the presentation of quantitative information have demonstrated that the interpretation of data is sensitive to the format used for its presentation. As noted earlier, Cleveland developed a hierarchy of perceptual skills that provides a theoretical basis for the accurate interpretation of graphical data.34 Feldman-Stewart33 compared participants' accuracy and speed of processing six different presentation formats and noted that pie charts and random ovals caused slower and less accurate performance compared with other formats for the task of choosing between two quantities. Elting et al38 demonstrated that experienced physicians interpreting identical data might draw different conclusions depending on the presentation format used. Differences in the interpretation of data as a function of the graphical format have also been described in other contexts.39,40
Although we were successful in demonstrating patients' consistent accuracy in format interpretation, and were rigorous in our approach to the analyses, our conclusions must be considered subject to the limitations inherent in this study. For example, we did not explore more textured descriptions of HRQL data (eg, the presentation of results in several HRQL domains simultaneously) and we have not tested the interpretation of data in patients considering a real decision (when additional stresses reduce cognitive resources). Nonetheless, our findings suggest strongly that a simple linear representation of mean scores over time represents an appropriate initial strategy for communicating HRQL data. This simple strategy may be at odds with more complex methods for HRQL data analysis, and in particular, complex strategies for dealing with data not missing at random41 or other methodologic challenges in data interpretation.42 It remains to be seen how complex analyses might be distilled down into the simpler display formats that patients seem to prefer, or alternatively, how new formats can accommodate complexities in data analysis without compromising the interpretation accuracy and the usefulness of the graphical displays. It may also be possible, in the context of Web-based information provision, to make available alternative data displays and in doing so accommodate consumers' preferences for information display. Such strategies, however, should not accommodate preference at the expense of the accuracy in how they are interpreted.
A further limitation of our method concerns the limited scope of the scenario under consideration, namely, that of adjuvant therapy following resection of a lung cancer. This scenario, which we selected based on our previous method development in focus groups, depicts changes in HRQL over time. In the setting of adjuvant treatment, some HRQL changes result from the adverse effects of treatment rather than from the improvements in HRQL conferred by treatment. It is possible that in a setting of symptom palliation, response change scores (format 4), for example, illustrating the proportion of improvement of HRQL attributed to treatment, may be valued differently. These issues require exploration with further research.
We conclude that cancer patients generally prefer a simple linear representation of group mean HRQL scores over time, and can accurately interpret data presented in this format more than 98% of the time irrespective of their age group and educational level. In general, graphic formats were deemed preferable to textual descriptions of the data. Research designed to further explore more complex issues of HRQL data communication in a variety of clinical scenarios is required.
Appendix
Description of clinical scenario used in the study: Card with case scenario.
Imagine the following: You went to your doctor 6 weeks ago and had a routine chest x-ray, which showed a cancer in your right lung. You were referred to a surgeon who admitted you to the hospital and performed an operation, which removed all visible cancer. You were then referred to a cancer specialist who explained to you that two new treatments were available to patients in your situation.
The new treatments are each intended to increase your chances of being cured. In other words, the surgery is not always successful—it is known that for every 10 patients that have the operation, four will be cured of the cancer. The new treatments increase the number of patients who are cured by treating any cancer that may have spread to other parts of the body.
The two new treatments are very similar, and require the same number of treatment visits. The treatments are equally successful at increasing the chances of the cancer being cured. The only difference that has been noted is the number of side effects with each treatment, which can impact how people feel about their quality of life.
The participant is then shown the questions (items) that were used in the clinical trial to collect data on quality of life.
For example, for global quality of life, participants would be told that patients in clinical studies responded to the following two questions:
How would you rate your overall health during the past week?
1 2 3 4 5 6 7
Very Poor to Excellent
How would you rate your overall quality of life during the past week?
1 2 3 4 5 6 7
Very Poor to Excellent
The global scores are described as follows: When a patient responded with the highest possible responses ("7" for each question), their quality of life was scored as "100"; when a patient responded with the lowest possible responses ("1" for each question), their quality of life was scored as a "0." Other combinations are scored accordingly (for instance, a "4" to each question would be scored as "50").
Authors' Disclosures of Potential Conflicts of Interest
The authors indicated no potential conflicts of interest.
NOTES
Supported by an operating grant from the National Cancer Institute of Canada. The funding agreement ensured the authors' independence in designing the study, interpreting the data, writing and publishing the report.
Presented at the 39th annual meeting of the American Society of Clinical Oncology, Chicago, IL, May 31-June 3, 2003.
Authors' disclosures of potential conflicts of interest are found at the end of this article.
REFERENCES
Davison J, Degner LF, Morgan TR: Information and decision-making preferences of men with prostate cancer. Oncol Nurs Forum 22:1401-1408, 1995
Sutherland HJ, Llewellyn-Thomas HA, Lockwood GA, et al: Cancer patients: Their desire for information and participation in treatment decisions. J R Soc Med 82:260-263, 1989
Wiggers JH, Donovan KO, Redman S, et al: Cancer patient satisfaction with care. Cancer 66:610-616, 1990
Cassileth BR, Zupkis RV, Sutton-Smith K, et al: Information and participation preferences among cancer patients. Ann Intern Med 92:832-836, 1980
Chaitchik S, Kreitler S, Shaked S, et al: Doctor-patient communication in a cancer ward. J Cancer Educ 7:41-54, 1992
Feldman-Stewart D, Brundage MD, Hayter C, et al: What questions do patients with curable prostate cancer want answered? Med Decis Making 20:7-19, 2000
Bezjak A, Taylor KM, Ng P, et al: Quality-of-life information and clinical practice: The oncologist's perspective. Cancer Prev Control 2:230-235, 1998
Greenfield S, Kaplan S, Ware JE: Expanding patient involvement in care. Ann Intern Med 102:520-528, 1985
Jacobs C, Ross RD, Walker IM, et al: Behavior of cancer patients: A randomized study of the effects of education and peer support groups. Am J Clin Oncol (CCT) 6:347-350, 1983
Meyerowitz BE, Watkins IK, Sparks FC: Psychosocial implications of adjuvant chemotherapy. Cancer 52:1541-1545, 1983
Molleman E, Krabbendam PJ, Annyas AA, et al: The significance of the doctor-patient relationship in coping with cancer. Soc Sci Med 18:475-480, 1984
Wallace LM: Informed consent to elective surgery: The ‘therapeutic’ value? Soc Sci Med 22:29-33, 1986
Anderson RT, Aaronson NK, Wilkin D: Critical review of the international assessments of health-related quality of life. Qual Life Res 2:369-395, 1993
Till JE, Osoba D, Pater JL, et al: Research on health-related quality of life: Dissemination into practical applications. Qual Life Res 3:279-283, 1994
Kiebert G, Wait S, Bernhard J, et al: Practice and policy of measuring quality of life and health economics in cancer clinical trials: A survey among co-operative trial groups. Qual Life Res 9:1073-1080, 2000
Osoba D, Dancey J, Zee B, et al: Health-related quality-of-life studies of the National Cancer Institute of Canada Clinical Trials Group. J Natl Cancer Inst Monographs:107-111, 1996
Sprangers MA, Moinpour CM, Moynihan TJ, et al: Assessing meaningful change in quality of life over time: A users' guide for clinicians. Mayo Clin Proc 77:561-571, 2002
Aaronson NK, Ahmedzai S, Bergman B: The European Organization for Research and Treatment of Cancer QLQ-C30: A quality-of-life instrument for use in international clinical trials in oncology. J Natl Cancer Inst 85:365-376, 1993
de Haes JC, van Knippenberg FC: Quality of life instruments for cancer patients: "Babel's Tower revisited." J Clin Epidemiol 42:1239-1241, 1989
Cohen SR, Mount BM, Bruera E, et al: Validity of the McGill Quality of Life Questionnaire in the palliative care setting: A multi-centre Canadian study demonstrating the importance of the existential domain. Palliative Medicine 11:3-20, 1997
Gill TM, Feinstein AR: A critical appraisal of the quality of quality-of-life measurements. JAMA 272:619-626, 1994
Guyatt GH, Feeny DH, Patrick DL: Measuring health-related quality of life. Ann Intern Med 118:622-629, 1993
Neymark N, Kiebert W, Torfs K, et al: Methodological and statistical issues of quality of life (QoL) and economic evaluation in cancer clinical trials: Report of a workshop. Eur J Cancer 34:1317-1333, 1998
Curran CR: Data display techniques. Appl Nurs Res 12:153-158, 1999
Berton J: Semiology of Graphs. Madison, WI, University of Wisconsin Press, 1983
Carpenter PA, Shaw PA: A model of the perceptual and conceptual processes in graph comprehension. J of Exp Psych 4:75-100, 1998
Cleveland WS: The elements of graphing data. Belmont, CA, Wadsworth, 1994
Shaw PA, Mayer RE, Hegarty M: Graphs as aids to knowledge construction: Signaling techniques for guiding the process of graph comprehension. J Ed Psych 91:690-702,1999
Standing L, Conezio J, Haber R: Perception and memory for pictures: Single-trial learning of 2500 visual stimuli. Psychonomic Sci 19:73-74, 1970
Jarvenpaa SL, Dickson GW: Graphics and managerial decision making: Research based guidelines. Comm ACM 31:764-774, 1988
Machin D, Weeden S: Suggestions for the presentation of quality of life data from clinical trials. Stat Med 17:711-724, 1998
Tufte ER: The visual display of quantitative information. Cheshire, CT, Graphics Press, 1983
Feldman-Stewart D, Kocovski N, McConnell B, et al: Perception of quantitative information for treatment decisions. Med Decis Making 20:228-238, 2000
Cleveland WS, McGill R: Graphical perception: Theory, experimentation, and application to the development of graphical methods. J Am Stat Assoc 79:531-554, 1984
Brundage MD, Leis A, Bezjak A, et al: Cancer patients' preferences for communicating clinical trial quality of life information: A qualitative study. Qual Life Res 12:395-404, 2002
Skeel RT: Quality of life dimensions that are most important to cancer patients. Oncology 7:55-61, 1969
Osoba D, Till JE, Pater JL, et al: Health-related quality of life: Measurement and clinical application: A workshop report. Can J Oncol 5:338-343, 1995
Elting LS, Martin CG, Cantor SB, et al: Influence of data display formats on physician investigators' decisions to stop clinical trials: Prospective trial with repeated measures. BMJ 318:1527-1531, 1999
Ottenbacher KJ: When is a picture worth a thousand p values? A comparison of visual and quantitative methods to analyze single subject data. J Spec Ed 23:436-449, 1990
Carswell CM: The art and psychological science of graphical communication. Psych Sci Agenda 1995;8-9
Fairclough DL, Gagnon DD, Zagari MJ, et al: Evaluation of quality of life in a clinical trial with non-random dropout: The effect of epoetin alfa in anemic cancer patients. Qual Life Res 12:1013-1027, 2003
Sprangers MA: Quality-of-life assessment in oncology: Acheivements and challenges. Acta Oncol 41:229-237, 2002(M. Brundage, D. Feldman-S)