Quality of Care in U.S. Hospitals as Reflected by Standardized Measures, 2002–2004
http://www.100md.com
《新英格兰医药杂志》
ABSTRACT
Background In July 2002, the Joint Commission on Accreditation of Healthcare Organizations implemented standardized performance measures that were designed to track the performance of accredited hospitals and encourage improvement in the quality of health care.
Methods We examined hospitals' performance on 18 standardized indicators of the quality of care for acute myocardial infarction, heart failure, and pneumonia. One measure assessed a clinical outcome (death in the hospital after acute myocardial infarction), and the other 17 measures assessed processes of care. Data were collected over a two-year period in more than 3000 accredited hospitals. All participating hospitals received quarterly feedback in the form of comparative reports throughout the study.
Results Descriptive analysis revealed a significant improvement (P<0.01) in the performance of U.S. hospitals on 15 of 18 measures, and no measure showed a significant deterioration. The magnitude of improvement ranged from 3 percent to 33 percent during the eight quarters studied. For 16 of the 17 process-of-care measures, hospitals with a low level of performance at baseline had greater improvements over the subsequent two years than hospitals with a high level of performance at baseline.
Conclusions Over a two-year period, we observed consistent improvement in measures reflecting the process of care for acute myocardial infarction, heart failure, and pneumonia. Both quantitative and qualitative research are needed to explore the reasons for these improvements.
The 2001 report of the Institute of Medicine, Crossing the Quality Chasm, indicated that the American health care system "is currently functioning at far lower levels than it could and should."1 To address shortcomings in the health care system, the report called for the "establishment of monitoring and tracking processes for use in evaluating the progress of the health system." The report emphasized the need to improve the effectiveness of health care through the consistent provision of services that are based on current scientific knowledge.
In 2003, the Agency for Healthcare Research and Quality (AHRQ) released the National Healthcare Quality Report (NHQR), which included results on a broad set of 57 performance measures that provided data on the trend in the quality of services for several clinical conditions.2 Although improvement was reported in 20 of the 57 measures for which trend data were available, the use of disparate, preexisting data sources limited the analysis and skewed the representativeness of some samples. In 2003, Jencks et al. reported on the positive changes in care delivered to Medicare beneficiaries, on the basis of data collected during two periods.3 These results, however, could not be confidently generalized beyond the Medicare population. In addition, neither the AHRQ nor the Centers for Medicare and Medicaid Services (CMS) provided feedback to contributing hospitals on their performance as a tool for continual quality improvement.
In 2002, the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) implemented evidence-based standardized measures of performance in over 3000 accredited hospitals. The measures were designed to track hospitals' performance over time and encourage improvement through quarterly feedback in the form of comparative reports to all participating hospitals. Both qualitative and quantitative studies have demonstrated benefits associated with providing hospitals regular feedback on their performance on quality measures.4,5,6,7 Comparative feedback has been particularly useful at an organizational level as a guide for improvement-oriented activities.4,8,9
While focusing on accredited hospitals, this report expands on the earlier work of the NHQR and CMS in three important ways. First, we did not limit patient populations to Medicare beneficiaries; rather, we included all patients. Second, data collected during the study were made available to hospitals through formal quarterly feedback reports, allowing hospitals to monitor their performance over time in comparison to national rates. Third, our use of hospital-level longitudinal analysis allowed us to compare the rates of change between hospitals that began the study with a low level of performance and those that began with a high level of performance.
Methods
Participants
JCAHO accreditation accounts for more than 90 percent of the acute care medical–surgical hospital beds in the United States.10 In July 2002, the JCAHO required 3377 of its 4644 accredited hospitals to submit data on standardized performance measures on their choice of at least two of the four initially available sets of measures: acute myocardial infarction, heart failure, pneumonia, and pregnancy and related conditions. (The pregnancy measures were not included in this analysis because two of the three measures address rare events and the third is a subject of controversy.) The 27 percent of accredited hospitals exempted from this requirement either did not provide services addressed by any of the measure sets (e.g., psychiatric hospitals or specialty hospitals) or had an average daily census of fewer than 10 patients.
Measures and Data Collection
The standardized performance measures for the quality of care for acute myocardial infarction, heart failure, and pneumonia (Table 1) were composed of precisely defined specifications and standardized data-collection protocols based on uniform medical language. The measures were designed to permit valid comparisons of health care organizations through the establishment of a national comparative database. Measures were identified with respect to published scientific evidence and consistency with established clinical-practice guidelines, since considerable gaps still exist between the practices described in clinical guidelines and the degree to which these practices are implemented during actual treatment.11,12,13,14
Table 1. Core Measures of the JCAHO for the Quality of Care for Acute Myocardial Infarction, Heart Failure, and Pneumonia.
The JCAHO performed a pilot test of the measures from January through December 2001 through a collaborative effort among five state hospital associations and 83 hospitals in nine states.15 After the national implementation of these measures, the reliability of the approach was reassessed with the use of on-site reabstraction of medical records to evaluate the accuracy of the individual data elements collected in the third quarter of 2002. The average rates of agreement exceeded 90 percent.16
The JCAHO required accredited hospitals to collect data on performance measures for all eligible patients through the abstraction of medical records and, where applicable, the use of administrative or billing data. In a small number of hospitals (approximately 2 percent each of hospitals submitting data on acute myocardial infarction, heart failure, and pneumonia), patients' records were randomly sampled from all eligible patients. Only hospitals with at least 75 eligible patients per month were allowed to use sampling. Once these data were collected, hospitals submitted patient-level data to a third-party vendor, which compiled and transmitted hospital-level data to the JCAHO on a quarterly basis.17,18 All participating hospitals received comparative feedback reports meeting standardized specifications on a quarterly basis. These reports were supplied by the third-party vendors and included, at a minimum, control charts to track variations in a hospital's performance over time and comparison charts to compare a hospital's rates for each measure against the national averages. Vendors often provided additional feedback for their hospital clients.
Statistical Analysis
For each measure, quarterly rates or means were calculated. Rates were based on a quarterly aggregation of data from all eligible patients. Rate-based measures are presented as a proportion in which the number of patients meeting the criteria for a specific measure is divided by the total number of patients. For continuous variables, national means were based on an aggregation of hospital means, weighted according to the number of patients, rather than a simple grand mean. Continuous variables are presented as a mean value (i.e., the number of minutes) for all patients who qualified for a given measure. For these measures, 2 percent of the aggregated data were identified as extreme outliers (i.e., monthly data points exceeding a threshold in which the mean time to thrombolysis was greater than 6 hours, the mean time to percutaneous coronary intervention was greater than 24 hours, or the mean time to the administration of antibiotics for pneumonia was greater than 36 hours) and were removed from the analysis. The national trends were analyzed with the use of ordinary least-squares regression analysis to quantify the linear change over time, expressed as change per quarter on the percent scale. For rate-based measures that have a high overall rate of performance, there was evidence that rates approached an upper asymptote over time. For these measures, a nonlinear three-parameter logistic curve was fitted to the data to quantify this upper asymptote. To compare the time trend on the measure of smoking-cessation counseling or advice, an analysis of covariance was used.19
For the analysis of hospital temporal trends, a mixed random-coefficients model analysis was used to assess the time trend, with the specific form of the analysis depending on the type of measure being analyzed: generalized linear mixed models were used to analyze binomial counts for the rate-based measures,20 normal mixed random-coefficient models were weighted with the use of the variance of each data point for the continuous variables,21 and a Poisson general linear mixed model was used for the risk-adjusted measures.22 The influence of baseline values of a measure on the change in hospitals' performance over time was assessed by adding an independent baseline variable to the model, as well as an interaction between this baseline variable and the linear effect of time. The interpretation of this interaction is the change in the linear effect of time per unit change in the baseline value of the measure. All reported P values are two-sided and are not adjusted for multiple testing.
Results
To ensure longitudinal comparability, we limited the analysis to hospitals that submitted data from the third quarter of 2002 (the first quarter of the study) through the second quarter of 2004 (the eighth and final quarter of the study). As a result, of the 3377 hospitals initially identified as participants, 3087 were included in the analysis; 1473 of the 3087 hospitals submitted data on acute myocardial infarction (only 1258 submitted data for the mean time to thrombolysis, and only 688 submitted data for the mean time to percutaneous coronary intervention, since not all hospitals submitting data on acute myocardial infarction provided these services), 1946 hospitals submitted data on heart failure, and 1797 submitted data on pneumonia. The decision to limit our analysis to hospitals with complete data sets led to the removal of 69 hospitals from the analysis of data on acute myocardial infarction, 82 hospitals from the analysis of data on heart failure, and 95 hospitals from the analysis of data on pneumonia. These hospitals had very small samples, leading to particular quarters in which no cases of these conditions were reported.
National Analysis
National rates for each measure are shown according to quarter in Figure 1 and Table 2. Rate-based measures are shown for each condition. The three measures expressed as continuous variables are displayed together in Figure 1D. The national rates and means include data from all participating hospitals. On a national scale, performance for 15 of the 18 standardized measures demonstrated a significant trend of improvement (change per quarter) over the eight-quarter period (P<0.01), and no measure showed significant deterioration. The overall rates for four of the measures for acute myocardial infarction and one of the pneumonia measures approached an upper limit: aspirin at admission (96 percent), aspirin at discharge (96 percent), beta-blocker at admission (95 percent), beta-blocker at discharge (96 percent), and oxygenation assessment (99 percent).
Figure 1. Trends in the Measures of the Quality of Care for Acute Myocardial Infarction (Panels A and D), Heart Failure (Panel B), and Pneumonia (Panels C and D) at U.S. Hospitals from July 2002 to June 2004.
Data in Panels A, B, and C are based on aggregate calculations for rate-based measures (dividing the number of patients who met the criterion by the total number of patients) for all participating hospitals. An improvement is reflected by a positive slope for all rate-based measures except inpatient death after acute myocardial infarction, for which an improvement is reflected by a negative slope. Panel D shows weighted mean values (in minutes) for the continuous variables. Mean values for each hospital were weighted by the total number of patients included by each hospital. For this panel, an improvement is reflected by a negative slope. ACE denotes angiotensin-converting enzyme, LV left ventricular, and PCI percutaneous coronary intervention.
Table 2. Mean Values and Overall Changes in Measures of the Quality of Care during the Eight Quarters.
Among the 18 measures studied, the most dramatic improvement occurred in the three measures of counseling for smoking cessation. A 19 percent, 32 percent, and 33 percent absolute difference from the first to the last quarter studied was observed for acute myocardial infarction, heart failure, and pneumonia, respectively. Interestingly, comparisons of the measures of counseling for smoking cessation, which were identically defined in the three categories, revealed significant differences in the national performance rates in each. Performance on the measure of smoking-cessation counseling for acute myocardial infarction was superior to that for heart failure, which was, in turn, better than that for pneumonia (P<0.001). The rates of change (slope) over the eight quarters were also significantly different (P<0.001) between the measures of counseling for smoking cessation for acute myocardial infarction (3 percent per quarter) and the measures of counseling for smoking cessation for heart failure and pneumonia (4 percent and 5 percent per quarter, respectively).
Hospital-Level Analysis
The analyses involving a mixed random-coefficients model demonstrated that the degree of hospitals' improvement was significantly positively associated with baseline performance and linear time for all rate-based process measures and two of the three measures expressed as continuous variables (P<0.05). Only the mean time to thrombolysis did not reveal a significant relationship between baseline performance and linear time. More simply stated, the performance of hospitals generally tended to improve over time, and hospitals that began the study with lower baseline rates tended to improve at faster rates than hospitals with higher baseline rates. Table 3 illustrates this trend by stratifying hospitals into three groups according to their baseline percentile ranks for each measure.
Table 3. Average Change in Measures of the Quality of Care over Time, According to Baseline Performance.
Discussion
Our data demonstrate a steady improvement in the performance of U.S. hospitals over a period of eight quarters in measures reflecting the quality of care for acute myocardial infarction, heart failure, and pneumonia. Improvement was observed in 15 of 18 measures, including 3 measures that already had mean performance rates of over 90 percent in the first quarter (e.g., the rate for aspirin at admission was 93 percent in the third quarter of 2002). Moreover, hospital-level analysis revealed that, for 16 of the 17 process measures, hospitals that began the study as low-level performers tended to improve at faster rates than those that started the study with higher levels of performance. With each passing quarter, low-level performers improved more quickly. In contrast, high-level performers generally maintained their high level of performance or improved at slower rates.
The faster rate of improvement among low-level performers represents an important finding. Whereas low-level performers have the most room for improvement, one might have expected different results, since such hospitals may be less likely to focus on quality or make an effort to improve performance than their counterparts with a higher level of performance. Our results support the results of previous work and lend support to the use of these measures as a means for encouraging improvement in hospitals and as tools for monitoring hospitals' performance, as called for by the Institute of Medicine.1,3,7,8,9,23 Receiving quarterly national comparative data may have been an added stimulus for poor-performing hospitals to improve.
The improvement observed in some measures may have resulted in part from increased attention to documentation, rather than better patient care. Although this possibility cannot be definitively discounted, it could not explain the reductions noted in the time to thrombolysis and time to percutaneous coronary intervention.
A review of the measures of the quality of care for acute myocardial infarction reveals an apparent contradiction between improvement in the process measures and the lack of improvement in the inpatient-death measure. This discrepancy is misleading for two important reasons. First, it is highly unlikely that improvements in these process measures would have an important effect on inpatient death. In fact, four of the eight process measures for acute myocardial infarction address discharge activities, and the patient populations targeted by the measures for the timing of thrombolytic therapy and percutaneous coronary intervention, which one might reasonably expect to be correlated with inpatient death, represent only a small fraction of the patients included in the inpatient-death measure. Second, the process measures were selected because they had a scientific evidence base, established through randomized clinical trials, that demonstrated their relationship to multiple measures of outcome. In such instances, process measures can be more sensitive to differences in quality than comparisons of outcomes.24 The inpatient-death measure would therefore be expected to provide important information about an individual hospital's performance but would not be expected to mirror trends observed for the process measures.
This study has several limitations. First, it is dependent on self-reported data from hospitals. Although the reliability of the data was evaluated twice, once in the pilot test of the measures and a second time after national implementation of the approach,15,16 the nature of self-reported data provides an opportunity to introduce bias into the results. However, the results are consistent with findings reported in studies that used data collected by independent sources.3,6,7,23 The Veterans Affairs health care system, for example, used similar measures, collected by independent abstractors, to track changes in performance after the implementation of a systemwide reengineering program. Over a four-year period, dramatic improvements in the quality of care were observed.25 Second, although improvements must have been due to hospital-based efforts to upgrade the quality of care, there is no way to determine the degree to which feedback on performance measures stimulated these improvement initiatives. Certainly, the national attention directed at these high-risk, problem-prone patient populations by the CMS, the National Quality Forum, the JCAHO, and others contributed to the observed improvement, independently of the availability of feedback on performance measures. In this issue of the Journal, Jha et al., who analyzed 10 measures from the CMS Hospital Quality Alliance, also reported modest differences in the performance of hospitals on the basis of specific characteristics, such as geographic location, teaching status, and for-profit or not-for-profit status.26 These easily observable demographic characteristics, however, did not account for the majority of the variation in quality observed on the measures. Both quantitative and qualitative research are needed to evaluate the reasons for differences in hospital performance and improvement. Our study included only hospitals that were accredited by the JCAHO and excluded hospitals with an average daily census of fewer than 10 patients. As a result, our findings may underrepresent the performance of very small hospitals.
Although the impetus for the improvement cannot be pinpointed, the improvement in measures reflecting the quality of care for acute myocardial infarction, heart failure, and pneumonia remains very encouraging. If the current rates of change were to be maintained (which is by no means a certainty), the mean performance for hospitals that began the study in the lowest performance quartile would be expected to reach rates of 90 percent for 11 of the 14 rate-based process measures by the first quarter of 2006. Given the very low starting point or slow rate of improvement observed for the remaining three rate-based process measures (discharge instructions, pneumococcal screening or vaccination, and blood cultures), the 90 percent mark would probably not be reached until 2007 or 2008. It is also important to note that the data were largely collected before the widespread public reporting of hospital data or the implementation of pay-for-performance initiatives. The role and influence of public reporting is a widely debated subject that varies depending on the intended audience and the purpose of the reporting.9,23,27,28 As the JCAHO's database of performance measures expands with each passing quarter, it will offer an opportunity to track the effect of national public reporting and pay-for-performance initiatives to a degree that once was not possible.
Source Information
From the Joint Commission on Accreditation of Healthcare Organizations, Division of Research, Oakbrook Terrace, Ill.
Address reprint requests to Dr. Williams at the Joint Commission on Accreditation of Healthcare Organizations, 1 Renaissance Blvd., Oakbrook Terrace, IL 60181, or at swilliams@jcaho.org.
References
Institute of Medicine. Crossing the quality chasm: a new health system for the 21st century. Washington, D.C.: National Academies Press, 2001.
Agency for Healthcare Research and Quality. 2004 National Healthcare Quality Report. (Accessed June 27, 2005, at http://www.qualitytools.ahrq.gov/qualityreport/download_report.aspx.)
Jencks SF, Huff ED, Cuerdon T. Change in the quality of care delivered to Medicare beneficiaries, 1998-1999 to 2000-2001. JAMA 2003;289:305-312.
Bradley EH, Holmboe ES, Mattera JA, Roumanis SA, Radford MJ, Krumholz HM. Data feedback efforts in quality improvement: lessons learned from US hospitals. Qual Saf Health Care 2004;13:26-31.
Bradley EH, Holmboe ES, Mattera JA, Roumanis SA, Radford MJ, Krumholz HM. A qualitative study of increasing beta-blocker use after myocardial infarction: why do some hospitals succeed? JAMA 2001;285:2604-2611.
Schade CP, Cochran BF, Stephens MK. Using statewide audit and feedback to improve hospital care in West Virginia. Jt Comm J Qual Saf 2004;30:143-151.
Marciniak TA, Ellerbeck EF, Radford MJ, et al. Improving the quality of care for Medicare patients with acute myocardial infarction: results from the Cooperative Cardiovascular Project. JAMA 1998;279:1351-1357.
Gibberd R, Hancock S, Howley P, Richards K. Using indicators to quantify the potential to improve the quality of health care. Int J Qual Health Care 2004;16:Suppl 1:i37-i43.
Marshall MN, Shekelle PG, Leatherman S, Brook RH. The public release of performance data: what do we expect to gain? A review of the evidence. JAMA 2000;283:1866-1874.
Healthcare Quick Disk 2005: AHA annual survey database for fiscal year 2003. Chicago: American Hospital Association, 2005.
Antman EM, Anbe DT, Armstrong PW, et al. ACC/AHA guidelines for the management of patients with ST elevation myocardial infarction -- executive summary: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Writing Committee to revise the 1999 guidelines for the management of patients with acute myocardial infarction). J Am Coll Cardiol 2004;44:671-719.
Hunt SA, Baker DW, Chin MH, et al. ACC/AHA guidelines for the evaluation and management of chronic heart failure in the adult: executive summary: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee to Revise the 1995 Guidelines for the Evaluation and Management of Heart Failure). J Am Coll Cardiol 2001;38:2101-2113.
Bartlett JG, Breiman RF, Mandell LA, File TM Jr. Community-acquired pneumonia in adults: guidelines for management. Clin Infect Dis 1998;26:811-838.
McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med 2003;348:2635-2645.
Joint Commission on Accreditation of Healthcare Organizations. A comprehensive review of development and testing for national implementation of hospital core measures. (Accessed June 27, 2005, at http://www.jcaho.org/pms/core+measures/cr_hos_cm.htm.)
Watt A, Williams S, Lee K, Robertson J, Koss RG, Loeb JM. Keen eye on core measures: Joint Commission data quality study offers insights into data collection, abstracting processes. J AHIMA 2003;74:20-25.
Braun BI, Koss RG, Loeb JM. Integrating performance measure data into the Joint Commission accreditation process. Eval Health Prof 1999;22:283-297.
Joint Commission on Accreditation of Healthcare Organizations. Performance measurement system requirements for ORYX listing. (Accessed June 27, 2005, at http://www.jcaho.org/pms/reference+materials/pms+requirements.htm.)
Winer BJ. Statistical principles in experimental design. 2nd ed. New York: McGraw-Hill, 1971.
Breslow NE, Clayton DG. Approximate inference in generalized linear mixed models. J Am Stat Assoc 1993;88:9-25.
SAS/STAT user's guide, version 8. Cary, N.C.: SAS Institute, 2002.
Clayton D, Kaldor J. Empirical Bayes estimates of age-standardized relative risks for use in disease mapping. Biometrics 1987;43:671-681.
Hibbard JH, Stockard J, Tusler M. Does publicizing hospital performance stimulate quality improvement efforts? Health Aff (Millwood) 2003;22:84-94.
Palmer RH. Using health outcomes data to compare plans, networks and providers. Int J Qual Health Care 1998;10:477-483.
Jha AK, Perlin JB, Kizer KW, Dudley RA. Effect of the transformation of the Veterans Affairs health care system on the quality of care. N Engl J Med 2003;348:2218-2227.
Jha AK, Zhonghe L, Orav EJ, et al. Care in U.S. hospitals -- the Hospital Quality Alliance program. N Engl J Med 2005;353:265-274.(Scott C. Williams, Psy.D.)
Background In July 2002, the Joint Commission on Accreditation of Healthcare Organizations implemented standardized performance measures that were designed to track the performance of accredited hospitals and encourage improvement in the quality of health care.
Methods We examined hospitals' performance on 18 standardized indicators of the quality of care for acute myocardial infarction, heart failure, and pneumonia. One measure assessed a clinical outcome (death in the hospital after acute myocardial infarction), and the other 17 measures assessed processes of care. Data were collected over a two-year period in more than 3000 accredited hospitals. All participating hospitals received quarterly feedback in the form of comparative reports throughout the study.
Results Descriptive analysis revealed a significant improvement (P<0.01) in the performance of U.S. hospitals on 15 of 18 measures, and no measure showed a significant deterioration. The magnitude of improvement ranged from 3 percent to 33 percent during the eight quarters studied. For 16 of the 17 process-of-care measures, hospitals with a low level of performance at baseline had greater improvements over the subsequent two years than hospitals with a high level of performance at baseline.
Conclusions Over a two-year period, we observed consistent improvement in measures reflecting the process of care for acute myocardial infarction, heart failure, and pneumonia. Both quantitative and qualitative research are needed to explore the reasons for these improvements.
The 2001 report of the Institute of Medicine, Crossing the Quality Chasm, indicated that the American health care system "is currently functioning at far lower levels than it could and should."1 To address shortcomings in the health care system, the report called for the "establishment of monitoring and tracking processes for use in evaluating the progress of the health system." The report emphasized the need to improve the effectiveness of health care through the consistent provision of services that are based on current scientific knowledge.
In 2003, the Agency for Healthcare Research and Quality (AHRQ) released the National Healthcare Quality Report (NHQR), which included results on a broad set of 57 performance measures that provided data on the trend in the quality of services for several clinical conditions.2 Although improvement was reported in 20 of the 57 measures for which trend data were available, the use of disparate, preexisting data sources limited the analysis and skewed the representativeness of some samples. In 2003, Jencks et al. reported on the positive changes in care delivered to Medicare beneficiaries, on the basis of data collected during two periods.3 These results, however, could not be confidently generalized beyond the Medicare population. In addition, neither the AHRQ nor the Centers for Medicare and Medicaid Services (CMS) provided feedback to contributing hospitals on their performance as a tool for continual quality improvement.
In 2002, the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) implemented evidence-based standardized measures of performance in over 3000 accredited hospitals. The measures were designed to track hospitals' performance over time and encourage improvement through quarterly feedback in the form of comparative reports to all participating hospitals. Both qualitative and quantitative studies have demonstrated benefits associated with providing hospitals regular feedback on their performance on quality measures.4,5,6,7 Comparative feedback has been particularly useful at an organizational level as a guide for improvement-oriented activities.4,8,9
While focusing on accredited hospitals, this report expands on the earlier work of the NHQR and CMS in three important ways. First, we did not limit patient populations to Medicare beneficiaries; rather, we included all patients. Second, data collected during the study were made available to hospitals through formal quarterly feedback reports, allowing hospitals to monitor their performance over time in comparison to national rates. Third, our use of hospital-level longitudinal analysis allowed us to compare the rates of change between hospitals that began the study with a low level of performance and those that began with a high level of performance.
Methods
Participants
JCAHO accreditation accounts for more than 90 percent of the acute care medical–surgical hospital beds in the United States.10 In July 2002, the JCAHO required 3377 of its 4644 accredited hospitals to submit data on standardized performance measures on their choice of at least two of the four initially available sets of measures: acute myocardial infarction, heart failure, pneumonia, and pregnancy and related conditions. (The pregnancy measures were not included in this analysis because two of the three measures address rare events and the third is a subject of controversy.) The 27 percent of accredited hospitals exempted from this requirement either did not provide services addressed by any of the measure sets (e.g., psychiatric hospitals or specialty hospitals) or had an average daily census of fewer than 10 patients.
Measures and Data Collection
The standardized performance measures for the quality of care for acute myocardial infarction, heart failure, and pneumonia (Table 1) were composed of precisely defined specifications and standardized data-collection protocols based on uniform medical language. The measures were designed to permit valid comparisons of health care organizations through the establishment of a national comparative database. Measures were identified with respect to published scientific evidence and consistency with established clinical-practice guidelines, since considerable gaps still exist between the practices described in clinical guidelines and the degree to which these practices are implemented during actual treatment.11,12,13,14
Table 1. Core Measures of the JCAHO for the Quality of Care for Acute Myocardial Infarction, Heart Failure, and Pneumonia.
The JCAHO performed a pilot test of the measures from January through December 2001 through a collaborative effort among five state hospital associations and 83 hospitals in nine states.15 After the national implementation of these measures, the reliability of the approach was reassessed with the use of on-site reabstraction of medical records to evaluate the accuracy of the individual data elements collected in the third quarter of 2002. The average rates of agreement exceeded 90 percent.16
The JCAHO required accredited hospitals to collect data on performance measures for all eligible patients through the abstraction of medical records and, where applicable, the use of administrative or billing data. In a small number of hospitals (approximately 2 percent each of hospitals submitting data on acute myocardial infarction, heart failure, and pneumonia), patients' records were randomly sampled from all eligible patients. Only hospitals with at least 75 eligible patients per month were allowed to use sampling. Once these data were collected, hospitals submitted patient-level data to a third-party vendor, which compiled and transmitted hospital-level data to the JCAHO on a quarterly basis.17,18 All participating hospitals received comparative feedback reports meeting standardized specifications on a quarterly basis. These reports were supplied by the third-party vendors and included, at a minimum, control charts to track variations in a hospital's performance over time and comparison charts to compare a hospital's rates for each measure against the national averages. Vendors often provided additional feedback for their hospital clients.
Statistical Analysis
For each measure, quarterly rates or means were calculated. Rates were based on a quarterly aggregation of data from all eligible patients. Rate-based measures are presented as a proportion in which the number of patients meeting the criteria for a specific measure is divided by the total number of patients. For continuous variables, national means were based on an aggregation of hospital means, weighted according to the number of patients, rather than a simple grand mean. Continuous variables are presented as a mean value (i.e., the number of minutes) for all patients who qualified for a given measure. For these measures, 2 percent of the aggregated data were identified as extreme outliers (i.e., monthly data points exceeding a threshold in which the mean time to thrombolysis was greater than 6 hours, the mean time to percutaneous coronary intervention was greater than 24 hours, or the mean time to the administration of antibiotics for pneumonia was greater than 36 hours) and were removed from the analysis. The national trends were analyzed with the use of ordinary least-squares regression analysis to quantify the linear change over time, expressed as change per quarter on the percent scale. For rate-based measures that have a high overall rate of performance, there was evidence that rates approached an upper asymptote over time. For these measures, a nonlinear three-parameter logistic curve was fitted to the data to quantify this upper asymptote. To compare the time trend on the measure of smoking-cessation counseling or advice, an analysis of covariance was used.19
For the analysis of hospital temporal trends, a mixed random-coefficients model analysis was used to assess the time trend, with the specific form of the analysis depending on the type of measure being analyzed: generalized linear mixed models were used to analyze binomial counts for the rate-based measures,20 normal mixed random-coefficient models were weighted with the use of the variance of each data point for the continuous variables,21 and a Poisson general linear mixed model was used for the risk-adjusted measures.22 The influence of baseline values of a measure on the change in hospitals' performance over time was assessed by adding an independent baseline variable to the model, as well as an interaction between this baseline variable and the linear effect of time. The interpretation of this interaction is the change in the linear effect of time per unit change in the baseline value of the measure. All reported P values are two-sided and are not adjusted for multiple testing.
Results
To ensure longitudinal comparability, we limited the analysis to hospitals that submitted data from the third quarter of 2002 (the first quarter of the study) through the second quarter of 2004 (the eighth and final quarter of the study). As a result, of the 3377 hospitals initially identified as participants, 3087 were included in the analysis; 1473 of the 3087 hospitals submitted data on acute myocardial infarction (only 1258 submitted data for the mean time to thrombolysis, and only 688 submitted data for the mean time to percutaneous coronary intervention, since not all hospitals submitting data on acute myocardial infarction provided these services), 1946 hospitals submitted data on heart failure, and 1797 submitted data on pneumonia. The decision to limit our analysis to hospitals with complete data sets led to the removal of 69 hospitals from the analysis of data on acute myocardial infarction, 82 hospitals from the analysis of data on heart failure, and 95 hospitals from the analysis of data on pneumonia. These hospitals had very small samples, leading to particular quarters in which no cases of these conditions were reported.
National Analysis
National rates for each measure are shown according to quarter in Figure 1 and Table 2. Rate-based measures are shown for each condition. The three measures expressed as continuous variables are displayed together in Figure 1D. The national rates and means include data from all participating hospitals. On a national scale, performance for 15 of the 18 standardized measures demonstrated a significant trend of improvement (change per quarter) over the eight-quarter period (P<0.01), and no measure showed significant deterioration. The overall rates for four of the measures for acute myocardial infarction and one of the pneumonia measures approached an upper limit: aspirin at admission (96 percent), aspirin at discharge (96 percent), beta-blocker at admission (95 percent), beta-blocker at discharge (96 percent), and oxygenation assessment (99 percent).
Figure 1. Trends in the Measures of the Quality of Care for Acute Myocardial Infarction (Panels A and D), Heart Failure (Panel B), and Pneumonia (Panels C and D) at U.S. Hospitals from July 2002 to June 2004.
Data in Panels A, B, and C are based on aggregate calculations for rate-based measures (dividing the number of patients who met the criterion by the total number of patients) for all participating hospitals. An improvement is reflected by a positive slope for all rate-based measures except inpatient death after acute myocardial infarction, for which an improvement is reflected by a negative slope. Panel D shows weighted mean values (in minutes) for the continuous variables. Mean values for each hospital were weighted by the total number of patients included by each hospital. For this panel, an improvement is reflected by a negative slope. ACE denotes angiotensin-converting enzyme, LV left ventricular, and PCI percutaneous coronary intervention.
Table 2. Mean Values and Overall Changes in Measures of the Quality of Care during the Eight Quarters.
Among the 18 measures studied, the most dramatic improvement occurred in the three measures of counseling for smoking cessation. A 19 percent, 32 percent, and 33 percent absolute difference from the first to the last quarter studied was observed for acute myocardial infarction, heart failure, and pneumonia, respectively. Interestingly, comparisons of the measures of counseling for smoking cessation, which were identically defined in the three categories, revealed significant differences in the national performance rates in each. Performance on the measure of smoking-cessation counseling for acute myocardial infarction was superior to that for heart failure, which was, in turn, better than that for pneumonia (P<0.001). The rates of change (slope) over the eight quarters were also significantly different (P<0.001) between the measures of counseling for smoking cessation for acute myocardial infarction (3 percent per quarter) and the measures of counseling for smoking cessation for heart failure and pneumonia (4 percent and 5 percent per quarter, respectively).
Hospital-Level Analysis
The analyses involving a mixed random-coefficients model demonstrated that the degree of hospitals' improvement was significantly positively associated with baseline performance and linear time for all rate-based process measures and two of the three measures expressed as continuous variables (P<0.05). Only the mean time to thrombolysis did not reveal a significant relationship between baseline performance and linear time. More simply stated, the performance of hospitals generally tended to improve over time, and hospitals that began the study with lower baseline rates tended to improve at faster rates than hospitals with higher baseline rates. Table 3 illustrates this trend by stratifying hospitals into three groups according to their baseline percentile ranks for each measure.
Table 3. Average Change in Measures of the Quality of Care over Time, According to Baseline Performance.
Discussion
Our data demonstrate a steady improvement in the performance of U.S. hospitals over a period of eight quarters in measures reflecting the quality of care for acute myocardial infarction, heart failure, and pneumonia. Improvement was observed in 15 of 18 measures, including 3 measures that already had mean performance rates of over 90 percent in the first quarter (e.g., the rate for aspirin at admission was 93 percent in the third quarter of 2002). Moreover, hospital-level analysis revealed that, for 16 of the 17 process measures, hospitals that began the study as low-level performers tended to improve at faster rates than those that started the study with higher levels of performance. With each passing quarter, low-level performers improved more quickly. In contrast, high-level performers generally maintained their high level of performance or improved at slower rates.
The faster rate of improvement among low-level performers represents an important finding. Whereas low-level performers have the most room for improvement, one might have expected different results, since such hospitals may be less likely to focus on quality or make an effort to improve performance than their counterparts with a higher level of performance. Our results support the results of previous work and lend support to the use of these measures as a means for encouraging improvement in hospitals and as tools for monitoring hospitals' performance, as called for by the Institute of Medicine.1,3,7,8,9,23 Receiving quarterly national comparative data may have been an added stimulus for poor-performing hospitals to improve.
The improvement observed in some measures may have resulted in part from increased attention to documentation, rather than better patient care. Although this possibility cannot be definitively discounted, it could not explain the reductions noted in the time to thrombolysis and time to percutaneous coronary intervention.
A review of the measures of the quality of care for acute myocardial infarction reveals an apparent contradiction between improvement in the process measures and the lack of improvement in the inpatient-death measure. This discrepancy is misleading for two important reasons. First, it is highly unlikely that improvements in these process measures would have an important effect on inpatient death. In fact, four of the eight process measures for acute myocardial infarction address discharge activities, and the patient populations targeted by the measures for the timing of thrombolytic therapy and percutaneous coronary intervention, which one might reasonably expect to be correlated with inpatient death, represent only a small fraction of the patients included in the inpatient-death measure. Second, the process measures were selected because they had a scientific evidence base, established through randomized clinical trials, that demonstrated their relationship to multiple measures of outcome. In such instances, process measures can be more sensitive to differences in quality than comparisons of outcomes.24 The inpatient-death measure would therefore be expected to provide important information about an individual hospital's performance but would not be expected to mirror trends observed for the process measures.
This study has several limitations. First, it is dependent on self-reported data from hospitals. Although the reliability of the data was evaluated twice, once in the pilot test of the measures and a second time after national implementation of the approach,15,16 the nature of self-reported data provides an opportunity to introduce bias into the results. However, the results are consistent with findings reported in studies that used data collected by independent sources.3,6,7,23 The Veterans Affairs health care system, for example, used similar measures, collected by independent abstractors, to track changes in performance after the implementation of a systemwide reengineering program. Over a four-year period, dramatic improvements in the quality of care were observed.25 Second, although improvements must have been due to hospital-based efforts to upgrade the quality of care, there is no way to determine the degree to which feedback on performance measures stimulated these improvement initiatives. Certainly, the national attention directed at these high-risk, problem-prone patient populations by the CMS, the National Quality Forum, the JCAHO, and others contributed to the observed improvement, independently of the availability of feedback on performance measures. In this issue of the Journal, Jha et al., who analyzed 10 measures from the CMS Hospital Quality Alliance, also reported modest differences in the performance of hospitals on the basis of specific characteristics, such as geographic location, teaching status, and for-profit or not-for-profit status.26 These easily observable demographic characteristics, however, did not account for the majority of the variation in quality observed on the measures. Both quantitative and qualitative research are needed to evaluate the reasons for differences in hospital performance and improvement. Our study included only hospitals that were accredited by the JCAHO and excluded hospitals with an average daily census of fewer than 10 patients. As a result, our findings may underrepresent the performance of very small hospitals.
Although the impetus for the improvement cannot be pinpointed, the improvement in measures reflecting the quality of care for acute myocardial infarction, heart failure, and pneumonia remains very encouraging. If the current rates of change were to be maintained (which is by no means a certainty), the mean performance for hospitals that began the study in the lowest performance quartile would be expected to reach rates of 90 percent for 11 of the 14 rate-based process measures by the first quarter of 2006. Given the very low starting point or slow rate of improvement observed for the remaining three rate-based process measures (discharge instructions, pneumococcal screening or vaccination, and blood cultures), the 90 percent mark would probably not be reached until 2007 or 2008. It is also important to note that the data were largely collected before the widespread public reporting of hospital data or the implementation of pay-for-performance initiatives. The role and influence of public reporting is a widely debated subject that varies depending on the intended audience and the purpose of the reporting.9,23,27,28 As the JCAHO's database of performance measures expands with each passing quarter, it will offer an opportunity to track the effect of national public reporting and pay-for-performance initiatives to a degree that once was not possible.
Source Information
From the Joint Commission on Accreditation of Healthcare Organizations, Division of Research, Oakbrook Terrace, Ill.
Address reprint requests to Dr. Williams at the Joint Commission on Accreditation of Healthcare Organizations, 1 Renaissance Blvd., Oakbrook Terrace, IL 60181, or at swilliams@jcaho.org.
References
Institute of Medicine. Crossing the quality chasm: a new health system for the 21st century. Washington, D.C.: National Academies Press, 2001.
Agency for Healthcare Research and Quality. 2004 National Healthcare Quality Report. (Accessed June 27, 2005, at http://www.qualitytools.ahrq.gov/qualityreport/download_report.aspx.)
Jencks SF, Huff ED, Cuerdon T. Change in the quality of care delivered to Medicare beneficiaries, 1998-1999 to 2000-2001. JAMA 2003;289:305-312.
Bradley EH, Holmboe ES, Mattera JA, Roumanis SA, Radford MJ, Krumholz HM. Data feedback efforts in quality improvement: lessons learned from US hospitals. Qual Saf Health Care 2004;13:26-31.
Bradley EH, Holmboe ES, Mattera JA, Roumanis SA, Radford MJ, Krumholz HM. A qualitative study of increasing beta-blocker use after myocardial infarction: why do some hospitals succeed? JAMA 2001;285:2604-2611.
Schade CP, Cochran BF, Stephens MK. Using statewide audit and feedback to improve hospital care in West Virginia. Jt Comm J Qual Saf 2004;30:143-151.
Marciniak TA, Ellerbeck EF, Radford MJ, et al. Improving the quality of care for Medicare patients with acute myocardial infarction: results from the Cooperative Cardiovascular Project. JAMA 1998;279:1351-1357.
Gibberd R, Hancock S, Howley P, Richards K. Using indicators to quantify the potential to improve the quality of health care. Int J Qual Health Care 2004;16:Suppl 1:i37-i43.
Marshall MN, Shekelle PG, Leatherman S, Brook RH. The public release of performance data: what do we expect to gain? A review of the evidence. JAMA 2000;283:1866-1874.
Healthcare Quick Disk 2005: AHA annual survey database for fiscal year 2003. Chicago: American Hospital Association, 2005.
Antman EM, Anbe DT, Armstrong PW, et al. ACC/AHA guidelines for the management of patients with ST elevation myocardial infarction -- executive summary: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Writing Committee to revise the 1999 guidelines for the management of patients with acute myocardial infarction). J Am Coll Cardiol 2004;44:671-719.
Hunt SA, Baker DW, Chin MH, et al. ACC/AHA guidelines for the evaluation and management of chronic heart failure in the adult: executive summary: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee to Revise the 1995 Guidelines for the Evaluation and Management of Heart Failure). J Am Coll Cardiol 2001;38:2101-2113.
Bartlett JG, Breiman RF, Mandell LA, File TM Jr. Community-acquired pneumonia in adults: guidelines for management. Clin Infect Dis 1998;26:811-838.
McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med 2003;348:2635-2645.
Joint Commission on Accreditation of Healthcare Organizations. A comprehensive review of development and testing for national implementation of hospital core measures. (Accessed June 27, 2005, at http://www.jcaho.org/pms/core+measures/cr_hos_cm.htm.)
Watt A, Williams S, Lee K, Robertson J, Koss RG, Loeb JM. Keen eye on core measures: Joint Commission data quality study offers insights into data collection, abstracting processes. J AHIMA 2003;74:20-25.
Braun BI, Koss RG, Loeb JM. Integrating performance measure data into the Joint Commission accreditation process. Eval Health Prof 1999;22:283-297.
Joint Commission on Accreditation of Healthcare Organizations. Performance measurement system requirements for ORYX listing. (Accessed June 27, 2005, at http://www.jcaho.org/pms/reference+materials/pms+requirements.htm.)
Winer BJ. Statistical principles in experimental design. 2nd ed. New York: McGraw-Hill, 1971.
Breslow NE, Clayton DG. Approximate inference in generalized linear mixed models. J Am Stat Assoc 1993;88:9-25.
SAS/STAT user's guide, version 8. Cary, N.C.: SAS Institute, 2002.
Clayton D, Kaldor J. Empirical Bayes estimates of age-standardized relative risks for use in disease mapping. Biometrics 1987;43:671-681.
Hibbard JH, Stockard J, Tusler M. Does publicizing hospital performance stimulate quality improvement efforts? Health Aff (Millwood) 2003;22:84-94.
Palmer RH. Using health outcomes data to compare plans, networks and providers. Int J Qual Health Care 1998;10:477-483.
Jha AK, Perlin JB, Kizer KW, Dudley RA. Effect of the transformation of the Veterans Affairs health care system on the quality of care. N Engl J Med 2003;348:2218-2227.
Jha AK, Zhonghe L, Orav EJ, et al. Care in U.S. hospitals -- the Hospital Quality Alliance program. N Engl J Med 2005;353:265-274.(Scott C. Williams, Psy.D.)