Relevance of the Agency for Healthcare Research and Quality Patient Safety Indicators for Children's Hospitals
http://www.100md.com
《小儿科》
University of Michigan Health System, Ann Arbor, Michigan
National Association of Children's Hospitals and Related Institutions, Alexandria, Virginia
Agency for Healthcare Research and Quality, Rockville, Maryland
Children's Hospital of Wisconsin, Wauwatosa, Wisconsin
ABSTRACT
Objectives. Patient safety indicators (PSIs) were developed by the Agency for Healthcare Research and Quality. Our objectives were (1) to apply these algorithms to the National Association of Children's Hospitals and Related Institutions (NACHRI) Aggregate Case Mix Comparative Database for 1999–2002, (2) to establish mean rates for each of the PSI events in children's hospitals, (3) to investigate the inadequacies of PSIs in relation to pediatric diagnoses, and (4) to express the data in such a way that children's hospitals could use the PSIs determined to be appropriate for pediatric use for comparison with their own data. In addition, we wanted to use the data to set priorities for ongoing clinical investigations and to propose interventions if the indicators demonstrated preventable errors.
Methods. The Agency for Healthcare Research and Quality PSI algorithms (version 2.1, revision 1) were applied to children's hospital administrative data (1.92 million discharges) from the NACHRI Aggregate Case Mix Comparative Database for 1999–2002. Rates were measured for the following events: complications of anesthesia, death in low-mortality diagnosis-related groups (DRGs), decubitus ulcer, failure to rescue (ie, death resulting from a complication, rather than the primary diagnosis), foreign body left in during a procedure, iatrogenic pneumothorax, infection attributable to medical care (ie, infections related to surgery or device placement), postoperative hemorrhage or hematoma, postoperative pulmonary embolism or venous thrombosis, postoperative wound dehiscence, and accidental puncture/laceration.
Results. Across the 4 years of data, the mean risk-adjusted rates of PSI events ranged from 0.01% (0.1 event per 1000 discharges) for a foreign body left in during a procedure to 14.0% (140 events per 1000 discharges) for failure to rescue. Review of International Classification of Diseases, Ninth Revision, Clinical Modification codes associated with each PSI category showed that the failure to rescue and death in low-mortality DRG indicators involved very complex cases and did not predict preventable events in the majority of cases. The PSI for infection attributable to medical care appeared to be accurate the majority of the time. Incident risk-adjusted rates of infections attributable to medical care averaged 0.35% (3.5 events per 1000 discharges) and varied up to fivefold from the lowest rate to the highest rate. The highest rates were up to 1.8 times the average.
Conclusions. PSIs derived from administrative data are indicators of patient safety concerns and can be relevant as screening tools for children's hospitals; however, cases identified by these indicators do not always represent preventable events. Some, such as a foreign body left in during a procedure, iatrogenic pneumothorax, infection attributable to medical care, decubitus ulcer, and venous thrombosis, seem to be appropriate for pediatric care and may be directly amenable to system changes. Evidence-based practices regarding those particular indicators that have been reported in the adult literature need to be investigated in the pediatric population. In their present form, 2 of the indicators, namely, failure to rescue and death in low-mortality DRGs, are inaccurate for the pediatric population, do not represent preventable errors in the majority of pediatric cases, and should not be used to estimate quality of care or preventable deaths in children's hospitals. The PSIs can assist institutions in prioritizing chart review-based investigations; if clusters of validated events emerge in reviews, then improvement activities can be initiated. Large aggregate databases, such as the NACHRI Case Mix Database, can help establish mean rates of potential pediatric events, giving children's hospitals a context within which to examine their own data.
Key Words: patient safety indicators Agency for Healthcare Research and Quality medical error National Association of Children's Hospitals and Related Institutions
Abbreviations: AHRQ, Agency for Healthcare Research and Quality PSI, patient safety indicator NACHRI, National Association of Children's Hospitals and Related Institutions DRG, diagnosis-related group ICD-9-CM, International Classification of Diseases, Ninth Revision, Clinical Modification HCUP, Healthcare Cost and Utilization Project KID, Kids' Inpatient Database
In 1999, the Institute of Medicine defined patient safety as "freedom from accidental injury because of medical care or medical errors" and charged hospitals to create a culture of safety in their institutions.1 Studies of pediatric medication errors have been published2; however, there is a paucity of data on pediatric patient injuries in children's hospitals.
The Agency for Healthcare Research and Quality (AHRQ) was charged by Congress to promote patient safety and to reduce medical errors through research and collaborative partnerships. One result of this mandate was the development of the AHRQ patient safety indicators (PSIs), ie, software that can identify potential adverse events from International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes in hospital inpatient administrative data. The University of California, San Francisco-Stanford University Evidence-Based Practice Center, under a contract with AHRQ, developed the PSIs in a 5-stage process.3 First, the literature was reviewed to identify a list of candidate indicators. Second, clinical panels evaluated the face validity or "sensibility" of the indicators, as judged by practicing physicians. Third, ICD-9-CM coding experts were consulted to confirm that the definition of each indicator reflected the intended clinical situation. Fourth, an empirical analysis of the chosen indicators was conducted with the Healthcare Cost and Utilization Project (HCUP) data.4 Fifth, PSI software and documentation were produced for public release by AHRQ. Since its initial development, this software has undergone several modifications, including refinement of the indicator definitions. Therefore, in comparisons of PSIs, it is very important to note whether the versions of the PSIs are the same.
The indicators were risk adjusted on the basis of patient age, patient gender, and AHRQ comorbidity groups, as determined with AHRQ comorbidity software, which uses ICD-9-CM codes to assign up to 30 comorbidity groups (such as hypertension and diabetes mellitus) to patient records.5
Extensive work was published in 2003 by Miller et al,6 describing pediatric patient safety with the use of the PSIs ( version) and administrative data from the 1997 HCUP.4 That study identified the high rates of birth trauma among newborns, with a rate of 1.5 cases per 100 births. Multivariate logistic regression analyses with the PSIs showed that patients with higher acuity exhibited higher rates of events. The study also demonstrated significant increases in length of stay and cost per case in relation to PSIs. The authors concluded "... that hospitalized children experience significant numbers of patient safety events, that these event rates are comparable to hospitalized adults, and that attention clearly needs to be paid to the unique event of childbirth."6(p1365) A follow-up study by Miller and Zhan7 with HCUP 2000 data confirmed the enormous burden of medical errors in pediatric care.
The HCUP database used by Miller et al6 is different from the National Association of Children's Hospitals and Related Institutions (NACHRI) database (Table 1). Specifically, the HCUP 2000 Kids' Inpatient Database (KID) includes a sample of pediatric discharges from community hospitals (short-term, nonfederal, general and specialty hospitals) drawn from 27 state inpatient databases on children 20 years of age (www.ahrq.gov/data/hcup). In comparison, the NACHRI Case Mix Database is a children's hospital-specific database (freestanding acute care and specialty children's hospitals and acute care children's hospitals located within larger institutions) that contains 100% of pediatric discharges from those institutions, excluding normal newborn and obstetric cases. Because of the differences in the databases and their potential effects on PSI rates, we considered it worthwhile to investigate the PSI algorithms specifically from a children's hospital perspective with the NACHRI Case Mix Database. Our objectives were (1) to apply the AHRQ PSI algorithms to the NACHRI Case Mix Database for 1999-2002, (2) to investigate PSI rates specifically in children's hospitals (with the major utility of the trend being to increase confidence in the data, because of the uniformity of the rates with time for each PSI), (3) to investigate any inadequacies of PSIs in relation to pediatric diagnoses, and (4) to express the data in such a way that individual children's hospitals could use the PSIs determined to be pediatric appropriate for comparison with their own data. In addition, we wished to use the data to set priorities for ongoing clinical investigation and to propose interventions.
METHODS
To understand patient safety events in children's hospitals and to prioritize future research, we applied the PSIs to the NACHRI Case Mix Database. The database contains administrative data at the ICD-9-CM level for inpatient discharges, with standard indicators such as cost per case, length of stay, readmission rate, mortality rate, and case mixture index (acuity level). It includes 4 years of data (1999–2002) and contains 1.92 million discharge records from children's hospitals representing 31 states. The number of hospitals included in the sample varied each year because new institutions joined the NACHRI Case Mix Program, with 50 hospitals in 1999, 56 in 2000, 65 in 2001, and 67 in 2002.
The AHRQ PSI software (version 2.1, revision 1) was used to analyze the data, and the resulting output (numerator, denominator, mean, observed rate, risk-adjusted rate, and smoothed rate) for each of the 11 PSIs with relevant pediatric volume was coupled with additional analyses with SPSS statistical software.8 Rates were measured for the following events: complications of anesthesia, death in a low-mortality diagnosis-related group (DRG), decubitus ulcer, failure to rescue (ie, death resulting from a complication rather than the primary diagnosis), foreign body left in during a procedure, iatrogenic pneumothorax, infection attributable to medical care (ie, infections related to surgery or device placement), postoperative hemorrhage or hematoma, postoperative pulmonary embolism or venous thrombosis, postoperative wound dehiscence, and accidental puncture/laceration.
The PSI software is very specific in defining the numerator and denominator for each indicator (see figure legends) and strictly limits the discharges that are considered for each patient safety event. For example, patients with paraplegia and quadriplegia are exempted from the decubitus measure, and patients with cancer and/or immunosuppression are excluded from the infection rate measure. The indicators are risk-adjusted on the basis of patient age, patient gender, and AHRQ comorbidity groups (ICD-9-CM codes are used to assign up to 30 comorbidity groups, such as hypertension and diabetes mellitus, to a patient record5). Depending on the covariate weight of these comorbidity groups, the rate of PSIs is adjusted up or down for each group analyzed. In this study, for example, the observed rate of decubiti was adjusted upward during the risk adjustment, in comparison with adult patients, who, because of diagnoses of vascular compromise and diabetes, are considered at higher risk for decubiti than are children.
The ICD-9-CM codes for each discharge identified as having a patient safety event were reviewed. This investigation of all of the principal and secondary diagnoses associated with each case was used to characterize complexities that influenced the indicators and allowed us to ascertain whether the patient safety events were potentially preventable.
RESULTS
The PSIs are also presented in Figs 1 to 11, which show the rate of the specific PSI per 1000 patient discharges on the left y-axis, the years of the events on the x-axis, and the total number of patient records that fulfilled the criteria for the analysis (ie, the denominator) on the right y-axis. The rate per 1000 discharges is also indicated above each bar, followed by the observed number of events for all hospitals combined for that year (in parentheses).
Across the 4 years of data, the mean PSI risk-adjusted rates of events ranged from 0.01% (0.1 event per 1000 discharges) for a foreign body left in during a procedure to 14.0% (140 events per 1000 discharges) for failure to rescue, which is counted when a patient dies as a result of a complication, such as acute renal failure, shock, or sepsis present as a secondary diagnosis, rather than a primary diagnosis such as leukemia or a congenital heart defect. Definitions of the numerator and denominator, an explanation of the diagnoses, and additional information about the possible pediatric relevance of the indicator are presented in the legends to Figs 1 to 11; the numerator and denominator indicated are the observed values, whereas the rate presented is risk adjusted.
With the high rate of infection incidents (>1000 incidents per year) allowing additional statistical analysis among hospitals, an analysis of ICD-9-CM codes indicating that many of these cases were potentially preventable, and national evidence-based literature regarding the prevention of device infection being available, a special examination of rates of infections attributable to medical care was performed for the 43 hospitals that provided data for each of the 4 years studied. A combined 4-year infection rate was calculated for each hospital and the overall sample.
The aggregated risk-adjusted rate for the 43 hospitals was 3.54 events per 1000 discharges (ie, 4469 cases with events in a total population of 1 252 768). As Fig 12 illustrates, 11 hospitals had rates that were significantly above the mean (P values ranging between .0001 and .038), and 19 hospitals had rates that were significantly below the mean (P values between .0001 and .046).
DISCUSSION
There can be significant limitations in attempting to identify preventable medical errors on the basis of administrative data. The data depend on medical record coders, whose major focus is billing. There are problems related to coding inaccuracies, coding variations among coders and among institutions (sometimes because of payer reimbursement variations), limited ability to adjust for risk with adult-based comorbidity groups and the Centers for Medicare and Medicaid Services DRGs, and difficulty in defining chronic illnesses that might have caused acute problems.9–11 These issues warrant additional investigation, because the variance in hospital rates we observed might be attributable simply to differences in coding. For meaningful validation of individual hospital PSIs, careful chart analysis by trained clinicians must be performed in a standardized manner. However, given the absence of a national reporting system for adverse events and medical errors and the enormous cost of individual chart reviews, large administrative database studies may be the best current, universally available tool for examination of rates of events, tracking of changes with time, or comparison of rates among settings.
The NACHRI data demonstrated PSI rates that were relatively consistent for the 4 years, with aggregate means that did not vary significantly from year to year. Examination of diagnosis and procedure codes associated with each case led us to think that many of these pediatric patient safety events were preventable, although the complexities and characteristics of these pediatric cases might be different from those for events in adult populations. For example, the risk-adjusted rate of decubiti was surprisingly high for children (17.2 events per 1000 discharges), and decubiti were noted frequently for children with poor perfusion, ie, those undergoing extracorporeal membrane oxygenation or cardiac patients who might require adrenergic agents to support their blood pressure, with resultant poor perfusion of the skin. Perhaps different methods to protect against decubiti in this population need to be investigated, compared with methods used for children without perfusion problems. Our goal should be that no child experiences decubiti. Three of the indicators, ie, infections attributable to medical care (ie, associated with procedures or device placements), postoperative pulmonary embolism and venous thrombosis, and decubitus ulcers, are subjects of evidence-based best-practice reports in the adult literature and are highlighted by AHRQ12 as events that should be investigated, with institution of standardized protocols.
Perhaps more importantly, examination of diagnosis and procedure codes associated with identified patient safety events led us to think that, in their current form, some of the PSIs are not sensitive for pediatric use, do not predict preventable events the majority of the time, and should not be used to compare institutions or judge the adequacy of care in children's hospitals. The majority of cases that the AHRQ software identified as deaths in low-mortality DRGs had primary diagnoses of pneumonia, seizures, or abdominal procedures. However, review of the associated ICD-9-CM codes for those cases showed that several patients who were listed as dying as a result of pneumonia actually had cystic fibrosis, some patients with seizures actually died as a result of terminal metabolic disease, and children who died as a result of abdominal procedures often had severe congenital defects, such as gastroschisis. For the failure to rescue indicator, the majority of cases identified by the software were very complex cases for which death was nonpreventable. A combination of diagnoses (eg, acute renal failure, respiratory failure, sepsis, and cardiac arrest) listed with the primary diagnosis (eg, congenital heart disease, leukemia, chronic renal failure, or lupus), rather than a preventable complication, often caused the death.
Our major goal was to identify events that are potentially preventable and to perform clinical redesign when evidence-based literature is available regarding improvement protocols. The infection rate variance analysis demonstrated hospital rates up to 5 times the lowest hospital rate and 1.78 times the mean rate. This variation may indicate that we can investigate the hospitals with the highest and lowest rates to identify best practices, which can be used when appropriate. For example, for the indicator of infections attributable to procedures and device placements, there is randomized, controlled trial evidence that rates of line infections can be reduced significantly with the use of antibiotic-coated central lines,13,14 preparation of the patient's skin with chlorhexidine instead of betadine,15,16 and the use of maximal barrier precautions (ie, sterile gloves, mask, long-sleeved gown, and full-sized sterile drapes) during insertion.17 Although these procedures have not been studied among pediatric patients in a randomized, controlled trial, a number of children's hospitals have already put some of these measures into place, on the basis of adult literature reports, and have observed decreases in infection rates (M. Miller, Johns Hopkins Hospital, personal communication, 2004). The goal should be to use these data to improve children's care. It is hoped that future findings can demonstrate progress toward decreasing the mean number of events observed and decreasing the variance among hospital performances.
PSI events are relatively rare in children's hospitals; therefore, adequate analysis of true variance among hospitals is difficult. As Miller et al previously noted, this underscores the appropriate use of the PSIs as "institutional case-finding tools aimed at internal quality improvement, as opposed to use for directly comparing individual institutions especially in public reports."6(p1364)
CONCLUSIONS
Analysis of PSIs in children's hospitals for 4 years, with data representing 1.92 million discharges, demonstrated a significant number of potentially preventable pediatric patient safety events. A number of these events might be amenable to evidence-based practices that could theoretically decrease the overall rates. Of specific interest in this work is the device infection rate, for which randomized, controlled trial data that demonstrate methods to decrease the number of events are available.
In their current form, the PSIs death in low-mortality DRGs and failure to rescue are inaccurate in relation to the pediatric population, do not predict preventable events the majority of the time, and should not be used to compare institutions, to judge adequacy of care, or to calculate potentially preventable deaths in children's hospitals. Research focused on improving indicators of this type by making them more sensitive for pediatric use is critically important and is one of the goals of future work.
ACKNOWLEDGMENTS
We thank Marlene Miller, MD, MSc, for comments, Akkeneel Talsma, PhD, RN, for data assistance, and Patricia Kneeland for preparation of the manuscript.
We also acknowledge the state data organizations that participate in the HCUP 2000 KID: Arizona Department of Health Services; California Office of Statewide Health Planning and Development; Colorado Health and Hospital Association; Connecticut—Chime, Inc; Florida Agency for Health Care Administration; Georgia—An Association of Hospitals and Health Systems; Hawaii Health Information Corp; Iowa Hospital Association; Kansas Hospital Association; Kentucky Department for Public Health; Maine Health Data Organization; Massachusetts Division of Health Care Finance and Policy; Maryland Health Services Cost Review Commission; Missouri Hospital Industry Data Institute; New Jersey Department of Health and Senior Services; New York State Department of Health; North Carolina Department of Health and Human Services; Oregon Association of Hospitals and Health Systems; Pennsylvania Health Care Cost Containment Council; South Carolina State Budget and Control Board; Tennessee Hospital Association; Texas Health Care Information Council; Utah Department of Health; Virginia Health Information; Washington State Department of Health; West Virginia Health Care Authority; and Wisconsin Department of Health and Family Services.
FOOTNOTES
Accepted Aug 31, 2004.
Reprint requests to (A.S.) University of Michigan Health System, C201 Med Inn Building, Box 0825, 1500 E Medical Center Dr, Ann Arbor, MI 48109-0825. asedman@umich.edu
The views expressed in this article are those of the authors and do not necessarily reflect those of the Agency for Healthcare Research and Quality or the US Department of Health and Human Services.
Conflict of interest: Dr Sedman is a paid medical advisor for the National Association of Children’s Hospitals and Related Institutions.
REFERENCES
Institute of Medicine. To Err Is Human: Building a Safer Health Care System. Washington, DC: National Academy Press; 1999
Kaushal R, Bates D, Landrigan C, et al. Medication errors and adverse drug events in pediatric inpatients. JAMA. 2001;285 :2114 –2120
University of California, San Francisco-Stanford University Evidence-Based Practice Center. Evidence Research Measures of Patient Safety Based on Hospital Administrative Data: The Patient Safety Indicators. Rockville, MD: Agency for Healthcare Research and Quality; 2002
Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care. 1998;36 :8 –27
Miller M, Elixhauser A, Zhan C. Patient safety events during pediatric hospitalizations. Pediatrics. 2003;111 :1358 –1366
Miller M, Zhan C. Pediatric patient safety in hospitals: a national picture in 2000. Pediatrics. 2004;113 :1741 –1746
SPSS. SPSS Base 12.0 User's Guide. Chicago, IL: SPSS; 2003
Johantgen M, Elixhauser A, Ball JK, Goldfarb M, Harris DR. Quality indicators using hospital discharge data: state and national applications. Jt Comm J Qual Improv. 1998;24 :88 –105
Forest CF, Shipman SA, Dougherty D, Miller MR. Outcomes research in pediatric settings: recent trends and future directions. Pediatrics. 2003;111 :171 –178
Fried VM, Prager K, MacKay AP, Xai H. Chartbook on Trends in the Health of Americans, Health United States, 2003. Hyattsville, MD: National Center for Health Statistics; 2003
Shojiana KG, Duncan BW, McDonald KM, Wachter RM, eds. Making Health Care Safer: A Critical Analysis of Patient Safety Practices: Evidence Report/Technology Assessment 43. AHRQ Publication No. 01-E058. Rockville, MD: Agency for Healthcare Research and Quality; 2001
Veenstra DL, Saint S, Saha S, Lumley T, Sullivan SD. Efficacy of antiseptic-impregnated central venous catheters in preventing catheter-related bloodstream infection: a meta-analysis. JAMA. 1999;281 :261 –267
Raad I, Darouiche R, Dupuis J, et al. Central venous catheters coated with minocyclin and ritampin for the prevention of catheter-related colonization and bloodstream infections: a randomized double-blind trial. Ann Intern Med. 1997;127 :267 –274
Mimoz O, Pieroni L, Lawrence C, et al. Prospective randomized trial of two antiseptic solutions for prevention of central venous or arterial catheter colonization and infection in intensive care unit patients. Crit Care Med. 1996;24 :1818 –1823
Humar A, Ostromecki A, Direnfeld J, et al. Prospective randomized trial at 10% povidone-iodine vs. 0.5% tincture of chlorhexidine as cutaneous antisepsis for prevention of control venous catheter infection. Clin Infect Dis. 2000;31 :1001 –1007
Raad I, Hohn DC, Gilbraith BJ, et al. Prevention of central venous catheter-related infections by using maximal sterile barrier precautions during insertion. Infect Control Hosp Epidemiol. 1994;15 :231 –238(Aileen Sedman, MD, J. Mit)
National Association of Children's Hospitals and Related Institutions, Alexandria, Virginia
Agency for Healthcare Research and Quality, Rockville, Maryland
Children's Hospital of Wisconsin, Wauwatosa, Wisconsin
ABSTRACT
Objectives. Patient safety indicators (PSIs) were developed by the Agency for Healthcare Research and Quality. Our objectives were (1) to apply these algorithms to the National Association of Children's Hospitals and Related Institutions (NACHRI) Aggregate Case Mix Comparative Database for 1999–2002, (2) to establish mean rates for each of the PSI events in children's hospitals, (3) to investigate the inadequacies of PSIs in relation to pediatric diagnoses, and (4) to express the data in such a way that children's hospitals could use the PSIs determined to be appropriate for pediatric use for comparison with their own data. In addition, we wanted to use the data to set priorities for ongoing clinical investigations and to propose interventions if the indicators demonstrated preventable errors.
Methods. The Agency for Healthcare Research and Quality PSI algorithms (version 2.1, revision 1) were applied to children's hospital administrative data (1.92 million discharges) from the NACHRI Aggregate Case Mix Comparative Database for 1999–2002. Rates were measured for the following events: complications of anesthesia, death in low-mortality diagnosis-related groups (DRGs), decubitus ulcer, failure to rescue (ie, death resulting from a complication, rather than the primary diagnosis), foreign body left in during a procedure, iatrogenic pneumothorax, infection attributable to medical care (ie, infections related to surgery or device placement), postoperative hemorrhage or hematoma, postoperative pulmonary embolism or venous thrombosis, postoperative wound dehiscence, and accidental puncture/laceration.
Results. Across the 4 years of data, the mean risk-adjusted rates of PSI events ranged from 0.01% (0.1 event per 1000 discharges) for a foreign body left in during a procedure to 14.0% (140 events per 1000 discharges) for failure to rescue. Review of International Classification of Diseases, Ninth Revision, Clinical Modification codes associated with each PSI category showed that the failure to rescue and death in low-mortality DRG indicators involved very complex cases and did not predict preventable events in the majority of cases. The PSI for infection attributable to medical care appeared to be accurate the majority of the time. Incident risk-adjusted rates of infections attributable to medical care averaged 0.35% (3.5 events per 1000 discharges) and varied up to fivefold from the lowest rate to the highest rate. The highest rates were up to 1.8 times the average.
Conclusions. PSIs derived from administrative data are indicators of patient safety concerns and can be relevant as screening tools for children's hospitals; however, cases identified by these indicators do not always represent preventable events. Some, such as a foreign body left in during a procedure, iatrogenic pneumothorax, infection attributable to medical care, decubitus ulcer, and venous thrombosis, seem to be appropriate for pediatric care and may be directly amenable to system changes. Evidence-based practices regarding those particular indicators that have been reported in the adult literature need to be investigated in the pediatric population. In their present form, 2 of the indicators, namely, failure to rescue and death in low-mortality DRGs, are inaccurate for the pediatric population, do not represent preventable errors in the majority of pediatric cases, and should not be used to estimate quality of care or preventable deaths in children's hospitals. The PSIs can assist institutions in prioritizing chart review-based investigations; if clusters of validated events emerge in reviews, then improvement activities can be initiated. Large aggregate databases, such as the NACHRI Case Mix Database, can help establish mean rates of potential pediatric events, giving children's hospitals a context within which to examine their own data.
Key Words: patient safety indicators Agency for Healthcare Research and Quality medical error National Association of Children's Hospitals and Related Institutions
Abbreviations: AHRQ, Agency for Healthcare Research and Quality PSI, patient safety indicator NACHRI, National Association of Children's Hospitals and Related Institutions DRG, diagnosis-related group ICD-9-CM, International Classification of Diseases, Ninth Revision, Clinical Modification HCUP, Healthcare Cost and Utilization Project KID, Kids' Inpatient Database
In 1999, the Institute of Medicine defined patient safety as "freedom from accidental injury because of medical care or medical errors" and charged hospitals to create a culture of safety in their institutions.1 Studies of pediatric medication errors have been published2; however, there is a paucity of data on pediatric patient injuries in children's hospitals.
The Agency for Healthcare Research and Quality (AHRQ) was charged by Congress to promote patient safety and to reduce medical errors through research and collaborative partnerships. One result of this mandate was the development of the AHRQ patient safety indicators (PSIs), ie, software that can identify potential adverse events from International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes in hospital inpatient administrative data. The University of California, San Francisco-Stanford University Evidence-Based Practice Center, under a contract with AHRQ, developed the PSIs in a 5-stage process.3 First, the literature was reviewed to identify a list of candidate indicators. Second, clinical panels evaluated the face validity or "sensibility" of the indicators, as judged by practicing physicians. Third, ICD-9-CM coding experts were consulted to confirm that the definition of each indicator reflected the intended clinical situation. Fourth, an empirical analysis of the chosen indicators was conducted with the Healthcare Cost and Utilization Project (HCUP) data.4 Fifth, PSI software and documentation were produced for public release by AHRQ. Since its initial development, this software has undergone several modifications, including refinement of the indicator definitions. Therefore, in comparisons of PSIs, it is very important to note whether the versions of the PSIs are the same.
The indicators were risk adjusted on the basis of patient age, patient gender, and AHRQ comorbidity groups, as determined with AHRQ comorbidity software, which uses ICD-9-CM codes to assign up to 30 comorbidity groups (such as hypertension and diabetes mellitus) to patient records.5
Extensive work was published in 2003 by Miller et al,6 describing pediatric patient safety with the use of the PSIs ( version) and administrative data from the 1997 HCUP.4 That study identified the high rates of birth trauma among newborns, with a rate of 1.5 cases per 100 births. Multivariate logistic regression analyses with the PSIs showed that patients with higher acuity exhibited higher rates of events. The study also demonstrated significant increases in length of stay and cost per case in relation to PSIs. The authors concluded "... that hospitalized children experience significant numbers of patient safety events, that these event rates are comparable to hospitalized adults, and that attention clearly needs to be paid to the unique event of childbirth."6(p1365) A follow-up study by Miller and Zhan7 with HCUP 2000 data confirmed the enormous burden of medical errors in pediatric care.
The HCUP database used by Miller et al6 is different from the National Association of Children's Hospitals and Related Institutions (NACHRI) database (Table 1). Specifically, the HCUP 2000 Kids' Inpatient Database (KID) includes a sample of pediatric discharges from community hospitals (short-term, nonfederal, general and specialty hospitals) drawn from 27 state inpatient databases on children 20 years of age (www.ahrq.gov/data/hcup). In comparison, the NACHRI Case Mix Database is a children's hospital-specific database (freestanding acute care and specialty children's hospitals and acute care children's hospitals located within larger institutions) that contains 100% of pediatric discharges from those institutions, excluding normal newborn and obstetric cases. Because of the differences in the databases and their potential effects on PSI rates, we considered it worthwhile to investigate the PSI algorithms specifically from a children's hospital perspective with the NACHRI Case Mix Database. Our objectives were (1) to apply the AHRQ PSI algorithms to the NACHRI Case Mix Database for 1999-2002, (2) to investigate PSI rates specifically in children's hospitals (with the major utility of the trend being to increase confidence in the data, because of the uniformity of the rates with time for each PSI), (3) to investigate any inadequacies of PSIs in relation to pediatric diagnoses, and (4) to express the data in such a way that individual children's hospitals could use the PSIs determined to be pediatric appropriate for comparison with their own data. In addition, we wished to use the data to set priorities for ongoing clinical investigation and to propose interventions.
METHODS
To understand patient safety events in children's hospitals and to prioritize future research, we applied the PSIs to the NACHRI Case Mix Database. The database contains administrative data at the ICD-9-CM level for inpatient discharges, with standard indicators such as cost per case, length of stay, readmission rate, mortality rate, and case mixture index (acuity level). It includes 4 years of data (1999–2002) and contains 1.92 million discharge records from children's hospitals representing 31 states. The number of hospitals included in the sample varied each year because new institutions joined the NACHRI Case Mix Program, with 50 hospitals in 1999, 56 in 2000, 65 in 2001, and 67 in 2002.
The AHRQ PSI software (version 2.1, revision 1) was used to analyze the data, and the resulting output (numerator, denominator, mean, observed rate, risk-adjusted rate, and smoothed rate) for each of the 11 PSIs with relevant pediatric volume was coupled with additional analyses with SPSS statistical software.8 Rates were measured for the following events: complications of anesthesia, death in a low-mortality diagnosis-related group (DRG), decubitus ulcer, failure to rescue (ie, death resulting from a complication rather than the primary diagnosis), foreign body left in during a procedure, iatrogenic pneumothorax, infection attributable to medical care (ie, infections related to surgery or device placement), postoperative hemorrhage or hematoma, postoperative pulmonary embolism or venous thrombosis, postoperative wound dehiscence, and accidental puncture/laceration.
The PSI software is very specific in defining the numerator and denominator for each indicator (see figure legends) and strictly limits the discharges that are considered for each patient safety event. For example, patients with paraplegia and quadriplegia are exempted from the decubitus measure, and patients with cancer and/or immunosuppression are excluded from the infection rate measure. The indicators are risk-adjusted on the basis of patient age, patient gender, and AHRQ comorbidity groups (ICD-9-CM codes are used to assign up to 30 comorbidity groups, such as hypertension and diabetes mellitus, to a patient record5). Depending on the covariate weight of these comorbidity groups, the rate of PSIs is adjusted up or down for each group analyzed. In this study, for example, the observed rate of decubiti was adjusted upward during the risk adjustment, in comparison with adult patients, who, because of diagnoses of vascular compromise and diabetes, are considered at higher risk for decubiti than are children.
The ICD-9-CM codes for each discharge identified as having a patient safety event were reviewed. This investigation of all of the principal and secondary diagnoses associated with each case was used to characterize complexities that influenced the indicators and allowed us to ascertain whether the patient safety events were potentially preventable.
RESULTS
The PSIs are also presented in Figs 1 to 11, which show the rate of the specific PSI per 1000 patient discharges on the left y-axis, the years of the events on the x-axis, and the total number of patient records that fulfilled the criteria for the analysis (ie, the denominator) on the right y-axis. The rate per 1000 discharges is also indicated above each bar, followed by the observed number of events for all hospitals combined for that year (in parentheses).
Across the 4 years of data, the mean PSI risk-adjusted rates of events ranged from 0.01% (0.1 event per 1000 discharges) for a foreign body left in during a procedure to 14.0% (140 events per 1000 discharges) for failure to rescue, which is counted when a patient dies as a result of a complication, such as acute renal failure, shock, or sepsis present as a secondary diagnosis, rather than a primary diagnosis such as leukemia or a congenital heart defect. Definitions of the numerator and denominator, an explanation of the diagnoses, and additional information about the possible pediatric relevance of the indicator are presented in the legends to Figs 1 to 11; the numerator and denominator indicated are the observed values, whereas the rate presented is risk adjusted.
With the high rate of infection incidents (>1000 incidents per year) allowing additional statistical analysis among hospitals, an analysis of ICD-9-CM codes indicating that many of these cases were potentially preventable, and national evidence-based literature regarding the prevention of device infection being available, a special examination of rates of infections attributable to medical care was performed for the 43 hospitals that provided data for each of the 4 years studied. A combined 4-year infection rate was calculated for each hospital and the overall sample.
The aggregated risk-adjusted rate for the 43 hospitals was 3.54 events per 1000 discharges (ie, 4469 cases with events in a total population of 1 252 768). As Fig 12 illustrates, 11 hospitals had rates that were significantly above the mean (P values ranging between .0001 and .038), and 19 hospitals had rates that were significantly below the mean (P values between .0001 and .046).
DISCUSSION
There can be significant limitations in attempting to identify preventable medical errors on the basis of administrative data. The data depend on medical record coders, whose major focus is billing. There are problems related to coding inaccuracies, coding variations among coders and among institutions (sometimes because of payer reimbursement variations), limited ability to adjust for risk with adult-based comorbidity groups and the Centers for Medicare and Medicaid Services DRGs, and difficulty in defining chronic illnesses that might have caused acute problems.9–11 These issues warrant additional investigation, because the variance in hospital rates we observed might be attributable simply to differences in coding. For meaningful validation of individual hospital PSIs, careful chart analysis by trained clinicians must be performed in a standardized manner. However, given the absence of a national reporting system for adverse events and medical errors and the enormous cost of individual chart reviews, large administrative database studies may be the best current, universally available tool for examination of rates of events, tracking of changes with time, or comparison of rates among settings.
The NACHRI data demonstrated PSI rates that were relatively consistent for the 4 years, with aggregate means that did not vary significantly from year to year. Examination of diagnosis and procedure codes associated with each case led us to think that many of these pediatric patient safety events were preventable, although the complexities and characteristics of these pediatric cases might be different from those for events in adult populations. For example, the risk-adjusted rate of decubiti was surprisingly high for children (17.2 events per 1000 discharges), and decubiti were noted frequently for children with poor perfusion, ie, those undergoing extracorporeal membrane oxygenation or cardiac patients who might require adrenergic agents to support their blood pressure, with resultant poor perfusion of the skin. Perhaps different methods to protect against decubiti in this population need to be investigated, compared with methods used for children without perfusion problems. Our goal should be that no child experiences decubiti. Three of the indicators, ie, infections attributable to medical care (ie, associated with procedures or device placements), postoperative pulmonary embolism and venous thrombosis, and decubitus ulcers, are subjects of evidence-based best-practice reports in the adult literature and are highlighted by AHRQ12 as events that should be investigated, with institution of standardized protocols.
Perhaps more importantly, examination of diagnosis and procedure codes associated with identified patient safety events led us to think that, in their current form, some of the PSIs are not sensitive for pediatric use, do not predict preventable events the majority of the time, and should not be used to compare institutions or judge the adequacy of care in children's hospitals. The majority of cases that the AHRQ software identified as deaths in low-mortality DRGs had primary diagnoses of pneumonia, seizures, or abdominal procedures. However, review of the associated ICD-9-CM codes for those cases showed that several patients who were listed as dying as a result of pneumonia actually had cystic fibrosis, some patients with seizures actually died as a result of terminal metabolic disease, and children who died as a result of abdominal procedures often had severe congenital defects, such as gastroschisis. For the failure to rescue indicator, the majority of cases identified by the software were very complex cases for which death was nonpreventable. A combination of diagnoses (eg, acute renal failure, respiratory failure, sepsis, and cardiac arrest) listed with the primary diagnosis (eg, congenital heart disease, leukemia, chronic renal failure, or lupus), rather than a preventable complication, often caused the death.
Our major goal was to identify events that are potentially preventable and to perform clinical redesign when evidence-based literature is available regarding improvement protocols. The infection rate variance analysis demonstrated hospital rates up to 5 times the lowest hospital rate and 1.78 times the mean rate. This variation may indicate that we can investigate the hospitals with the highest and lowest rates to identify best practices, which can be used when appropriate. For example, for the indicator of infections attributable to procedures and device placements, there is randomized, controlled trial evidence that rates of line infections can be reduced significantly with the use of antibiotic-coated central lines,13,14 preparation of the patient's skin with chlorhexidine instead of betadine,15,16 and the use of maximal barrier precautions (ie, sterile gloves, mask, long-sleeved gown, and full-sized sterile drapes) during insertion.17 Although these procedures have not been studied among pediatric patients in a randomized, controlled trial, a number of children's hospitals have already put some of these measures into place, on the basis of adult literature reports, and have observed decreases in infection rates (M. Miller, Johns Hopkins Hospital, personal communication, 2004). The goal should be to use these data to improve children's care. It is hoped that future findings can demonstrate progress toward decreasing the mean number of events observed and decreasing the variance among hospital performances.
PSI events are relatively rare in children's hospitals; therefore, adequate analysis of true variance among hospitals is difficult. As Miller et al previously noted, this underscores the appropriate use of the PSIs as "institutional case-finding tools aimed at internal quality improvement, as opposed to use for directly comparing individual institutions especially in public reports."6(p1364)
CONCLUSIONS
Analysis of PSIs in children's hospitals for 4 years, with data representing 1.92 million discharges, demonstrated a significant number of potentially preventable pediatric patient safety events. A number of these events might be amenable to evidence-based practices that could theoretically decrease the overall rates. Of specific interest in this work is the device infection rate, for which randomized, controlled trial data that demonstrate methods to decrease the number of events are available.
In their current form, the PSIs death in low-mortality DRGs and failure to rescue are inaccurate in relation to the pediatric population, do not predict preventable events the majority of the time, and should not be used to compare institutions, to judge adequacy of care, or to calculate potentially preventable deaths in children's hospitals. Research focused on improving indicators of this type by making them more sensitive for pediatric use is critically important and is one of the goals of future work.
ACKNOWLEDGMENTS
We thank Marlene Miller, MD, MSc, for comments, Akkeneel Talsma, PhD, RN, for data assistance, and Patricia Kneeland for preparation of the manuscript.
We also acknowledge the state data organizations that participate in the HCUP 2000 KID: Arizona Department of Health Services; California Office of Statewide Health Planning and Development; Colorado Health and Hospital Association; Connecticut—Chime, Inc; Florida Agency for Health Care Administration; Georgia—An Association of Hospitals and Health Systems; Hawaii Health Information Corp; Iowa Hospital Association; Kansas Hospital Association; Kentucky Department for Public Health; Maine Health Data Organization; Massachusetts Division of Health Care Finance and Policy; Maryland Health Services Cost Review Commission; Missouri Hospital Industry Data Institute; New Jersey Department of Health and Senior Services; New York State Department of Health; North Carolina Department of Health and Human Services; Oregon Association of Hospitals and Health Systems; Pennsylvania Health Care Cost Containment Council; South Carolina State Budget and Control Board; Tennessee Hospital Association; Texas Health Care Information Council; Utah Department of Health; Virginia Health Information; Washington State Department of Health; West Virginia Health Care Authority; and Wisconsin Department of Health and Family Services.
FOOTNOTES
Accepted Aug 31, 2004.
Reprint requests to (A.S.) University of Michigan Health System, C201 Med Inn Building, Box 0825, 1500 E Medical Center Dr, Ann Arbor, MI 48109-0825. asedman@umich.edu
The views expressed in this article are those of the authors and do not necessarily reflect those of the Agency for Healthcare Research and Quality or the US Department of Health and Human Services.
Conflict of interest: Dr Sedman is a paid medical advisor for the National Association of Children’s Hospitals and Related Institutions.
REFERENCES
Institute of Medicine. To Err Is Human: Building a Safer Health Care System. Washington, DC: National Academy Press; 1999
Kaushal R, Bates D, Landrigan C, et al. Medication errors and adverse drug events in pediatric inpatients. JAMA. 2001;285 :2114 –2120
University of California, San Francisco-Stanford University Evidence-Based Practice Center. Evidence Research Measures of Patient Safety Based on Hospital Administrative Data: The Patient Safety Indicators. Rockville, MD: Agency for Healthcare Research and Quality; 2002
Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care. 1998;36 :8 –27
Miller M, Elixhauser A, Zhan C. Patient safety events during pediatric hospitalizations. Pediatrics. 2003;111 :1358 –1366
Miller M, Zhan C. Pediatric patient safety in hospitals: a national picture in 2000. Pediatrics. 2004;113 :1741 –1746
SPSS. SPSS Base 12.0 User's Guide. Chicago, IL: SPSS; 2003
Johantgen M, Elixhauser A, Ball JK, Goldfarb M, Harris DR. Quality indicators using hospital discharge data: state and national applications. Jt Comm J Qual Improv. 1998;24 :88 –105
Forest CF, Shipman SA, Dougherty D, Miller MR. Outcomes research in pediatric settings: recent trends and future directions. Pediatrics. 2003;111 :171 –178
Fried VM, Prager K, MacKay AP, Xai H. Chartbook on Trends in the Health of Americans, Health United States, 2003. Hyattsville, MD: National Center for Health Statistics; 2003
Shojiana KG, Duncan BW, McDonald KM, Wachter RM, eds. Making Health Care Safer: A Critical Analysis of Patient Safety Practices: Evidence Report/Technology Assessment 43. AHRQ Publication No. 01-E058. Rockville, MD: Agency for Healthcare Research and Quality; 2001
Veenstra DL, Saint S, Saha S, Lumley T, Sullivan SD. Efficacy of antiseptic-impregnated central venous catheters in preventing catheter-related bloodstream infection: a meta-analysis. JAMA. 1999;281 :261 –267
Raad I, Darouiche R, Dupuis J, et al. Central venous catheters coated with minocyclin and ritampin for the prevention of catheter-related colonization and bloodstream infections: a randomized double-blind trial. Ann Intern Med. 1997;127 :267 –274
Mimoz O, Pieroni L, Lawrence C, et al. Prospective randomized trial of two antiseptic solutions for prevention of central venous or arterial catheter colonization and infection in intensive care unit patients. Crit Care Med. 1996;24 :1818 –1823
Humar A, Ostromecki A, Direnfeld J, et al. Prospective randomized trial at 10% povidone-iodine vs. 0.5% tincture of chlorhexidine as cutaneous antisepsis for prevention of control venous catheter infection. Clin Infect Dis. 2000;31 :1001 –1007
Raad I, Hohn DC, Gilbraith BJ, et al. Prevention of central venous catheter-related infections by using maximal sterile barrier precautions during insertion. Infect Control Hosp Epidemiol. 1994;15 :231 –238(Aileen Sedman, MD, J. Mit)