The Pediatric Risk of Hospital Admission Score: A Second-Generation Severity-of-Illness Score for Pediatric Emergency Patients
http://www.100md.com
《小儿科》
Department of Pediatrics, George Washington University School of Medicine, Washington, DC
Department of Emergency Medicine Center for Health Services and Community Research, Children's Research Institute
Center for Hospital-Based Specialties and Department of Critical Care Medicine, Children's National Medical Center, Washington, DC
ABSTRACT
Objective. To develop and validate a second-generation severity-of-illness score that is applicable to pediatric emergency patients. The Pediatric Risk of Admission (PRISA) score was developed in a single hospital and was recalibrated and validated in 2, previous, small studies from academic pediatric hospitals. This study was performed to develop and validate a score in a larger sample of diverse hospitals.
Methods. Emergency departments (EDs) were block randomly selected as part of a study on ED quality on the basis of 3 care characteristics: annual patient volume (high or low compared with national median), presence or absence of a pediatric emergency medicine subspecialist, and presence or absence of residents. Patients were selected randomly on the basis of daily arrival logs. Medical records were photocopied, and abstracted data included demographic, historical, physiologic, and therapeutic information. The total sample was randomly divided into a 75% development sample and a 25% validation sample. Univariate and multivariate analyses were used to model the risk of mandatory admission, admissions for which preidentified, inpatient medical resources were used. The resulting multiple logistic regression model coefficients were converted to integer scores. Calibration (Hosmer-Lemeshow goodness of fit) and discrimination (area under the ROC curve) were used to measure performance. As a measure of construct validity, proportions of patients in ordered risk intervals were correlated with the outcomes of admission, mandatory admission, and ICU admission.
Results. Sixteen EDs enrolled 11664 patients. Mean patient age (±SD) was 6.8 ± 5.8 years, and 53% were male. Nine percent arrived by emergency medical services, and 6.9% were admitted. The most common diagnoses were minor injuries, otitis media, and fever. The multivariate analysis yielded a score with 7 historical variables, 8 physiologic variables, 1 therapy (oxygen) term, and 1 interaction term. Calibration was excellent. In the development sample, 442 mandatory admissions were predicted and 442 were observed (total 2 = 2.275), and in the validation sample, 136.6 were predicted and 145 were observed (2 = 8.575). The area under the receiver operator characteristic curve was 0.82 ± 0.01 (SE) in the development sample and 0.77 ± 0.02 in the validation sample. In ordered predicted risk intervals, the proportion of patients with admissions, mandatory admissions, and ICU admissions increased in a linear manner.
Conclusions. The second-generation PRISA II score for pediatric ED patients has been developed and validated in a large sample of diverse hospitals. Performance characteristics indicate that PRISA II will be useful for institutional comparisons, benchmarking, and controlling for severity of illness when enrolling patients in clinical trials.
Key Words: emergency medicine health services research severity-of-illness index
Abbreviations: ED, emergency department PRISA, Pediatric Risk of Admission PEM, pediatric emergency medicine ROC, receiver operator characteristic
Specific components of the Emergency Medical Services system and the Emergency Medical Services–Children system have benefited from the development and use of severity assessment methods. For example, PICU severity methods have enabled evaluation of quality of PICU care using severity-adjusted mortality rates,1 identification of PICU care characteristics associated with quality of care,2,3 measurement of efficiency of bed use,4 and measurement of severity-adjusted length of stay.5 Such methods have not been applied to pediatric or adult emergency departments (EDs). Although severity scales exist for specific diseases, there is no widely accepted method that is applicable to all pediatric ED patients.
Previous severity-of-illness methods applied to hospital and ED environments (eg, trauma patients) have used mortality as the outcome for quality-of-care studies.6 However, mortality is not an appropriate outcome for the predominant ED activities and is so rare in the general pediatric ED population that it cannot be used as a meaningful outcome measure. One of the most important functions of ED practice is the correct disposition of patients to either inpatient or outpatient care settings. Overadmission results in wasted resources and the risk of iatrogenic injury, whereas failure to admit when necessary results in delayed care and avoidable complications. We propose that the appropriate outcomes that are indicative of pediatric ED quality are correct rates of hospital admission and ED discharge.
We developed a severity-of-illness model using hospital admission as the outcome. Initially developed and validated at a single institution, the Pediatric Risk of Admission (PRISA) score uses components of acute and chronic medical history, physiology, and ED therapies to model the risk of hospital admission. Higher PRISA scores were associated with higher probability of medically mandatory admissions, ICU admission, and mortality.7 The PRISA score subsequently has been validated in 2 other studies, including 1 multisite study.8,9
The purpose of this study was to develop a second-generation pediatric ED severity-of-illness (PRISA II) score in a large national sample of hospitals. We used a sample of 16 EDs block randomly selected to represent large and small EDs, those with and without pediatric emergency medicine (PEM) specialists, and those with and without residents providing ED care.
METHODS
Site Selection
Sites were selected as part of a study to evaluate the association of hospital care factors with quality of pediatric emergency medical care.10 In brief, EDs were randomly selected to represent the care factors of high or low volume ( or > the national median of pediatric visits per year), presence or absence of a PEM specialist, and presence or absence of residents delivering ED care such that there were 2 EDs representing each of the 8 possible combinations of factors. Volume was determined from the American Hospital Association 1997 Annual Survey Database (Chicago, IL) and used to randomly select EDs for telephone survey concerning the other 2 care factors. All selected hospitals had PICUs. The Institutional Review Boards of the participating hospitals approved the study.
Patient Selection
A random-number generator was used to select 2 patients per day, using the daily consecutive log of patient arrivals routinely kept at each site. We attempted to enroll 750 patients at each site. Actual patient enrollment ranged from 729 to 753 patients. For sites that enrolled >729, patients were randomly eliminated from the sample to achieve a balanced sample of 729 patients from each site, for a total study enrollment of 11664 patients.
Data Management
After personal identifiers were masked, photocopied records were submitted to the coordinating center. The entire ED record and the first 24 hours of the inpatient record, where applicable, were submitted. Data reliability was determined by interrater reliability between the data abstractor and 1 of the authors (J.M.C.) to maintain an intraclass correlation coefficient R > 0.75.
Outcomes
Admission was defined as admission to an inpatient ward or admission to an observation area for >12 hours. The primary outcome for this study was mandatory admission. Admission was classified as mandatory on the basis of a previously validated list of services usually provided in acute care hospitals.7 Mandatory admissions are considered those that receive a therapeutic resource that because of the therapy or its intensity is usually delivered as an inpatient (Table 1). Nonmandatory admissions did not receive one of these resources.
Identification of Potential Predictor Variables
Potential predictor variables included historical, physiologic, and other risk factors. Unmeasured variables were assumed to be normal. Several variable types were excluded from the potential predictor variables. First, variables that are likely to be influenced by hospital practice, individual practice patterns, or medical recording practices were not included. Thus, we minimized the consideration of therapeutic variables. Second, socioeconomic factors were not included as predictor variables. Third, we considered ranges of physiologic variables rather then absolute values to reduce the effect of sampling frequency.
Multiple logistic regression was used to model the probability of the mandatory admission using a process similar to development of the PRISM III score.1 The patient sample was randomly divided into a 75% development sample and a 25% validation sample. The independent variables for possible inclusion in the predictive model were selected using a multistep process. First, variables were chosen for clinical relevance on the basis of clinical experience and previous work in developing severity adjustment models. Second, univariate analyses were used to determine associations with the dependent variable. For categorical variables, this was performed using unadjusted odds ratios. Some categorical variables were uncommon in ED populations and therefore were combined with other, clinically similar variables. For example, the chronic disease variables "HIV" and "oncologic disease" were combined with other conditions into a single category "immunodeficiency." For continuous variables, patients were stratified into the age groups of neonates (0–1 month), infants (>1–12 months), children (>12–144 months), and adolescents (>144 months). Univariate logistic regression analysis was used to determine which variables and subranges significantly contributed to the probability of hospital admission. The risk of hospital admission relative to the midrange (40th–60th percentiles) of discharged patients was computed for the physiologic variables in each age stratum. We assumed that deviations from the midrange contributed to risk, with larger deviations reflecting higher risk. We initially used deciles of physiologic values, then readjusted these cutpoints on the basis of clinical relevance and performance in the multivariate model. Neighboring deciles were combined when the regression coefficients were similar.1 Variables with a liberal significance level (P < .30) in univariate analysis were included in the multivariate model. Third, forward and backward stepwise inclusion of variables was used in the development of the multiple logistic regression model. Variables were retained as long as they contributed to a lower Akaike's Information Criteria and were statistically significant at P < .10. The regression coefficients in the model were expressed as relative percentages by dividing each of the coefficients by the sum of the regression coefficients and multiplying by 100. The rounded percentages for each variable and variable range are the components of the PRISA II score, and the total for all variables is the PRISA II score for each patient. Although most variables contribute to a higher risk of hospital admission, a variable may have a negative regression coefficient (and thus a corresponding negative integer score), indicating that the presence of that variable in an individual patient contributes to a lower risk of hospital admission.
Standard performance measures were used to assess the PRISA II score. The area under the curve for the receiver operator characteristic (ROC) curve was used to measure discrimination. Calibration compared observed outcomes with predicted outcomes, where the predicted outcomes were the number predicted by the PRISA II score. The Hosmer-Lemeshow goodness-of-fit tests (2) were used to measure calibration by comparing observed with predicted admissions in 8 ordered risk intervals of predicted admission probability. Risk intervals were chosen to be clinically relevant, to ensure at least 10 predicted admissions in each interval in the development sample, and to attempt to achieve a balanced distribution of admitted patients across the intervals. Overall performance was also assessed with the standardized admission ratio, the ratio of observed mandatory admissions to predicted mandatory admissions. Comparison of mean PRISA II score among different severity subgroups used the Kruskal-Wallis test. Construct validity was assessed by comparing proportions of patients with admission, mandatory admission, and ICU admission in the ordered PRISA II risk intervals using the Cochran-Armitage test for linear trend in proportion.11
RESULTS
Sample and Patient Data
Institutional and patient characteristics are depicted in Table 2. The 16 sites ranged in annual pediatric ED volumes from 1740 to 31000 patients (median: 13905). Four sites had pediatric volumes <8000, 4 sites had volumes between 8000 and 13900, 4 sites had volumes between 13900 and 17500, and 4 sites had volumes >17500. Consistent with the study design for site enrollment, 8 sites had residents caring for patients in the ED, and 8 sites had at least 1 PEM specialist in the practice. The most common chief complaints were fever (17.5%), nonspecific respiratory complaints (14.5%), and injuries to the extremities (10.4%) and head (9.4%). The most common diagnoses were minor injuries (24.2%), otitis media (9.2%), and fever without a source (5.0%). There was significant variability among the EDs for all institutional and patient population characteristics (P < .01 for all variables).
Of the 11664 patients, 808 (6.9%) were hospitalized and 93 (0.8%) patients were admitted to ICUs. Three patients died in the ED and were included in the hospitalized group. Of the 808 admitted patients, 587 (73%) were mandatory admissions. Admission rates ranged from 1.4% to 17% (P < .001), and mandatory admission rates ranged from 1.0% to 13.7% (P < .001) of ED patients.
Severity-of-Illness Model
Univariate analyses revealed 10 historical and 29 physiologic variables associated with mandatory hospital admission. One therapeutic variable, oxygen, was substantially more strongly associated with the outcome than associated physiologic variables, including pulse oximetry and respiratory rate, and thus was included for testing in multivariate modeling. One variable, low temperature, was not statistically significant but was considered clinically significant and thus was retained in the model.
The component variables identified in multivariate analysis are shown in Table 3. There are 16 variables and 1 interaction term. Seven of the variables are historical, 8 are physiologic, and 1 is a therapeutic variable. For the physiologic variables, none required multiple ranges and 3 (temperature, systolic blood pressure, and diastolic blood pressure) are age-adjusted. The 1 therapeutic variable, oxygen, is 1 of the most important predictor variables. Two of the variables, minor injury and the interaction term, were negative. Thus, minor injury reduces the likelihood of admission, whereas the interaction term attenuates the cumulative effects of having both elevated potassium and low serum bicarbonate.
Overall, the PRISA II score performed very well. Figure 1 illustrates the increasing PRISA II score with cohorts of increasing illness severity: ED discharges, nonmandatory hospital admissions, mandatory non-ICU hospital admissions, and ICU admissions. There was a significant trend in increasing PRISA II scores, and each group's score was significantly different from the others with the exception of ICU admissions compared with mandatory non-ICU admissions (P = .085).
The Hosmer-Lemeshow goodness-of-fit test indicated excellent calibration in both the development sample (2 = 2.275, degrees of freedom, df = 6, P = .8928; Table 5, Fig 2) and validation sample (2 = 8.575, df = 7, P = .2846; Table 6, Fig 2). The standardized mandatory admission ratios for the total samples were 1.00 ± 0.04 and 1.06 ± 0.07 in the development and validation samples. The area under the ROC curve was 0.822 ± 0.012 in the development sample and 0.774 ± 0.023 in the validation sample (Fig 3).
Construct validity is depicted in Fig. 4. The figure illustrates the increasing proportion of patients with admissions, mandatory admissions, and ICU admissions as PRISA II risk intervals increase (P < .0001).
DISCUSSION
Severity-of-illness methods are important for measuring and controlling for case-mix variability in many situations, including scientific studies and institutional comparisons.12–16 Such tools have been used extensively in pediatric and neonatal critical care and have allowed comparisons among individual hospitals and among cohorts of hospitals. Although there are clinical severity scores for ED patients with specific diseases,17–19 there are no widely accepted general severity measures that are applicable to all pediatric emergency patients. The PRISA score was initially developed in a single institution7 and validated in 2 small samples of large academic centers8,9; thus, it was necessary to revise the first-generation PRISA score based on a contemporary set of more diverse PEM patients.
The development of a severity measure that is applicable to pediatric ED patients has been difficult. A major barrier to developing an effective severity method that is applicable to all ED patients has been lack of a relevant outcome measure that occurs with sufficient frequency. Because one of the most important functions of ED practice is the correct disposition of patients to either inpatient or outpatient care settings, we used this outcome in developing a severity-of-illness method. In contrast to mortality and its tight linkage with physiologic derangement, hospital admission is influenced by several nonphysiologic factors, including patient age, chronic history, acute history, and variations in individual practice and health system factors. Therefore, we calibrated the score to mandatory admissions, a group of patients for whom most clinicians would agree under most circumstances would be best cared for as inpatients. We recognize, however, that there is still some practice pattern variability in the use of the inpatient therapies that compose the mandatory admission list. Most of the criteria for mandatory admission can be assessed on disposition from the ED, making this a practical assessment for future use.
Despite these difficulties, PRISA II performs well. First, mean PRISA II scores increased in patients with increasing clinical severity, and the PRISA II scores were ordered correctly for the patient groups by increasing severity. Second, the goodness-of-fit data for the development and validation samples were excellent; throughout the range of admission risks, the observed numbers of mandatory admissions were very similar to the expected numbers. Third, the discrimination of the score was very good with an area under the curve of 0.82 for the development sample and 0.77 for the validation sample.
The component variables of PRISA II are consistent with the clinical assessment of children. The score is composed of 16 variables and 1 adjustment for physiologic interactions. Eight of the variables are physiologic with relative values ranging from 3 to 14, and 3 have age-adjusted scoring. There are 2 chief complaint variables, minor injury and abdominal pain in an adolescent. Minor injury has a score of –2, indicating reduced admission risk for a given set of historical and physiologic variables, and abdominal pain in an adolescent is a patient condition that, regardless of other variables, is more likely to result in admission. Other historical variables are important in our model. Very young age by itself is a risk factor, as are immunodeficiency, indwelling medical devices, chronic asthma medications, and referral status. One therapeutic variable, oxygen, was sufficiently important to be included; its use is generally well governed by other physiologic variables such as respiratory distress and oxygen saturation. Of note, the score of each variable is determined by its importance in the presence of all variables. Thus, scores may be lower or higher than might be expected if that variable were assessed alone, out of the context of other variables.
There are 2 limitations to this study. First, our sample is not nationally representative of all hospitals. The block random selection was designed to include equal representation of academic and nonacademic EDs, large and small EDs, hospitals with and without PEM specialists, and all possible combinations of these factors. Some may believe that different comparison groups would be better. We believe that although this is not a random sample of national EDs, it perhaps is a better representation of the diversity of the types of EDs, which might not be otherwise well captured in a relatively small random national sample. Most important, our sample is well defined so that future researchers will be able to consider the impact of sample selection. Second, although the PRISA II score is the best effort to date to develop a general severity tool, the discrimination is very good but is not sufficient for use in individual patients. However, the use of PRISA II for large groups of patients to compare performance and for controlling for severity of illness and other case-mix factors for studies is appropriate.
CONCLUSION
Severity-of-illness methods have been useful for clinical research, for performance assessment, and for comparing institutional performance with external benchmarks. We have developed and validated a second-generation severity score for application in EDs that treat children. In the study sample of 16 EDs, there were wide variations in severity-adjusted admission rates, suggesting that this tool will be useful for quality assessment.
ACKNOWLEDGMENTS
This study was supported by the Agency for Health Care Quality and Research, Grant RO1 HS10238-02.
FOOTNOTES
Accepted Jul 6, 2004.
No conflict of interest declared.
REFERENCES
Pollack MM, Patel KM, Ruttimann UE. PRISM III: an updated Pediatric Risk of Mortality Score. Crit Care Med. 1996;24 :743 –752
Pollack MM, Cuerdon TC, Patel KM, Ruttimann UE, Getson PR, Levetown M. Impact of quality-of-care factors on pediatric Intensive Care Unit mortality. JAMA. 1994;272 :941 –946
Pollack MM, Yeh TS, Ruttimann UE, Holbrook PR, Fields AI. Evaluation of pediatric intensive care. Crit Care Med. 1984;12 :376 –383
Pollack MM, Getson PR, Ruttimann UE, et al. Efficiency intensive care. A comparative analysis of eight pediatric intensive care units. JAMA. 1987;258 :1481 –1486
Ruttimann UE, Pollack MM. Variability in duration of stay in pediatric intensive care units: a multiinstitutional study. J Pediatr. 1996;128 :35 –44
Knudson MM, Shagoury C, Lewis FR. Can adult trauma surgeons care for injured children J Trauma. 1992;32 :729 –737
Chamberlain JM, Patel KM, Ruttiman UE, Pollack MM. Pediatric Risk of Admission (PRISA): a measure of severity of illness for assessing the risk of hospitalization from the emergency department. Ann Emerg Med. 1998;32 :161 –169
Chamberlain JM, Patel KM, Pollack MM, et al, for the Collaborative Research Committee of the Emergency Medicine Section of the American Academy of Pediatrics. Recalibration of the pediatric risk of admission score using a multi-institutional sample. Ann Emerg Med. 2004;43 :461 –468
Gravel J, Gouin S, Amre D, Bergeron S, Lacroix J. Evaluation of the pediatric risk of admission score in a pediatric emergency department. Ann Emerg Med. 2003;41 :630 –638
Chamberlain JM, Patel KM, Pollack MM. Emergency department residents are associated with lower quality of care. Presented at the 2003 Annual Meeting of The Pediatric Academic Societies; Seattle, WA; May 5, 2003
Armitage P. Statistical Methods in Medical Research. New York, NY: John Wiley and Sons; 1971:363–365
Keeler EB, Rubenstein LV, Kahn KL, et al. Hospital characteristics and quality of care. JAMA. 1992;268 :1709 –1714
Brennan TA, Hebert LE, Laird NM, et al. Hospital characteristics associated with adverse events and substandard care. JAMA. 1991;265 :3265 –3269
Hartz AJ, Krakauer H, Kuhn EM, et al. Hospital characteristics and mortality rates. N Engl J Med. 1989;321 :1720 –1725
Laffel GL, Barnett AI, Finkelstein S, Kaye MP. The relationship between experience and outcome in heart transplant patients. N Engl J Med. 1992;327 :1220 –1225
Fink A, Yano EL, Brook RH. The condition of the literature on differences in hospital mortality. Med Care. 1989;27 :315 –336
Ramenofsky ML, Ramenofsky MB, Jurkovich GJ, et al. The predictive value of the Pediatric Trauma Score. J Trauma. 1988;28 :1038 –1042
Smith SR, Baty JD. Hodge D 3rd. Validation of the pulmonary score: an asthma severity score for children. Acad Emerg Med. 2002;9 :99 –104
Westley CR, Cotton EK, Brook JG. Nebulized racemic epinephrine by IPPB for the treatment of croup: a double-blind study. Am J Dis Child. 1978;132 :484 –487(James M. Chamberlain, MD,)
Department of Emergency Medicine Center for Health Services and Community Research, Children's Research Institute
Center for Hospital-Based Specialties and Department of Critical Care Medicine, Children's National Medical Center, Washington, DC
ABSTRACT
Objective. To develop and validate a second-generation severity-of-illness score that is applicable to pediatric emergency patients. The Pediatric Risk of Admission (PRISA) score was developed in a single hospital and was recalibrated and validated in 2, previous, small studies from academic pediatric hospitals. This study was performed to develop and validate a score in a larger sample of diverse hospitals.
Methods. Emergency departments (EDs) were block randomly selected as part of a study on ED quality on the basis of 3 care characteristics: annual patient volume (high or low compared with national median), presence or absence of a pediatric emergency medicine subspecialist, and presence or absence of residents. Patients were selected randomly on the basis of daily arrival logs. Medical records were photocopied, and abstracted data included demographic, historical, physiologic, and therapeutic information. The total sample was randomly divided into a 75% development sample and a 25% validation sample. Univariate and multivariate analyses were used to model the risk of mandatory admission, admissions for which preidentified, inpatient medical resources were used. The resulting multiple logistic regression model coefficients were converted to integer scores. Calibration (Hosmer-Lemeshow goodness of fit) and discrimination (area under the ROC curve) were used to measure performance. As a measure of construct validity, proportions of patients in ordered risk intervals were correlated with the outcomes of admission, mandatory admission, and ICU admission.
Results. Sixteen EDs enrolled 11664 patients. Mean patient age (±SD) was 6.8 ± 5.8 years, and 53% were male. Nine percent arrived by emergency medical services, and 6.9% were admitted. The most common diagnoses were minor injuries, otitis media, and fever. The multivariate analysis yielded a score with 7 historical variables, 8 physiologic variables, 1 therapy (oxygen) term, and 1 interaction term. Calibration was excellent. In the development sample, 442 mandatory admissions were predicted and 442 were observed (total 2 = 2.275), and in the validation sample, 136.6 were predicted and 145 were observed (2 = 8.575). The area under the receiver operator characteristic curve was 0.82 ± 0.01 (SE) in the development sample and 0.77 ± 0.02 in the validation sample. In ordered predicted risk intervals, the proportion of patients with admissions, mandatory admissions, and ICU admissions increased in a linear manner.
Conclusions. The second-generation PRISA II score for pediatric ED patients has been developed and validated in a large sample of diverse hospitals. Performance characteristics indicate that PRISA II will be useful for institutional comparisons, benchmarking, and controlling for severity of illness when enrolling patients in clinical trials.
Key Words: emergency medicine health services research severity-of-illness index
Abbreviations: ED, emergency department PRISA, Pediatric Risk of Admission PEM, pediatric emergency medicine ROC, receiver operator characteristic
Specific components of the Emergency Medical Services system and the Emergency Medical Services–Children system have benefited from the development and use of severity assessment methods. For example, PICU severity methods have enabled evaluation of quality of PICU care using severity-adjusted mortality rates,1 identification of PICU care characteristics associated with quality of care,2,3 measurement of efficiency of bed use,4 and measurement of severity-adjusted length of stay.5 Such methods have not been applied to pediatric or adult emergency departments (EDs). Although severity scales exist for specific diseases, there is no widely accepted method that is applicable to all pediatric ED patients.
Previous severity-of-illness methods applied to hospital and ED environments (eg, trauma patients) have used mortality as the outcome for quality-of-care studies.6 However, mortality is not an appropriate outcome for the predominant ED activities and is so rare in the general pediatric ED population that it cannot be used as a meaningful outcome measure. One of the most important functions of ED practice is the correct disposition of patients to either inpatient or outpatient care settings. Overadmission results in wasted resources and the risk of iatrogenic injury, whereas failure to admit when necessary results in delayed care and avoidable complications. We propose that the appropriate outcomes that are indicative of pediatric ED quality are correct rates of hospital admission and ED discharge.
We developed a severity-of-illness model using hospital admission as the outcome. Initially developed and validated at a single institution, the Pediatric Risk of Admission (PRISA) score uses components of acute and chronic medical history, physiology, and ED therapies to model the risk of hospital admission. Higher PRISA scores were associated with higher probability of medically mandatory admissions, ICU admission, and mortality.7 The PRISA score subsequently has been validated in 2 other studies, including 1 multisite study.8,9
The purpose of this study was to develop a second-generation pediatric ED severity-of-illness (PRISA II) score in a large national sample of hospitals. We used a sample of 16 EDs block randomly selected to represent large and small EDs, those with and without pediatric emergency medicine (PEM) specialists, and those with and without residents providing ED care.
METHODS
Site Selection
Sites were selected as part of a study to evaluate the association of hospital care factors with quality of pediatric emergency medical care.10 In brief, EDs were randomly selected to represent the care factors of high or low volume ( or > the national median of pediatric visits per year), presence or absence of a PEM specialist, and presence or absence of residents delivering ED care such that there were 2 EDs representing each of the 8 possible combinations of factors. Volume was determined from the American Hospital Association 1997 Annual Survey Database (Chicago, IL) and used to randomly select EDs for telephone survey concerning the other 2 care factors. All selected hospitals had PICUs. The Institutional Review Boards of the participating hospitals approved the study.
Patient Selection
A random-number generator was used to select 2 patients per day, using the daily consecutive log of patient arrivals routinely kept at each site. We attempted to enroll 750 patients at each site. Actual patient enrollment ranged from 729 to 753 patients. For sites that enrolled >729, patients were randomly eliminated from the sample to achieve a balanced sample of 729 patients from each site, for a total study enrollment of 11664 patients.
Data Management
After personal identifiers were masked, photocopied records were submitted to the coordinating center. The entire ED record and the first 24 hours of the inpatient record, where applicable, were submitted. Data reliability was determined by interrater reliability between the data abstractor and 1 of the authors (J.M.C.) to maintain an intraclass correlation coefficient R > 0.75.
Outcomes
Admission was defined as admission to an inpatient ward or admission to an observation area for >12 hours. The primary outcome for this study was mandatory admission. Admission was classified as mandatory on the basis of a previously validated list of services usually provided in acute care hospitals.7 Mandatory admissions are considered those that receive a therapeutic resource that because of the therapy or its intensity is usually delivered as an inpatient (Table 1). Nonmandatory admissions did not receive one of these resources.
Identification of Potential Predictor Variables
Potential predictor variables included historical, physiologic, and other risk factors. Unmeasured variables were assumed to be normal. Several variable types were excluded from the potential predictor variables. First, variables that are likely to be influenced by hospital practice, individual practice patterns, or medical recording practices were not included. Thus, we minimized the consideration of therapeutic variables. Second, socioeconomic factors were not included as predictor variables. Third, we considered ranges of physiologic variables rather then absolute values to reduce the effect of sampling frequency.
Multiple logistic regression was used to model the probability of the mandatory admission using a process similar to development of the PRISM III score.1 The patient sample was randomly divided into a 75% development sample and a 25% validation sample. The independent variables for possible inclusion in the predictive model were selected using a multistep process. First, variables were chosen for clinical relevance on the basis of clinical experience and previous work in developing severity adjustment models. Second, univariate analyses were used to determine associations with the dependent variable. For categorical variables, this was performed using unadjusted odds ratios. Some categorical variables were uncommon in ED populations and therefore were combined with other, clinically similar variables. For example, the chronic disease variables "HIV" and "oncologic disease" were combined with other conditions into a single category "immunodeficiency." For continuous variables, patients were stratified into the age groups of neonates (0–1 month), infants (>1–12 months), children (>12–144 months), and adolescents (>144 months). Univariate logistic regression analysis was used to determine which variables and subranges significantly contributed to the probability of hospital admission. The risk of hospital admission relative to the midrange (40th–60th percentiles) of discharged patients was computed for the physiologic variables in each age stratum. We assumed that deviations from the midrange contributed to risk, with larger deviations reflecting higher risk. We initially used deciles of physiologic values, then readjusted these cutpoints on the basis of clinical relevance and performance in the multivariate model. Neighboring deciles were combined when the regression coefficients were similar.1 Variables with a liberal significance level (P < .30) in univariate analysis were included in the multivariate model. Third, forward and backward stepwise inclusion of variables was used in the development of the multiple logistic regression model. Variables were retained as long as they contributed to a lower Akaike's Information Criteria and were statistically significant at P < .10. The regression coefficients in the model were expressed as relative percentages by dividing each of the coefficients by the sum of the regression coefficients and multiplying by 100. The rounded percentages for each variable and variable range are the components of the PRISA II score, and the total for all variables is the PRISA II score for each patient. Although most variables contribute to a higher risk of hospital admission, a variable may have a negative regression coefficient (and thus a corresponding negative integer score), indicating that the presence of that variable in an individual patient contributes to a lower risk of hospital admission.
Standard performance measures were used to assess the PRISA II score. The area under the curve for the receiver operator characteristic (ROC) curve was used to measure discrimination. Calibration compared observed outcomes with predicted outcomes, where the predicted outcomes were the number predicted by the PRISA II score. The Hosmer-Lemeshow goodness-of-fit tests (2) were used to measure calibration by comparing observed with predicted admissions in 8 ordered risk intervals of predicted admission probability. Risk intervals were chosen to be clinically relevant, to ensure at least 10 predicted admissions in each interval in the development sample, and to attempt to achieve a balanced distribution of admitted patients across the intervals. Overall performance was also assessed with the standardized admission ratio, the ratio of observed mandatory admissions to predicted mandatory admissions. Comparison of mean PRISA II score among different severity subgroups used the Kruskal-Wallis test. Construct validity was assessed by comparing proportions of patients with admission, mandatory admission, and ICU admission in the ordered PRISA II risk intervals using the Cochran-Armitage test for linear trend in proportion.11
RESULTS
Sample and Patient Data
Institutional and patient characteristics are depicted in Table 2. The 16 sites ranged in annual pediatric ED volumes from 1740 to 31000 patients (median: 13905). Four sites had pediatric volumes <8000, 4 sites had volumes between 8000 and 13900, 4 sites had volumes between 13900 and 17500, and 4 sites had volumes >17500. Consistent with the study design for site enrollment, 8 sites had residents caring for patients in the ED, and 8 sites had at least 1 PEM specialist in the practice. The most common chief complaints were fever (17.5%), nonspecific respiratory complaints (14.5%), and injuries to the extremities (10.4%) and head (9.4%). The most common diagnoses were minor injuries (24.2%), otitis media (9.2%), and fever without a source (5.0%). There was significant variability among the EDs for all institutional and patient population characteristics (P < .01 for all variables).
Of the 11664 patients, 808 (6.9%) were hospitalized and 93 (0.8%) patients were admitted to ICUs. Three patients died in the ED and were included in the hospitalized group. Of the 808 admitted patients, 587 (73%) were mandatory admissions. Admission rates ranged from 1.4% to 17% (P < .001), and mandatory admission rates ranged from 1.0% to 13.7% (P < .001) of ED patients.
Severity-of-Illness Model
Univariate analyses revealed 10 historical and 29 physiologic variables associated with mandatory hospital admission. One therapeutic variable, oxygen, was substantially more strongly associated with the outcome than associated physiologic variables, including pulse oximetry and respiratory rate, and thus was included for testing in multivariate modeling. One variable, low temperature, was not statistically significant but was considered clinically significant and thus was retained in the model.
The component variables identified in multivariate analysis are shown in Table 3. There are 16 variables and 1 interaction term. Seven of the variables are historical, 8 are physiologic, and 1 is a therapeutic variable. For the physiologic variables, none required multiple ranges and 3 (temperature, systolic blood pressure, and diastolic blood pressure) are age-adjusted. The 1 therapeutic variable, oxygen, is 1 of the most important predictor variables. Two of the variables, minor injury and the interaction term, were negative. Thus, minor injury reduces the likelihood of admission, whereas the interaction term attenuates the cumulative effects of having both elevated potassium and low serum bicarbonate.
Overall, the PRISA II score performed very well. Figure 1 illustrates the increasing PRISA II score with cohorts of increasing illness severity: ED discharges, nonmandatory hospital admissions, mandatory non-ICU hospital admissions, and ICU admissions. There was a significant trend in increasing PRISA II scores, and each group's score was significantly different from the others with the exception of ICU admissions compared with mandatory non-ICU admissions (P = .085).
The Hosmer-Lemeshow goodness-of-fit test indicated excellent calibration in both the development sample (2 = 2.275, degrees of freedom, df = 6, P = .8928; Table 5, Fig 2) and validation sample (2 = 8.575, df = 7, P = .2846; Table 6, Fig 2). The standardized mandatory admission ratios for the total samples were 1.00 ± 0.04 and 1.06 ± 0.07 in the development and validation samples. The area under the ROC curve was 0.822 ± 0.012 in the development sample and 0.774 ± 0.023 in the validation sample (Fig 3).
Construct validity is depicted in Fig. 4. The figure illustrates the increasing proportion of patients with admissions, mandatory admissions, and ICU admissions as PRISA II risk intervals increase (P < .0001).
DISCUSSION
Severity-of-illness methods are important for measuring and controlling for case-mix variability in many situations, including scientific studies and institutional comparisons.12–16 Such tools have been used extensively in pediatric and neonatal critical care and have allowed comparisons among individual hospitals and among cohorts of hospitals. Although there are clinical severity scores for ED patients with specific diseases,17–19 there are no widely accepted general severity measures that are applicable to all pediatric emergency patients. The PRISA score was initially developed in a single institution7 and validated in 2 small samples of large academic centers8,9; thus, it was necessary to revise the first-generation PRISA score based on a contemporary set of more diverse PEM patients.
The development of a severity measure that is applicable to pediatric ED patients has been difficult. A major barrier to developing an effective severity method that is applicable to all ED patients has been lack of a relevant outcome measure that occurs with sufficient frequency. Because one of the most important functions of ED practice is the correct disposition of patients to either inpatient or outpatient care settings, we used this outcome in developing a severity-of-illness method. In contrast to mortality and its tight linkage with physiologic derangement, hospital admission is influenced by several nonphysiologic factors, including patient age, chronic history, acute history, and variations in individual practice and health system factors. Therefore, we calibrated the score to mandatory admissions, a group of patients for whom most clinicians would agree under most circumstances would be best cared for as inpatients. We recognize, however, that there is still some practice pattern variability in the use of the inpatient therapies that compose the mandatory admission list. Most of the criteria for mandatory admission can be assessed on disposition from the ED, making this a practical assessment for future use.
Despite these difficulties, PRISA II performs well. First, mean PRISA II scores increased in patients with increasing clinical severity, and the PRISA II scores were ordered correctly for the patient groups by increasing severity. Second, the goodness-of-fit data for the development and validation samples were excellent; throughout the range of admission risks, the observed numbers of mandatory admissions were very similar to the expected numbers. Third, the discrimination of the score was very good with an area under the curve of 0.82 for the development sample and 0.77 for the validation sample.
The component variables of PRISA II are consistent with the clinical assessment of children. The score is composed of 16 variables and 1 adjustment for physiologic interactions. Eight of the variables are physiologic with relative values ranging from 3 to 14, and 3 have age-adjusted scoring. There are 2 chief complaint variables, minor injury and abdominal pain in an adolescent. Minor injury has a score of –2, indicating reduced admission risk for a given set of historical and physiologic variables, and abdominal pain in an adolescent is a patient condition that, regardless of other variables, is more likely to result in admission. Other historical variables are important in our model. Very young age by itself is a risk factor, as are immunodeficiency, indwelling medical devices, chronic asthma medications, and referral status. One therapeutic variable, oxygen, was sufficiently important to be included; its use is generally well governed by other physiologic variables such as respiratory distress and oxygen saturation. Of note, the score of each variable is determined by its importance in the presence of all variables. Thus, scores may be lower or higher than might be expected if that variable were assessed alone, out of the context of other variables.
There are 2 limitations to this study. First, our sample is not nationally representative of all hospitals. The block random selection was designed to include equal representation of academic and nonacademic EDs, large and small EDs, hospitals with and without PEM specialists, and all possible combinations of these factors. Some may believe that different comparison groups would be better. We believe that although this is not a random sample of national EDs, it perhaps is a better representation of the diversity of the types of EDs, which might not be otherwise well captured in a relatively small random national sample. Most important, our sample is well defined so that future researchers will be able to consider the impact of sample selection. Second, although the PRISA II score is the best effort to date to develop a general severity tool, the discrimination is very good but is not sufficient for use in individual patients. However, the use of PRISA II for large groups of patients to compare performance and for controlling for severity of illness and other case-mix factors for studies is appropriate.
CONCLUSION
Severity-of-illness methods have been useful for clinical research, for performance assessment, and for comparing institutional performance with external benchmarks. We have developed and validated a second-generation severity score for application in EDs that treat children. In the study sample of 16 EDs, there were wide variations in severity-adjusted admission rates, suggesting that this tool will be useful for quality assessment.
ACKNOWLEDGMENTS
This study was supported by the Agency for Health Care Quality and Research, Grant RO1 HS10238-02.
FOOTNOTES
Accepted Jul 6, 2004.
No conflict of interest declared.
REFERENCES
Pollack MM, Patel KM, Ruttimann UE. PRISM III: an updated Pediatric Risk of Mortality Score. Crit Care Med. 1996;24 :743 –752
Pollack MM, Cuerdon TC, Patel KM, Ruttimann UE, Getson PR, Levetown M. Impact of quality-of-care factors on pediatric Intensive Care Unit mortality. JAMA. 1994;272 :941 –946
Pollack MM, Yeh TS, Ruttimann UE, Holbrook PR, Fields AI. Evaluation of pediatric intensive care. Crit Care Med. 1984;12 :376 –383
Pollack MM, Getson PR, Ruttimann UE, et al. Efficiency intensive care. A comparative analysis of eight pediatric intensive care units. JAMA. 1987;258 :1481 –1486
Ruttimann UE, Pollack MM. Variability in duration of stay in pediatric intensive care units: a multiinstitutional study. J Pediatr. 1996;128 :35 –44
Knudson MM, Shagoury C, Lewis FR. Can adult trauma surgeons care for injured children J Trauma. 1992;32 :729 –737
Chamberlain JM, Patel KM, Ruttiman UE, Pollack MM. Pediatric Risk of Admission (PRISA): a measure of severity of illness for assessing the risk of hospitalization from the emergency department. Ann Emerg Med. 1998;32 :161 –169
Chamberlain JM, Patel KM, Pollack MM, et al, for the Collaborative Research Committee of the Emergency Medicine Section of the American Academy of Pediatrics. Recalibration of the pediatric risk of admission score using a multi-institutional sample. Ann Emerg Med. 2004;43 :461 –468
Gravel J, Gouin S, Amre D, Bergeron S, Lacroix J. Evaluation of the pediatric risk of admission score in a pediatric emergency department. Ann Emerg Med. 2003;41 :630 –638
Chamberlain JM, Patel KM, Pollack MM. Emergency department residents are associated with lower quality of care. Presented at the 2003 Annual Meeting of The Pediatric Academic Societies; Seattle, WA; May 5, 2003
Armitage P. Statistical Methods in Medical Research. New York, NY: John Wiley and Sons; 1971:363–365
Keeler EB, Rubenstein LV, Kahn KL, et al. Hospital characteristics and quality of care. JAMA. 1992;268 :1709 –1714
Brennan TA, Hebert LE, Laird NM, et al. Hospital characteristics associated with adverse events and substandard care. JAMA. 1991;265 :3265 –3269
Hartz AJ, Krakauer H, Kuhn EM, et al. Hospital characteristics and mortality rates. N Engl J Med. 1989;321 :1720 –1725
Laffel GL, Barnett AI, Finkelstein S, Kaye MP. The relationship between experience and outcome in heart transplant patients. N Engl J Med. 1992;327 :1220 –1225
Fink A, Yano EL, Brook RH. The condition of the literature on differences in hospital mortality. Med Care. 1989;27 :315 –336
Ramenofsky ML, Ramenofsky MB, Jurkovich GJ, et al. The predictive value of the Pediatric Trauma Score. J Trauma. 1988;28 :1038 –1042
Smith SR, Baty JD. Hodge D 3rd. Validation of the pulmonary score: an asthma severity score for children. Acad Emerg Med. 2002;9 :99 –104
Westley CR, Cotton EK, Brook JG. Nebulized racemic epinephrine by IPPB for the treatment of croup: a double-blind study. Am J Dis Child. 1978;132 :484 –487(James M. Chamberlain, MD,)