Problems in assessing rates of infection with methicillin resistant Staphylococcus aureus
1 MRC Biostatistics Unit, Institute of Public Health, Cambridge CB2 2SR
Chance variability makes it impossible to assess reliably whether individual trusts are meeting annual targets for reduction in the risk of MRSA infection
Introduction
One of the core standards set by the Department of Health is to achieve year on year reductions in rates of infection with methicillin resistant Staphylococcus aureus (MRSA).1 This was clarified in November 2004 by the (then) health secretary John Reid, who said that he expected "MRSA bloodstream infection rates to be halved in our hospitals by 2008," that "NHS Acute Trusts will be tasked with achieving a year on year reduction,"2 and that such a target was "achievable, measurable, and not too burdensome." Several problems can arise, however, when measuring change in rates, particularly when the observed number of events is fairly low. These include the effects of chance variability, regression to the mean, and low power to detect genuine underlying changes. These problems are accentuated with an infectious disease, since cases tend to cluster and hence rates are "over-dispersed" relative to chance variation.3 So how should we interpret government targets on MRSA infections
, 百拇医药
What is the target
The government target of a 50% reduction in cases over three years essentially corresponds to a 20% annual reduction. But the term target is ambiguous: does it refer to an observed change in rates or a true underlying reduction in risk At a national level this distinction may be unnecessary because of the large numbers involved, but for individual trusts the play of chance can mean that an observed rate reduction that meets the target may not accurately reflect a corresponding change in underlying risk and, conversely, that a true risk reduction may not be reflected in the observed rates, which might even increase. This lack of clarity can give rise to a range of possible criteria to determine whether a target has been met, as illustrated below.
, http://www.100md.com
Role of chance
The Department of Health publishes data on MRSA infection in individual trusts obtained through its mandatory reporting scheme.4 Using data for financial years 2001-2, 2002-3, and 2003-4, I estimated how close the variation was to that expected by chance from the Poisson regression residuals around an individual trend line for each trust.5 This showed a significant over-dispersion of 1.76 (Pearson 2 = 304.3, degrees of freedom = 167; P < 0.001). For example, Aintree Hospitals NHS Trust had 34 cases in 2001-2, rising to 66 cases in 2002-3, and falling to 48 in 2004-4, far more variability than would be expected by chance alone. Such over-dispersion is expected with an infectious disease, and means that all interval estimates of MRSA rates should be 33% wider (1.33 = 1.76) than those that assume simple random variability.
, 百拇医药
Improving measures of performance
Performance monitoring presents many challenges, and it may need a protocol10 similar to that required before a clinical trial, including careful definitions, power calculations, and so on. I have not dealt with the considerable difficulty in defining the numerator and denominator of the MRSA infection rate, but any more restrictive definition of MRSA will reduce the counts and hence only exaggerate the issues raised.
, 百拇医药
The analysis suggests that, although MRSA rates show some systematic differences between trusts, recent observed variability in annual changes has been entirely explainable by chance (fig 2). Year on year changes are therefore extremely difficult to measure since they are strongly influenced by chance variability and exhibit substantial regression to the mean Even if an average trust is truly reducing risk according to the government target, it has little chance of showing a significant reduction in observed cases on a year on year basis. At the other extreme, most trusts will show results that are statistically compatible with the target risk reduction, even if they truly have not improved. The naive target of an observed annual reduction of 20% will be failed by half the trusts that have truly reduced their risk by that extent.
, 百拇医药
Summary points
The Department of Health has set targets for reduction in MRSA rates for individual trusts
It is not clear whether the targets refer to an observed rate reduction or a true reduction in underlying risk
Recent annual changes in MRSA rates have been dominated by chance variability
Reliable annual monitoring of reductions in MRSA rates in individual trusts is not generally feasible
, 百拇医药
MRSA infection rates formed part of the 2002-3 star ratings for NHS trusts11 with annual change as a performance measure. They were excluded from the 2003-4 ratings12 but reintroduced in 2004-5 with a composite indicator based on current rate, change over a two year baseline, and presence of hand cleaning facilities.13 This seems appropriate, but if MSRA rates are to be used to assess future performance the analysis above suggests further changes should be included:
, 百拇医药
Although statistical techniques could be used to adjust for regression to the mean,14 it would be more straightforward to measure change against at least a three year baseline. This will increase the power to detect genuine change
Trusts should be rated highly if they show either an improvement over a three year baseline or a consistently low rate. It is unreasonable for trusts that already have low MRSA rates to prove further improvement, especially since they have little statistical power to provide such proof
, 百拇医药
Trusts should be penalised only if there is evidence (but not necessarily as stringent as P < 0.025) that the underlying risks have not been reduced according to the target rather than if the observed rate reduction simply fails to meet the target.
Finally, the government needs to be more precise about what it means by the term target. When it comes to assessing whether a target has been met, it is vital to distinguish between observed reduction in numbers of cases and reduction in true underlying risk. Underlying risk, though it cannot be precisely measured, is the appropriate interpretation when setting local targets.
, 百拇医药
Editorial by Duckworth and Charlett and Papers p 982
A table showing the chance of errors for different strategies in assessing targets is on bmj.com
I thank Adrian Cook, Martin Bardsley, and David Cromwell for useful discussions.
Contributors and sources: DJS works with the Healthcare Commission on performance assessment. The views expressed here are his alone. All data were obtained from publicly available sources.
, 百拇医药
Competing interests: None declared.
References
Department of Health. National standards, local action: health and social care standards and planning framework 2005/06-2007/08. London: Department of Health. 2004. www.dh.gov.uk/assetRoot/04/08/60/58/04086058.pdf (accessed 9 Aug 2005).
BBC News. Hospital superbug must be halved. BBC, 5 Nov 2004. http://news.bbc.co.uk/1/hi/health/3983077.stm (accessed 9 Aug 2005).
, 百拇医药
Health Protection Agency Communicable Disease Surveillance Centre. The second year of the Department of Health's mandatory MRSA bacteraemia surveillance scheme in acute trusts in England: April 2002-March 2003. CDR Weekly 2003;13(25):1-9. www.hpa.org.uk/cdr/PDFfiles/2003/cdr2503.pdf (accessed 9 Aug 2005).
Department of Health. Mandatory bacteraemia surveillance scheme—MRSA bacteraemia by NHS trust: April 2001-March 2004. www.dh.gov.uk/assetRoot/04/08/58/93/04085893.pdf (accessed 9 Aug 2005).
, http://www.100md.com
McCullagh P, Nelder J. Generalized linear models. 2nd ed. London: Chapman and Hall, 1989.
Spiegelhalter DJ. Funnel plots for comparing institutional performance. Stat Med 2005;24: 1185-202.
Mohammed MA, Cheng KK, Rouse A, Marshall T. Bristol, Shipman, and clinical governance: Shewhart's forgotten lessons. Lancet 2001;357: 463-7.
Bland M, Altman DG. Regression towards the mean. BMJ 1994;308: 1499.
, 百拇医药
BBC News. Superbug league tables published. London: BBC, 13 July 2004. http://news.bbc.co.uk/1/hi/health/3891459.stm (accessed 9 Aug 2005).
Bird S, Cox DR, Farewell VT, Goldstein H, Holt T, Smith PC. Performance indicators: good, bad and ugly. J R Stat Soc A 2005;168: 1-28.
Commission for Health Improvement. Methicillin resistant Staphylococcus aureus (MRSA) bacteraemia improvement score, 2003. www.chi.nhs.uk/Ratings/Trust/Indicator/indicatorDescriptionShort.aspindicatorId=1610 (accessed 9 Aug 2005).
, http://www.100md.com
Healthcare Commission. Indicator listings for acute trusts, 2004. http://ratings2004.healthcarecommission.org.uk/Trust/Indicator/indicators.asptrustType=1 (accessed 9 Aug 2005).
Healthcare Commission. Indicator listings for acute trusts: MRSA, 2005. http://ratings.healthcarecommission.org.uk/Indicators_2005/Trust/Indicator/ indicatorDescriptionShort.aspindicatorId=1348 (accessed 9 Aug 2005).
Burgess JF, Christiansen CL, Michalak SE, Morris CN. Medical profiling: improving standards and risk adjustments using hierarchical models. J Health Econ 2000;19: 291-309., 百拇医药(David J Spiegelhalter)
Chance variability makes it impossible to assess reliably whether individual trusts are meeting annual targets for reduction in the risk of MRSA infection
Introduction
One of the core standards set by the Department of Health is to achieve year on year reductions in rates of infection with methicillin resistant Staphylococcus aureus (MRSA).1 This was clarified in November 2004 by the (then) health secretary John Reid, who said that he expected "MRSA bloodstream infection rates to be halved in our hospitals by 2008," that "NHS Acute Trusts will be tasked with achieving a year on year reduction,"2 and that such a target was "achievable, measurable, and not too burdensome." Several problems can arise, however, when measuring change in rates, particularly when the observed number of events is fairly low. These include the effects of chance variability, regression to the mean, and low power to detect genuine underlying changes. These problems are accentuated with an infectious disease, since cases tend to cluster and hence rates are "over-dispersed" relative to chance variation.3 So how should we interpret government targets on MRSA infections
, 百拇医药
What is the target
The government target of a 50% reduction in cases over three years essentially corresponds to a 20% annual reduction. But the term target is ambiguous: does it refer to an observed change in rates or a true underlying reduction in risk At a national level this distinction may be unnecessary because of the large numbers involved, but for individual trusts the play of chance can mean that an observed rate reduction that meets the target may not accurately reflect a corresponding change in underlying risk and, conversely, that a true risk reduction may not be reflected in the observed rates, which might even increase. This lack of clarity can give rise to a range of possible criteria to determine whether a target has been met, as illustrated below.
, http://www.100md.com
Role of chance
The Department of Health publishes data on MRSA infection in individual trusts obtained through its mandatory reporting scheme.4 Using data for financial years 2001-2, 2002-3, and 2003-4, I estimated how close the variation was to that expected by chance from the Poisson regression residuals around an individual trend line for each trust.5 This showed a significant over-dispersion of 1.76 (Pearson 2 = 304.3, degrees of freedom = 167; P < 0.001). For example, Aintree Hospitals NHS Trust had 34 cases in 2001-2, rising to 66 cases in 2002-3, and falling to 48 in 2004-4, far more variability than would be expected by chance alone. Such over-dispersion is expected with an infectious disease, and means that all interval estimates of MRSA rates should be 33% wider (1.33 = 1.76) than those that assume simple random variability.
, 百拇医药
Improving measures of performance
Performance monitoring presents many challenges, and it may need a protocol10 similar to that required before a clinical trial, including careful definitions, power calculations, and so on. I have not dealt with the considerable difficulty in defining the numerator and denominator of the MRSA infection rate, but any more restrictive definition of MRSA will reduce the counts and hence only exaggerate the issues raised.
, 百拇医药
The analysis suggests that, although MRSA rates show some systematic differences between trusts, recent observed variability in annual changes has been entirely explainable by chance (fig 2). Year on year changes are therefore extremely difficult to measure since they are strongly influenced by chance variability and exhibit substantial regression to the mean Even if an average trust is truly reducing risk according to the government target, it has little chance of showing a significant reduction in observed cases on a year on year basis. At the other extreme, most trusts will show results that are statistically compatible with the target risk reduction, even if they truly have not improved. The naive target of an observed annual reduction of 20% will be failed by half the trusts that have truly reduced their risk by that extent.
, 百拇医药
Summary points
The Department of Health has set targets for reduction in MRSA rates for individual trusts
It is not clear whether the targets refer to an observed rate reduction or a true reduction in underlying risk
Recent annual changes in MRSA rates have been dominated by chance variability
Reliable annual monitoring of reductions in MRSA rates in individual trusts is not generally feasible
, 百拇医药
MRSA infection rates formed part of the 2002-3 star ratings for NHS trusts11 with annual change as a performance measure. They were excluded from the 2003-4 ratings12 but reintroduced in 2004-5 with a composite indicator based on current rate, change over a two year baseline, and presence of hand cleaning facilities.13 This seems appropriate, but if MSRA rates are to be used to assess future performance the analysis above suggests further changes should be included:
, 百拇医药
Although statistical techniques could be used to adjust for regression to the mean,14 it would be more straightforward to measure change against at least a three year baseline. This will increase the power to detect genuine change
Trusts should be rated highly if they show either an improvement over a three year baseline or a consistently low rate. It is unreasonable for trusts that already have low MRSA rates to prove further improvement, especially since they have little statistical power to provide such proof
, 百拇医药
Trusts should be penalised only if there is evidence (but not necessarily as stringent as P < 0.025) that the underlying risks have not been reduced according to the target rather than if the observed rate reduction simply fails to meet the target.
Finally, the government needs to be more precise about what it means by the term target. When it comes to assessing whether a target has been met, it is vital to distinguish between observed reduction in numbers of cases and reduction in true underlying risk. Underlying risk, though it cannot be precisely measured, is the appropriate interpretation when setting local targets.
, 百拇医药
Editorial by Duckworth and Charlett and Papers p 982
A table showing the chance of errors for different strategies in assessing targets is on bmj.com
I thank Adrian Cook, Martin Bardsley, and David Cromwell for useful discussions.
Contributors and sources: DJS works with the Healthcare Commission on performance assessment. The views expressed here are his alone. All data were obtained from publicly available sources.
, 百拇医药
Competing interests: None declared.
References
Department of Health. National standards, local action: health and social care standards and planning framework 2005/06-2007/08. London: Department of Health. 2004. www.dh.gov.uk/assetRoot/04/08/60/58/04086058.pdf (accessed 9 Aug 2005).
BBC News. Hospital superbug must be halved. BBC, 5 Nov 2004. http://news.bbc.co.uk/1/hi/health/3983077.stm (accessed 9 Aug 2005).
, 百拇医药
Health Protection Agency Communicable Disease Surveillance Centre. The second year of the Department of Health's mandatory MRSA bacteraemia surveillance scheme in acute trusts in England: April 2002-March 2003. CDR Weekly 2003;13(25):1-9. www.hpa.org.uk/cdr/PDFfiles/2003/cdr2503.pdf (accessed 9 Aug 2005).
Department of Health. Mandatory bacteraemia surveillance scheme—MRSA bacteraemia by NHS trust: April 2001-March 2004. www.dh.gov.uk/assetRoot/04/08/58/93/04085893.pdf (accessed 9 Aug 2005).
, http://www.100md.com
McCullagh P, Nelder J. Generalized linear models. 2nd ed. London: Chapman and Hall, 1989.
Spiegelhalter DJ. Funnel plots for comparing institutional performance. Stat Med 2005;24: 1185-202.
Mohammed MA, Cheng KK, Rouse A, Marshall T. Bristol, Shipman, and clinical governance: Shewhart's forgotten lessons. Lancet 2001;357: 463-7.
Bland M, Altman DG. Regression towards the mean. BMJ 1994;308: 1499.
, 百拇医药
BBC News. Superbug league tables published. London: BBC, 13 July 2004. http://news.bbc.co.uk/1/hi/health/3891459.stm (accessed 9 Aug 2005).
Bird S, Cox DR, Farewell VT, Goldstein H, Holt T, Smith PC. Performance indicators: good, bad and ugly. J R Stat Soc A 2005;168: 1-28.
Commission for Health Improvement. Methicillin resistant Staphylococcus aureus (MRSA) bacteraemia improvement score, 2003. www.chi.nhs.uk/Ratings/Trust/Indicator/indicatorDescriptionShort.aspindicatorId=1610 (accessed 9 Aug 2005).
, http://www.100md.com
Healthcare Commission. Indicator listings for acute trusts, 2004. http://ratings2004.healthcarecommission.org.uk/Trust/Indicator/indicators.asptrustType=1 (accessed 9 Aug 2005).
Healthcare Commission. Indicator listings for acute trusts: MRSA, 2005. http://ratings.healthcarecommission.org.uk/Indicators_2005/Trust/Indicator/ indicatorDescriptionShort.aspindicatorId=1348 (accessed 9 Aug 2005).
Burgess JF, Christiansen CL, Michalak SE, Morris CN. Medical profiling: improving standards and risk adjustments using hierarchical models. J Health Econ 2000;19: 291-309., 百拇医药(David J Spiegelhalter)