Retrospective cohort study of false alarm rates associated with a series of heart operations: the case for hospital mortality monitoring gro
http://www.100md.com
《英国医生杂志》
1 Community Health Sciences, St George's Hospital Medical School, London SW17 0RE, 2 St George's Healthcare NHS Trust, London SW17 0QT
Correspondence to: J Poloniecki j.poloniecki@sghms.ac.uk
Abstract
In September 2000 heart transplantation at St George's Hospital, London, was suspended because of concern that more patients were dying than previously. The newspapers reported that 80% mortality in the last 10 cases had been of particular concern because this was "more than five times the national average."1 We tested these assumptions—that surgical results had been satisfactory but later became unsatisfactory—against numerical criteria.
Methods
Average mortality
The death rate exceeded the benchmark of 15% from the fourth operation onwards (fig 1) but did not become significant—that is, P value below 5%—until operation number 16. For the complete series, the observed mortality was 21% (P = 0.0015, two tailed). The probability of a type I error from repeated significance testing throughout the series is 0.17 (see table 3)—that is, this test has a false positive rate of about 1 in 6.
Fig 1 Average mortality in 371 heart transplantations in one hospital compared with national average (15%)
Table 3 Summary of test results to detect excess mortality in series of heart transplantations
Excess mortality
The death rate was above 20% by the fourth operation but this was not significant (fig 2). By operation number 19 excess mortality was significant (P < 0.05, one tailed). At the end of the series there was no significant evidence for excess mortality.
Fig 2 Excess mortality in 371 heart transplantations in one hospital compared with national average plus margin of 5%
Moving average
The death rate as a moving average of 10 operations reached 80% only once, at operation 230 (fig 3). At other times, the moving average was not significant for deaths within 30 days, as defined here, including at the end of the series, when the moving average was 50%. The newspaper account of eight deaths in the last 10 cases was presumably based on a different period of survival or sequencing of cases.
Fig 3 Moving average of 10 heart transplantations in which there were eight or more deaths
Runs of deaths
The longest run of consecutive deaths was five, and this occurred only once, at operation number 230 (operation 230 being the fifth in the run) (fig 4). Only two deaths occurred in the last five cases. The type I error rate for repeated examination for a run of five or more deaths in a series of 371 operations with 15% event rate is 0.023 or 1 in 43.
Fig 4 Run length of deaths when five consecutive patients undergoing heart transplantation died
Sequential probability ratio test (SPRT)
At operation number 56 the sequential probability ratio test indicates that the death rate was 20% rather than 15% (fig 5). The type I error rate for the repeated test was set to 5%. Strictly speaking, the plot and the test are not relevant after one of the control lines has been crossed, because once the decision between a benchmark death rate of 15% or 20% has been taken the test does not allow for a reversal of the decision. The final point on the plot was above the 20% limit.
Fig 5 Sequential probability ratio test in 371 heart transplantations in one hospital
Cusum graph with v-mask
A truncated v-mask is shown in figure 6 at operation number 57, which was the first occasion that a mortality greater than 15% was signalled. The mask is shown again at the end of the series. If we assuming no change in death rate from 15%, the average run length before a different death rate is signalled would be 3662 operations. By contrast, if the death rate increased to 20% it would be 29 operations.
CRAM chart
Not adjusted for risk factors—Individual target risk estimates were not available for the early part of the series, so we used a uniform external risk estimate of 15% mortality to draw figure 7. The upper control limit was crossed at the first determination of the control limits, which was at operation number 104. The test result was positive in the sense that the control limits were reached and a change in death rate was signalled, but the change was towards a lower mortality than had occurred earlier in the programme. As we did not adjust for risk factors, one reason for the improvement may have been a shift to lower risk patients. As with the sequential probability ratio test, there are some uncertainties in interpretation of control limits once they have been crossed; however, it seems reasonable to infer from figure 7 that the death rate was within the limits at the end of the series.
Fig 7 CRAM chart with uniform external risk estimate of 15% for all patients, showing performance ratio of observed number of deaths to number of deaths predicted by external estimate
Risk adjusted—Data on risk factors (see box and table 2) were available from the 240th transplant, but we could not calculate control limits as there were not enough operations. The CRAM chart, however, provides prospective risk estimates for individual patients after the first 16 deaths even in the absence of control limits (fig 8). The 16th death in cases with data on risk factors was transplant number 338. For all the remaining operations the performance ratio was close to 2—that is, the observed number of deaths remained at about twice the number of deaths predicted by the risk factors in table 2.
Fig 8 CRAM chart with uniform external risk estimates of 3%, 5%, 13%, and 20% and showing performance ratio of observed numbers of deaths to number of deaths predicted by external estimates
Summary of results
At the end of the series the average mortality, sequential probability ratio, and cusum tests indicated a level of deaths higher than the benchmark, and the remaining four of the seven statistical tests yielded negative results (table 3). Six of the tests showed that the transplant programme had a level of deaths above benchmark at some point. The point at which an alarm would first have occurred varied with the choice of method. With the CRAM chart, the only change detected was a decrease in the death rate early in the programme.
Discussion
Booth J. Heart transplant deaths trigger official inquiry. Sunday Telegraph 2000 Oct 15.
Wald A. Sequential analysis. New York: Wiley, 1947.
Barnard GA. Control charts and stochastic processes. J Roy Stat Soc Ser B 1959;21: 239-71.
Poloniecki J, Valencia O, Littlejohns P. Cumulative risk adjusted mortality chart for detecting changes in death rate: observational study of heart surgery. BMJ 1998;316: 1697-700.
Basseville M, Nikiforov IV. Detection of abrupt changes: theory and application. New Jersey: Prentice-Hall, 1993.
Poloniecki J. Half of all doctors are below average. BMJ 1998;316: 1734-6.
General Medical Council. Determination of the Professional Conduct committee in the case of Mr Wisheart, Mr Dhasmana, and Dr Roylance, 18 June 1998.
De Leval MR, Francois K, Bull C, Brawn W, Spiegelhalter DJ. Analysis of a cluster of surgical failures. J Thoracic Cardiovasc Surg 1994;107: 914-24.
British Transplantation Society. Towards standards for organ and tissue transplantation in the United Kingdom. British Transplantation Society, 1998. (accessed Nov 2003).
Anyanwu AC, Rogers CA, Murday AJ. A simple approach to risk stratification in adult heart transplantation. Eur J Cardiothorac Surg 1999;16: 424-8.
Death certification and investigation in England, Wales and Northern Ireland—the report of a fundamental review 2003. London: Stationery Office, 2003. (accessed Nov 2003).(Jan Poloniecki, senior le)
Correspondence to: J Poloniecki j.poloniecki@sghms.ac.uk
Abstract
In September 2000 heart transplantation at St George's Hospital, London, was suspended because of concern that more patients were dying than previously. The newspapers reported that 80% mortality in the last 10 cases had been of particular concern because this was "more than five times the national average."1 We tested these assumptions—that surgical results had been satisfactory but later became unsatisfactory—against numerical criteria.
Methods
Average mortality
The death rate exceeded the benchmark of 15% from the fourth operation onwards (fig 1) but did not become significant—that is, P value below 5%—until operation number 16. For the complete series, the observed mortality was 21% (P = 0.0015, two tailed). The probability of a type I error from repeated significance testing throughout the series is 0.17 (see table 3)—that is, this test has a false positive rate of about 1 in 6.
Fig 1 Average mortality in 371 heart transplantations in one hospital compared with national average (15%)
Table 3 Summary of test results to detect excess mortality in series of heart transplantations
Excess mortality
The death rate was above 20% by the fourth operation but this was not significant (fig 2). By operation number 19 excess mortality was significant (P < 0.05, one tailed). At the end of the series there was no significant evidence for excess mortality.
Fig 2 Excess mortality in 371 heart transplantations in one hospital compared with national average plus margin of 5%
Moving average
The death rate as a moving average of 10 operations reached 80% only once, at operation 230 (fig 3). At other times, the moving average was not significant for deaths within 30 days, as defined here, including at the end of the series, when the moving average was 50%. The newspaper account of eight deaths in the last 10 cases was presumably based on a different period of survival or sequencing of cases.
Fig 3 Moving average of 10 heart transplantations in which there were eight or more deaths
Runs of deaths
The longest run of consecutive deaths was five, and this occurred only once, at operation number 230 (operation 230 being the fifth in the run) (fig 4). Only two deaths occurred in the last five cases. The type I error rate for repeated examination for a run of five or more deaths in a series of 371 operations with 15% event rate is 0.023 or 1 in 43.
Fig 4 Run length of deaths when five consecutive patients undergoing heart transplantation died
Sequential probability ratio test (SPRT)
At operation number 56 the sequential probability ratio test indicates that the death rate was 20% rather than 15% (fig 5). The type I error rate for the repeated test was set to 5%. Strictly speaking, the plot and the test are not relevant after one of the control lines has been crossed, because once the decision between a benchmark death rate of 15% or 20% has been taken the test does not allow for a reversal of the decision. The final point on the plot was above the 20% limit.
Fig 5 Sequential probability ratio test in 371 heart transplantations in one hospital
Cusum graph with v-mask
A truncated v-mask is shown in figure 6 at operation number 57, which was the first occasion that a mortality greater than 15% was signalled. The mask is shown again at the end of the series. If we assuming no change in death rate from 15%, the average run length before a different death rate is signalled would be 3662 operations. By contrast, if the death rate increased to 20% it would be 29 operations.
CRAM chart
Not adjusted for risk factors—Individual target risk estimates were not available for the early part of the series, so we used a uniform external risk estimate of 15% mortality to draw figure 7. The upper control limit was crossed at the first determination of the control limits, which was at operation number 104. The test result was positive in the sense that the control limits were reached and a change in death rate was signalled, but the change was towards a lower mortality than had occurred earlier in the programme. As we did not adjust for risk factors, one reason for the improvement may have been a shift to lower risk patients. As with the sequential probability ratio test, there are some uncertainties in interpretation of control limits once they have been crossed; however, it seems reasonable to infer from figure 7 that the death rate was within the limits at the end of the series.
Fig 7 CRAM chart with uniform external risk estimate of 15% for all patients, showing performance ratio of observed number of deaths to number of deaths predicted by external estimate
Risk adjusted—Data on risk factors (see box and table 2) were available from the 240th transplant, but we could not calculate control limits as there were not enough operations. The CRAM chart, however, provides prospective risk estimates for individual patients after the first 16 deaths even in the absence of control limits (fig 8). The 16th death in cases with data on risk factors was transplant number 338. For all the remaining operations the performance ratio was close to 2—that is, the observed number of deaths remained at about twice the number of deaths predicted by the risk factors in table 2.
Fig 8 CRAM chart with uniform external risk estimates of 3%, 5%, 13%, and 20% and showing performance ratio of observed numbers of deaths to number of deaths predicted by external estimates
Summary of results
At the end of the series the average mortality, sequential probability ratio, and cusum tests indicated a level of deaths higher than the benchmark, and the remaining four of the seven statistical tests yielded negative results (table 3). Six of the tests showed that the transplant programme had a level of deaths above benchmark at some point. The point at which an alarm would first have occurred varied with the choice of method. With the CRAM chart, the only change detected was a decrease in the death rate early in the programme.
Discussion
Booth J. Heart transplant deaths trigger official inquiry. Sunday Telegraph 2000 Oct 15.
Wald A. Sequential analysis. New York: Wiley, 1947.
Barnard GA. Control charts and stochastic processes. J Roy Stat Soc Ser B 1959;21: 239-71.
Poloniecki J, Valencia O, Littlejohns P. Cumulative risk adjusted mortality chart for detecting changes in death rate: observational study of heart surgery. BMJ 1998;316: 1697-700.
Basseville M, Nikiforov IV. Detection of abrupt changes: theory and application. New Jersey: Prentice-Hall, 1993.
Poloniecki J. Half of all doctors are below average. BMJ 1998;316: 1734-6.
General Medical Council. Determination of the Professional Conduct committee in the case of Mr Wisheart, Mr Dhasmana, and Dr Roylance, 18 June 1998.
De Leval MR, Francois K, Bull C, Brawn W, Spiegelhalter DJ. Analysis of a cluster of surgical failures. J Thoracic Cardiovasc Surg 1994;107: 914-24.
British Transplantation Society. Towards standards for organ and tissue transplantation in the United Kingdom. British Transplantation Society, 1998. (accessed Nov 2003).
Anyanwu AC, Rogers CA, Murday AJ. A simple approach to risk stratification in adult heart transplantation. Eur J Cardiothorac Surg 1999;16: 424-8.
Death certification and investigation in England, Wales and Northern Ireland—the report of a fundamental review 2003. London: Stationery Office, 2003. (accessed Nov 2003).(Jan Poloniecki, senior le)