Real-time monitoring of hospital performance: A practical application of the hospital and critical care outcome prediction equations (HOPE & COPE) for monitoring clinical performance in acute hospitals. Researcher: Dr Graeme Duke Software and analysis assistance: Dr. David Cook The Northern Clinical Research Centre 1
Table of Contents Abstract...2 Introduction...2 Index Procedure and Outcomes...2 Data Display Formats...2 Boundary or Control Limits...2 Multi-disciplinary Review Panel...2 Methods...2 Stage 1...2 Stage 2...2 Stage 3....2 Stage 4...2 Results...2 Conclusions...2 References...2 Table 1 Frequency of alerts during simulation...2 Figure 1. Average run length estimates for EWMA and COPE model...2 Figure 2. Average run length estimates for EWMA and HOPE model....2 Figure 3. Frequency of Level 1 (yellow bars) and Level 2 (red bars) alerts for Group 1...2 Appendix 1: FLOWCHART of proposed outlier investigation process...2 Appendix 2: Example of risk-adjusted EWMA control chart...2 2
Abstract OBJECTIVE: To present a real-time monitoring and governance process for assessment of hospital-wide clinical performance, and to investigate its performance by simulation. DESIGN: A four-stage governance process incorporating three graded alert levels, based on the Critical Care Outcome Prediction Equation (COPE) model and the Hospital Outcome Prediction Equation (HOPE) developed by the Northern Clinical Research Centre. SETTING: Twenty-three, major Victorian public hospitals, Australia. PATIENTS: Two patient groups: Group 1 all hospital inpatient admissions; Group 2 all patients admitted to ICU during their hospital stay. MAIN OUTCOME MEASURES: Risk-adjusted hospital outcome (death) analysed by SMR (95% confidence intervals) and process control chart with control limits set and 2 and 3 standard deviations of the predicted outcome. The frequency of three pre-determined alert levels were measured. Alerts were defined as Level 1 alert >2SD from predicted average; Level 2 alert >3SD from predicted average; and Level 3 alert >3SD from predicted average for two or more consecutive months. RESULTS: Group 1 comprised 311,541 patients (2.35% mortality) and Group 2 comprised 17,522 patients (12.30% mortality). Simulated monitoring of Groups 1 and 2 for high outliers (poor performers) revealed a Level 1 alert rate of 2.7 and 1.6 per month, and Level 2 alert rates of 0.8 and 0.4 per month, respectively. No Level 3 alerts occurred, indicating none of the 23 hospitals were designated as true outliers. CONCLUSION: A practical four-stage process for real-time monitoring of hospital performance appears feasible using the Critical Care Outcome Prediction Equation (COPE) model and the Hospital Outcome Prediction Equation (HOPE). This process appears to provide a practical application for real-time monitoring of hospital performance. 3
Introduction Performance monitoring compares system outputs to a given standard. Its unavoidable consequence is the identification of outliers those whose performance appears to be outside the benchmark. No monitoring system can reliably separate true inliers from the true outliers. This is because of the complexity and variations inherent in patients and healthcare services and the inaccuracy of monitoring methodologies (1). A well constructed system for monitoring clinical performance will allow for randomly occurring variations arising from natural (common cause) variations and still be able to identify non-random (special cause) variation due to change in quality of care so that true outliers can be identified. A process for separating true outliers from apparent outliers is necessary for several reasons, chiefly to avoid misclassification and unnecessary investigations. An outlier identification process should provide a transparent, logical, and sequential approach to identifying and investigating potential outliers. This process has several goals: 1. To demonstrate stability within the system (true inliers) 2. To prevent misinterpretation of results and minimise the risk of incorrectly classifying hospitals as outliers (false outliers) 3. To identify true outliers who perform above the benchmark (State average) and may be an example of best practice ; and 4. To identify true outliers who perform below the benchmark that may require additional resources to improve performance. Several elements are required for a functional monitoring system. These include a clearly defined index procedure and outcome(s) to be measured, a data analysis and display process with clinically relevant and statistically appropriate boundary or control limits that separates inliers from outliers, and a multidisciplinary review panel with appropriate expertise. 4
Index Procedure and Outcomes The proposed index procedure is hospital separation. The outcome measure proposed is risk-adjusted mortality rates using the HOPE model for acute hospital separations and the COPE model for those whose includes any period of time in an intensive care unit (ICU). Four cohorts, based on increasing severity of illness, are suggested: 1. All adult hospital separations (excluding day-case only separations; Group 1.) 2. The subset of Group 1 based on the top-ten diagnostic groups with the highest statewide mortality. 3. The subset of Group 1 whom are admitted to ICU (Group 2) 4. The subset of Group 2 whom are mechanically ventilated in ICU. Poor quality of care is unlikely to lead to fatalities in low-risk patients (Group 1) who are likely to survive despite suboptimal care. Poor quality of care is more likely to increase mortality rates in high-risk patients (Groups 2) because they have less reserve to withstand these deficiencies and are more likely to respond favourably to improvements in care. Data Display Formats Several data display methods are readily available (2,4). Broadly these can be divided into (a) cross-sectional displays such as a SMR and funnel plots, (b) and longitudinal timebased analyses known as process control charts. The later are available in a number of different formats - CUSUM, RASPRT, VLAD, EWMA, etc. Each has its own strengths and limitations. All need to be used with caution to avoid misinterpretation of data and incorrect classification of an inlier or an outlier hospital. The risk-adjusted EWMA chart (Appendix 3) is here proposed as the preferred format. Like others squential process control charts the EWMA chart: Provides a method for continuous real-time monitoring. Has a high sensitivity and flags outliers at the same time as other methods Is sufficiently sensitive to flag small variations but less sensitive to sudden fluctuations. Allows the identification of better and poor performers on the one chart. Incorporates risk-adjustment models Provides indications of poorly calibrated risk-adjustment model Has been used in industry since the 1950s Allows control limits to be adjusted Provides a simple visual representation of complex data. 5
Like other control charts the EWMA chart is dependent upon the quality of the available data, the calibration of risk-adjustment model, and its preset parameters, such as the weighting factor ( w ), confidence intervals, and run length characteristics (Appendix 4). The Standardised Mortality Ratio (SMR) parameter provides a cross-sectional analysis of data. It aggregates data over time and thus provides an average measure over time for the entire cohort. The confidence intervals for an SMR reflect the number of observed events (death), whereas the confidence intervals in a control chart reflect the sample size. The SMR may be considered as less sensitive in flagging outliers than control charts but is more specific. Boundary or Control Limits When the monitored outcome rate crosses a boundary level this event is the trigger for further investigation of the source data. It is important to distinguish between false alarms and true alarms. The most common source of false alarms is data error. In this situation a common-cause variation in the data can be misinterpreted as a special-cause variation and the hospital is incorrectly classified as an outlier. A true change in performance ( special-cause variation ) is less common but is equally important to correctly identify. Control limits or boundaries are set on both the higher and the lower sides of the average to provide a range for common cause variation (and a benchmark) and facilitate the identification of poor performers (mortality above the benchmark) and better performers (mortality below the benchmark). The selection of boundary levels is complex and is usually based on clinical judgement and/or the level of resources available to investigate outliers. There are very few data available to guide these decisions. As an example, Queensland Health (3) have chosen to incorporate control limits (or alert levels) at 30%, 75% and 95% relative risk above or below the benchmark. These control limits are based on the minimum number of alarms desirable. At this point it is worth asking the question Why monitor? as this can inform the selection of control limits. For example, if the aim of monitoring is simply to demonstrate stability and reassure health-care administrators or the community that a minimum standard of 6
care is being provided in the most hospitals then wide control limits may be sufficient. If the aim is to identify all deviations from the benchmark as soon as possible then narrow control limits will be required. If highly sensitive or narrow control limits are chosen then an additional verification process will be required to improve the specificity of the alerts. Wide control limits are more likely to correctly identify the true inliers (high specificity) but risk missing true outliers (low sensitivity), whilst narrow control limits are likely to identify the true outliers (high sensitivity) but risk falsely classifying outliers (low specificity). Wide control limits may lead to a false sense of security, whereas narrow control limits may lead to unnecessary anxiety and doubt regarding the validity of the results. The quality of the data and the robustness of the risk-adjustment method need to be taken into account. High quality data analysed with a robust risk-adjustment model will reduce common cause variation, the risk of false classification, and make it safer to apply narrow control limits. Conversely, poor quality data or incomplete risk-adjustment is likely to lead to greater variation and the risk of falsely classifying outliers unless wide control limits are applied. Methods for monitoring the quality of the VAED and the performance of the risk-adjustment models and the performance of control charts are available. In conclusion, there are a number of possible solutions for the selection of control limits. These options include: Select a narrow control limits and investigate all outliers. Select control limits based on a pre-determined level of statistical significance, e.g. 2 or 3 standard deviations from the mean. Set pragmatic control limits according to the pre-determined number of alarms that can be managed and/or investigated. Select multiple control limits. The narrower control limits provide a warning level to flag hospitals that require further attention. Wider control limits provide an alarm level to flag hospitals that require further investigation. For example, a warning level could be set at two standard deviations from the mean and an alarm level set at three standard deviations from the mean. 7
Multi-disciplinary Review Panel The Victorian Intensive Care Data Review Committee (VICDRC) is an example of a multidisciplinary group that would be appropriate for the analysis and interpretation of the results relating to Intensive Care performance (and this is within its terms of reference). No similar group currently exists for the analysis and interpretation of hospital-wide performance data and this should be addressed before such monitoring is activated. This group should include to clinicians, statistical advisors, data analysts, DHS Safety & Quality representatives, and administrative support. The panel s activity might include the following scheduled tasks: 1.Monthly review by DHS data analyst of each hospital s EWMA charts. 2.Quarterly review by the multidisciplinary panel of each hospital s EWMA charts. 3.Annual comparison of SMR results for all hospitals. 4.Annual review of re-calibration of the COPE and HOPE models. Model coefficients should be compared from year to year to check for coding and casemix shifts and model stability. Reporting to hospitals could be in the form of an Annual Clinical Performance Report including 1.SMR results for designated patient subgroups for all State hospitals. 2.Detailed analysis and explanation of trends, seasonal changes, areas of clinical interest, frequent and high mortality diagnoses, etc. 3.Incorporate other applicable State-wide audit readily available audit reports eg. AusPSI, Registries, VICNISS, Cardiac Surgery, Vascular Surgery, Surgical Outcomes. 8
Methods. A four-stage graded response system is proposed with escalation to the next stage if an alert has occurred during the previous stage. See flowchart in Figure 1. A detailed description is provided below. This process was tested by simulation in 23 major Victorian hospitals. Average run lengths estimates were calculated for each hospital. EWMA control charts for Group 1 and Group 2 were constructed from the relevant dataset. The number and type of alerts according to the process outlines below were counted. Each hospital was restricted to a maximum of one alert per month for each of the separate alert levels. Stage 1 This is a continuous monitoring process using separate risk-adjusted EWMA control charts (see example in Figure 2) for each of the four cohorts. Risk-adjustment is based on the relevant model - the HOPE model for Group 1, the COPE model for Group 2. Three alert levels are proposed for the control chart. The Level 1 alert or warning trigger set to flag at a statistically significant change from the benchmark of two standard deviations from the mean. If a Level 1 alert is reached a watching brief is set for that hospital and no further action is taken until a Level 2 alert is flagged. A Level 2 alert or investigation trigger is set at three standard deviations from the mean. If a Level 2 is reached then monitoring proceeds to Stage 2. A Level 3 alert is flagged If the Level 2 alert persists for more than two consecutive months and prompts the monitoring process to proceed to Stage 3. Stage 2 This Stage is initiated by a Level 2 alert arising from any one of the control charts. The aim of this stage is to identify and exclude common factors that may have led to a false alarm. Here are examples of actions that may be selected by the Review Panel: 1.Check raw data for major errors e.g. Are the number of records correct? Are there missing records or empty fields present? 2.Check raw data for major shifts in that hospital s demographic and casemix. E.g Compare age, sex, casemix, elective and emergency workload, crude mortality rate, to historical data. 9
3.Are there casemix factors peculiar to this hospital (e.g. state-wide service) that may explain the observed variations? 4.Look at control chart for trends, e.g. seasonal trends. 5.Calculate SMR & 95% confidence. Do the 95% CI include unity? 6.Compare SMR with peer group hospitals. 7.Are the same trends found in subgroups 1, 2, etc? 8.Compare results with any other relevant benchmark, e.g. APACHE-III, Cardiac Surgery. Following this review there should be some form of confidential communication to the relevant health service (e.g. CEO and Clinical Director) with results of preliminary analysis and an explanatory report. Stage 3. This stage is activated when either (a) the Level 3 alert occurs (Level 2 alert for >2 consecutive months) or (b) the Stage 2 investigation does not provide a suitable explanation in the judgement of the review panel. Once again, the aim of this stage is to identify and exclude common factors that may have lead to a false alarm. Here is a suggested action plan: 1.All of Stage 2 is undertaken (if not already completed) PLUS 2.Check for calculation errors, such as an inadvertent model coefficient error or an incorrect risk-adjustment formula. 3.If necessary, extract raw data from VAED a second time and re-calculate riskadjustment models and compare calibration and discrimination parameters. 4.Compare SMRs in same hospital for the previous 1, 3, and 5 years. Using the larger dataset will decrease the size of the confidence intervals. 5.Has there been significant changes in the recalibrated model(s)? Following this review there should be notification of the health service with a written report. This might include an analysis of casemix and individual diagnostic subgroup mortality rates compared to the benchmark. Specific recommendations and suggestions for targeted investigation focusing of areas of interest are likely to be helpful, e.g. AMI and CCF risk-adjusted mortality rates appear to be higher than average, please investigate. 10
Stage 4 This stage is activated when the results of a Stage 3 investigation do not provide a suitable explanation for the alert signal(s). This is simply a checklist of Questions To Be Answered by the outlier Hospital: 1.Is there evidence of changes that might affect Data Quality? 2.Has there been a change in data coding practices (eg personnel changes in HIS, IT software/hardware changes, data submission)? 3.Has there been an internal coding audit in past 12months? 4.Is there significant variation in casemix? 5.Has the casemix shifted? Which clinical areas? 6.Has crude mortality shifted unexpectedly? Compare to previous year(s). 7.Have clinical services been substantially altered? 8.Have referral patterns changed? 9.Structure and resource availability. 10.Has there been a change in funding? 11.Has there been a change in resources? 12.Has there been a change in clinical services? 13.Has the throughput changed? Eg. numbers, casemix, emergency workload. 14.Clinical Peer Review and Audit. 15.Is there an internal clinical audit process? 16.Do these internal audit reports highlight areas of interest? 17.Can you provide results of internal audits or other external benchmarks? Following this review a discussion between hospital and review panel representatives is likely to be required to determine actions to be taken to address any identified problems, e.g. improvement to resources, staffing, training, clinical audit, peer review, etc. The following outlier classification is suggested as a nomenclature system for defining hospital status during an investigation whilst avoiding inappropriate or emotive labels e.g. bad or poor performer. If Stage 3 investigation is activated for high outlier the hospital is designated as Possible Area Of Need to avoid being stigmatised as poor performer. If Stage 4 investigation indicates a high outlier status the hospital is designated as Area Of Need 11
If Stage 3 investigation is activated for low outlier the hospital is designated as Possible Improved Practice Site (and not stigmatised as a good performer ) If Stage 4 investigation indicates a low outlier status the hospital is designated as Possible Best Practice Site. Results The simulated monitoring process was undertaken in 23 major hospitals, using EWMA charts (Appendix 2) and the HOPE model for Group 1 and the COPE model for Group 2. We used data from the 12-months, 1/7/2006-30/6/2007, for the Group 1 monitoring and, since Group 2 was a much smaller subset, we used all three years (1/7/2004-30/6/2007) data for the Group 2 monitoring simulation. The average run lengths (ARL) for the HOPE model and Group 1 are displayed in Figure 1, and ARLs for the COPE model and Group 2 are displayed in Figure 2. There was a wide separation of high outlier ARLs (OR=2) and low outlier ARLs (OR=0.5) from the inlier ARLs (OR=1). High outlier ARL (OR=2) were less then 100 for all hospitals and above 2,000 for inlier ARLs (OR=1). The number of alerts from the risk-adjusted EWMA charts are summarised in Table 1. The majority (52%) of hospitals had few if any alerts. Although a number of Level 1 and Level 2 alerts occurred, no Level 3 alerts were found and therefore there were no outliers found and all hospitals were within the set benchmarks. An example of the seasonal distribution of alerts is displayed in Figure 3. Conclusions A practical four-stage hospital monitoring process, based on the Critical Care Outcome Prediction Equation (COPE) model and the Hospital Outcome Prediction Equation (HOPE) model is presented and was tested by simulation. This process appears to provide a practical application for real-time monitoring of hospital performance. 12
References 1.Scott IA, Ward M. Public reporting of hospital outcomes based on administrative data: risks and opportunities. Med J Aust 2006; 184(11): 571-5 2.Cook DA, Duke GJ, Hart GH, Pilcher D, Mullany D. Review of the application of riskadjusted charts to analyse mortality outcomes in critical care. Critical care and resuscitation. 2008; 10(3):239-51 3.Coory M, Duckett S, Sketcher-baker K. Using control charts to monitor quality of hospital care with administrative data. International journal for quality in health care 2008; 20(1): 31-9 4.Lim T. Statistical process control charts for monitoring clinical performance. International journal for quality in health care. 2003; 15(1):3-4 Group 1 Hospitals with high outlier alerts (%) Average number Hospitals with of alerts per low outlier alerts month (%) Average number of alerts per month Level 1 11 (48%) 1.6 7 (30%) 0.9 Level 2 3 (13%) 0.4 2 (9%) 0.3 Level 3 0 0 0 0 Group 2 Level 1 11 (48%) 2.7 13 (57%) 1.6 Level 2 6 (26%) 0.8 7 (30%) 1 Level 3 0 0 0 0 Table 1 Frequency of alerts during simulation 13
Graph of ARLs 10000 ARLs in cases 1000 100 10 1 13 16 3 15 2 4 18 5 20 10 22 8 21 14 121 17 6 23 7 19 9 1 0 0.5 1 1.5 2 2.5 Altered OR Figure 1. Average run length estimates for EWMA and COPE model. Note that less than 100 cases are required to identify a doubling in odds ratio for death (outlier ARL) and many thousands of cases are required before a false positive signal is likley (OR=1; inlier ARL). Graph of HOPE model ARLs for 23 hospitals 100000 ARLs in case numbers 10000 1000 100 10 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 Odds Ratio Figure 2. Average run length estimates for EWMA and HOPE model. Note that less than 250 cases are required to identify a doubling in odds ratio for death (outlier ARL) and over 15,000 cases are required before a false positive signal is likley (OR=1; inlier ARL). 14
Number of HIGH Outlier Alerts - Victorian Major Hosp 2006-07 Warning Alert Frequency 10 9 8 7 6 5 4 3 2 1 0 Jul-06 Aug-06 Sep-06 Oct-06 Nov-06 Dec-06 Jan-07 Feb-07 Mar-07 Apr-07 Month May- 07 Jun-07 Figure 3. Frequency of Level 1 (yellow bars) and Level 2 (red bars) alerts for Group 1. 15
5. Appendix 1: FLOWCHART of proposed outlier investigation process. Monthly review of RA EWMA charts for each Group & hospital Stage 1 EWMA chart: x>2sd? No Status: inlier Yes = Level 1 Warning EWMA chart: x >3SD No Status: possible outlie Yes = Level 2 Alert Stage 2 Data Quality Check EWMA: x >3SD Duration > 2-mth? Submit report to Review Panel. No Status: Level 2 Alert Yes = Level 3 Alert Submit report to Review Panel and Health Service. Stage 3: Data analysi +/- Stage 4: Questionaire 16
Appendix 2: Example of risk-adjusted EWMA control chart 0.035 0.03 RA EWMA EWMA of observed mortality rate Upper 2SD of predicted Lower 2SD of predicted Upper 3SD of predicted Lower 3SD of predicted 0.025 EWMA statistic 0.02 0.015 0.01 0.005 0 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 case number 17