APACHE IVb White Paper Report. December 2016

Transcription

1 APACHE IVb White Paper Report December 2016

2 Introduction APACHE has a rich history in outcomes management for the ICU. First published in 1981 [1], it has undergone a number of major transformations and minor adjustments. Prior to the current update labeled APACHE IVb, the most recent adjustment was made based on data collected from 2006 to Changes in practice and overall population health warranted a review and recalibration of these previous models. This document is intended to provide details on why the models needed to be updated, the calibration process, the changes to the APACHE equations in APACHE IVb, and the impact one may expect to see in predictions. Why the model needed to change The 4-minute mile The concept of drift in outcome statistics can be better understood with a comparison in the world of sports. Foot races first became popular in 19th century Britain, and professional athlete Charles Westhall set the first world record for the mile at 4:28 in With improvements in nutrition and athleticism, the record time slowly decreased toward the supposedly unbreakable 4-minute barrier. That is, until in 1954 when British athlete Roger Bannister ran the event in 3:59.4. The men s world record today, held by Hicham El Guerrouj of Morocco, is 3: Let s say you re a prospective Olympian, and your personal best time is 4:00.0. To compare your performance, you could calculate a Standardized Mile Ratio (or SMR) at three points in time, based on your personal best versus the current benchmark. If you re living in 1855 your SMR would be 0.86 or significantly better than expected. Book a steamer for Athens! By 1954, you re still highly competitive with a SMR of But your 4:00 time today generates an SMR of You haven t changed; but the world has changed around you. And something similar is happening with medical care. Despite the insistent media drumbeat about medical errors costing hundreds of thousands of lives each year, the data from the ICU world actually demonstrates constant improvement in outcomes since the 1980 s, when benchmarked ICU outcomes were first measured. Some changes are technical we enjoy improved mechanical ventilators and monitors. Some are pharmaceutical stress ulcers were a common cause of fatal GI hemorrhage in the era before histamine-2 antagonists. But most of the changes are organizational teamwork, education, unit policies and procedures, closed units, 24-hour coverage and ICU telemedicine are several examples. There is no question that the benchmark has moved in a favorable direction, and yesterday s quality of care does not compare favorably with today s standard. Change occurs almost imperceptibly, but evidence from multiple generations of APACHE [2, 3, 4], MPM [5, 6, 7] and SAPS [8, 9, 10] suggests that benchmarking models need to be recalibrated at 5 to 10 year intervals. Page 2 of 23

3 How often should models be updated The question of when to update a statistical model has no easy answer in short, it depends. As stated Title above, in for Franklin the broad population Gothic mortality Demi rates tend 18pt to change slowly over time, suggesting a longer period between updates. Still, a new treatment for a particular subset of the population may make outdated models over-predict the mortality rates of that population. It s certainly not practical to track all such advancements in treatment and adjust the model accordingly. Furthermore, the process of getting a new model into production takes considerable time and effort. While research in model fade and the regular recalibration of models is ongoing, the best practice is simply to monitor the models performance over time and to commit the resources to refit the model when the accuracy of the model is no longer satisfactory. For APACHE this can be observed in the regular reporting of Standardized Mortality Ratios (SMRs) for all the hospitals in the database. The APACHE IVa models were developed on patients admitted to the ICU from 2006 to Using these models to predict ICU mortality for APACHE ICU patients admitted from January 1, 2014 to December 31, 2015 results in an expected mortality rate of 8.2%. Compare this to the observed mortality rate of 7.7%. In a little less than 10 years, we have seen a reduction in ICU mortality of roughly 0.5%. As a result, the SMR reported by APACHE IVa is 0.89 overall; the models need to be updated. Evaluating the existing equations Apache Outcomes It is important to recognize that the source of the data has changed since the APACHE IVa models were introduced in APACHE Outcomes, with embedded APACHE IV methodology and predictive equations, was introduced in These same APACHE IV predictive equations were developed with data captured in an earlier version of the solution known as APACHE Client Server/APACHE for ICU. APACHE IVb is the first version of the APACHE models built with data from the APACHE Outcomes solution. The APACHE methodology remained the same across both solutions. Data abstractors were trained on the APACHE methodology, and held the integrity of the data to the highest standards. However, APACHE Outcomes afforded the abstractors more robust validation checking of the data that was both manually added and integrated from the EHR. Validating the data allowed the system to monitor and draw to attention real-world documentation errors, such as the inadvertent documentation of 0 for vital signs. Table 1 History of APACHE. Year # hospitals # admissions APACHE APACHE II ,815 APACHE III ,440 Page 3 of 23

4 APACHE III-i ,264 APACHE III-j ,000 APACHE IV ,618 APACHE IVa ,846 APACHE IVb ,319 Characteristics of the data To evaluate the APACHE IVa models, data was collected from the APACHE Outcomes database. This consisted of 186,319 patients admitted to an ICU between January 1, 2014 and December 31, Patients from a total of 148 ICUs at 70 hospitals are included. Table 2 shows that all four regions of the country were represented, and the hospitals were a mix in terms of teaching status and bed size. Table 2 Hospital characteristics for APACHE IVb dataset. Region # % Midwest Northeast Southeast West Teaching status # % Council of Teaching Hospitals (COTH) Small teaching Non-teaching Bed size # % < The distribution of ICU types is shown in Table 3. Table 3 Characteristics of the ICUs in the APACHE IVb dataset. ICU type # % APACHE IVa % ( ) Cardio-thoracic Surgery ICU Only Coronary/Cardiac Care ICU Only Medical ICU Only Page 4 of 23

5 Mixed Neurologic/Neurosurgical ICU Combined Surgical ICU Only Trauma ICU (Trauma Only, Surgical/Trauma, Trauma/Burn) Table 4 shows the demographic information for the subencounters. There were 54% male admissions and a large portion, 74%, identified as white. There was a slight increase in the number of encounters from 2014 to 2015, approximately 84,000 and 99,000 respectively. Almost 6% were ICU readmissions and only 3% were post-operative coronary artery bypass graft (CABG) patients. Table 4 Subencounter demographics for the APACHE IVb dataset. Gender # % Male 101, Female 85, Race # % White 138, Black 30, Asian 3, Hispanic 1, Other/Unknown 12, Admission date # % , , ICU Readmission # % 10, CABG* # % 5, *CABG is defined by a diagnoses of S-CABG, S-CABGREDO, S-CABGROTH, or S- CABGWOTH. In Table 5, the most frequent diagnosis (top 25) are given along with the proportion in the APACHE IVa and IVb datasets. These 25 diagnoses account for 52% of the population. Finally, Table 6 shows the outcomes for both CABG and non-cabg patients. While the length of stay and ventilator usage remained basically unchanged from IVa, the risk of ICU mortality for non-cabg patients dropped by nearly half a percentage point (8.5% for IVa). When comparing Page 5 of 23

6 only those hospitals that contributed to both datasets, the change is less drastic, from 8.3% for APACHE IVa to 8.1% for non-cabg patients in the IVb dataset. The improved outcomes for the more recent population demonstrate the change in care discussed previously, and it explains the need for the APACHE models to be updated in order to bring predictions more in line with observed mortality rates. The mean APS and APACHE score for the IVb dataset were 41.9 (± 0.1) and 54 (± 0.1), respectively. The distribution of the day 1 APS is shown in Figure 1, and it is highly skewed with the majority of patients having a lower APS. No significant change in this distribution was observed from the data used for APACHE IVa; which suggests using the same spline variables in the updated model. Table 5 Most frequently occurring diagnosis in the APACHE IVb dataset. Diagnosis # APACHE IVb % ( ) APACHE IVa % ( ) Sepsis, pulmonary 6, CVA, cerebrovascular accident/stroke 6, Infarction, acute myocardial (MI) 6, Cardiac arrest (with or without respiratory 5, arrest; for respiratory arrest see Respiratory System) CHF, congestive heart failure 5, Emphysema/bronchitis 5, Diabetic ketoacidosis 5, Sepsis, renal/uti (including bladder) 4, Respiratory- medical, other 4, CABG alone, coronary artery bypass grafting 4, Hemorrhage/hematoma, intracranial 4, Sepsis, unknown 3, Head (CNS) only trauma 3, Bleeding, upper GI 3, Sepsis, GI 3, Pneumonia, bacterial 3, Seizures (primary-no structural brain disease) 2, Rhythm disturbance (atrial, supraventricular) 2, Hypertension, uncontrolled (for 2, cerebrovascular accident-see Neurological System) Sepsis, other 2, Hypovolemia (including dehydration. Do NOT 2, include shock states.) Bleeding, lower GI 1, Renal failure, acute 1, Sepsis, cutaneous/soft tissue 1, Page 6 of 23

7 Neoplasm-cranial, surgery for (excluding transphenoidal) 1, Table 6 Observed outcomes (mean) for CABG and non-cabg patients in the APACHE IVb dataset. Outcome Non-CABG CABG Hospital Mortality 11.5% 1.6% ICU Mortality 7.9% 1.2% Hospital Length of Stay ICU Length of Stay Duration of Mechanical Ventilation % Monitor* patients actively treated 4.5% N/A * Monitor patients are defined as patients not actively treated on day 1. The definition of low risk monitor patients has been modified (see below). Figure 1 Distribution of Day 1 Acute Physiology Score in APACHE IVa (left) and APACHE IVb (right) datasets Model validation To determine which equations in particular needed to be recalibrated, the observed outcomes were compared to the expected/predicted outcomes for the appropriate population. For equations predicting mortality, the Standardized Mortality Ratio (SMR) was used; if the SMR was substantially different from 1.00, then the equation was updated. For example, the observed ICU mortality rate for non-cabg patients in the APACHE IVb data set ( ) was reported above as 7.9%. Meanwhile the APACHE IVa version of Equation 1 predicted a mortality rate of 8.9% for this same population; an SMR of 0.87, indicating that Equation 1 needed to be updated. For equations predicting continuous outcomes, such as length of stay, the main determinant was the ratio of mean observed to mean predicted values; an analogue to Page 7 of 23

8 the SMR. The full list of updated equations is provided in Table 7. The decision was made to focus on the Day 1 models. This was done to reduce the impact of the model update. While we understand the daily models may also be in need of recalibration, we chose instead to devote more resources to building the next generation of APACHE predictive models discussed in the conclusion of this paper. Table 7 Equations recalibrated in APACHE IVb. Eq. Description Eq. Description # # 1 ICU Mortality Day 1 42 Active Treatment, Day 1 Monitored patients 8 Hospital Mortality Day 1 64 Ventilator Days 15 CABG ICU Mortality Day 1, National 66 CABG Ventilator Days 18 CABG Hospital Mortality Day 1, National 83 ICU Mortality Day 1, Similar 21 CABG ICU Length of Stay, National 84 Hospital Mortality Day 1, Similar 22 CABG Hospital Length of Stay, National 85 ICU Length of Stay, National 23 CABG Discharged Alive Next 48 hrs, Day 1 86 Hospital Length of Stay, National 29 ICU Length of Stay, Similar 90 Hospital Mortality, National 32 Discharged Alive Next 48 hrs, Day 1 93 Active Treatment, Day 1, National 40 Hospital Length of Stay, survivors Modeling the new equations Design decision The basic APACHE methodology remains unchanged for APACHE IVb. When APACHE IV was introduced in 2005 it introduced several changes to the APACHE III-j models in use at the time: carryover labs, exclusion of ICU transfers, continuous (instead of integer) length of stay, and the inclusion of a variable indicating whether a Glasgow Coma Score could be assessed due to sedation. The most significant change was a new categorization of disease groups from 94 groups to 116 [11]. No new variables were added to the APACHE IVb models, and no existing variables were removed. This decision was based partially on the desire to limit the changes to the codebase, but also because the current collection of APACHE variables proved to be adequate in predicting hospital and ICU outcomes. Introducing new variables would also have required significant engineering effort in order to extract these new elements from Cerner, and non-cerner, EHRs; further extending the timeline for APACHE IVb update. Statistical techniques Logistic regression was used to model binary outcomes (Y/N) and continuous outcomes (e.g. length of stay) were modeled with linear regression. As in the previous version of APACHE, cubic splines were used for age, APS, prior length of stay (lead time bias), creatinine level, and ejection fraction to allow for nonlinear relationships with the outcomes. Splines for creatinine level and ejection fraction were used only in the CABG equations. Splines are commonly used Page 8 of 23

9 when the relationship between continuous variables and outcomes is not linear. In the simplest case, linear splines represent a piecewise linear relationship between predictor and outcome; lines with different slopes are constructed between specific points (knots) over the range of the predictor variables. More advanced nonlinear relationships can also be defined between these knots, such as the cubic splines used here. A more thorough analysis of splines and their use in modeling can be found in the book by Harrell [12]. Evaluating the logistic regression models was based on two primary measures: SMR and the area under the receiver operating characteristic curve (AUROC) [13, 14]. The AUROC assesses how well the model discriminates between patients who died and patients who survived. An AUROC of 0.5 indicates that the model does not discriminate between patients any better than chance alone; a score of 1.0 indicates perfect discrimination. Depending on the setting, AURO C values above 0.8 are generally considered acceptable [15]; APACHE has traditionally been in close to In addition, calibration over across risk deciles was evaluated by comparing observed and expected outcomes in each decile. For the equations modeling continuous outcomes, assessment was based on the ratio of the mean observed to mean predicted value, as well as the coefficient of determination (R 2 ). The coefficient of determination measures how well the model fits the observed outcome over the entire range of outcomes. It can range from 0 to 1, with higher values being better. The APACHE equations have typically been in the range of 0.2. Data was partitioned, randomly, into training and validation datasets comprised of 60% and 40% of the patients, respectively. The selection of data for each dataset was done independently for each equation. The model was then built on the dataset and its accuracy in predicting the modeled outcome was assessed on the hold out validation dataset. Results Results for the recalibration of the equations predicting binary outcomes are shown in Table 8. Overall the non-cabg mortality equations perform quite well, achieving AUROC values above for the validation dataset in all cases. In general, ICU predictions are more accurate than hospital predictions; which is expected considering the source of the data used to make predictions comes from ICUs only. The prediction for ICU mortality is also rather accurate across risk deciles, as shown in Figure 2; the difference between observed and expected mortality is less than 1% for all deciles of risk, as can be seen in the corresponding data in Table 9. The CABG models are not as accurate for the validation dataset. This is due mostly to the limited amount of data; as reported above only 5,595 CABG patients were included in the analysis. It is also more difficult for models to accurately predict rarer events, such as mortality Page 9 of 23

10 rates around 1%. While the active treatment equation for monitored patients (Eq. 42) appears to perform poorly in the validation dataset, we show in the next section a more appropriate evaluation of this model based on the classification of low-risk monitor (LRM) patients. The results for the equations predicting continuous outcomes (durations) are given in Table 10. Again the non-cabg models do better than the CABG models, and for the same reasons. Even for the CABG equations though, ratios of the mean observed to mean expected values are close to The R 2 values are good for the non-cabg equations as well. The only equation that doesn t achieve this level is the prediction for ventilator days. Figure 3 shows the prediction for ICU LOS (Eq. 85) accurate across risk deciles; the observed and predicted values compare well. The largest discrepancy can be seen in the lowest decile (patients with an expected LOS less than 1.35 days) with an observed mean LOS of 1.50 days compared to a predicted 1.02 days. Table 8 Statistics for APACHE IVb models predicting binary outcomes. Eq. # Description APACHE IVa Training Validation SMR SMR AUROC Observed Predicted SMR AUROC 1 ICU Mortality % 7.9% ICU Mortality, % 7.9% Similar 8 Hospital % 10.7% Mortality 84 Hospital % 10.7% Mortality, Similar 90 Hospital Mortality, National % 10.7% Page 10 of 23

11 32 Discharged Alive Next 48 hrs % 63.6% CABG ICU Mortality, National 18 CABG Hospital Mortality, National 23 CABG Discharged Alive Next 48 hrs 42 Active Treatment (monitored) 93 Active Treatment, National % 0.8% % 1.1% % 74.5% % 9.1% % 56.8% Table 9 Observed and expected ICU mortality by risk decile for APACHE IVb. Decile # patients Observed mortality Expected mortality # % # % 1 17, , , , , , , ,811 1, , ,811 2, , ,811 8, , Page 11 of 23

12 Figure 2 APACHE IVb expected versus observed ICU mortality by decile of predicted risk. Table 10 Statistics for APACHE IVb models predicting continuous outcomes. Eq. # Description APACHE IVa Training Validation Ratio LOS R 2 Mean Mean LOS R 2 Ratio Observed Predicted Ratio 85 ICU Length of Stay, National 29 ICU Length of Stay, Similar 86 Hospital Length of Stay, National 40 Hospital Length of Stay, survivors 64 Ventilator Days CABG ICU Length of Stay, National 22 CABG Hospital Length of Stay, National Page 12 of 23

13 66 CABG Ventilator Days Figure 3 APACHE IVb expected versus observed ICU length of stay by decile. Low risk monitor patients Philosophy of Low Risk Monitoring Intensive care units were designed to cohort sicker patients in a location that provided more intense nursing supervision (typically 1:1 or 1:2 nurse to patient ratios) and specialized technology (physiologic monitors, mechanical ventilators, infusion pumps). Although the ICU without walls now allows intensive services to be delivered elsewhere, the ICU remains a location of choice for two types of patients: those who are being actively treated in a high - technology environment, and those who are at risk for needing intervention. This latter group is termed monitored patients, and while the vast majority will not go on to need ICU intervention, there is a level of comfort in having the patients closely watched with technology an d nursing support readily available. Typical admission diagnoses for low risk monitoring include diabetic ketoacidosis, stroke with or without thrombolytic therapy, acute myocardial infarction, head trauma, intracranial hemorrhage, postop neurosurgery, gastrointestinal bleeding, and sepsis. The expected mortality rate in low risk monitor patients should be low (<2%), which produces a challenge both in developing good predictive models, and in assessing standardized mortality ratios, since death will be rare, and very large numbers of patients must be evaluated to determine statistical significance. Page 13 of 23

14 Definition of active treatments During 2012, Title the in Cerner Franklin Critical Care Gothic Task Force, Demi comprised 18pt of non-cerner physicians and nurses, reviewed what was contemporary active treatment (AT) versus the original APACHE definitions from the 1970 s. Several interventions formerly included on the AT list from 1970 (Gsuits for trauma, balloon tamponade for esophageal varices, and NG lavage as a therapy for upper gastrointestinal bleeding) have been supplanted by more contemporary practices. Patients admitted to the ICU for, or following pericardiocentesis, bronchoscopy, continuous/intermittent mannitol infusion, fresh tracheostomy or treatment of seizures or complex metabolic derangements were previously considered AT, but the advisory board s opinion was that monitoring needs were more relevant than the procedure itself with regard to classifying these patients. On the other hand, a number of procedures uncommon 40 years ago were added to reflect contemporary AT. These include Continuous Neuromuscular Blockade, ECMO, HFOV, prone positioning, pharmacologic treatment of ongoing status epilepticus, use of ventricular assist devices and hemodynamic intervention requiring use of a pulmonary artery catheter. Continuous renal replacement therapy and intermittent dialysis were placed in separate categories, the term NIPPV replaced multiple BiPAP modes, and IPPV replaced the term mechanical ventilation. Table 11 lists all of the treatments currently considered active ICU treatments for APACHE Outcomes. Overall shift in monitor patients As a result of the changes to the list, as well as other changes in medical practice, the population of actively treated patients changed in the APACHE Outcomes database. By definition, monitored patients are those that did not receive an active treatment on their first day in the ICU. In the APACHE IVb dataset we see an increase in the proportion of monitor patients to 40%, from 36% in the APACHE IVa. More striking is the change observed in specific diagnostic categories. Table 12 shows the proportion of monitor patients in each dataset for the top 10 most frequent diagnoses in the APACHE IVb dataset. Nearly 80% of ICU encounters with the diagnosis of DKA were monitor patients (i.e. did not receive an active treatment on their first day in the ICU), much higher than the less than 50% of DKA patients in the APACHE IVa dataset. Table 11 List of active treatments in APACHE Outcomes. A/V Pacing Barbiturate Anesthesia Cardioversion Continuous Antiarrhythmic Continuous Arterial Drug Infusion Continuous Neuromuscular Blockade CRRT IRRT IV Replacement Excessive Fluid Loss IV Vasopressin Naso/Orotracheal Intubation in ICU NIPPV (BiPAP) PA Catheter (with or w/o CO measurement) Post Arrest (48 hours) Page 14 of 23

15 Extracorporeal membrane oxygenation (ECMO) Emergency Op Procedures Inside ICU Emergency Op Procedures Outside ICU Endoscopies High Frequency Oscillation Ventilation (HFOV) Induced Hypothermia Intra-Aortic Balloon Pump IPPV Prone Positioning Rapid Blood Transfusion Reintubation Within 24 Hours Single Vasoactive Drug Infusion Tx of Status Epilepticus VAD Vasoactive > One Ventriculostomy In turn, these changes in the monitored population contribute to an update in the prediction of low risk monitor patients. Low risk monitor (LRM) patients are those predicted (using Eq. 42) to have less than or equal to 10% risk of ever receiving an active treatment. The modeled outcome for Eq. 42 is monitored patients that receive active treatment; i.e. a monitor patient who does receive any active treatment on subsequent days is identified as a positive response. To better understand the change in LRM identification from APACHE IVa it is important to compare the rate of monitor patients that eventually receive an active treatment. For day one monitor patients in the APACHE IVa dataset, 18.9% receive an active treatment during their ICU encounter, compared to only 9.3% in the APACHE IVb dataset. As a result, the IVa equations severely overestimate the risk for monitor patients; evidence of this can be seen in Table 8, where the SMR for the APACHE IVa version of Eq. 42 was The change in rate of monitor patients that eventually receive an active treatment, a reduction of over 50%, is the main contributing factor to the observed increase in the average predictions for LRM. The LRM identification The results for Equation 42 were shown in Table 8 above. However, a better way to evaluate the equation is to consider the sensitivity and specificity. Sensitivity, or true positive rate, is the proportion of correctly identified positive events. In the case of LRM that means a monitor patient who never receives an active treatment is correctly identified as LRM. Specificity, or true negative rate, measures the same ratio for negative events; monitor patients who do receive active treatments being identified as non-lrm. Table 13 shows the results for APACHE IVa and IVb versions of Eq. 42 and the number of correctly/incorrectly identified LRM/non-LRM patients. The sensitivity and specificity are also given in the table. It is clear that the updated model correctly identifies LRM patients with a sensitivity of 69.8% compared to only 52.0% for the APACHE IVa model. Even more striking is the improvement for non-lrm patients, where the outdated model only correctly identified 24.9% of the patients, the APACHE IVb model correctly identifies 61.6% of the population as non-lrm. In short, the APACHE IVb model does better at identifying both LRM and non-lrm patients, making correct identifications in roughly 60% of the cases. Page 15 of 23

16 Table 12 Change in monitor status by diagnostic category. Top 10 most frequent diagnoses in the APACHE IVb dataset. Diagnosis # patients (IVb model) Monitor % (IVb model) Sepsis, pulmonary 7, % 12.0% CVA, cerebrovascular accident/stroke 6, % 49.2% Infarction, acute myocardial (MI) 6, % 44.0% Monitor % (IVa model) Cardiac arrest (with or without respiratory arrest; 6, % 2.9% for respiratory arrest see Respiratory System) CHF, congestive heart failure 5, % 21.5% Emphysema/bronchitis 5, % 17.6% Diabetic ketoacidosis (DKA) 5, % 46.8% Sepsis, renal/uti (including bladder) 4, % 23.9% Respiratory- medical, other 4, % 27.7% CABG alone, coronary artery bypass grafting 4, % 0.4% Table 13 Monitor patients subsequently receiving active treatments and their LRM status for the APACHE IVb dataset. Does not receive active treatment Receives active treatment APACHE IVa APACHE IVb # of patients LRM all others LRM all others 25,660 13,340 12,320 17,922 7,738 (52.0%) (69.8%) 2,573 1, ,586 (24.9%) (61.6%) In Table 14, LRM results are reported by ICU type. Trauma ICUs had the highest predicted rate of LRM patients at 41.0%, while Cardiothoracic ICUs had the lowest at 22.7%. Table 14 Breakdown of LRM rates by ICU type. ICU Type % LRM Cardiothoracic Surgery ICU Only 22.7% Coronary/Cardiac Care ICU Only 28.5% Medical ICU Only 23.9% Mixed 27.6% Neurologic/Neurosurgical ICU Combined 34.4% Surgical ICU Only 28.0% Trauma ICU (Trauma Only, Surgical/Trauma, Trauma/Burn) 41.0% Table 15 presents a breakdown of LRM rates by diagnosis for the top 10 most frequently occurring diagnoses for monitor patients, based on the APACHE IVb dataset. LRM rates are given for both the IVa and IVb versions of the model. A significant change can be observed for several diagnostic categories. In Head (CNS) only trauma, for example, 92.4% of monitor Page 16 of 23

17 patients are low-risk using the IVb model, compared to only 64.2% with the IVa model. This is validated by the observed rates of actively treated (after day 1) monitor patients in the two datasets; in the IVb dataset, only 6.6% of monitor patients with Head (CNS) only trauma diagnosis ever receive an active treatment, while 13.9% received an active treatment in the IVa dataset. Page 17 of 23

18 Table 15 Change in LRM rates by diagnostic category. Top 10 most frequent monitor diagnoses. Diagnosis # monitor % LRM % LRM % actively % actively Title in Franklin Gothic patients Demi (IVb) * 18pt (IVa) * treated treated (IVb) (IVb) (IVa) Diabetic ketoacidosis 3, % 93.6% 2.1% 8.1% CVA, cerebrovascular 3, % 45.6% 7.8% 19.1% accident/stroke Infarction, acute myocardial 3, % 70.2% 7.6% 12.9% (MI) Head (CNS) only trauma 2, % 64.2% 6.6% 13.9% Sepsis, renal/uti (including 1, % 12.9% 11.9% 24.8% bladder) Sepsis, pulmonary 1, % 9.6% 16.7% 28.6% Bleeding, upper GI 1, % 8.6% 13.2% 27.1% Hemorrhage/hematoma, 1, % 38.5% 10.1% 19.3% intracranial Neoplasm-cranial, surgery for 1, % 94.4% 3.6% 7.3% (excluding transphenoidal) Bleeding, lower GI 1, % 28.3% 11.8% 23.4% *Rates are for monitor patients only, i.e. the percentage of monitor patients that are predicted to be low-risk. How the SMR changed As mentioned above, the overall SMR for APACHE ICUs had drifted to 0.89 prior to the APACHE IVb update. The new equations bring that ratio back to 1.00 for the entire APACHE population. Individual hospital organizations and specific ICUs may observe more or less of a change in their ratios however, and of course values above or below 1.00 are expected for most. Below we investigate how this change is realized across the contributing APACHE ICUs. Overall comparison of hospital classification IVa vs IVb Table 16 shows the classification of ICUs by whether they had an SMR above, below, or equal to one. Immediately one can see that more than twice as many hospitals had an SMR less than one using APACHE IVa (38) compared to IVb (18). The same thing is observed on the other end of the spectrum; almost three times as many ICUs have an SMR greater than one using the IVb model (11) compared to IVa (4). The distribution for APACHE IVa is also very skewed toward less than one, whereas the APACHE IVb model is more uniformly distributed. Table 16: ICUs grouped by SMR with the APACHE IVa and IVb models. IVa SMR<1.0 IVb SMR<1.0 SMR=1.0 SMR>1.0 Total Page 18 of 23

19 SMR= SMR>1.0 Title in 0 Franklin 0 Gothic 4 Demi 4 18pt Total The same conclusions can be drawn from Figure 4 that shows the distribution of SMRs for all ICUs plotted for both APACHE IVa and APACHE IVb equations. In general the IVa ratios are lower than IVb SMR, and the distribution of ICUs is centered around the value of one for APACHE IVb. Figure 4 Distribution of SMRs across ICUs: APACHE IVb (solid) and APACHE IVa (dashed). Contributing factors In addition to the changes in patient population discussed above, a number of factors contribute to changes in SMR observed at each institution and in the aggregate. Operating procedures Individual ICUs and hospitals have implemented numerous procedural changes since the release of APACHE IVa. These improvements in efficiency and treatment of patients is a factor in the observed mortality rates. New recommendations for specific diagnoses means patients receive more appropriate care. Changes in admission and discharge practices mean more efficient use of ICU resources and better care available for the most severe population. Page 19 of 23

20 Changes in treatment for specific diagnoses Over time new procedures or medications become available, improving the outcomes for certain subgroups of the population, such as specific diagnoses. Enumerating all such cases in the previous 10 years would require a much more detailed study. Looking at the mortality rates for specific APACHE diagnosis groups we have identified a few cases where mortality rates have appeared to decrease significantly from the introduction of APACHE IVa. Table 17 shows the mortality rate for the APACHE IVa and IVb populations for these diagnoses. Table 17 Mortality rates for selected diagnoses in the APACHE IVa and IVb datasets. Diagnosis Mortality rate ( ) Mortality rate ( ) Effusions, pleural 17.7 % 10.3 % GI Vascular ischemia, surgery for (resection) 18.9 % 13.8 % Hemorrhage/hematoma-intracranial, surgery for 24.2 % 16.7 % Head (CNS) only trauma, surgery for 18.9 % 14.2 % Conclusion In order to deliver the most value to clients, the day 1 APACHE IV equations were evaluated to determine if they were outdated. The analysis indicated that all day 1 equations needed to be adjusted. No major methodological changes were made, existing equations were modified to more accurately reflect the practice and expectations for ICU patients. Future of APACHE While the new APACHE IVb version of the APACHE predictive (or risk-adjusted) equations should maintain applicability for years to come, the plans for the next incarnation of APACHE predictive models are already in development. The next update will see major changes to the methodology and scoring systems used, predictions will be available outside the ICU setting alone, and data will be weighted more appropriately in time, all while maintaining the same reporting suite and outcomes management tools that have made APACHE the standard for ICU benchmarking. Further Information To obtain further information about APACHE Outcomes or the APACHE IVb models, please contact: Melany Blakemore (mblakemore@cerner.com) Corey Bryant (corey.bryant@cerner.com) Page 20 of 23

21 Laura Freeseman-Freeman Kathy Henson Maureen Stark Acknowledgement Cerner is grateful to Dr. Thomas L Higgins, MD, MBA, Chief Medical Officer of Baystate Franklin Medical Center, Baystate Health Northern Region, and Baystate Noble Hospital, for graciously contributing to this white paper. Page 21 of 23

22 References [1] W. A. Knaus, J. E. Zimmerman, D. P. Wagner, E. A. Draper and D. E. Lawrence, "APACHE-acute physiology and chronic health evaluation: a physiologically based classification system," Critical Care Medicine, vol. 9, pp , [2] W. A. Knaus, E. A. Draper, D. P. Wagner and J. E. Zimmerman, "APACHE II: a severity of disease classification system," Crit Care Med, vol. 13, no. 10, pp , [3] W. A. Knaus, D. P. Wagner, E. A. Draper, J. E. Zimmerman, M. Bergner, P. G. Bastos, C. A. Sirio, D. J. Murphy, T. Lotring and A. Damiano, "The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults," Chest, vol. 100, no. 6, pp , [4] J. E. Zimmerman, A. A. Kramer, D. S. McNair and F. M. Malila, "Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today's critically ill patients.," Crit Care Med, vol. 34, no. 5, pp , [5] S. Lemeshow, D. Teres, H. Pastides, J. S. Avrunin and J. S. Steingrub, "A method for predicting survival and mortality of ICU patients using objectively derived weights.," Crit Care Med, vol. 13, no. 7, pp , [6] S. Lemeshow, D. Teres, J. Klar, J. S. Avrunin, S. H. Gehlbach and J. Rapoport, "Mortality Probability Models (MPM II) based on an international cohort of intensive care unit patients.," JAMA, vol. 270, no. 20, pp , [7] T. L. Higgins, D. Teres, W. S. Copes, B. H. Nathanson, M. Stark and A. A. Kramer, "Assessing contemporary intensive care unit outcome: an updated Mortality Probability Admission Model (MPM0-III).," Crit Care Med, vol. 35, no. 3, pp , [8] J. R. Le Gall, P. Loirat, A. Alperovitch, P. Glaser, C. Granthil, D. Mathieu, P. Mercier, R. Thomas and D. Vilers, "A simplified acute physiology score for ICU patients.," Crit Care Med, vol. 12, no. 11, pp , [9] J. R. Le Gall, S. Lemeshow and F. Saulnier, "A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study.," JAMA, vol. 270, no. 24, pp , [10] R. P. Moreno, P. G. Metnitz, E. Almeida, B. Jordan, P. Bauer, R. A. Campos, G. Iapichino, D. Edbrooke, M. Capuzzo, J. R. Le Gall and SAPS 3 Investigators, "SAPS 3--From evaluation of the Page 22 of 23

23 patient to evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital mortality at ICU admission.," Intensive Care Med, vol. 31, no. 10s, pp , [11] J. E. Zimmerman, A. A. Kramer, D. S. McNair and F. M. Malila, "Acute Physiology and Chronic Health Evaluation (APACHE) IV: Hospital mortality assessment for today's critically ill patients.," Crit Care Med, vol. 34, no. 5, pp , [12] F. Harrell, Regression Modeling Strategies With Applications to Linear Models, Logistic Regression, and Survival Analysis., New York, NY: Springer-Verlag, [13] J. A. Swets, "Measuring the accuracy of diagnostic systems.," Science, vol. 240, no. 4857, pp , [14] J. A. Hanley and B. J. McNeil, "The meaning and use of the area under a receiver operating characteristic (ROC) curve.," Radiology, vol. 143, no. 1, pp , [15] B. H. Nathanson and T. L. Higgins, An Introduction to Statistical Methods Used in Binary Outcome Modeling., 2nd ed., Sage Publications, Page 23 of 23