Dashboards, Data and Information Overload: Using ACS NSQIP as a Reality Check Mary Hawn MD MPH Professor and Chief, Division of Gastrointestinal Surgery Associate Quality Officer: Perioperative Services The University of Alabama at Birmingham
Why Dashboards? Public reporting of mortality, PSIs, HACs, readmissions. is the latest paradigm of quality measurement. 100% 90% 80% 70% 60% 50% 40% CMS has published the 30% plan for value based 20% purchasing (VBP) which 10% is upon us. 0% VBP by Year Process (SCIP) Experience( HCAHPS) Outcome( mort/psi) Efficiency($ /pt)
Dashboards for Surgical Quality Surgical Care Improvement Project University HealthSystem Consortium State Surgical Site Infection reporting Patient Safety Indicators Hospital Acquired Conditions Never Events Readmissions HCAPS CGCAPS And so on and so on..
Hospital Compare Data SCIP Core Measures UAB Natl Avg Top 10% 100 90 80 70 60 50 40 30 20 10 0 86 86 98 99 98 100 100 90 92 92 95 98 97 90 84 85 82 84 82 81 75 ABX within 1hr ABX Selection ABX D/C time Card pts controll ed 6am Post Op Glucos e A ppropriate Hair Removal Patients w/ recommended VTE prophy l axis Patient who receiv ed appropri ate VTE prophylaxis
UAB Hospital Jan - Mar 2010 (Q1) Po st - Su r g ical Mo r t alit y ( Ex cl u d in g Tr au m a/ Bu r n ) n it io n - Po st - Su r g ical M o r t a lit y ( Ex clu d in g Tr au m a / Bu r n ) m ortality O/ E ratio for inpatient s age 18 and older undergoing any surgical procedure excluding t raum a, m aj o 265, 326-358, 405-425, 453-517, 573-585, 614-630, 652-675, 707-718, 734-750, 799-804, 820-830, 853-8 6, 113-123, 134-140, 146-171, 187-191, 199-204, 209-217, 227-232, 237-244, 270-271, 277-280, 287-288 nt quart er. Bad data, nonviable neonates, organ harvest cases, and records with a null expected m ort alit y ar im e. Rel at iv e Per f o r m an ce Ob ser v ed Tar g et UHC Med ian Ran k nt Quarter 1.22 0.71 0.82 102/ 109 t Year 1. 11 0.76 0. 84 99/ 109 Cu r r en t Qu ar t er Last Qu ar t er Recen t Year ( denom.) 3,269 3,300 13,538 ved Deat hs 116 85 406 ted Deaths 94.73 83.29 365.31 ved/ Expected Rat io 1.22 1.02 1.11 ved Mort ality(% ) 3.55 2.58 3.00 ted Mortality ( % ) 2.90 2.52 2.70 t (A) 0.71 0.73 0.76 4.5 3.5 2.5 1.5 0 Po st - Su r g ical Mor t alit y ( Ex clu d i n g Tr au m a/ Bu r n ) 2008 Q1 2008 Q2 2008 Q3 2008 Q42009 Q1 2009 Q2 2009 Q32009 Q4 2010 Q1 Expected Mortality ( % ) Observed Mortality( % )
University of Alabama at Birmingham Hospital General/Vascular Report No. 3995 PAGE 3 Site No. 67 07/06-06/07 01/07-12/07 07/07-06/08 01/08-12/08 07/08-06/09 01/09-12/09 07/09-06/10 01/10-12/10 07/10-06/11 01/11-12/11 GV Mortality 1.09 1.25 1.24 0.89 0.97 0.82 0.77 0.84 0.80 0.86 GV Morbidity 0.93 0.83 0.80 0.93 1.02 1.13 H 1.08 0.83 L 0.76 L 0.73 L GV Cardiac 1.14 0.68 2.11 H 2.18 H 1.21 1.09 1.00 1.04 1.07 0.97 GV Pneumonia 0.34 L 0.30 L 0.20 L 0.75 1.92 H 2.09 H 1.54 H 0.67 0.57 L 0.65 GV Unplanned intubation 1.05 0.95 1.13 0.80 0.97 1.05 0.83 0.79 0.87 1.02 GV Ventilator > 48 hours 0.82 0.64 0.67 0.61 0.75 0.94 0.94 0.84 0.71 0.76 GV DVT/PE 0.69 0.61 0.12 L 0.31 0.56 0.52 L 0.72 0.94 0.69 0.67 GV Renal Failure 0.99 1.40 1.09 0.90 1.62 H 1.08 0.83 1.13 1.17 1.02 GV UTI 0.80 0.86 0.62 0.87 1.19 1.10 1.08 1.05 0.85 0.86 GV SSI 0.97 0.84 0.84 1.09 1.03 0.98 0.81 0.75 L 0.80 0.77 GV ROR 0.91
What is the Truth? Process versus Outcome? SCIP versus SSI/VTE/MI Different data definitions NHSN versus CDC SSI Clinical versus Administrative data Risk adjustment HAC/PSI versus outcome
Surgical Care Improvement Project (SCIP) Implemented in 2006 by Centers for Medicare and Medicaid Services 3 Focus Areas: Surgical site infections Adverse cardiac events Venous thromboembolism (VTE) Reduce Surgical Complications by 25% in 2010 Bratzler DW, Hunt DR. Clin Infect Dis. Aug 1 2006;43(3):322-330.
Surgical Care Improvement Program 6 SCIP Infection Measures (at the time of assessment) Timely Antibiotic Administration Appropriate Coverage Timely Discontinuation Glucose control for cardiac patients Normothermia for colon cases Appropriate Hair removal Two VTE measures VTE Prophylaxis ordered VTE Prophylaxis received Cardiac Measure Beta blocker given within 24 hours prior to surgery
SCIP Measures: By definition, they re measureable Denominator specified Which surgical cases Exclusion Criteria Numerator is number of cases receiving measure Publically reported on Hospital Compare Are used for Value Based Purchasing 11
Association between Adherence and SSI SSI Unadjusted Adjusted SCIP Measure Met % OR 95% C.I. OR* 95% C.I. Timely Yes 5.0 No 7.8 0.67 0.58-0.77 0.90 0.76-1.07 Appropriate Yes 5.3 No 13.4 0.36 0.30-0.43 0.89 0.72-1.09 Discontinue Yes 5.6 No 5.3 1.07 0.95-1.22 1.07 0.93-1.24 Hair Removal Yes 6.3 No 4.8 1.32 0.85-2.05 1.04 0.62-1.75 Normothermia Yes 15.8 No 14.6 1.09 0.95-1.25 1.02 0.88-1.18 Composite Yes 4.7 No 8.3 0.55 0.49-0.62 0.92 0.80-1.06 *Adjusted for age, work RVU, operative time, specialty, diabetes, COPD, steroid use, ASA class, wound class, smoking status, dyspnea, alcohol, history of radiation therapy and gender
100% Hospital Infection Rate by Adherence 112 VA Hospitals R² = 0.03 Hospital Adherence (%) 80% 60% 40% 20% 0% 0% 2% 4% 6% 8% 10% 12% 14% 16% Infection Rate (%) * Bubble size represents
Adherence and Adjusted SSI Over Time 100% SCIP Adherence 80% 60% 40% 20% 0% 2004 2005 2006 2007 2008 2009 2010 Time Period (6 Month Intervals) Composite Timely Discontinue Appropriate Hair Removal Normothermia
100% Adherence and Adjusted SSI Over Time 25% SCIP Adherence 80% 60% 40% 20% 20% 15% 10% 5% Adjusted SSI Rate 0% 0% 2004 2005 2006 2007 2008 2009 2010 Time Period (6 Month Intervals) Composite Timely Discontinue Appropriate Hair Removal Normothermia Adjusted SSI Rate Hawn et al Ann Surg2011 254:494-501
Association between Adherence and VTE: Patient Level Total Unadjusted Adjusted N 30,531 % OR 95% CI OR 95% CI c-index 89. VTE Adherence 27,438 9 1.06 0.77-1.47 1.07 0.77-1.49 0.65 *Adjusted for age, gender, BMI, smoking status, weight loss, emergent case status, operative time Composite SCIP-VTE adherence was not associated with VTE outcomes even after adjusting for patient factors Altomet al Am J Surg. 2012 204:591-7
Association between Adherence and VTE: Hospital Level Altomet al Am J Surg. 2012 204:591-7
Association between Adherence and VTE: Secular Trends Altomet al Am J Surg. 2012 204:591-7
Richman et al JAMA Surgery in press Whole Cohort
Conclusion SCIP Performance Has Little Correlation with Outcomes
What about Patient Safety Indicators and Hospital Acquired Conditions? UAB Hospital 12/2012 through 8/2013 11,899 pts admitted to or discharged from DOS 552 total adverse outcomes in 448 patients 235 deaths (2.0%) 253 PSI (at least one) 26 HAC (at least one)
PSI by Preventability and Reason Table 3a: PSI Outcomes by Admission Type and Preventability Overall Not Preventable Possibly Preventable Preventable N (%) N (%) N (%) N (%) p-value Overall 290 89 (30.7) 171 (59.0) 30 (10.3) Admission Type Emergent 107 (36.9) 39 (43.8) 58 (33.9) 10 (33.3) 0.47 Urgent 26 (9.0) 9 (10.1) 15 (8.8) 2 (6.7) Elective 157 (54.1) 41 (46.1) 98 (57.3) 18 (60.0) % of total PSI 100 80 60 40 20 0 n = Respiratory 27 DVT/PTE 27 Hemorrhage 23 Puncture 27 Escalation Protocol Preop Opt Judgment Supervision Technical
Do Patient Safety Indicators Correlate with Clinical Outcomes? 10 AHRQ PSI Post-op Physiologic and Metabolic Derangement 11 Post-op Respiratory Failure 12 Post-op PulmonaryEmbolism (PE)/ Deep Vein Thrombosis (DVT) VASQIP Surgical AE Actual Renal Failure -Failure to Wean - Reintubation -PE -DVT 13 Post-op Sepsis Systemic Sepsis 14 Post-op Wound Dehiscence Dehiscence
Data Sources VA Patient Treatment File (PTF) VASQIP Database Data Element Source Data Element Source Demographic information (age, sex); Diagnostic information (ICD-9 CM); Summary information on episode of care (dates, setting of care, DRG, discharge status) PTF-Main Sex, principal disease and comorbidities Surgical Clinical Nurse Reviewer (SCNR) collected pre-operative data One primary and up to four secondary bedsectiondiagnoses, DRG and length of stay information for each bedsection stay PTF- Bedsection Emergency operations, principal anesthesia technique, intra operative blood loss SCNR collected intra operative data ICD-9-CM procedure codes, date, time, and site of all procedures during the inpatient stay PTF- Procedure Systemic sepsis, pneumonia, unplanned intubation, pulmonary embolism SCNR collected post operative data ICD-9-CM surgery codes and surgical specialty for each hospitalization PTF-Surgery 24 selected adverse events including morbidity/mortality SCNR collected 30 days post surgery data
Example: Respiratory Failure Example of PSI#11 Events eligible for PSI #11, Respiratory Failure N=109,916 VASQIP AEs N = 2,151 surgeries VASQIP only n=683 Overlap n=1,468 PSI AEs N = 2,047 surgeries PSI only n=579 Comparingthe Chart Chart Reviewed Reviewed 20 15 PSI VASQIPonly VASQIP gold Casesstandard PSI-only to the Differences in in AE A 13 (65%) Sensitivity definition -PSI codes 68.3% Specificity PSI for reintubationhave false negative 6 (30%) 1 (7%) 99.4% different timing criteria Unableto Positive than VASQIP determine without predictive value input Unableto from determine VASQIP SCNR without input from VASQIP SCNR 71.7% 1 (5%) 14 (93%)
Results PSI Validity Comparison Group Results for PSI version 4.1a, FY03-07 Data PSI VASQIP AE #10 PMD ARF #11 RF #12 PE/DVT #13 Sepsis SS #14 WD Dehiscence Cases eligible for PSI(%)* 131,711 (49%) FW R/UI 109,916 (41%) PE 253,090 DVT (95%) 47,523 (18%) 74,251 (28%) VASQIP- Detected AEs (%)** 442 (0.3%) 2,151 (2.0%) 1,588 (0.6%) 570 (1.2%) 1,254 (1.7%) *Percent of the PTF-VASQIP matched sample of 268,771 hospitalizations. **Percent of the hospitalizations eligible for the corresponding PSI. PSI- Detected AEs (%)** 322 (0.2%) 2,047 (1.9%) 3,323 (1.3%) 397 (0.8%) 544 (0.7%) PSI Sensitivity (95% CI) 48% (44-53%) 68% (66-70%) 65% (63-67%) 31% (27-35%) 31% (29-34%) PSI PPV (95% CI) 66% (61-72%) 72% (70-74%) 31% (30-33%) 44% (39-49%) 72% (68-76%)
Opportunities for Improvement: PSI Hemorrhage (9): Need more data: balance between the enemy of good is perfect, anticoagulation to prevent DVT and never leaving the OR Respiratory failure (11): Escalation: (previous slide conclusions) Documentation: Elective extubation within 48 hours of operation NOT a PSI DVT/PTE (12): Protocol: need to work together on high risk services to balance bleeding vs anti-coagulation Protocol: need to ensure that each patient is on appropriate prophylaxis Accidental Puncture/Perforation (15): Education: rules for documentation of inherent conditions and therefore NOT a PSI
Conclusions PSI and Outcomes Measures Have Little Overlap
UAB O and E Mortality by Quarter 140 120 100 80 60 Expected Observed 40 20 0 Q3CY12 Q4CY12 Q1CY13 Q2CY13 Q3CY13 Q4CY13
UHC Mortality Ratios UHC data based on discharge service only Many surgical services transferred patients to other services within the hospital where they died. palliative care MICU Unclear the contribution of the surgical services to the patients morbidity or mortality Reviewing mortality based on whether a patient was admitted to or discharged from a surgical service captures all aspects of surgical care.
Understanding the O/E ratio Observed Deaths 10 patients Expected Death O (actual number of people who died) E (Each ptassigned a risk of dying b/w 0 and 1 based on coding) Sum all 10 ptsrisk to create total expected risk of dying (denominator) 3 E If E > 3 (i.e. 3.5) that means lesspeople died than expected (Good News) If E < 3 (i.e. 2.5) that means morepeople died than expected (Bad News)
DOS OE Mortality Ratio by Quarter 1.6 1.4 1.2 Education M&M change POA Statements More CDS Quality Restructure POA App 1 0.8 0.6 OE ratio 0.4 0.2 0 p<0.05 Q3CY12 Q4CY12 Q1CY13 Q2CY13 Q3CY13 Q4CY13
UHC IQI90 %ilerank by Quarter 0 Q3CY12 Q4CY12 Q1CY13 Q2CY13 Q3CY13 10 20 30 40 50 IQI90 60 70 80 90 100
Accurate documentation with service specific present on admission (POA) statements (increased E ) We have compiled the top 10 DRGs for each service for the FY 2012. From this we have extracted the important drivers of the UHC model for determining the individual patients mortality risk by DRG. Each category has language that the coders need to appropriately code.
Conclusions Surgeon led, systematic quality improvement efforts improve patient care and documentation. These efforts positively effect our external measures of quality. Significant leadership support, political will and perseverance are critical. this is not going away Teamwork with the physicians, administration, data resources and the coding team are essential.
Understanding the differences among the various measures will help direct QI efforts ACS NSQIP provides the gold standard to double check your other measures against.
Lack of Cohesive Snapshot of your Hospital? Disparate surgical populations Different definitions of outcomes Different assessment/screening Coding versus clinical data Time point of measure Index hospitalization