Identifying and Defining Improvement Measures

Identifying and Defining Improvement Measures M1 December 8, 2014 Following the CAUTI Case P2 1. Baselines, Gaps, Aims, Outcomes Where are we now, and what are we trying to accomplish? 2. Building a Theory of Improvement (Driver Diagram) What should we measure and why? 3. Mapping the measures (Measure Tree) How will we calculate the measures? 4. Defining the Measures Attributes of Useful Improvement Measures 5. Collecting Data and Testing Changes 1

What Are We Trying to Accomplish? Dimensions of Quality Baselines, gaps, outcomes Aim Statement Three Types of Measures Outcome Measures Point to qualities that stakeholders value. Is this system meeting the needs of those who care about its operation? Is our improvement work making a meaningful impact? Process Measures Voice of the process. Are the parts/steps in the system performing as planned? Are processes reliable? Efficient? Patient-Centered? Are we on track to improve? Balancing Measures Are we producing perverse unintended consequences in our efforts to improve? What other factors may be affecting results? 2

Balancing Outcomes: IHI Triple Aim P5 IOM Report: Dimensions of Care Quality Safe - as safe in healthcare as in our homes Effective - matching care to science; avoiding overuse of ineffective care and underuse of effective care Patient-centered - honoring the individual and respecting choice Timely - less waiting for both patients and those who give care Efficient - reducing waste Equitable - closing racial and ethnic disparities in access and health status Institute Of Medicine (2001). Crossing the quality chasm : a new health system for the 21st century. Washington, D.C., National Academy Press. 3

IHI Triple Aim: Examples of Measures Systems of Care P8 D: The environment (policy, payment, accreditation, etc.) C: Organizations that support microsystems B: Microsystems A: Lives of Patients 4

Case Background: Reducing CAUTIs A medium sized acute care hospital has noticed that there has been an increasing occurrence of catheter associated urinary tract infections (CAUTIs) over the past year. Not only has the occurrence of CAUTIs been gradually going up but also the severity of the infections has been increasing. Indwelling urinary catheters are commonly used medical devices within acute and non-acute settings. But their use does increase the risk of CAUTIs by: Enabling organisms to gain entry to the bladder via external surface or opened connections Reducing the body's defense of flushing out organisms during urination Facilitating biofilm formation Reducing CAUTIs would contribute to: Improving the patient experience Reducing the cost of antibiotic prescribing Reducing inpatient length of stay Reducing readmissions Improving patient outcomes Baseline Data Key Outcome P10 5

Aim P11 AIM: Reduce CAUTI infections in all units below 1.6 (10 th percentile) within 12 months and to zero within 24 months. Exercise P12 1. Case Discussion Why are catheter-associated infections measured as Number of CAUTIs per 1000 Foley catheter days? What is the evidence that the rate of infections has actually been increasing? 2. Own Project: Reflect and discuss in pairs What are you trying to accomplish (your aim?) What is the outcome measure that best captures the aim of your project? What is the baseline level of performance on the outcome? How much does the outcome need to improve? 3. Share with the group 6

Building a Theory of Improvement Driver diagrams Prioritization Linking drivers and measures Theory Drives Improvement P14 Without theory, there are no questions; without questions, there is no learning. 7

A Theory of How to Improve a System P15 Effect Drives Cause Theory for CAUTI Reduction Version: 11/26/2014 Outcomes Primary Drivers Secondary Drivers Changes / Interventions S1 Clear policies for infection control P1 Leadership and aligned policy for catheter use S2 Transparent reporting of process failures S3 Staff training, with feedback on observed protocol compliance Reduce catheter associated urinary tract infections by 50% in one year P2 Eliminate unnecessary catheter insertions S4 Insert catheters only for appropriate indications S5 Consider alternative methods S6 Minimize use of catheters for patients at risk for infections P3 Reliable compliance with catheter insertion protocol S7 Remove when no longer required S8 Insertion only by trained staff S9 Standard insertion procedure Insertion Bundle: A Hand Hygiene B Sterile gloves, materials C Aseptic insertion D Unobstructed flow P4 Reliable compliance with catheter maintenance protocol S10 Daily assessment of need, removal at earliest opportunity S11 Standard cleaning and maintenance procedure Maintenance Bundle: A Tamper seal intact B Secured in place C Hand hygiene D Meatal hygiene E Disposal & clean container F Maintain unobstructed flow 8

Needed Measures for CAUTI Reduction Outcomes Primary Drivers Secondary Drivers Changes / Interventions Version: 11/26/2014 S1 Clear policies for infection control Reduce catheter associated urinary tract infections by 50% in one year P1 Leadership and aligned policy for catheter use S2 Transparent reporting of process failures S3 Staff training, with feedback on observed protocol compliance A fundamental assumption of S4 Insert catheters only for appropriate indications clinical QI: P2 Eliminate unnecessary catheter S5 Consider alternative methods insertions Reliable execution of key clinical S6 Minimize use of catheters for driver processes improves patients at risk for infections P3 Reliable outcomes compliance measured S7 Remove when no longer at required the with catheter insertion protocol Insertion Bundle: A Hand Hygiene population S8 Insertion only level by trained staff B Sterile gloves, materials C Aseptic insertion D Unobstructed flow S9 Standard insertion procedure P4 Reliable compliance with catheter maintenance protocol S10 Daily assessment of need, removal at earliest opportunity S11 Standard cleaning and maintenance procedure Maintenance Bundle: A Tamper seal intact B Secured in place C Hand hygiene D Meatal hygiene E Disposal & clean container F Maintain unobstructed flow Measuring Improvement Measures let us Monitor progress in improving the system Identify effective changes 9

Prioritizing Drivers P20 Limitations of resources, attention or will usually mean we cannot work on (or measure!) everything. Priorities: Where is the Bang for Buck? Which drivers do we believe will deliver the biggest impact? Which ones will be easiest to work on? Most difficult? Are some beyond our control? What is our current level of performance on these drivers? CAUTI Driver Rankings P21 HIGH Impact LOW LOW Difficulty HIGH 10

CAUTI Priority Measure Concepts P22 S4: Insert catheters only for appropriate indications. The most effective way to eliminate the possibility of a CAUTI is to eliminate an unneeded catheter. S7: Remove when no longer required. Since the risk of infection is roughly proportional to the time the catheter is in place, removing catheters as soon as possible will reduce the risk. S9: Standard insertion procedure. If trained staff follow strict protocols for aseptic insertion of catheters, the risk of bacterial infection will be minimized. S11: Standard cleaning and maintenance procedure. Similarly, careful adherence to the components of the maintenance bundle will reduce risk. Exercise P23 Use the Driver Diagram Rubric to guide the following: Case Discussion Do you have questions or issues about the CAUTI driver diagram? Discuss and resolve. If you get stuck, raise the question to the group. Own Project Discussion Review (or create) the driver diagram for your project Discuss in pairs (or to table) Be prepared to share with the group 11

Driver Diagram Rubric P24 1. Does the aim of the diagram focus on OUTCOMES? 2. Do the driver labels refer to the improvements needed to accomplish the aim? 3. Are all of the secondary drivers necessary for achieving the aim? 4. Are the secondary drivers sufficient to achieve the aim? 5. Do the drivers consider needed process, leadership, cultural, and structural changes? 6. (Optional) Does the diagram include change concepts or specific change ideas that might be tested as part of an improvement initiative? Testing the diagram: Show the driver diagram to a knowledgeable person who is naïve to the system you want to improve. Ask them to explain what you are trying to accomplish and how. Identify areas of confusion, and consider revising your diagram. Mapping the Measures Measure Tree Diagram 12

Measures for CAUTI Reduction Version: 11/26/2014 Outcomes Primary Drivers Secondary Drivers Changes / Interventions S1 Clear policies for infection control S2 Transparent reporting of process failures P1 Leadership and aligned policy for catheter use S3 Staff training, with feedback on observed protocol compliance Reduce catheter associated urinary tract infections by 50% in one year P2 Eliminate unnecessary catheter insertions S4 Insert catheters only for appropriate indications S5 Consider alternative methods M1 S6 Minimize use of catheters for patients at risk for infections M3 M4 P3 Reliable compliance with catheter insertion protocol M2 S7 Remove when no longer required S8 Insertion only by trained staff M5 S9 Standard insertion procedure Insertion Bundle: A Hand Hygiene B Sterile gloves, materials C Aseptic insertion D Unobstructed flow P4 Reliable compliance with catheter maintenance protocol S10 Daily assessment of need, removal at earliest opportunity S11 Standard cleaning and maintenance procedure M6 Maintenance Bundle: A Tamper seal intact B Secured in place C Hand hygiene D Meatal hygiene E Disposal & clean container F Maintain unobstructed flow CAUTI Measures P27 13

CAUTI Reduction Measures P28 Denominators Numerators Measures D1 Count of patients with catheters in situ in measurement month D3, N2 Sum of days with catheters in situ D2 Count of catheters inserted in measurement month N1 Count of patients meeting critiera for catheter insertion N3, M3 Count of CAUTIs in measurement month N4 Count of catheter insertions with all insertion bundle elements in compliance N5 Count of catheters with all maintenance bundle elements in compliance M1 Percent of patients with appropriate catheter placements M2 Average catheter duration M4 CAUTIs per 1000 patient days M4 (alternate) Catheter days between CAUTI events M5 Percent of catheter insertions with all insertion bundle elements in compliance M6 Percent of catheter insertions with all maintenance bundle elements in compliance Exercise P29 Case Discussion Do you have questions or issues about the CAUTI measure tree? Discuss and resolve. If you get stuck, raise the question to the group. Own Project Discussion Based on your own driver diagram, identify the outcome and key process measures you will need (1 outcome, no more than 4 process) Create a measure tree that shows the numerators and denominators for your measures. Be sure your process measures are linked to drivers. 14

Attributes of Useful Improvement Measures Responsive Valid Comprehensible Timely Feasible Relevant Attributes of Useful Improvement Measures Responsive Valid The measure issensitive to changes in the system state. When the system improves, the measure says so. The measure aligns with the theory of changes (driver diagram). Improvement in the measure means improvement in the system. Comprehensible The intended audience understandsthe meaning of the measure for system improvement. Timely Feasible Relevant Consistent Ownership The data are available soonenough to inform improvement decisions (project planning, PDSA testing). The data can be collectedwith minimum effort and cost, and in a timely fashion. Themeasure supports problem identification and testing at the appropriate level of management. Themeasure has a clear definition: it yields consistent results when applied in different places and at different times. Someone is explicitly assigned to monitor the measure on a regular basis, detect problems, and initiate change. 15

Why Time Is Important for Measuring Improvement Improvement is temporal! Lloyd Provost Displaying data over time (using run charts or control charts) allows us to make informed predictions, and thus manage effectively Did We Improve? P35 Percent of ER patients with Chest Pain Seen by a Cardiologist within 10 min Did we improve? What will happen next? Should we do something? Source: R. Lloyd 16

Validity: Aligned with Improvement Theory Outcomes Primary Drivers Secondary Drivers Changes / Interventions Version: 11/26/2014 S1 Clear policies for infection control P1 Leadership and aligned policy for catheter use S2 Transparent reporting of process failures S3 Staff training, with feedback on observed protocol compliance Reduce catheter associated urinary tract infections by 50% in one year P2 Eliminate unnecessary catheter insertions S4 Insert catheters only for appropriate indications S5 Consider alternative methods S6 Minimize use of catheters for patients at risk for infections M1 M3 M4 P3 Reliable compliance with catheter insertion protocol M2 S7 Remove when no longer required S8 Insertion only by trained staff M5 S9 Standard insertion procedure Insertion Bundle: A Hand Hygiene B Sterile gloves, materials C Aseptic insertion D Unobstructed flow P4 Reliable compliance with catheter maintenance protocol S10 Daily assessment of need, removal at earliest opportunity S11 Standard cleaning and maintenance procedure M6 Maintenance Bundle: A Tamper seal intact B Secured in place C Hand hygiene D Meatal hygiene E Disposal & clean container F Maintain unobstructed flow Validity: Alignment with Improvement Work P39 Total Population Measurement Pilot Unit Sample Improvement in a pilot population (1 practice, 1 unit, etc.) will not be evident in measures based on the total population (city, hospital system) 17

Validity: Alignment with Improvement Work P40 Total Population Target population To track improvement, we must measure in the same target population where we are working to improve. Comprehensible? Percentage of patients discharged in the measurement month that suffered a CAUTI Number of CAUTIs per 1000 Foley catheter days during measurement month Number of CAUTIs per 1000 inpatient days during the measurement month Count of CAUTIs in the measurement month Number of catheter days since the last CAUTI event 18

Importance of Timely Data Consistency: Operational Definition A procedural description of what to measure and the steps to follow to measure it consistently Gives communicable meaning to a concept Tells what you need to count or measure, and how to do it Specifies measurement methods and equipment Provides guidance on sampling Identifies detailed criteria for inclusion and exclusion is the basis for reliable measurement Source: R. Lloyd 19

Operational Definition Example Measure: Percentage of patients undergoing hip and knee replacement surgery during the measurement period who have had preoperative nasal swabs to screen for Staphylococcus aureus carriage Goal: 95% Measurement Period Length: Monthly Numerator Definition: Number of patients undergoing hip or knee replacement surgery who have had a nasal swab specimen processed to screen for Staphylococcus aureus carriage prior to surgery Denominator Definition: Number of patients undergoing elective hip or knee replacement surgery Numerator and Denominator Exclusions: Patients who are less than 18 years of age Patients who had a principal or admission diagnosis suggestive of preoperative infectious diseases Patients with physician-documented infection prior to surgical procedures Patients undergoing non-elective hip or knee replacement surgery Definition of Terms: Hip or knee replacement surgery includes operations involving placement of a nonhuman-derived device into the hip or knee joint space. ICD-9 Codes include 00.70-00.73, 00.85-00.87, 81.51-81.53, 00.80-00.84, 81.54, and 81.55. Calculate as: (numerator/denominator*100) Responsive Percent of catheters removed during the measurement month within 2 days of insertion Average catheter duration by month Which measure better reflects the improvement work of teams trying to reduce unnecessary catheter placement? Which measure better reflects protocol compliance?` 20

January April Average Time versus Percent Conforming* Specification = 30 min or less Which measure is most useful to an improvement team? % of cases with Abx within 30 min Average time to Abx admin June August September *Simulated data via @Risk Population Measures Throughput = visits P50 with reliable care process Population: who s health are we responsible for? have an incremental impact on population. DM pts with LDL<100 Active DM pts in practice panel 21

Typical Population Questions What is the current state of the population for whom we are responsible (even those we haven t seen for awhile?) re: Health status? Pt. Experience? Cost of care? How do our population s risk factors and outcomes compare with those of other provider organizations? How should we plan for the long term? What has the impact of our improvement work been on the population? Are there other factors effecting changes in outcomes? Outpatient Look-Back Measures Percent of population with current self-management plan as of most recent visit within the past 12 months. Current test No current test 12 months Each measurement contains mostly the same patients as the previous month. These measures are slow to show improvement, but reflect the state of care for the population! 22

Current Care Measures Percent of patients seen last monthwho lacked an up-to-date A1C and who got the test during the visit or were referred. Current test No current test 1 mo Each subgroup contains different patients & represents current work These measures are great for tracking process changes! Exercise P59 Case Discussion The worksheet titled Attributes of Useful Improvement Measures shows alternative measures that might be used to capture the key measure concepts in the case. Compare the alternative measures with respect to the attributes of useful improvement measures Own Project Discussion Refer to your proposed project measures How do your measures relate to the attributes of useful improvement measures? Discuss at your table Share your insights 23

P60 Data Collection Data for PDSA testing Concurrent data collection Segmentation Sampling 24

Project Data Collection Existing EMR system PRO: data collected as component of routine care CON: needed process measures may not be included; data may lag by weeks or months; process failures lack context; usually requires custom reports Paper chart review PRO: notes may provide useful context; may be necessary if no electronic system CON: labor intensive (but sampling helps); data may lag by days or weeks Concurrent log or registry PRO: ad hoc data can target PDSAs, project measures; no lag; context available; CON: extra work for caregivers; special data process necessary Measuring Reliability Reliability = Number of Actions That Achieve The Intended Result Total Number of Actions Taken or Intended = Percent Conforming 25

Proposal that patient requires urinary catheter Indications are appropriate? Yes Alternative methods available? No Check pt. for past problems, allergies, etc. % of cases with appropriate indication No Yes Stop Explain procedure to pt. and/or caregivers Decontaminate hands % of cases with proper hand hygiene Clean and prepare the work area, assemble materials Put on personal protection equipment (PPE) and sterile gloves % of males with proper insertion procedure Prepare patient Follow male procedures for urinary catheter insertion A Is the patient male? Yes No Dispose of equipment and materials in designated bag. Remove PPE and wash hands Record patient experience, document technical specifications and time of completion into the chart % of females with proper insertion procedure Follow female procedures for urinary catheter insertion B Measuring Process: Total Joint Arthroplasty Aim: Pre-screen all total hip or knee replacement patients for nasal Staph; those who test positive will complete a course of mupirocin. Population: All patients undergoing TKA or HKA in our hospital (with exclusions) Process: Screening and decolonization Measurement interval: monthly Process reliability measure: Percent of patients who screened positive for SA who report they had completed a course of mupirocin prior to day or surgery. 26

Staph aureus (SA)Screening and Decolonization Process Example 1-4 weeks pre-procedure 2-3 weeks pre-procedure Day of surgery Scheduling Schedule procedure TKA or THA? Yes Insert lab request for SA culture Inform patient of SA screening % of cases with missing lab order No % of no-shows for swab Pt presents for nasal swab Notify hospital Lab Process specimen Positive for SA? No Time to receive lab results Results to surgeon & hospital Yes Time to notify patient % of positive results not acted on Contact patient Hospital /Surgeon Prescribe 5 day mupirocin Confirm Rx complete Document in record Suggested measures KEY RELIABILITY MEASURE % of colonized patients with completed Rx Surgery Individual Patient Data to Assess Process Improvement Time to Antibiotic Medication Sept 2010 - Feb 2012 160 Mean = 46 Mean = 38 Mean = 35 Guidelines #1 Guidelines #2 Mean = 27.3 Mean = 29.9 Updated: March 7, 2012 Shift = 6; Trend = 5 ED Comm Excel Dose Calc 140 120 100 Time(min) 80 60 40 Goal: 20 ED Comm Stickies/ QI VOE Guidelines 9/16/10 10/1/10 10/11/10 10/31/10 11/4/10 12/2/10 12/8/10 12/19/10 12/27/10 1/17/11 2/2/11 2/10/11 2/10/11 2/15/11 2/25/11 2/28/11 3/2/11 3/5/11 3/24/11 3/28/11 4/28/11 5/8/11 5/13/11 5/17/11 5/23/11 5/23/11 6/15/11 6/25/11 6/27/11 7/3/11 7/8/11 7/10/11 7/18/11 7/28/11 7/30/11 8/4/11 8/10/11 8/18/11 8/25/11 8/28/11 8/31/11 9/5/11 9/8/11 9/21/11 9/25/11 9/27/11 10/2/11 10/5/11 10/18/11 10/21/11 10/25/11 10/31/11 11/10/11 11/16/11 11/26/11 12/5/11 12/14/11 1/2/12 1/5/12 1/28/12 1/30/12 2/8/12 2/14/12 2/20/12 Individual Patients Over Time 0 Source: James Moses, BMC 27

Sampling when you can t measure the entire population, you can estimate its characteristics by sampling Systematic sampling Random sampling Stratified sampling Convenience sampling Judgment sampling Sampling Methods Convenience Sample Gosh I m in a hurry. Why don t I just review these? Source: R. Lloyd 28

Sampling Methods Simple Random Sample Every element has an equal chance of being selected Sampling Methods Systematic Sample First element selected at random then select every n th element You might survey every 10th patient who arrives at a clinic beginning at a randomly selected time Possible bias if there are patterns in the sequence of elements 29

Sampling Methods Judgment Sample Especially for PDSA testing, someone expert with the process selects items to measure: To include a range of conditions Selection criteria may change as understanding increases Successive small samples instead of one large sample What Sample Size? To be useful, samples should be large enough to reveal improvement shifts and trends. This also depends on magnitude of the change, and the inherent variability of the measure. 30 is a good rule of thumb for current care measures You can approach this issue empirically Don t sample unless you need to Small samples ok for PDSA testing 30

Tracking Change Segment by Segment Jan 10 Mar 11 Segment 1 - Pilot Segment 2 Segment 3 Segmentation By Organization Site Provider Region Diagnosis Patient process trajectory R. Scoville 76 31

Exercise How did the CAUTI team approach their data collection task? Own Project What data are available to support your improvement measurement plan? Is it possible to gather concurrent data? What are some of the change ideas that you will test? What data will be needed to assess their impact? How will you gather those data? 32