Selecting Measures Presented by: Rebecca Lash, PhD, RN Collaborative Outcomes Council July 2016 Content adapted from Robert Martin, PsyD, Performance Excellence & Dr. Moira Inkelas Methods for Improvement and Implementation Science course UCLA School of Public Health 2
Common elements to consider for all approaches to clinical improvement Planning Buy-in and continued engagement of stakeholders and participants Selecting and measuring the appropriate outcomes Sustainability
Types of Measures Purpose of Measurement Key question Penalty for being wrong Research EBP QI/PI What is the truth? Misdirection for the profession Are we better or worse than? Impact to reputation, penalty, resources Are we getting better? Misdirection for an initiative Measurement requirements and characteristics Complete, accurate, controlled, glacial pace, expensive Risk adjusted, with denominators, attributable to individuals or orgs, validity Real time, raw counts, consistent operational definitions, utility Typical displays Comparison of control and experimental populations Performance relative to benchmarks and standards Run charts, control charts, time between events Adapted from Solberg,Mosser, McDonald Jt Comm J Qual Improv. 1997 Mar;23(3):135-47.
Types of Measures Outcome Measures Process Measures Balancing Measures How the system impacts values of patient health, experience, satisfaction, etc. How the processes in the system are performing per plan What affect are our changes having on other parts of the system? (unintended consequences) Examples Avg hemoblobin A1c level for diabetic pts Risk adjusted mortality Falls per 1000 pt days CAUTI rates Examples % of pts whose hemoglobin A1c level was measured 2x last year % of Falls Mobility Assessments completed per schedule Bundle Compliance Examples If we are reducing time pts spent on ventilators after surgery, measure increase of re-intubation rates If we improve discharge by noon, check increase of readmission rates You should consider including all three types of metrics http://www.ihi.org/resources/pages/howtoimprove/scienceofimprovementestablishingmeasures.aspx
*A3 is a UCLA Operating System 11x17 template used to document and communicate complex problem-solving using the Plan Do Check Act (PDCA) method: Steps 1-4 (Plan), Step 5 (Do), Step 6 (Check), Step 7(Act) A3* Project Title Project Lead: Project Champion(s): Date Updated: Project Team: 1) Problem Statement: (description &quantification of the problem and effect) In the X children's psychiatry clinic, a chart review study showed that monitoring guidelines to assess youth cardio-metabolic symptoms from atypical antipsychotics are only followed 9% of the time, whereas CAMESA guidelines suggest 80%. Our knowledge of patients cardiometabolic side-effects is generally unknown, which increases side-effects such as obesity and impacts patient safety. 2) Current State: (depiction of the current state, its processes, and problem(s) - Chart review showed only 9% full compliance - Nurses unable to height, weight, and blood pressure 100% of the time Best Practices/Literature Search: 1 - Practice Parameter for the Assessment and Treatment of Children and Adolescents with Schizophrenia (2013, in press) 2. American Diabetes Association (2004) 3. Canadian Alliance for Monitoring Effectiveness and Safety of Antipsychotics in Children (CAMESA) Guidelines (2011) 3) Goal: (how will we know the project is successful; standard/basis for comparison) Obtain 80% compliance with recommended metabolic monitoring rates of atypical antipsychotic medications by June 2013. 4) Root Cause Analysis: (investigation depicting the problems root causes) 5) Solutions: (action plan and findings of tested solutions) Root Cause Tested Solution Responsible Due Finding Not aware of specific recommendations in guidelines No standardized data collection instrument Blood pressure cuffs in one size No boards under scales in offices (Ensure accuracy of scales) 6) Check: (Summary of the solutions results, overall goal success, and any supporting metrics) Implementation of solutions have increased compliance with recommended metabolic monitoring rates of atypical antipsychotic medications. Goal & Metrics Baseline Target Current Average of % monitoring rate for first year on atypical antipsychotic medications Use of standardized data collection instrument Presence of blood pressure cuffs in two sizes in each office Boards placed under scales in each office Presentation of CAMESA guidelines for monitoring and management of side effects Training in use of CAMESA excel files Obtain adult sized blood pressure cuffs Obtain boards to place under scales in offices Jessica Jeffrey Jessica Jeffrey Karina Amaya, administrative assistant/ Jessica Jeffrey Karina Amaya, administrative assistant/ Jessica Jeffrey February 1, 2014 February 1, 2014 May 1, 2014 May 1, 2014 9% 80% 29% 0% 80% 70% (Range: 30%-90%) 0% 100% 0% 14% 100% 100% Eliminates root cause Eliminates root cause Eliminates root cause Eliminates root cause 7) Act: (Action taken as a result of the Check, and the plan to sustain results) Act
Measurement for Improvement Use standardized, evidence-based process or outcome measures for goals when available e.g. falls per 1000 patient days vs. count of falls per month For QI projects Create meaningful measures as needed for smaller PDCA cycles To improve, measurement does not need to be perfect (small samples are often ok in QI) Consider the effects of day of the week, shift, etc. N=1 (patient tracer, deep dive into one example, etc.)
Pitfalls to avoid Too many measures (4 to 10 is usually ok) Measures are vaguely defined, subject to multiple interpretations, not at the right level Measures are too complicated for others to understand Data collection process is too complicated Collection takes too much time Display of measures does not quickly help tell the story 8
Guiding Questions in Selecting Measures What processes can we move at all? What intermediate outcomes can we move? What processes/experiences will shift the outcome, have the greatest impact? What measures will motivate people to focused and collective action? (For goal targets) What is the best result that any system has achieved?
Evaluating Nurse Education Interventions Common component of EBP/QI/Research Project Examples Ventilator Weaning Protocol (ICU) Dealing with disruptive behaviors Selecting evaluation measures: What change do you expect to see in the nurse as a result of the learning activity and how will you measure it? Knowledge, Skills or Practice What patient or nurse level outcomes can be measured? Bundle compliance (process measure measuring practice?) Patient Outcome (medication errors, fall rates, restraint use) Nurse Outcome (satisfaction, retention, injury)
Discussion: What measures are you using? Current projects What is the plan to measure outcomes? What type of outcomes are they? Are the others to consider? Good or Bad examples? 11