Today s presenters have nothing to declare. 18 IHI National Forum Better Quality Through Better Measurement Worksheets IHI Faculty Dr. Robert Lloyd, Sue Butts-Dion & Dr. Todd Hatley Minicourse Q4 11 December 18 Measurement Self-Assessment Source: R. Lloyd, Quality Health Care: A Guide to Developing and Using Indicators. 2 nd edition, Jones & Bartlett Publishers, 17. Measurement Topic or Skill Help people in my organization determine why they are measuring (improvement, judgment or research) Move teams from concepts to specific quantifiable measures Building clear and unambiguous operational definitions for our measures Develop data collection plans (including stratification and sampling strategies) Explain why plotting data over time (dynamic display) is preferable to using aggregated data and summary statistics (static display) Explain the differences between random and non-random variation Construct run charts (including locating the median) Explain the reasoning behind the run chart rules Interpret run charts by applying the run chart rules Explain the statistical theory behind Shewhart control charts (e.g., sigma limits, zones, special cause rules) Describe the basic 7 Shewhart charts and when to use each one Help teams select the most appropriate Shewhart chart for their measures Describe the rules for special cause variation on a Shewhart chart Help teams link measurement to their improvement efforts Response Scale 1 2 3 4 5 1. I'd definitely have to call in an outside expert to explain and apply this topic/method. 2. I'm not sure I could apply this appropriately to a project. 3. I am familiar with this topic but would have to study it further before applying it to a project. 4. I have knowledge about this topic, could apply it to a project but would not want to be asked to teach it to others. 5. I consider myself an expert in this area, could apply it easily to a project and could teach this topic/method to others. 17 /R. Lloyd 1
Aim Statement Worksheet Project Topic: Aim statement (What s the problem? Why is it important? What are you going to do about it?) How good? By when? 3 16 /R. Lloyd Organizing Your Measures Worksheet Topic for Improvement: Concept Potential Measure(s) Outcome Process Balancing Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. 2 nd Edition, Jones & Bartlett Learning,17. 16 /R. Lloyd 2
Operational Definition Worksheet 5 Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. 2 nd Edition, Jones & Bartlett Learning,17. Measure Name: (Remember this should be specific and quantifiable, e.g., the time it takes to,the number of, the percent of or the rate of ) Operational Definition Define the specific components of this measure. Specify the numerator and denominator if it is a percent or a rate. If it is an average, identify the calculation for deriving the average. Include any special equipment needed to capture the data. If it is a score (such as a patient satisfaction score) describe how the score is derived. When a measure reflects concepts such as accuracy, complete, timely, or an error, describe the criteria to be used to determine accuracy. 16 /R. Lloyd Data Collection Plan Worksheet Project name & location: Measure Name Is Stratification appropriate? If Yes, list the levels of stratification Will you use sampling? If Yes, describe the sampling method you will use Frequency of data collection (e.g., hourly, daily weekly?) Duration of data collection (i.e., how long do you plan to collect the data?) Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. 2 nd edition, Jones and Bartlett, 17 15 /R. Lloyd 3
Measurement Dashboard Worksheet Name of team: Date: Measure Name (Be sure to indicate if it is a count, percent, rate, days between, etc.) Operational Definition (Define the measure in very specific terms. Provide the numerator and the denominator if a percentage or rate. Be as clear and unambiguous as possible) Data Collection Plan (How will the data be collected? Who will do it? Frequency? Duration? What is to be excluded?) Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. 2 nd Edition, Jones and Bartlett, 17. 16 /R. Lloyd After plotting the dots, calculate the median position and then the median value 1. Find the Median Position (n+1)/2 2. Then slide a piece of paper down the page to reveal the dots in descending order. 3. At the point where you find the Median Position draw a horizontal line across the chart. This is the median line. 4. Determine where this line intersects the Y axis and find the Median Value. Number of Clinic Visits Visits 90 80 70 60 50 40 30 Do NOT use a calculator or Excel to find the median! 0 1/6/09 2/6/09 3/6/09 4/6/09 5/6/09 6/6/09 7/6/09 8/6/09 9/6/09 /6/09 11/6/09 12/6/09 1/7/09 2/7/09 3/7/09 4/7/09 5/7/09 6/7/09 7/7/09 8/7/09 9/7/09 (n+1)/2 = median position /7/09 11/7/09 12/7/09 1/8/09 2/8/09 3/8/09 4/8/09 5/8/09 17 /R. Lloyd 4
How many runs on this chart? 30 25 Run = series consecutive points above or below the Median, ignore points equal to Median 15 9 5 0 Jan-97 Mar-97 May-97 Jul-97 Sep-97 Nov-97 Jan-98 Mar-98 Points on the Median (don t count these when counting the number of runs) May-98 Jul-98 Sep-98 Nov-98 Jan-99 Mar-99 17 /R. Lloyd Run Chart: Medical Waste Determine the number of runs on this chart 6.00 5.75 5.50 5.25 Pounds of Red Bag Waste 5.00 4.75 4.50 4.25 4.00 Median=4.6 3.75 3.50 3.25 Points on the Median (don t count these when counting the number of runs) 1 2 3 4 5 6 7 8 9 11 12 13 14 15 16 17 18 19 21 22 23 24 25 26 27 28 29 Point Number 17 /R. Lloyd 5
Non-Random Rules for Run Charts A Shift: 6 or more A Trend 5 or more Too many or too few runs An astronomical data point Source: The Data Guide by L. Provost and S. Murray, Jossey-Bass Publishers, 11. 17 /R. Lloyd Rule #3: Too few or too many runs Use this table by first calculating the number of "useful observations" in your data set. This is done by subtracting the number of data points on the Median from the total number of data points. Then, find this number in the first column. The lower number of runs is found in the second column. The upper number of runs can be found in the third column. If the number of runs in your data falls below the lower limit or above the upper limit then this is a signal of a special cause. # of Useful Lower Number Upper Number Observations of Runs of Runs 15 5 12 16 5 13 17 5 13 18 6 14 19 6 15 6 16 21 7 16 22 7 17 23 7 17 24 8 18 25 8 18 26 9 19 27Total useful observations 19 28 29Total data points 30 12 11 21 The key point here is that in any data set you can have too much or too little variation. In either case, it does not produce a normal distribution. 16 /R. Lloyd 6
Run Chart Interpretation: Medical Waste 6.00 5.75 Total data points = 29 Data points on the Median = 2 Number of useful observations = 27 (should have between &19 runs) The number of runs = 14 Number of times the data line crosses the Median = 13 + 1 = 14 5.50 5.25 Pounds of Red Bag Waste 5.00 4.75 4.50 4.25 4.00 3.75 3.50 Points on the Median (don t count these as useful observations ) Median=4.6 Are there any non-random patterns present in this chart? 3.25 1 2 3 4 5 6 7 8 9 11 12 13 14 15 16 17 18 19 21 22 23 24 25 26 27 28 29 Point Number 17 /R. Lloyd Determine the number of runs on this chart and then apply the 4 Run Chart Rules Time of Registration of Critically Ill U5 to Initiation of Treatment in the Ward Time in Minutes 160 140 1 0 80 60 40 Fast track at Lab Triage at OPD, Fast track at Registration and Billing NOTE: 17 total data points with 0 on the Median = 17 useful observations. Should be between 5 and 13 runs. How many do you find? Median=79 0 Nov-09 Jan- Mar- May- Jul- Sep- Nov- Jan-11 Mar-11 May-11 Jul-11 Sep-11 Nov-11 7
Determine the number of runs on this chart and then apply the 4 Run Chart Rules Time of Registration of Critically Ill U5 to Initiation of Treatment in the Ward Time in Minutes 400 350 300 250 0 150 0 50 Additional nurse to triage Triage at OPD, Fast track at Lab NOTE: 18 total data points with 3 on the Median = 15 useful observations. Should be between 5 and 12 runs. How many do you find? 0 Aug-09 Nov-09 Feb- May- Aug- Nov- Feb-11 May-11 Aug-11 Nov-11 Median=196.5 Early ANC Registration: % of ANC Registrants in 1 st Trim at Registration Percentage 90 80 70 60 50 40 30 0 NOTE: 22 total data points with 0 on the Median = 22 useful observations. Should be between 7and 17 runs. How many do you find? Change made 8
Neonatal Deaths Rate of Neonatal deaths per 00 live births 60 50 40 NOTE: 22 total data points with 0 on the Median = 22 useful observations. Should be between 7 and 17 runs. How many do you find? Change made Rate 30 0 1. The Median Position is 15 and the Median Value is 30. 2. It is useful to mark the data points that fall on the Median so you do not count them when determining the number of runs. 3. All the data points not on the Median are the useful data points for analysis using the run chart rules. 4. 29 total data points with 1 on the Median for 28 useful observations 5. Now count the number of runs and apply the rules! Visits 90 80 Run Chart Median 70 60 50 40 30 0 1/6/09 2/6/09 3/6/09 4/6/09 5/6/09 6/6/09 7/6/09 8/6/09 9/6/09 /6/09 11/6/09 12/6/09 1/7/09 2/7/09 M M M M M M Day 3/7/09 4/7/09 5/7/09 6/7/09 7/7/09 8/7/09 9/7/09 /7/09 11/7/09 12/7/09 1/8/09 2/8/09 3/8/09 4/8/09 5/8/09 Now, let s return to the Clinic Visits Run Chart 9
Let s find the Median Median Position: (27+1) = 28/2 = 14 th data point Median Value =? 90 Percent Compliance Percent Compliance 85 80 75 70 65 60 Now, how many runs are on this chart? 1 2 3 4 5 6 7 8 9 11 12 13 14 15 16 17 18 19 21 22 23 24 25 26 27 Week Rules for Detecting Special Causes on A Shewhart Chart 17 /R. Lloyd
Month ED /0 Returns Is there a Special Cause on this chart? M 41.78 17 1.2 A 43.89 26 M 39.86 13 J 40.03 16 J 38.01 24 A 43.43 27 Unplanned Returns to Ed w/in 72 Hours S 39.21 19 O 41.90 14 N 41.78 33 D 43.00 u ch a rt J 39.66 17 F 40.03 22 M 48.21 29 A 43.89 17 M 39.86 36 J 36.21 19 J 41.78 22 A 43.89 24 S 31.45 22 1.0 U C L = 0.88 Rate per 0 ED Patients 0.8 0.6 Mean = 0.54 0.4 0.2 LC L = 0.19 0.0 1 2 3 4 5 6 7 8 9 11 12 13 14 15 16 17 18 19 Copyright 13 /R. Lloyd Percent Patients Seen Minutes 1% 1% 0% 90% 80% 70% 60% 50% 40% 30% % % 0% XYZ Medical Center Performance Improvement Report March 25, 04 PERCENT PATIENTS C/O CHEST PAIN SEEN BY CARDIOLOGIST WITHIN MINUTES OF ARRIVAL TO ED EXAMPLE CHART 1 2 3 4 5 6 7 8 9 11 12 13 14 15 16 17 18 19 21 22 23 24 Fictitious data for educational purposes Are there special causes on this chart? Target Goal / Desired Direction: INCREASE in the PERCENT of patients c/o chest pain seen by cardiologist within minutes of arrival to Emergency Department. Interpretation: Current performance shows (desirable) upward trend. 24 Weeks: October 03 -- March 04 81.5 % P UCL Average LCL p-chart, possible range 0-0% 22 Copyright 13 /R. Lloyd 11
Number of Patient Complaints by Month (XmR chart) Are there any special causes present? If so, what are they? 50.0 45.0 40.0 UCL=44.855 A Number of Complaints 35.0 30.0 25.0.0 15.0 B C CL=29.250 C B A LCL=13.645.0 5.0 Jan01 Mar01 May01 July01 Sept01 Nov01 Jan02 Mar02 May02 July02 Sept02 Nov02 23 Month Copyright 13 /R. Lloyd You Make the Call! Is it an XmR (I) or X bar & S? Measure 1. Time to clean an inpatient room (in minutes) Subgroup XmR ( I chart) X bar & S chart 2. Patient satisfaction scores for a sample of 15 outpatients collected every 2 weeks 3. Avg. turnaround time for all STAT labs done each day and stratified by shift 4. Cost for each normal delivery 5. A diabetic patient s 3x a day blood sugar readings 6. Average length of stay for a random sample of ICU patients pulled each month 7. The distance (in feet) that a sample of knee replacement patients can walk in 15 seconds 12
The Control Chart Decision Tree Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. 2 nd edition, Jones and Bartlett, 17. Variables Data Decide on the type of data Attributes Data Yes More than one observation per subgroup? No No Occurrences & Nonoccurrences? Yes Yes Is there an equal area of opportunity? No X bar & S Average and Standard Deviation XmR Individual Measurement c-chart The number of Defects u-chart The Defect Rate p-chart The percent of Defective Units You Make the Call! Is it a p, c or u-chart? Measure 1. The number of central line insertions each week during which all elements of the bundle were followed divided by the total number of central line insertions that week Subgroup p- chart c- chart u- chart 2. The number of catheter-associated urinary tract infections is placed over the total number of urinary catheter days each month 3. The total number of patient falls each month (with or without injury to the patient and whether or not assisted by a staff member) is divided by the total patient days for the month 4. An analyst pulls a sample of 50 orthopedic surgery charts each week and counts all discrepancies from standard documentation practice. 5. Each medication order is checked against five potential types of errors. You also have the total number of orders placed each week 6. Each day the number of home healthcare visits that are more than 15 minutes late on arrival are recorded and compared with the total number of visits scheduled for that day. 7. The number of outpatients not showing up for an appointment is recorded each week. The total volume of outpatient appointments is also recorded. 13
Selecting the most appropriate chart for your measures Measure Name Outcome (O) Process (P) Balancing (B) Subgroup? Type of Data? Chart of Choice? V or A V or A V or A V or A V or A 14