Basic Skills for CAH Quality Managers MARCH 20, 2014 THE BASICS OF DATA MANAGEMENT Data Management Systems COLLECTION AGGREGATION ASSESSMENT REPORTING 1
Some Data Management Terminology Objective data Subjective data quantitative qualitative Semi-quantitative data Aggregation Assessment Statistical significance Data Collection WHO WHAT WHEN HOW 2
Who Collects the Data? SOM, Appendix W, C-0337 All patient care services and other services affecting patient health and safety are evaluated Survey procedures: Who is responsible to evaluate CAH patient care services? What Data to Collect KEEP IT SIMPLE! 3
Five Keep It Simple Data Collection Steps 1. Develop a list of potential data to collect 2. Use objective criteria to identify the vital few 3. Define specific performance measures 4. Clarify data collection cycles 5. Clarify data collection responsibilities work through this with your quality management team to build consensus, get buy-in 1. Develop a List of Potential Data You Could Collect What do we have to collect What should we collect What do we want to collect 4
Data We Have To Collect SOM for CAHs, Appendix W OSHA Life Safety Code Contracts, liability carriers Voluntary accreditation organizations CAH SOM: Data We Have to Collect C-150 C-154 C-200 C-220 C-227 C-231 Compliance with federal, state and local laws; includes EMTALA Staff licensing and certifications Emergency Services - Blood use and therapeutic gases Building and equipment maintenance Emergency Preparedness Life Safety 5
CMS SOM: Data We Have To Collect C-251 C-263 C-276 C-277 C-278 C-279 Physicians: quality of tx and dx Mid-levels: quality of tx and dx Medication Use Adverse drug events Nosocomial Infections Dietary services and nutrition CMS SOM: Data We Have To Collect C-280 C-281 C-285 C-294 C-300 C-320 C-322 Policies and Procedures review Outpatient services - lab, imaging, rehab, infusions Contracted services Nursing services Medical records Surgical services Anesthesia services 6
CMS SOM: Data We Have To Collect C-330 C-336 C-344 C-350 Annual CAH Program evaluation QA/PI Program Organ Donation Swing Beds Data We Should Collect Strategic and Operational Work Plans o o o Customer needs and expectations Quality of clinical care, including national performance measures data Hospital Operations 7
Data We Want to Collect High risk patient care systems, processes ED, OB, surgery, anesthesia, non-op invasive procedures, meds High volume processes Registration, admission, patient ID, med use, billing Problem prone processes Current patient info, transfers Drill down data, active improvement Reduce preventable events, injuries 2. Identify the Vital Few Sample criteria for identifying the vital few Specifically required by a regulator Specifically identified in the strategic plan High risk patient care systems, processes High volume patient care systems, processes Problem-prone patient care systems, processes Current focus for active improvement 8
2. Identify the Vital Few, Practice MR ADEs Noso Infect Customer Sat Service Volume CMS Strateg Plan High Risk High Vol Prob Prone Total X X X X X 5 X X X 3 X X 2 X X 2 X 1 3. Define Performance Measures Why? So everyone collects the data the same way Numerator Denominator 9
4. Clarify Data Collection Cycles How stable, or volatile, is the process? How accessible is the data? Are there costs other than staff time /materials involved in collecting the data? - customer satisfaction surveys - employee satisfaction surveys Common Data Collection & Reporting Cycles Daily, case by case Weekly Monthly Quarterly Semi-annually Annually active improvement, low volumehigh risk active improvement high risk, active, strategic moderate risk, strategic low risk, stable low risk, stable 10
Examples: Data Collection & Reporting Cycles Measures: MR ED Provider arrives in 30 min Verbal orders authenticated Why Collecting Report Cycle Collect Cycle active imp weekly daily survey def per POC Increases w/ compliance MR Delinquency stable, CEO semi-ann semi-ann emr: post install strategic quarterly quarterly 5. Clarify Data Collection Responsibilities Who has easy access to the data? Administration, managers, staff, others Your role in the facility PI, risk management, infection control, medical records, HIPAA, other duties. Who is attending the end-users meeting? Board, med staff, dept managers, dept staff meetings, other committees, community health planning meetings, etc 11
Clarify Responsibilities Measures: MR Who Collects End User Who Reports Provider arrival ED staff QMT Med Staff DON Verbal orders Nursing QMT, Nursing, MS DON Delinquency rate Med Records Med Staff Board PI Coord emr post install IT, CFO, CEO IT, execs, Board, MS IT, CFO, CEO Simple Data Collection Tools Log sheets Table (matrix) Dot Plots Surveys fast and easy easy, great for QA, more efficient than several log sheets if collecting data on related measures from same source great for collecting same data over a long period of time satisfaction, needs, opinions www.surveymonkey.com 12
Data Aggregation and Assessment Why do it? The quality of decision-making improves when it is based on objective data. Turn collected data into useful information Improve buy-in, build consensus for focused improvement Evaluate progress toward improvement and organization goals Improve the effectiveness and value of the overall quality management program How to Aggregate Data A group or mass of distinct things gathered into or considered as a total or a whole. (New World Dictionary) Group like kinds of data together into a data set Qualitative (male, female, young, old, etc) Ordinal (first, second, third, etc) Metric (1,2,3,4,5. measurements, continuous scales) Frequency- counts Tools for grouping tables (matrix); graphs; charts 13
Use a Matrix to Aggregate Data Data set name Column headings Time or point when data was collected; measurement interval Row labels Data class Totals Admit Mon Tue Wed Total Adult Teens Peds Total Data Aggregation What do we know so far about the value or importance of the data collected? Can we determine if it is significant? If our data is only a sample, what can we accurately say or infer about the population our sample represents? How do we know that what we are saying is really valid? 14
How to Assess Data To estimate or determine the significance, importance or value of; to evaluate. (New World Dictionary) 1. Tools for Assessing Data Calculate descriptive measures (rate, average, percent, mean, median) Add control limits, means, benchmarks to graphs and/or charts 2. Draw valid conclusions about the data set collected Evaluate variation- is it common cause or special cause? 3. Draw valid conclusions about the population a sample represents when you are able to do so Charts & Graphs: Add Mean & Control Limits Glucose Control Values mg/dl 94 93 92 91 90 89 88 87 86 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Day Glucose Control Values 94 92 90 88 86 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Day Upper Control Limit, + 3SD Mean, average Lower Control Limit, - 3SD 15
Charts & Graphs: Add Threshold for Intervention Mortality Rate % Inpatients 12 Month Rolling Av in % 2.5 2.0 1.5 1.0 0.5 0.0 Q4 '04 Q1 '05 Q2 '05 Q3 '05 Q4 '05 Q1 '06 Q2 '06 Q3 '06 Q4 '06 Threshold: a predetermined point at which action will be taken Source: internal discussions Charts & Graphs: Add Benchmarks Heart Failure Clinical Care Guidelines 120 LVS Assess ACEI/ARB Discharge Ins Smoke Cess Success Rate in % 100 80 60 40 20 Benchmark: a pre-determined level of desired performance Source: internal or external 0 Q4 '06 16
Charts & Graphs: Look for Trends, Relationships Percent All Falls By Day of Week Percent 25 20 15 10 5 0 Mon Tue Wed Thur Fri Sat Sun More falls: why? Fewer falls: why? Charts & Graphs: Look for Trends, Relationships 8% 3% 2% CAH Admission Sources, 2004 12% 16% 59% Emergency room Physician clinic Scheduled surgery OB SNF Unexpected post op 17
Assessment: evaluate the variation present 1. Identify what normal looks like 2. Identify what is not normal: outliers, unusual, unexpected process/system events 3. Evaluate the relative severity or importance of multiple factors when more than one is present 4. Identify trends: better? Or worse? Data Assessment: Normal Distribution Normal Distribution 13 12 11 10 9 8 7 6 5 4 1 2 3 4 5 6 7 8 9 10 11 12 13 Series1 +3 SD - 3 SD mean - even and varied distribution of points on both sides of the mean, all within control limits - common cause variation - the process is said to be in control and/or stable. 18
Evaluate Variation: Westgard Rules for Control Charts 1 Point Outside Control Limits 2-2SD Rule 4SD Rule 1-3SD Warning 6 point trend 7 + point trend Sawtooth Westgard Rules: 1 Point Outside Control Limits 1 Point Outside Control Limits 14 13 12 11 10 9 8 7 6 5 4 1 2 3 4 5 6 7 8 9 10 Series1 +3 SD - 3 SD mean 1 point exceeding the upper or lower control limit is special cause variation Source: http://www.westgard.com/mltirule.htm 19
Westgard Rules: Two 2SD Rule (2:2SD) 2:2 SD Rule 13 12 11 10 9 8 7 6 5 4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Series1 UCL LCL mean 2 consecutive points greater than or less than 2 SD; special cause variation Source: http://www.westgard.com/mltirule.htm Westgard Rules: 3 SD Warning 1: 3SD Warning, Cross Center Line 13 12 11 10 9 8 7 6 5 4 1 2 3 4 5 6 7 8 9 10 11 Series1 +3 SD - 3 SD mean Change of 3SD crosses the center line; special cause variation may be present; investigate Source: http://www.westgard.com/mltirule.htm 20
Westgard Rules: 1:4SD Rule 1:4SD Rule 13 12 11 10 9 8 7 6 5 4 1 2 3 4 5 6 7 8 9 10 11 12 Series1 +3 SD - 3 SD mean Change between 2 points of 4SD up or down is special cause variation Source: http://www.westgard.com/mltirule.htm Westgard Rules: 6 Points on One Side of Mean 6 Points on One Side of Mean 13 12 11 10 9 8 7 6 5 4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Series1 UCL LCL mean 6 consecutive points on one side of the mean is special cause variation Source: http://www.westgard.com/mltirule.htm 21
Westgard Rules: 7 Ascending, Descending Points 7 Point Trend, Ascending or Descending 13 12 11 10 9 8 7 6 5 4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Series1 UCL LCL mean 7 consecutive ascending or descending points is special cause variation Source: http://www.westgard.com/mltirule.htm Westgard Rules: Sawtooth Sawtooth 13 12 11 10 9 8 7 6 5 4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Series1 +3 SD - 3 SD mean A sawtooth pattern is not normal; it is special cause variation Source: http://www.westgard.com/mltirule.htm 22
Step Three: Resolve Data Quality Issues Before Reporting: Resolve Data Quality Issues Is the data valid? How do you know? Data collection: small numbers; random samples; populations Are your conclusions valid? Is the data accurate Is the data reliable 23
Resolve Data Quality Issues: Accuracy Data Accuracy Precision: how close is the measured value to the true value? Confidence intervals: how confident can you be that a measured value really is the true value? Resolve Data Quality Issues: Reliability Reliability do repeated measurements produce the same results? How do you know? population sample size confidence intervals 24
Resolve Data Quality Issues: Samples 30 data points approximates the standard normal curve For a large population- sample = 10% >/= 300 in the whole population: 10% of 300 = 30 For a small population- sample = 100% But no less than 10 data points unless that is the full 100% We are not conducting scientific research! Data needs to be accurate and reliable, so it is actionable, but does not need to achieve statistical significance! 25