11/26/2014. Measurement for Improvement. Pre-work. Introduced three faces of performance measurement: Research Accountability/Judgment Improvement

Transcription

1 Measurement for Improvement Pre-work P2 Introduced three faces of performance measurement: Research Accountability/Judgment Improvement 1

2 Measurement for Improvement There must be a method for improvement associated with the measure(s) to be considered measurement for improvement. You can use the Model for Improvement, A3 Problem-solving, Six Sigma's Define-Measure-Analyze- Improve-Control (DMAIC) cycle or some other approach that is built on the basic scientific method. In terms of the Model for Improvement, improvement requires an aim that defines success, measures that track progress toward the aim, and changes to appropriate systems. Measures help clarify the aim (the metrics for "how much, by when") but are subordinate to the aim and distinct from the changes. What's the logic of Measurement for Improvement? P4 (1) "If we measure, we have to improve." or (2) "If we improve, we have to measure."? 2

3 Three ways to get better numbers P5 1. Improve the system. 2. Distort the system. 3. Distort the numbers. Brian Joiner, Ph.D., statistician, founder of Joiner Associates, my boss, teacher and friend. He sketched his "better numbers" analysis 30 years ago. Campbell's Insight on Distortion The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor. P6 "Campbell's Law" from Assessing the Impact of Planned Social Change, 1976 Donald T. Campbell, Ph.D., social psychologist ( ) s/08%20assessing%20the%20impact%20of%20planned%20social%20ch ange.pdf 3

4 Distorting the system & the numbers P7 Measurement Use: Judge schools and students as successful based on the scores from a competency test. "Unless fifty-eight per cent of students passed the math portion of the test and sixty-seven per cent passed in language arts, the state could shut down the school." /07/21/wrong-answer P8 CMS Five-Star Nursing Home Ratings: Measurement Challenges 4

5 P9 Self-reported by Nursing Homes before accessed 19 Nov 2014 P10 "Receiving a high star rating has never been more important to nursing homes. When nurses and doctors discharge patients from hospitals, they often use the ratings in referral decisions, and insurers consider them when setting up preferred networks. The ratings are also often a first stop for investors and lenders, who consult them to decide whether a nursing home company is a safe bet." 5

6 Visual Story: Text, numbers and repeated stacked bar graphs, over time P11 More Background-1 Up through 2014, staff ratings are derived from an annual self-report, completed at same time as the annual inspection. P12 Raising staff levels for the month when the annual inspection occurs can yield a higher score on the staff component and so boost the star rating. CMS announced in October 2014 that starting in 2015 they will change the staffing component to a quarterly report, linked to electronic payroll data, potentially auditable. (based on New York Times articles) 6

7 More Background-2 P13 "Advocates for patients and for large employers that pay for healthcare services argue that the government should not slow the rollout. They acknowledge that performance ratings are an imperfect science, but counter that more transparency and better tools are needed to help consumers make informed choices." "Dr. Patrick Conway, deputy administrator for innovation and quality for the CMS, said in a June blog post that ratings empower consumers and encourage providers to strive for higher levels of quality." S/ accessed 19 Nov 2014 More Background-3 P14 Riskadjusted Certification/CertificationandComplianc/Downloads/usersguide.pdf p. 10 accessed 19 Nov

8 Exercise: CMS Five-Star Ratings P15 CMS describes the Five-Star ratings this way: The primary goal of this rating system is to provide residents and their families with an easy way to understand assessment of nursing home quality, making meaningful distinctions between high and low performing nursing homes. (Design for Nursing Home Compare Five-Star Quality Rating System: Technical Users Guide,July 2012, p. 1) 1. Our view: the CMS Five-Star rating is a measurement system primarily for judgment/accountability. Do you agree or disagree? Why? 2. Some people and groups argue that transparent reporting of measures can drive better performance, reflecting improvements in nursing homes. What features should the Five-Star rating system have so that people will improve nursing homes rather than distort nursing homes or distort the reported numbers? P16 Question about Three Faces of Measurement and Your Work 8

9 Your work with measures Click using web access (share) P17 9