Measuring for Improvement: From Toyota to Thoracic Surgery James M. Levett, MD, and Raymond G. Carey, PhD Department of Surgery, Lutheran General Hospital, Park Ridge, Illinois Background. Measuring quality has become a high priority in the era of managed care. Nevertheless, it can be counterproductive to use the same methods for measuring improvement in surgical procedures and processes as we use for measurement in basic research. Techniques of statistical process control have been used for many years to measure process improvement in industry and are now being applied to health care. Methods. Examples of using statistical process control charts to monitor coronary artery bypass grafting mortality, intensive care unit admission time, and length of stay are reviewed. Results. The major advantage of using control chart methodology is that it allows one to determine whether the process being evaluated is in fact stable and to detect when significant or special cause variation occurs. Conclusions. Summary statistics currently provided to purchasers of care and regulatory agencies do not ensure that the processes being evaluated are stable. We need to look at data over time with statistically validated methods such as control charts to better monitor our processes of care and thereby provide accurate statistics. (Ann Thorac Surg 1999;68:353 8) 1999 by The Society of Thoracic Surgeons In a recent article in the Joint Commission s Journal on Quality Improvement, submitted by the Institute for Clinical Systems Integration in Minneapolis, it was emphasized that physicians are increasingly realizing not only how critical measurement is to the quality improvement we seek, but also how counterproductive it can be to mix measurement for accountability or research with measurement for improvement [1]. Most surgeons have been trained in basic research and in the use of random clinical trials to analyze data, but even well-designed clinical trials only establish the efficacy of various procedures and will often not tell us how effective a procedure may be in our own practice. Currently we are being asked to provide data to many different users, including employers, managed care plans, health plans, and government agencies. To meet these requests, the entire field of so-called outcomes research is evolving, and as this occurs, it is important for clinicians to realize that meaningful data are not collected by simply taking before and after snapshot measurements at various points in time. For example, knowing that a complication rate is high does not tell us why it is Fig 1. Histogram of average coronary artery bypass grafting (CABG) mortality before and after implementation of new protocol. Was the new protocol effective in reducing deaths? Presented at the Thirty-fourth Annual Meeting of The Society of Thoracic Surgeons, New Orleans, LA, Jan 26 28, 1998. Address reprint requests to Dr Levett, Department of Surgical Specialists, Physicians Clinic of Iowa, PC, 830 Fourth Ave SE, Cedar Rapids, IA 52403. 1999 by The Society of Thoracic Surgeons 0003-4975/99/$20.00 Published by Elsevier Science Inc PII S0003-4975(99)00547-0
354 LEVETT AND CAREY Ann Thorac Surg MEASURING FOR IMPROVEMENT 1999;68:353 8 Fig 2. Control chart of coronary artery bypass grafting (CABG) mortality before and after new protocol. CL center line or mean; LCL lower central limit; UCL upper control limit. high. What we need are useful techniques and tools to help in determining where to begin our efforts to improve. Thus, we cannot conclude anything with certainty by comparing two numbers unless we understand the stability of the processes that produced them. This insight was obtained in industry many years ago by Walter A. Shewhart, and his methodology for measuring processes was later developed by W. Edwards Deming and others [2]. Shewhart s methods have application to surgical processes. For example, the board of directors at a local hospital is evaluating mortality rates for coronary artery bypass grafting. The bar graph in Figure 1 shows a decrease in the mortality from 5% in 1994 to 4% in 1995. When the data are analyzed monthly, however, we see that the mortality has actually increased in 1995 (Fig 2). The learning point here is that although the average mortality decreased from 5% in 1994 to 4% in 1995, it was not attributable to the new protocol, which seemed to have a negative effect. After the new protocol began, the monthly average actually increased from 2% to 6%. Thus, bar charts can be misleading unless the processes that produced the data are stable. This methodology of displaying data on a chart over time illustrates the concept of a time-series design. It is important to note that the technique of plotting data over time is useful in providing a way to display, think about, and analyze data in a process. The understanding derived will allow us to track changes that we make and thereby monitor improvement. The time-series design format is further illustrated in the next three figures. Hospital A and hospital B have both succeeded in decreasing the intensive care unit admission time after open heart procedures from an average of 39 to 26 minutes (Fig 3). This admission time is defined as the time it takes to transition a patient from the operating room until he or she is stabilized in the intensive care unit with line change-over and full monitoring in place. Both hospital A and hospital B have seen improvement in this admission time as noted on the histogram. Looking at the data over time, however, suggests that two different processes are occurring. In hospital A, the change that was implemented in January 1997, resulted in a positive shift in the process as is evident from the control chart in Figure 4. Admission time improved from an average of 39 minutes in 1996 to Fig 3. Histogram of decrease in average intensive care unit (ICU) admission time after the 1997 implementation plan, hospitals A and B. Was the decrease due to the implementation plan?
Ann Thorac Surg LEVETT AND CAREY 1999;68:353 8 MEASURING FOR IMPROVEMENT 355 Fig 4. Control chart of intensive care unit (ICU) admission time, hospital A. 26 minutes in 1997. Improvement was clearly due to the change effort introduced in January 1997 where a shift in the process (special cause) is noted. Hospital B also saw an improvement, but it would appear that the changes implemented in January did not really make any difference since there was a clear trend already in place (Fig 5). The improvement was not due to the change effort in January 1997, but to some other factor that began in 1996 and continued into 1997. Putting data into proper context according to the way in which the data are collected over time helps us understand when variation is significant, particularly since the most common type of error is to see a trend when none is present. To analyze variation, Shewhart [3] developed the control chart and published his methodology in 1931 in a landmark text entitled Economic Control of Quality of Manufactured Product. A simple control chart is illustrated in Figure 6, where the key quality characteristic is represented on the y (vertical) axis, and time is represented on the x (horizontal) axis. The upper and lower control limits are generally set at three sigmas, and the mean (x ) is set in the midpoint of the control limits. Data falling outside of three sigma limits are a signal that the process has significantly changed. Shewhart s tests for a special cause, and subsequent tests, which were later described as the Western Electric Tests, have helped industrial managers to analyze variation in manufacturing processes to distinguish noise from signals [4]. Fig 5. Control chart of intensive care unit (ICU) admission time, hospital B.
356 LEVETT AND CAREY Ann Thorac Surg MEASURING FOR IMPROVEMENT 1999;68:353 8 Fig 6. Basic elements of a control chart. Thus, one example of a significant change in the process would be when a data point falls outside of three sigma limits. These sigma limits are actually different from classic standard deviation in the sense that they measure the variability of a process over time rather than the variability of a static distribution. Another test states that when seven or eight consecutive data points fall on one side of the centerline, a significant shift in the process has occurred. With control charts, manufacturing processes have been greatly improved and, in fact, this methodology has been used to both improve the quality and reduce variation in manufactured products since the 1950s when they were first used in Japan [5]. These techniques are now being applied in health care and, we believe, will help in analyzing how our processes work for us in our own hospitals [6]. To illustrate this point, we will review three examples from Lutheran General Hospital. In the first example, we have analyzed our elective coronary artery bypass grafting mortality over a period of time from January 1995 through August 1997 (Fig 7). The data are plotted on a control chart with the number of successful surgical procedures between mortalities on the y-axis and each point on the x-axis representing a mortality. As one can see, there was a period from March through November 1995 when 171 consecutive cases were done without a mortality, and this certainly signaled a positive change in our process, which we decided to analyze. Since that time, the process has returned to a stable, but less positive, average of 26.5 open heart procedures being done between mortalities. The next figure illustrates a way of representing coronary artery bypass grafting complications on a control chart (Fig 8). Complications that we are interested in following include reoperation for bleeding, perioperative myocardial infarction, sternal infection or dehiscence, stroke, and renal failure. The vertical axis represents the number of procedures done between complications, and the horizontal axis represents time with each point being a patient with one or more complications. The time period ranges from January through August 1997. As one can see, there were three points where around 34 to 40 patients were operated upon between one of the complications. These, however, were within the control limits of the process and were not significantly different from the other points. They are an example of common cause variation. We see now, however, that six consecutive points have fallen below the centerline, which represents the mean of 14.25 successful operations done between complications. These points are an example of special cause variation Fig 7. Control chart of first/ elective coronary artery bypass grafting (CABG) mortality (DRG 106 and 107).
Ann Thorac Surg LEVETT AND CAREY 1999;68:353 8 MEASURING FOR IMPROVEMENT 357 Fig 8. Control chart of first/ elective coronary artery bypass grafting (CABG) complications. (because there are only 12 total points) and suggest a significant shift in the process. This discovery is now being analyzed, as it represents a negative finding that we had not previously identified. Our final illustration is taken from a bowel surgery quality improvement team. We have included this to illustrate the technique of measuring a change in a process. Several years ago, we established a process improvement team at Lutheran General Hospital to study patients undergoing bowel resection. First, we attempted to reduce the use of resources, as measured by postoperative length of stay. The control chart in Figure 9 illustrates that the average length of stay for 18- to 64-year-old patients decreased from 9.0 days during the baseline, prepathway period to 6.8 days after the pathway was put into place. The next question is whether the improvement in length of stay had an impact on clinical outcomes. The control chart in Figure 10 suggests that it has had a negative impact. On this chart, we are tracking a composite of complications including pulmonary embolus, pneumonia, wound infection, return to operating room, and readmission within 30 days after discharge. The number of operations between one of the complications was 8.7 during the prepathway period but decreased to 3.1 during the initial implementation of the pathway. The current pathway is a stable process with 6.4 operations being done between any one of the complications men- Fig 9. Control chart of bowel operation length of stay (LOS) (ages 18 to 64).
358 LEVETT AND CAREY Ann Thorac Surg MEASURING FOR IMPROVEMENT 1999;68:353 8 Fig 10. Control chart of complications from bowel operation. tioned. Thus, although we have reduced our length of stay, we also see complications more frequently. The pathway is currently under evaluation by the process improvement team to ascertain whether the increase is attributable to a flaw in the pathway, or to physicians not following the pathway. In conclusion, plotting data over time and using control chart techniques will tell us whether the variation in a surgical process is stable and predictable or whether variation signals a significant change in the process. If we really want to measure the effectiveness of our procedures with our own patients, random clinical trials and outcomes measurements alone will not accomplish this task. The summary statistics we currently provide to the purchasers of care and to regulatory agencies do not ensure that our processes are stable. In fact, such data may be misleading. We need to look at our data over time with statistically validated methods such as the control chart methodology developed by Shewhart for industry and which is now beginning to be used in health care. References 1. Solberg LI, Mosser G, McDonald S. The three faces of performance measurement: improvement, accountability, and research. Joint Comm J Qual Improv 1997;23:135 47. 2. Walton M. The Deming management method. New York: Perigee Books, 1986:3 21. 3. Shewhart WA. Economic control of quality of manufactured product. New York: Van Nostrand, 1931:351 404. 4. Carey RG, Lloyd RC. Measuring quality improvement in health care: a guide to statistical process control applications. New York: Quality Resources, 1995. 5. Wheeler DJ, Chambers DS. Understanding statistical process control. Knoxville, TN: SPC Press, 1992. 6. Shahian DM, Williamson WA, Svensson LG. Applications of statistical quality control to cardiac surgery. Ann Thorac Surg 1996;62:1351 9.