Q-Tracks. A College of American Pathologists Program of Continuous Laboratory Monitoring and Longitudinal Performance Tracking

Transcription

1 Q-Tracks A College of American Pathologists Program of Continuous Laboratory Monitoring and Longitudinal Performance Tracking Richard J. Zarbo, MD, DMD; Bruce A. Jones, MD; Richard C. Friedberg, MD, PhD; Paul N. Valenstein, MD; Stephen W. Renner, MD; Ron B. Schifman, MD; Molly K. Walsh, PhD; Peter J. Howanitz, MD; for the Quality Practices Committee, College of American Pathologists The College of American Pathologists (CAP) Q-Tracks program, initiated in late 1998, was designed to satisfy accreditation requirements for continuous monitoring and benchmarking in clinical and anatomic pathology. Q- Tracks became an ORYX-approved indicator monitoring system for the Joint Commission on Accreditation of Accepted for publication May 2, From the Department of Pathology, Henry Ford Hospital, Detroit, Mich (Dr Zarbo); the Department of Pathology, St John Hospital and Medical Center, Detroit, Mich (Dr Jones); the Department of Pathology, Baystate Medical Center, Springfield, Mass (Dr Friedberg); the Department of Pathology, St Joseph Mercy Health System, Ann Arbor, Mich (Dr Valenstein); the Department of Pathology, Veterans Administration Greater Los Angeles Healthcare System, Los Angeles, Calif (Dr Renner); the Department of Diagnostics, Veterans Administration Medical Center Tucson, Tucson, Ariz (Dr Schifman); the College of American Pa- Context. Continuous monitoring of key laboratory indicators of quality by hundreds of laboratories in a standardized measurement program affords an opportunity to document the influence of longitudinal tracking on performance improvement by participants focused on that outcome. Objective. To describe the results of the first 2 years of participation in a unique continuous performance assessment program for pathology and laboratory medicine. Design. Participants in any of 6 modules in the 1999 and 2000 College of American Pathologists (CAP) Q-Tracks program collected data according to defined methods and sampling intervals on standardized input forms. Data were submitted quarterly to CAP for statistical analysis. Interinstitutional comparison reports returned in 6 weeks provided each laboratory with its performance profile of key indicators and its percentile ranking compared with all participants in that quarter. This also included longitudinal comparisons of performance during previous cumulative quarters. Control charts graphically displayed data with flags identifying performance points that were out of statistical control. Setting. Hospital-based laboratories in the United States (98%), Canada, and Australia. Participants. Voluntary subscriber laboratories in the CAP Q-Tracks performance measurement program: roughly 70% from hospitals of 300 occupied beds or fewer, 65% from private, nonprofit institutions, slightly more than half located in cities, one third from teaching hospitals, and 20% with pathology residency training programs. Main Outcome Measures. Each module measured several major and additional minor quality indicators and unbenchmarked individualized data for internal use. Results. Participants in 4 of 6 Q-Tracks continuous monitors demonstrated statistically significant performance improvement trends in 1999 and 2000, which were most marked for laboratories that continued participation throughout both years. These monitors were wristband patient identification, laboratory specimen acceptability, blood product wastage, and intraoperative frozen section consultation. Conclusions. Key continuous indicators chosen on the basis of a decade s experience in the CAP Q-Probes quality improvement program are useful measurement and benchmarking tools for laboratories to improve performance. In general, measures in which there is a broad range of demonstrable performance initially are most optimal for subsequent improvement using continuous monitoring. These studies have shown that quality is not static, but rather is a moving benchmark of performance as seen in the redefinition of benchmarks over time by participants in the first 2 years of the CAP Q-Tracks program. (Arch Pathol Lab Med. 2002;126: ) Healthcare Organizations in This Q-Tracks approach to longitudinal tracking of key indicators of quality was ]developed from experience gained in the groundbreaking Q-Probes program. The precursor Q-Probes program was founded in 1989 to establish key benchmarks and standardized approaches to measurements of laboratory quality. The Q-Probes program of time-limited monitors has resulted in more than 100 peer-reviewed publications, dethologists, Northfield, Ill (Dr Walsh); and the Department of Pathology, State University of New York Health Science Center, Brooklyn, NY (Dr Howanitz). Corresponding author: Richard J. Zarbo, MD, DMD, Department of Pathology, Henry Ford Hospital, 2799 West Grand Blvd, Detroit, MI ( rzarbo1@hfhs.org). Reprints: See Archives of Pathology & Laboratory Medicine Web page at Arch Pathol Lab Med Vol 126, September 2002 Q-Tracks Zarbo et al

2 Figure 1. Individual Q-Tracks participant performance using an x-chart to show wristband error rate tracked during 1 year. The 2 points designated by stars indicate values exceeding the upper control limit, which are therefore unlikely to be due to background or unexplained variation. Overall average was 7.90; lower control limit, 0.00; and upper control limit, fining preanalytic, analytic, and postanalytic benchmarks for quality improvement in all disciplines of pathology and laboratory medicine. 1 Currently, the time-trended Q- Tracks program offers subscribers 12 continuous monitors for which Q-Probes data had previously defined important indicators and influential variables. These packaged, standardized Q-Tracks monitors allow users to track laboratory performance throughout a 12-month period with each subscription. Quarterly performance reports are issued to assist participants in documenting improvement over time and in gaining knowledge of practices and policies associated with improved performance of other participants. The Q-Tracks program provides professional statistical analysis and interpretation of data with benchmarking and graphical representations of a multi-institutional database up to eight quarters in duration. It also provides individualized profiling of participants to their closest comparator laboratory with matching practice characteristics. In addition, data are analyzed to identify best performers and derive best practices to share successful quality practices. Early in our Q-Probes quality monitoring experience, we recognized that quality is not static and that we were defining moving targets of improvement. In this article, we present data from the 6 initial Q-Tracks longitudinal monitors in anatomic and clinical pathology during the first 2 years of the program. These results underscore the value of continuous and focused monitoring of key indicators of quality as laboratories raise the performance bar over time. MATERIALS AND METHODS These databases were derived from laboratories that voluntarily subscribed to any of 6 modules in the CAP Q-Tracks quality monitoring program in 1999, 2000, or both years. The individual monitors covered the preanalytic testing phase (patient wristband identification, blood culture contamination, clinical laboratory specimen acceptability), the analytic phase (intraoperative frozen section consultation), the postanalytic phase (blood unit wastage in date), and all phases (Papanicolaou smear cytohistologic correlation). The basic method was the same, although specifics of monitoring differed in each of the 6 Q-Tracks studies as elaborated herein. Similar to each study was the collection of data according to defined method and sampling intervals on standardized input forms. Data were submitted quarterly to CAP for statistical analysis. Interinstitutional comparison reports were returned to laboratories in 6 weeks to provide each laboratory with its performance profile of key indicators with percentile ranking compared with all participants in that quarter. These reports also included longitudinal comparisons of performance over previous cumulative quarters. Control charts graphically displayed data with flags identifying performance points that were out of statistical control. Each participant received an annual summary with critique of the year s data file and identification of best practices sought from best performers that had significant associations with better or most improved performance. Statistical Analysis Assessment of an Individual Institution s Performance. Data from individual institutions participating in the Q-Tracks program were abstracted to illuminate changes in performance over time and performance relative to a group of appropriately selected peers. Both of these attributes are important but separate components of a comprehensive quality improvement program. Both were reported quarterly to individual program subscribers that submitted evaluable data. Change in performance was assessed with an x-chart, a statistical quality control chart that records the proportion of defective product or practices. Standard Shewhart control rules were used to assess special cause variation improvement or deterioration of performance that was unlikely to be due to background or unexplained variation. 2,3 Figure 1 illustrates performance of an individual Q-Tracks subscriber over time displayed using an x- chart; the points designated by the stars represent values that are out of control and therefore unlikely to be the result of background or unexplained variation. The performance of an institution relative to peers was ex- Arch Pathol Lab Med Vol 126, September 2002 Q-Tracks Zarbo et al 1037

3 Figure 2. Mean wristband error rate achieved in each quarter by laboratories that participated in single years 1999 or 2000 of the Q-Tracks program compared with those participating for 2 years. pressed as a percentile of the reference group. For example, an individual institution with a blood culture contamination rate of 2.83% would be in the 50th percentile of the 220 institutions that participated in the Q-Tracks program and reported data. For the most part, the reference group used to calculate percentiles consisted of all reporting institutions. However, in some circumstances individual participants were compared with a subset of other program participants. This occurred when an uncontrollable variable was known to be a strong influence on the quality or outcome measure of interest. Identification of Best Performing and Most Improved Institutions. Best Performer. For each study, at year s end, cut points were set to identify the top 25% of participants as best performers, based on the primary performance statistic. Each identified best performer institution was recognized with a certificate of achievement at year s end. Most Improved Performer. Most improved institutions were defined as participants that demonstrated statistically significant improvement in performance during their participation, after excluding data from the first quarter of participation. Improvement over time was assessed by linear regression; participants with a regression slope greater than 0 at the P.05 significance level were considered to have shown improvement. The first quarter data were excluded from this analysis because of concerns that subscribers may not have correctly understood data collection instructions during their first quarter of program participation. Best Practices. A survey tool was sent to each best performer and most improved performer laboratory with a list of questions regarding laboratory processes related to the study s focus. These participants also summarized various aspects of their practices that they believed contributed to their superior or improved performance. The responses were compiled and included in the yearend summaries for all participating laboratories to review and include in their discussions on process improvement. Impact of Program Participation on Performance. Q-Tracks is a voluntary subscription program. Subscribers typically submit data for 1 year and then make a decision whether to continue in the program and submit data for another year. Because some participants discontinue participation after 1 year whereas others join, comparison of average participant performance from one year to the next does not reveal the impact of program participation on performance. This phenomenon is illustrated clearly in Figure 2, which shows results of the mean institution participating in the wristband error Q-Tracks study. Although improvement over time is evident in 1999 and also in 2000, the overall performance of 1-year program participants during 2000 was no better than the overall performance of 1-year participants in This was likely due to the better performers leaving at the conclusion of 1999, to be replaced by new institutions that started 2000 with lesser performance. To correct for the impact of selective program disenrollment, performance for each institution was tabulated according to the number of quarters of continuous enrollment, regardless of whether participation began in 1999 or Figure 2 illustrates performance of the mean institution participating in the wristband error study as a function of the number of quarters of continuous program enrollment. Improvement over time is visually apparent. Trends were assessed by linear regression; if the median program participant had a performance-over-time regression slope greater than 0 at the P.05 confidence level, overall improvement over time was said to be associated with the duration of program participation. Longitudinal trends in each Q- Tracks monitor were also edited for partial-year participants who exited before the year end to ensure that any decline or trend was due to improvement in performance. RESULTS More than 98% of participants were from the United States, with the remainder from Canada and Australia. Participants in 4 of 6 Q-Tracks continuous monitors demonstrated statistically significant performance improvement trends in 1999 and 2000 that were most marked for laboratories that continued participation throughout both years. These monitors were wristband patient identification, laboratory specimen acceptability, blood product wastage discarded in date, and intraoperative frozen section consultation. In these 4 Q-Tracks studies, the duration of program participation showed a significant positive association with improved performance. There was no association between the length of program participation and performance improvement in the other 2 long-term monitors Papanicolaou smear cytohistologic correlation or the blood culture contamination rate. The specific design and results of each of the 6 Q-Tracks studies follow. Patient Identification Accuracy To calculate the primary indicator of a wristband error rate, participants in this hospital-based Q-Tracks study collected wristband data for 18 specific days per quarter (1 2 days per week) specified by a calendar schedule. Phlebotomists performed wristband data collections at the time of patient phlebotomy. Participants returned collected wristband data to CAP for analysis at the end of each month and data then were summarized per quarter. A secondary source of data collected was the type of wristband error. In addition, data to effect process improvement were collected by participants for local use to identify the specific hospital patient location for each type of wristband error encountered. In 1999, there were 143 participants; in 2000, there were 151, with 77 participating in the program in both years. Participants examined wristbands in 1999 and wristbands in 2000 and found and wristband errors in these years, respectively. Seen in Figure 2 are mean quarterly wristband error data for those participating in 1999, 2000, and both years of the program. Participants in 1999 and 2000 programs had 4 consecutive quarters in which their mean wristband error rate decreased, whereas those who participated in the monitor for both years had mean wristband error rates that decreased for the first 7 quarters. The performance improve Arch Pathol Lab Med Vol 126, September 2002 Q-Tracks Zarbo et al

4 Figure 3. Mean blood culture contamination rate obtained in each quarter by laboratories that participated in single years 1999 or 2000 of the Q-Tracks program compared with those that participated in both years. Figure 4. Mean chemistry and hematology specimen rejection rate achieved in each quarter by laboratories that participated in single years 1999 or 2000 of the Q-Tracks program compared with those participating for 2 years. ment suggestion derived from best and most improved performers was implementation of an institutional policy for phlebotomists to refuse to perform phlebotomy on a patient without a correct wristband. Blood Culture Contamination The blood culture contamination Q-Tracks study was designed to measure blood culture contamination rates using a uniform definition that facilitated interinstitutional comparisons. The definition used in this study (isolation of coagulase-negative staphylococcus species, diphtheroid organism, or Bacillus species) was appropriate for calculating institutional blood culture contamination rates, but was not appropriate for clinical decision making in individual cases, since rare isolates classified as contaminants in using the study definition may be associated with clinical infection. In 1999, there were 155 participants; in 2000, there were 196 participants, with 105 participating in both years. Participant s blood culture contamination rates varied widely, with a median of 2.83% (range, 1.34% at the 10th percentile to 4.43% at the 90th percentile). Those with high contamination rates in one quarter tended to report high contamination rates in other quarters, underscoring that the differences between institutions were real and not the result of random variation. Superior performance was associated with use of tincture of iodine as a disinfectant rather than iodophor-betadine-povidone iodine, a finding that has been reported previously, 4,5 and monitoring of blood culture contamination internally. There was no association between blood culture contamination rate and the duration of program participation, a finding that contrasted with the results of the other Q- Tracks studies (Figure 3). Best performing institutions (43 of 191) consistently achieved quarterly contamination rates that were less than 2.5%. Twenty-seven institutions identified as best performers that agreed to be surveyed reported having a training program for phlebotomists. Most (78%) of the institutions selected as best performers reported having a blood culture monitoring program. Several institutions indicated that they provided regular performance feedback and retraining to care units and phlebotomists with high contamination rates. Laboratory Specimen Acceptability The objective of this monitor was to identify and characterize unacceptable blood specimens submitted for testing to the chemistry and hematology sections of the clinical laboratory. Adequacy of a specimen is an important preanalytic factor that affects the accuracy and usefulness of laboratory results. For this reason, laboratories have guidelines for evaluating specimens submitted for testing. Specimens not meeting these criteria of acceptability are rejected and another specimen is requested. The Q-Tracks monitor of laboratory specimen acceptability was based on previous Q-Probes studies conducted in 1992 and ,7 The main outcome measure of this monitor was the overall specimen rejection rate. A secondary indicator detailed specific reasons for rejection provided in a summary of aggregate percentages. These included specimen lost or not received, unlabeled specimen, mislabeled specimen, incompletely labeled specimen or inadequately filled out requisition, specimen hemolyzed, specimen clotted, insufficient specimen quantity, unacceptable variance (delta check), or other reason. In 1999, there were 172 laboratories submitting data; in 2000, there were 198 participants, with 82 laboratories participating in both years. In 1999, the aggregate rejection rate was 0.83% ( specimens of total test requests were rejected for testing). This rate was similar to a lower aggregate rejection rate of 0.62% by participants in the 2000 module, with rejected tests of evaluated. There was a significant downward trend (P.05) in the specimen rejection rate for 1999 and 2000 participants, with the most significant decrease in specimen rejection rates observed for those participants who participated in the monitor for 2 years (Figure 4). Best performers were identified in 1999 on the basis of a cumulative specimen rejection rate of less than 0.30% and achieved quarterly Arch Pathol Lab Med Vol 126, September 2002 Q-Tracks Zarbo et al 1039

5 Figure 5. Mean overall blood product wastage rate shown for each quarter by laboratories that participated in single years 1999 or 2000 of the Q-Tracks program compared with those participating for 2 years. Figure 6. Mean positive predictive value of a positive cytology result achieved in each quarter by laboratories that participated in single years 1999 or 2000 of the Q-Tracks program compared with those participating for 2 years. rates of less than 0.37% for all quarters in which data were submitted. In 2000, best performers achieved quarterly rejection rates of less than 0.28% for all quarters in which data were submitted. Higher rejection rates were associated significantly with use of microcollection containers and specimen collection by nonlaboratory personnel. Blood Product Wastage The Q-Tracks monitor of blood product wastage was designed to compare the rates of blood products wasted (defined as discarded in date), track rates of improvement over time, and identify characteristics of best performers. The main outcome measure was the overall blood wastage rate. Secondary indicators included wastage rates of whole blood (allogeneic), red blood cell (allogeneic), fresh frozen plasma, platelet concentrate, single-donor platelet, cryoprecipitate, and breakdown of circumstances of wastage. Facilities recorded the total number of units transfused by component type (eg, packed red blood cells, fresh frozen plasma, cryoprecipitate). Participants were also questioned regarding their institutional practices and policies related to blood wastage. For each component wasted in date, the circumstances of wastage were also tallied for the following specific categories: miscommunication or failure to inform, physical breakage or contamination, patient not ready for transfusion, and improper handling. More than 100 institutions submitted data in each year of this Q-Tracks monitor, 116 in 1999 and 113 in During the 2 years, approximately 2 million units were monitored of which roughly units were wasted in date. Cancelled orders represented by far the most common circumstance associated with wastage, accounting for nearly half of all units lost in each year. Participants who stayed in this blood product wastage Q-Tracks module for 2 years experienced a significant downward trend (P.05) in the overall blood product wastage rate (Figure 5). Best performing institutions typically had active blood wastage monitors in place for 3 or more years and actively tracked the categories of individuals responsible for indate blood wastage. Most also provided periodic feedback concerning wastage, although no specific threshold was set for triggering corrective action. Best performers also listed policies and procedures instituted as a result of monitoring in-date wastage rates and techniques or processes considered likely to lead to low wastage rates. For example, low wastage rates were routinely found in institutions where medical staff was instructed in proper procedure for handling and policy for return of blood products to the laboratory (P.01). Gynecologic Cytology Outcomes The gynecologic cytology outcomes monitor was based on a previous Q-Probes study of 1994 to The correlation of cervicovaginal cytology (Papanicolaou smear) findings with cervical biopsies findings has been a common component of many cytopathology laboratories quality assurance programs. Documented cytologic-histologic review is a requirement of the CAP Laboratory Accreditation Program. 1 Correlation studies have become a mandatory component in all cytopathology laboratories in the United States as a result of the Clinical Laboratory Improvement Amendments of 1988 (CLIA 88). Laboratories participating in the Q-Tracks gynecologic cytology outcomes monitor prospectively identified patients for whom primary diagnostic surgical pathology material from the uterine cervix (ie, cervical biopsy specimens) was submitted to the laboratory. For each of these patients, cytopathology files were reviewed to identify any satisfactory or satisfactory but limited Papanicolaou smear(s), excluding unsatisfactory cases, that were submitted to the laboratory within 3 months before the biopsy or at the time of the biopsy. Participants correlated the biopsy and cytology findings, providing a summary of true-positive, false-positive, and false-negative results. True-negative results were not collected because they were not necessary for the statistical analysis. For all falsenegative cervical cytology results, the cytology and biopsy specimens were reexamined microscopically and categorized as cytology screening error, cytology interpretive error, cytology adequacy determination error, or cytology sampling error, according to definitions provided. The 1040 Arch Pathol Lab Med Vol 126, September 2002 Q-Tracks Zarbo et al

6 Performance Comparison Between Q-Probes and Q-Tracks Studies of Papanicolaou Smear Cytohistologic Correlation Variable 1995 Q-Probe 1999 Q-Track 2000 Q-Track No. of participating laboratories No. of correlated cytology and biopsy cases Aggregate positive predictive value 10th 90th percentile positive predictive value Sensitivity 10th 90th percentile sensitivity Screening and interpretive sensitivity 10th 90th percentile screening and interpretive sensitivity Sampling sensitivity 10th 90th percentile sampling sensitivity main outcome measure of this Q-Tracks monitor was the calculated predictive value of a positive cytology result. Secondary indicators included screening sensitivity and interpretive sensitivity, sampling sensitivity, percentage of diagnoses positive for atypical squamous cells of unknown significance, and percentage of diagnoses positive for atypical glandular cells of unknown significance. Best performers were identified on the basis of positive predictive value scores of 90% or more for each quarter for which data were submitted. There were 94 laboratories in the 1999 module and 99 participants in 2000, and of these 52 participated in both years of the program. No improvement trend was identified for those laboratories participating in 1999, 2000, or both years of this Q-Tracks monitor (Figure 6). In the Table, this continuous monitoring experience in Q-Tracks is compared with the Q-Probes time-limited monitor conducted in 1994 to 1995 using the same method. Intraoperative Consultations Assessment of intraoperative (frozen section) consultation diagnoses is the classic hospital-based quality improvement study in surgical pathology. In this Q-Tracks study, data collection was prospective, requiring pathologists to correlate each intraoperative diagnosis rendered with corresponding diagnoses from permanent sections according to defined guidelines and exclusions based on 3 previous time-limited Q-Probes experiences in hundreds of laboratories There were 2 main outcome measures: intraoperative frozen section diagnostic discordance rate and intraoperative frozen section deferred diagnosis rate. Secondary indicators were reasons for diagnostic discordance in each frozen section diagnosis; anatomic site of diagnostic discordance in frozen section and deferred cases; diagnostic mission of intraoperative consultation cases, both with frozen section and deferred; and tabulation of discordant and deferred intraoperative diagnoses by pathologist and consultant pathologist. Best performers were identified based on achievement of a cumulative diagnostic discordance rate of less than 2.0% and quarterly rates of less than 2.0% for all quarters of submitted data. There were 85 institutions submitting data in 1999 and 95 in 2000, with 45 participating in both years. Taken as separate cohorts, participants in both the 1999 and 2000 cycles generally lowered both the aggregate and mean diagnostic discordance rates below the 2% benchmark during their 1-year period of continuous monitoring (Figure 7). Although the cohort of surgical pathology laboratories participating for the first time in the 2000 monitor showed a decline in the mean diagnostic discordance rate for the Figure 7. Mean intraoperative consultation diagnostic discordance rate achieved in each quarter by laboratories that participated in single years 1999 or 2000 of the Q-Tracks program compared with those that participated both years. group that translated into a diagnostic concordance rate for the first quarter of 98.73%, this improved by the fourth quarter to 99.33%. Of more significance is the fact that the 45 laboratories with 2 full years of monitoring through this program made even more clinical improvement over this time (Figure 7). This trend to lower mean diagnostic discordance rates was significantly associated (P.05) with the amount of time invested in continuous monitoring through this program. The greatest improvement was seen after 1 year of participation. No characteristics of practices were found to have a statistically significant association with discordant diagnosis or deferred diagnosis rates. Best performers had been actively monitoring frozen section correlation before participation in Q-Tracks for more than 3 years, had established thresholds for corrective action for discordant diagnosis rates and appropriateness of deferred diagnoses, and had a specific pathologist or committee responsible for the review. In addition, best performing institutions routinely cut 2 levels on all frozen section blocks, had a histotechnologist cut those blocks, emphasized good preoperative and intraoperative communication with surgeons, and mandated intradepartmental consultations on all malignant diagnoses made on frozen section slides. COMMENT Significant performance improvement over time by participants in 4 of these 6 Q-Tracks continuous monitors un- Arch Pathol Lab Med Vol 126, September 2002 Q-Tracks Zarbo et al 1041

7 derscores the Q-Tracks philosophy that quality is not static, but can be viewed as a moving target of improvement. These 4 key indicators reflect aspects of laboratory service in which the laboratory has major influence over structure, process, and outcome. The exceptions were the Papanicolaou smear correlation and the blood culture contamination monitors, which, in our opinion, include broader health system protocols and effects of clinical screening and/or sampling techniques rather than purely laboratory performance. Since its inception, the CAP Q-Probes program had studied wristband errors in hospitals in 1991 and 1995 and in small hospitals of fewer than 200 beds When results of the latter 2 studies were compared with the initial 1991 Q-Probes study, mean wristband errors actually increased markedly during the baseline study. In contrast, when wristband errors were compared over time for 3 different cohorts of participants in the year-long Q-Tracks continuous monitoring program, mean wristband errors consistently decreased by more than 50% during the year. When one of these cohorts continued to collect wristband errors for a second year, consistent mean improvement continued for the first three quarters of another year, demonstrating that even into a second year further improvement continues. We believe that one of the study characteristics that has allowed improvement to continue during this period is that there are wide differences between the poorest performers entering the program and the best performers at the end of a year of data collection. These differences in current studies span almost 2 logs, extending from an error rate of almost 20% for some poor performers entering the Q-Tracks study to almost 0.2% for those best performers at year end of a study. Studying performance throughout 2 years and comparing best performers to the mean indicate that mean performance still can be improved, since the error rates between these groups was approximately 10-fold. This difference can be used to provide guidance describing how long an indicator of quality should be studied. The data suggest that as performance improves, sustained improvement at the initial rate was no longer possible but improvement was inconsistent. We postulate that changes in personnel or even the temporary absence of a key person in the process has marked effect on increasing error rates. Occasional contamination of blood cultures is probably unavoidable. Nevertheless, blood culture contamination has a significant cost as it is associated with increased length of hospital stay, excess expense, and the administration of unnecessary antibiotics. 15 Despite the lack of improvement demonstrated by participants in the first 2 years of this Q-Tracks monitor, continuous monitoring of the blood culture contamination rate is a common monitor in most laboratories. The clinical practices associated with lower contamination rates revealed in this Q-Tracks experience are consistent with those identified in previous Q-Probes studies 4,16 and should be considered if rates exceed the benchmark of 3% contamination. The practice of providing regular performance feedback and retraining to care units and phlebotomists with high contamination rates has been reported to reduce contamination rates in at least one other investigation. 17 Others reported using dedicated phlebotomy teams to collect blood cultures, in preference to phlebotomists who have other responsibilities. This practice has been associated with reduced contamination rates in published studies. 18,19 The accreditation standards of the CAP Laboratory General checklist now include a specific question about the measurement of preanalytic variables through quality improvement indicators. 20 Laboratories often perform this responsibility by defining specimen acceptability and rejection criteria for clinical users. Communication, education, and enforcement of these preanalytic requirements represent major opportunities for the laboratory to influence many aspects of patient care, including outcomes. We noted that rejection rates for the institutions participating in the laboratory specimen acceptability Q-Tracks module were higher than those in the previous 2 Q-Probes studies. 6,7 One possible explanation is that laboratories that recognized their specimen rejection rates as a problem subscribed to the Q-Tracks program as a structured method to monitor and provide process suggestions for improving. Nevertheless, participants in this Q-Tracks program experienced a significant improvement in this measured quality variable during the monitoring period. Preanalytic variables are critical elements that can have a significant effect on the outcome of clinical laboratory testing. 21 Laboratories appropriately invest a great deal of effort to control the preanalytic process and monitor specimen quality before testing. When a problem is identified, the specimen must be rejected for testing if results will be affected. Therefore, the appropriate training and competence of laboratory personnel are critical in accomplishing this goal and improving performance. 22,23 The wastage, or loss in date, of blood resources represents an opportunity to improve efficiency. Some circumstances that result in blood wastage may reflect systematic problems in patient safety. Efforts to decrease blood product wastage can reveal areas of hospital-wide practice that may benefit from continuing improvement efforts, including communication, timeliness, education, and patient assessment. Instances of blood wastage may indicate a need for investigation of problems in the process of distribution, shipment, handling, storage, ordering, release, administration, and retrieval. Although wastage may not directly harm patients, it may serve as an indictor of an out-ofcontrol environment of care. This blood product wastage Q-Tracks monitor allowed the repeated benchmarking of efforts to improve and documented the change and/or improvement for each hospital during the 2-year time span. There was a significant downward trend in overall blood product wastage for institutions that stayed in the program for 2 years. Compared with 1999, the first-time Q-Tracks participants for 2000 also showed a similar decline in wastage during the year-long course of the program. Those that had participated in 1999 maintained their improvement during The key remaining problem was canceled orders, where blood units were prepared for release but canceled before transfusion. Together with miscommunication, these 2 circumstances accounted for nearly two thirds of all blood products wasted. Clearly, renewed efforts at improved communication are warranted. Analysis of best performing institutions suggests several interventions that may reduce blood product wastage. Responses from best performer institutions indicate that proactive programs that target blood product wastage provide real benefits. Both the laboratory in general and blood banks in particular should recognize that avoidable wastage of a limited resource, such as blood, not only is 1042 Arch Pathol Lab Med Vol 126, September 2002 Q-Tracks Zarbo et al

8 wasteful in terms of money and time, but perhaps more significantly can have serious repercussions for other patients. Along the blood product collection and distribution continuum from donor vein to patient vein, the wastage of blood products is one of the few largely avoidable factors that unnecessarily consume this precious resource. Increasingly, the laboratory must take the lead role in controlling inefficient practices within their scope of influence. These aggregate blood wastage data from this Q-Tracks experience provide a starting point for quality improvement efforts for many participants. The results of the monitor raise a number of key questions that should be addressed in all blood banks that wish to control in-date blood wastage and highlight the need to specifically identify the system processes, providers, and products that contribute to excess wastage. For example, what provider cancels orders most often? Are certain components canceled more often than others? Is there a particular patient care area from which most orders are canceled? Is there a rationale for ordering and then canceling the order, that is, are the components ordered just in case they will be needed? If there are a few responsible providers, do they know about the associated wastage? What systems problems are present that cause the steps in the process to fail? Does avoidable wastage exacerbate or create other problems downstream (eg, shortages that delay treatment or increase patient morbidity)? There was no improvement trend for participants identified during the 2-year period of monitoring in the Q- Tracks module quantifying the correlation between cervicovaginal cytology (Papanicolaou smears) with corresponding cervical biopsy material. We postulate that this lack of improvement is related to the inclusion of broader health system effects of clinical screening with cytology and biopsy sampling in addition to cytology laboratory performance. This activity has been a common component of many cytopathology laboratories quality assurance programs. Documented cytologic-histologic review has also been a requirement of the CAP Laboratory Accreditation Program, 1 and with CLIA 88 legislation, correlative studies have become a mandatory component in all cytopathology laboratories in the United States. 2 This Q-Tracks cytohistologic correlation monitor is a true health system monitor, not just an evaluation of cytology laboratory performance. Therefore, the correlative statistics represent a summary of the performance of the entire process: cytology sampling, processing, screening, interpretation, biopsy sampling, processing, and interpretation. Some of these variables may also be influenced by biologic variation. Poor correlation may represent problems in any, or all, steps in the process. This Q-Tracks monitor was not designed to necessarily identify specific problems for individual participants. Rather, each laboratory should establish a structured method to identify and address specific causes for poor correlation. After examining cytology and biopsy evaluation performance, most laboratories discovered that most noncorrelation is the result of cytology and biopsy sampling problems. For the first 2 years of this study, laboratories probably concentrated on the in-laboratory components of this process that they have the potential to influence directly. However, after optimizing the laboratory evaluation of cervical cytology and biopsy specimens, we recommend enhanced communication with clinicians to provide them with feedback and information that will assist them in improving the cytology and biopsy sampling processes. Until this occurs, laboratories are unlikely to see significant improvement in the statistical performance of this correlation. From another perspective, the diagnostic variability demonstrated in the interpretation of gynecologic cytology and cervical biopsy pathology should be a signal that more introspection and review of these cases are needed in all laboratories to work toward a better understanding and standardization of diagnostic interpretation As previously pointed out by Woodhouse and Wagner, 27 The pathology community must continue to emphasize the value of cytohistologic correlation to those individuals involved in making health care decisions for large groups of patients. The appropriate and ultimately the most efficient and cost-effective care of the patient with an abnormal Pap smear depend on careful, consistent cytohistologic correlation. The CAP Q-Tracks program provides a standardized, structured, and consistent mechanism to collect this data and provide statistical analysis of performance, with interlaboratory performance comparison. Despite the lack of demonstrated performance improvement by participants in the first 2 years of this complex Q- Tracks monitor, we believe that cytology-histology correlation will remain one of several important pieces of a complete quality improvement program in the cytopathology laboratory. 28 Three initial Q-Probes studies 9 11 of frozen section correlations in 1989, 1991, and 1995 set the benchmark of a median error rate of 1.6% to 1.7%, with a wider range of actual performance achieved by participants in these timelimited studies. In the intraoperative consultation Q- Tracks monitor, surgical pathology laboratories that continued participation through a second year raised the performance expectation from time of enrollment by continually lowering their mean error rate past that achieved by the end of their first year s monitoring. For this group of 45 laboratories that participated in both years, this was a statistically significant downward trend (P.05) in diagnostic discordance to an even lower mean diagnostic discordance rate below 1%. Although the second cohort of enrollees in the 2000 program had a higher initial mean discordance rate of 2.4% on initial participation, this progressively decreased to a statistically significant (P.05) value of 1.6% by the end of 1 year of continuous monitoring in this program. Again, this corroborates that benchmark targets move, but that despite the point of initiation, continuous monitoring and resultant corrective actions are effective in quality improvement. Although we believe that these frozen section discordance data represent true improvement made by these participants over time, also pertinent are 2 considerations related to this Q-Tracks intraoperative consultation study that may influence the calculated diagnostic discordance rate. Both are uncontrolled variables in this type of multiinstitutional study and relate to the consistency in which participants handled the data collection and correlations. The first is that each diagnosis rendered, especially if more than one diagnosis was rendered on one frozen section diagnostic event, should have been tabulated in the total number of frozen section diagnoses rendered. The denominator for all calculations was thus the greater of the total number of frozen section diagnoses rendered, not the potentially smaller and more readily calculated number of cases in which a frozen section was performed. In addition, the basis on which intraoperative frozen section and Arch Pathol Lab Med Vol 126, September 2002 Q-Tracks Zarbo et al 1043

9 permanent sections were correlated may also vary between participants from lenient to strict interpretation. Participants were reminded that if broad and/or specific diagnostic terms were used in frozen section reporting, diagnostic discordance and concordance with the permanent diagnosis should have been judged on the basis of that same broad and/or specific diagnosis rendered on frozen section study. By integrating the identification of best performer and most improved performer laboratories with best practices, the Q-Tracks program assisted laboratories in closing the quality loop. Collecting, analyzing, graphically presenting, and interpreting performance data constituted only the first part of the quality improvement process. Process change decisions could then be made and implemented based on scientific benchmark data, with performance improvement demonstrated in subsequent data collection and analysis. As pointed out by Nevalainen et al, 29 quality monitors for laboratory services have considerable room for improvement that is accentuated when translated into six sigma or parts per million variance. For example, when interpreting the performance of the specimen acceptability of Q-Probes and Q-Tracks studies in terms of parts per million, the studies demonstrated variance of many thousand parts per million, with a six sigma goal of 3.4. Deficiencies such as these expressed in parts per million are staggering in this context, but considering the distance from the goal demonstrated by the average laboratory, all laboratories should focus significant effort toward continuous improvement in this preanalytic area. Merely being as good as or even better than the average laboratory should not distract laboratory managers from working to improve processes to achieve much higher levels of performance. Since the initial offering of these 6 Q-Tracks monitors, the program has developed 6 additional long-term tracking indicators that address clinician and customer performance, laboratory and pathologist performance, and patient perception of care, including satisfaction with outpatient specimen collection, stat test turnaround time outliers, morning rounds inpatient test availability, critical value reporting, small surgical specimen diagnosis turnaround time, and inpatient phlebotomy success rate. This CAP program now successfully serves more than 1200 laboratories in 7 countries as a resource for pathologists to positively influence the quality of laboratory and patient care services. References 1. Schifman RB, Howanitz PJ, Zarbo RJ. Q-Probes: a College of American Pathologists benchmarking program for quality management in pathology and laboratory medicine. In: Weinstein RS, ed. Advances in Pathology and Laboratory Medicine. Chicago, Ill: Mosby-Yearbook; 1996: Mohammed MA, Cheng KK, Rouse A, Marshall T. Bristol, Shipman and clinical governance: Shewhart s forgotten lessons. Lancet. 2001;357: Carey RG. Measuring health care quality: how do you know your care has improved? Eval Health Prof. 2000;23: Schifman RB, Strand CL, Meier FA, Howanitz PJ. Blood culture contamination: a College of American Pathologists Q-Probes study involving 640 institutions and specimens from adult patients. Arch Pathol Lab Med. 1998; 122: Little JR, Murray PR, Traynor PS, Spitznagel E. A randomized trial of povidone-iodine compared with iodine tincture for venipuncture site disinfection: effects on rates of blood culture contamination. Am J Med. 1999;107: Jones BA, Meier F, Howanitz PJ. Complete blood count specimen acceptability: a College of American Pathologists study of 703 laboratories. Arch Pathol Lab Med. 1995;119: Jones BA, Calam RR, Howanitz PJ. Chemistry specimen acceptability: a College of American Pathologists Q-Probes study of 453 laboratories. Arch Pathol Lab Med. 1997;121: Jones BA, Novis DA. Cervical biopsy cytology correlation: a College of American Pathologists Q-Probes study of correlations in 348 laboratories. Arch Pathol Lab Med. 1996;120: Zarbo RJ, Hoffman GG, Howanitz PJ. Interinstitutional comparison of frozen section consultation: a College of American Pathologists Q-Probe study of consultations in 297 North American institutions. Arch Pathol Lab Med. 1991; 115: Gephardt GN, Zarbo RJ. Interinstitutional comparison of frozen section consultations: a College of American Pathologists Q-Probes study of cases in 461 institutions. Arch Pathol Lab Med. 1996;120: Novis DA, Gephardt GN, Zarbo RJ. Interinstitutional comparison of frozen section consultation in small hospitals: a College of American Pathologists Q- Probes study of frozen section consultation diagnoses in 233 small hospitals. Arch Pathol Lab Med. 1996;120: Renner SW, Howanitz PJ, Bachner P. Wristband identification error reporting in 712 hospitals: a College of American Pathologists Q-Probes study of quality issues in transfusion practice. Arch Pathol Lab Med. 1993;117: Dale JC, Renner SW. Wristband errors in small hospitals: a College of American Pathologists Q-Probes study of quality issues in patient identification. Arch Pathol Lab Med. 1997;28: Renner SW, Dale JC. Wristband Identification Reporting Errors: Data Analysis and Critique. Q-Probes study 95-07, Series (ed). Northfield, Ill: College of American Pathologists; Bates DW, Goldman L, Lee TH. Contaminant blood cultures and resource utilization: the true consequences of false-positive results. JAMA. 1991;265: Schifman RB, Bachner P, Howanitz PJ. Blood culture quality improvement: a College of American Pathologists Q-Probes study involving 909 institutions and blood culture sets. Arch Pathol Lab Med. 1996;120: Gipp AP, Hill B, Chorel B, Brant R. Reduction in blood culture contamination rate by feedback to phlebotomists. Arch Pathol Lab Med. 1997;121: Weinbaum FI, Lavie S, Danek M, Sixsmith D, Heinrich GF, Mills SS. Doing it right the first time: quality improvement and the contaminant blood culture. J Clin Microbiol. 1997;35: Surdulescu S, Utamsingh D, Shekar R. Phlebotomy teams reduce bloodculture contamination rate and save money. Clin Perform Qual Health Care. 1998;6: College of American Pathologists, Laboratory Accreditation Program. Laboratory General Checklist Questions. GEN.20348, Series (ed). Northfield, Ill: College of American Pathologists; Narayanan S. The preanalytic phase: an important component of laboratory medicine. Am J Clin Pathol. 2000;113: Haun DE, Zeringue A, Leach A, Foley A. Assessing the competence of specimen processing personnel. Lab Med. 2000;31: Howanitz PJ, Valenstein PN, Fine G. Employee competence and performance based assessment: a College of American Pathologists study of laboratory personnel in 522 institutions. Arch Pathol Lab Med. 2000;124: Stoler MH, Schiffman M. Interobserver reproducibility of cervical cytologic and histologic interpretations: realistic estimates from the ASCUS-LSIL Triage Study. JAMA. 2001;285: Selvaggi SM. Implications of low diagnostic reproducibility of cervical cytologic and histologic diagnoses. JAMA. 2001;285: Cocchi V, Carretti D, Fanti S, et al. Intralaboratory quality assurance in cervical/vaginal cytology: evaluation of intercytologist diagnostic reproducibility. Diagn Cytopathol. 1997;16: Woodhouse SL, Wagner E. Managed discontinuity of care: the value and fate of cytohistologic correlation. Diagn Cytopathol. 1997;16: Jones BA, Davey DD. Quality management in gynecologic cytology using interlaboratory comparison. Arch Pathol Lab Med. 2000;124: Nevalainen D, Berte L, Kraft C, Leigh E, Picaso L, Morgan T. Evaluating laboratory performance on quality indicators with the six sigma scale. Arch Pathol Lab Med. 2000;124: Arch Pathol Lab Med Vol 126, September 2002 Q-Tracks Zarbo et al