QMP-LS: A Canadian Regional EQA Program How Labs Get In and Out of Trouble in Ontario Anne Raby Mayo/NASCOLA Coagulation Testing Quality Conference April 14 th, 2009
2 Disclosure Relevant Financial Relationship(s) None Off Label Usage None QMP-LS have no financial interests or other relationships with the manufacturers of commercial products, suppliers of commercial services, or commercial supporters. Presentations will not include any discussion of the unlabeled use of a product or a product under investigation.
3 Photo of Ontario Quality Management Program Laboratory Services 1510 250 Bloor Street East Toronto, Ontario, Canada, M4W 1E6 raby@qmpls.org
4 Learning Objectives Describe the general aspects of the QMP-LS EQA model Discuss the EQA assessment pathway for sub-standard performance Review why root cause analysis is important Recognize the problem-solving components included in a discordant finding investigation Strategize and formulate an action plan when multiple issues occur
5 Quality Management Program Laboratory Services (QMP LS) Deemed agent of the Ontario Ministry of Health and Long-Term Care Department of the Ontario Medical Association Provides peer review activities performed by volunteer membership made up of MLTs, lab physicians and scientists Scientific advisory committees
6 QMP LS Mission and Function Promote quality improvement of labs and other related services for the public good and benefit of health professionals OLA: Determines lab s conformance to standards and ensures QMS processes are effective, evidenced by self- and peer-assessment EQA: Provides performance improvement through assessment and education. Wet surveys and Patterns-of-Practice assess pre-analytical, analytical and post-analytical practice Institute for Quality Management Healthcare (IQMH): Provides Knowledge Transfer through publications, workshops and symposia
Ontario Laboratory Accreditation (OLA) Requirements (ISO 15189: Medical Labs) ISO 15189 developed to assess the competency of medical labs Standard on which OLA Requirements are based ISO 15189 requires participation in PT/EQA as a condition of satisfactory accreditation In Ontario, participation in QMP-LS EQA is mandated by government
8 Value of QMP-LS Dual program of accreditation and EQA and provides comprehensive review of lab QMS and overall performance Accreditation is mile-wide, inch-deep Ensures quality management processes are in place and ongoing competence EQA is inch-wide and mile-deep Monitors the output of those processes to ensure accurate, reliable and timely test results Identifies problems with process outcome
OLA Requirements Relating to PT/EQA VII.9.1 Lab shall participate in PT/EQA for all tests included within accreditation VII.9.2 Samples shall be treated as much as possible in the same manner as patient samples VII.9.3 There shall be no inter-lab communication on data before results are reported VII.9.4 Lab shall document and record results VII.9.5 Lab shall act upon problems or deficiencies and records of actions shall be retained VII.11 Management shall conduct periodic reviews of QC and PT/EQA data (min. monthly) and participate in implementation and documentation of corrective actions VII.11.2 There shall be documentation of corrective action taken as a result of review
10 International Standards for PT/EQA Providers QMP-LS observes: ISO 43:1: PT by inter-laboratory comparisons: Development & Operation of PT Schemes ISO/IEC CD 17043: Conformity Assessment General Requirements for Proficiency Testing ILAC G-13:2007: Guidelines for the Requirements for the Competence of Providers of Proficiency Testing Schemes All three contain requirements for QMS and technical activities
11 EQA Model Challenge Surveys Distribute stable, homogenous testing material for assessment of lab s analytical process Determine lab performance by means of inter-lab comparisons Compare peer group performance to assigned value (consensus mean, confirmatory testing) Feedback to participants summary of results and educational comments Follow-up with participants, results outside of allowable performance limits resulting in poor performance scores Not required by ISO/ILAC standard but required by Ontario government
Do you or members of your staff perform discordant finding investigations on coagulation PT/EQA outliers? 1. Yes 2. No 3. Not Applicable 93% 6% 2% Yes No Not Applicable
13 EQA Assessment Pathway for Sub-standard Performance What brings QMP-LS to the point of acting on sub-standard performance? Combination of: Unacceptable performance in two or more consecutive EQA surveys and Inability to implement effective corrective action Both indicate problems with lab s QMS
14 Sub-standard Performance All labs must investigate discordant findings and submit investigation Acceptable discordant finding investigation (DFI) response includes detailing complete root cause analysis including contributing factors and corrective actions SC reviews DFI and accepts or rejects If corrective action is ineffective or unsatisfactory, scrutiny is intensified: Lab must provide acceptable plan of corrective action Clinical and Laboratory Standards Institute. Using Proficiency Testing to Improve the Clinical Laboratory; Approved Guideline -Second Edition. CLSI document G27-A2, 2007.
15 Sub-standard Performance If plan indicates: Incorrect identification of contributing causes Inadequate in-depth root cause analysis Inappropriate or ineffective corrective actions Action plan is rejected and On-site Review is offered Team of experts problem-solve the lab s issues by identifying the problems/contributing factors/root causes and then recommending required corrective actions Recommendations linked to OLA Requirements Only successful if lab is prepared to be open and honest regarding their issues!
16 Criteria for On-site Review Combination of poor performance and indication of lack of understanding of problem as revealed by ineffective corrective action or lack of evidence. Accreditation (ISO 15189) vs. On-site Review: Process-based Universal requirements Scheduled assessment Observation, evidence Peer review Looking for what is right Procedure based Discipline focused Problem-driven visit Review, education Expert review Looking for what is wrong
17 OSR Team On-site Review Meet with medical director, owner/operator, administrators, relevant stakeholders and staff Hold discussions, review documents and observe staff performing analysis Either: - Identify issues and recommend corrective action Or - Considered a potential to compromise patient care. Recommend temporary suspension of test s licence OSR team and SC support, coach and monitor staff during this time To satisfy QMP LS recommendations, all issues must be corrected and an acceptable and timely action plan developed for outstanding items of resolution (90 days)
18 Why is Root Cause Analysis Important? What led to the need for On Site Review? Lack of identifying or correcting the root cause Root cause analysis is used where there is a gap between actual and desired performance Prevents the potential of compromised patient care Distinguishes contributing factor from root cause and allows development of appropriate corrective action Identifies not only what and how a problem occurred but why things went wrong Prevents the problem from recurring Prevents inadequate corrective actions from wasting valuable resources
19 Discordant Finding Form
20 5 Essential Elements of DFI Team-based approach Include individuals involved in sample receipt, handling, testing and reporting Identify the problem Review the correspondence and Committee Comments, confirm discordance is authentic Investigate the process only the FACTS Discover how the incident occurred not what didn t happen or what should have happened Systematically review the process (reconstruct what happened, use flowcharts) Ask staff a set of pre-determined questions, don t assume you know Analyze the process Brainstorm by asking WHY? 5 times Stop when the cause is beyond your control
21 Identify, Select and Monitor Once the root causes have been identified, validate: Ask: Would the problems have occurred if the causes had not been present? Ask: Will the problems recur if the causes are eliminated? Evaluate and select the corrective actions Identify staff responsible for implementation Monitor for repeat problems using performance indicators and assess effectiveness: Compare before and after using performance indicators Confirm elimination of root actually taken place Survey client base for improvement of service
In your lab, what is the most common root cause that results in reporting a discordant coagulation PT/EQA outlier? 1. Clerical/Entry 51% 2. Technical 3. Method 4. Instrument/Equipment 5. Organizational Clerical/Entry 21% 18% 10% Technical Method Instrument/Equipment 0% Organizational
23 Contributing Factors or Root Causes? Major Causes Identified in Ontario Technical EQA material improperly prepared or stored Reconstitution error Analysis of wrong sample Calculation or conversion error Failure to act on inappropriate QC results Method Inadequate written procedures QC material not at relevant clinical levels Method lacks sensitivity/specificity These are contributing factors, not root causes Lab needs to identify why contributing factors occurred, need to ask another WHY? Without training and education as part of the corrective action, the same error might recur
24 What are the Real Root Causes? Usually Organizational Issues Lack of human resource Inadequate space/workplace, design or inappropriate equipment Lack of theoretical understanding/technical expertise/necessary training They don t know what they don t know Inadequate educational in-service Lack of good document control process Inadequate/incorrect procedures Inadequate communication/supervision Inadequate coordination of tasks Thought someone else did it Lack of communication with other departments Pharmacy, Outpatient clinics
25 Placing Specialty Tests in Core Labs Lack of management/administrative understanding of medical/technical needs Inadequate medical oversight for complex tests Lack of communication with users Insufficient staff with required technical expertise Inadequate continuing education/competency assessment High throughput/volume/noisy environment not conducive to reflexive testing algorithms No validation of complex methods Lack of understanding of instrument platform capabilities and shortcomings Lack of access and inappropriate review of QC Lack of understanding how calibration curves are achieved
26 Incomplete DFI Discordant Result not reporting required test APTT and heparin assay performed in different areas and sample not forwarded, heparin assay not performed 1st time identified organizational root cause due to lack of communication Next survey, AGAIN did not report heparin assay, what went wrong? 2 nd investigation lab identified contributing factors that led to the root cause: Inadequate coordination of tasks Failure to follow instructions No review of worksheet prior to submission Inadequate SOP Inadequate training and sign-off Now lab knew what needed to be fixed
27 No Problem with Our QC. Basic QC principles not applied No clear defined accountability for QC review No clear QC action limits (CV or % difference) Results reported prior to entering QC or reviewing LJ graphs Not consistently participating in instrument vendor QC program Two months of QC signed off at one time Not following established QC protocols noted in procedure Fundamental lack of understanding of how to handle QC outliers, rely on LIS to alert staff Acceptable QC ranges set too wide therefore generating few outliers
28 Instrument Cannot Replace Expertise Factor VIII of 0.02 U/mL (assigned value: <0.01 U/mL) Contributing and Root Causes SOP not in accordance with standard of practice Relying only on QC to trouble-shoot Insufficient points on standard curve If warranted, not using a low standard curve Averaging values of multiple dilutions Lack of communication with staff and/or vendor Inadequate instrument/reagent orientation, inappropriate set-up, calibration issues Reconstitution error and use of graduated pipettes Inadequate validation - not validated for clinical levels Method lacks sensitivity and specificity Problem with reagent performance
When Multiple Issues Arise What to Do? PLAN ACT DO CHECK Example of an Ontario Lab s Action Plan DO - Review QC and QA activities Review root cause analysis and corrective actions, identify gaps what worked and what didn t CHECK - Ensure medical and technical accountability Ensure processes are in place and all policies and procedures are updated and communicated ACT - Partner with a lab in good EQA standing Conduct and/or facilitate continuing education sessions Perform competency assessments on staff PLAN - Arrange with the vendor for retraining or advance training and troubleshooting
30 Summary EQA provides a unique external perspective not available to individual sites EQA can reflect error in practice that is not apparent in QC program DFI provides tangible basis for Quality Improvement as root cause analysis is effective in prevention of discordant result recurrence QMP-LS adds value to poor performing labs by supporting, coaching and monitoring staff until performance has improved
Acknowledgments EQA Director Jane Gun Munro Hematology Scientific Committee Chair Mike Keeney Membership Len Burger, Mark Crowther, John Lafferty, Karen Moffat, Adam Ninos, Ruth Padmore, Ian Chin-Yee