Outcomes Framework Miller s Framework Description Data Sources and Methods Participation LEVEL 1 Number of learners who participate in the educational activity Attendance records Satisfaction LEVEL 2 Degree to which expectations of participants were met regarding the setting and delivery of the educational activity Questionnaires/surveys completed by attendees after an educational activity Learning: Declarative Knowledge LEVEL 3A Knows The degree to which participants state what the educational activity intended them to know Objective: Pre- and post-tests of knowledge Subjective: Self-report of knowledge gain Learning: Procedural Knowledge LEVEL 3B Knows how The degree to which participants state how to do what the educational activity intended them to know how to do Objective: Pre- and post-tests of knowledge Subjective: Self-reported gain in knowledge (e.g. reflective journal) Competence LEVEL 4 Shows how The degree to which participants show in an educational setting how to do what the educational activity intended them to be able to do Objective: Observation in educational setting (e.g. online peer assessment and EHR chart stimulated recall.) Subjective: Self-report of competence; intention to change Performance LEVEL 5 Patient health LEVEL 6 Community health LEVEL 7 Does The degree to which participants do what the educational activity intended them to be able to do in their practices The degree to which the health status of patients improves due to changes in the practice behavior of participants The degree to which the health status of a community of patients changes due to changes in the practice behavior of participants Objective: Observed performance in clinical setting; patient charts; administrative databases Subjective: Self-report of performance Objective: Health status measures recorded in patient charts or administrative databases Subjective: Patient self-report of health status Objective: Epidemiological data and reports Subjective: Community self-report Moore DE, Green JS, Gallis HA. Achieving desired results and improved outcomes: Integrating planning and assessment throughout learning activities. J Contin Educ Health Prof. 2009;29(1):1-15.
L&D Observational Process Tool Observer ID: OR ID: Date: Procedure: (C-section, low risk; C-section, high risk; premature <36) Process/task Team Roles HIT/Equipment Policies/checklists Setting Environment/Culture Teamwork/Communication Scrub tech, L&D Nurse e.g Transferring pt to elevator Confusion over appropriate sterilization procedures for main OR5 A lot of people in hallway made it difficult to move obese patient L&D nurse aggressively told scrub tech they were doing it wrong 2
VCU-OB SA SCALE Directions: Rate your agreement with each item in relation to the simulation you just participated in. Strongly Disagree Strongly Agree I was confused about where to go when the decision was made to transport the patient. I knew how to get to the OR once I arrived on the 5 th floor. I was confused about my role during the patient transfer. I was confused about my role during the operative delivery. I was confused about my role during the transport process after the operation. The availability of equipment and resources was similar to the L&D OR. 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 3
Using In-Situ Simulation with a Failure Mode Effects Analysis for Quality Improvement Moshe Feldman, PhD, MS Ellen Brock, MD, MPH Cheryl Bodamer PhD, RN, MPH Paul Mazmanian, PhD
Funding and Disclosures Dr. Mazmanian receives funding from the Center for Clinical and Translational Research at VCU. Collaborators: Moshe Feldman, PhD Ellen Brock MD, PhD Cheryl Bodamer PhD, RN, MPH 6
Context and Setting Joint Commission standard LD.5.2 Leaders ensure that an ongoing, proactive program for identifying risks to patient safety and reducing medical/health care errors is defined and implemented. 7
Context and Setting Identify and mitigate patient safety risks associated with new surgical patient workflows due to a renovation of the L&D unit Anticipate risks for new and unpredicted workflows Evaluate the use of in-situ simulation for identifying training needs and mitigating patient safety risks associated with new clinical workflows 8
Failure Mode Effects Analysis (FMEA) Interdisciplinary approach to prospectively evaluate patient safety risks 1. Identify and Map Processes 2. Identify Failure Modes 3. Estimate & Prioritize Risks 9
FMEA: Identify and map process GOAL 1 Develop a triage protocol to minimize risk GOAL 2 Develop standardized work process for patient transfer GOAL 3 Design the system to mitigate patient safety risks GOAL 4 Develop process for transferring back to L&D 1. Room checks at start of shifts 2. Decision to transport 3. Where to operate? 4. Notify team members 5. Prepare OR 6. Transfer Patient to OR 7. Perform operative delivery 8.Infant resuscitation 9. Transfer back to L&D,NICU, 10
FMEA: Identify Failure Modes Methods: Live case observations Workflow setting Communication Health informatics Team culture Structured interviews In-situ simulations Simulations occurring in the actual clinical setting 11
In-Situ Simulation What does it look like? What can go wrong? L&D 6 th floor Standardized Patient Part task trainer NICU Main 6 New Workflows Surgery 5 th floor Scenarios: Gestational diabetic with protracted second stage Postpartum hemorrhage Multiple emergencies 12
FMEA: Risk Assessment Failure Modes (what might happen) Failure to notify main 5 of operative delivery Causes (why it happened) Confusion about who is responsible Effects Severity Likelihood Detectability Delay in operative delivery Risk Priority Score (LXSXD) Data sources Live observations Structured interviews In-situ simulation 13
Risk Assessment: Prioritizing Needs Risk Priority Number (RPN) = Likelihood X Severity X Detectability Risk Priority Likelihood Severity Definition The perceived chance of the failure happening within a defined period. How severe the outcome is to the patient should failure occur. Detectability Is the area of failure readily known, or is it discovered only when a bad outcome occurs? Description of the rating scales Rating of 1-10: from failure is unlikely (1 in >5 years) to very likely or inevitable (1/day). Rating of 1-10: from no severity at all (would not affect individual or system) to moderate (significant effect with no injury) to major injury to death. Rating of 1-10: from almost certain the control will detect potential cause(s) to absolute uncertainty that the control will not detect potential cause(s) and subsequent failure mode(s). Source: Joint Commission Resources 2005, and Failure Modes and Effects Analysis (FMEA). An Advisors Guide, June 2004; Department of Defense Patient Safety Center. 14
Planned Change for Patient Safety Failure Mode Description of Action (Plan) Outcome Measure Person Responsible Failure to notify main 5 of operative delivery Redesign staff role with responsibility of constant coordination with main 5 clerk and main 6 charge nurse Evidence of intervention, observations, staff surveys Practice committee will redesign staff roles Nursing will post staff responsibilities 1. Identify Process 6.Implement (Do) 2. Identify Failure Modes 5. Action Plan (Plan) Rapid cycle improvement using simulation 7. Evaluate (Study) 3. Estimate & Prioritize Risks 4. Identify Intervention (Plan) 8. Revise (Act) 15
Results 25 structured interviews with nursing, residents, attendings, L&D teams, front desk clerks 27 potential failure modes were identified 11 members of the Obstetrical team ranked failure modes on likelihood, severity, and detectability Highest risk priority scores were associated with anesthesia resources, communication between team members, and availability of resources In-situ simulation revealed unique failure modes not detected with observations and interviews alone 16
Evidence of Translation Implementing the Action Plan Failure Mode CHSPS QI Recommendation Owner/ Contact Recommendations Plan of Action Anesthesia unable to staff cases on both floors simultaneously on nights and weekends Inability to efficiently communicate with NICU Increase staffing levels or backup for anesthesia. Achieve more predictable patient throughput through scheduling. Develop and disseminate formal workflow roles and process. (1,Nursing) Test the communication system from the Main 5 OR Mitigating risks B A meeting is scheduled on 10/24 with L&D leadership, physicians, Anesthesia leadership, and CRNAs Perinatal Practice Committee Standard language approved by Perinatal Practice Comm. To be used when calling for NICU resources. Language will include Location, Room #, Gestation, and brief reason for call. Staff will be education on this during Orientation Upgraded Ascom phones We did our first trial run of a scheduled uncomplicated C/S in the Main 5 OR this morning and with the exception of a printer problem and a confusion about a phone # everything went off without a problem. There was even a drop in the fetal heart and the NICU had to respond, but all went well. Not bragging, but great work group!!! Check out the pink footprints that now guide the way in the Main OR. 17
Conclusions In-situ simulation is feasible for identifying educational needs and apparently mitigating risks associated with new clinical workflows In-situ simulation allows for rapid training of interdisciplinary healthcare teams for new clinical workflows January 2013 follow up surveys and focus groups to assess value of the modified FMEA with in situ simulation 18
Implications for Learners and Planners Self assessment and practice based learning Work processes Identify educational needs Links CPD to patient safety and systems of care Tools for assessment, educational planning, and translation to change 19
Thank you for your attention Questions and Discussion Contact us mfeldman@vcu.edu 20