The Advisory Board and Planning Committee for

Transcription

1 CONSENSUS CONFERENCE Simulation-based Education to Ensure Provider Competency Within the Health Care System Sharon Griswold, MD, MPH, Alise Fralliccardi, MD, John Boulet, PhD, Tiffany Moadel, MD, Douglas Franzen, MD, Marc Auerbach, MD, Danielle Hart, MD, Varsha Goswami, MD, Joshua Hui, MD, and James A. Gordon, MD, MPA ABSTRACT The acquisition and maintenance of individual competency is a critical component of effective emergency care systems. This article summarizes consensus working group deliberations and recommendations focusing on the topic Simulation-based education to ensure provider competency within the healthcare system. The authors presented this work for discussion and feedback at the 2017 Academic Emergency Medicine Consensus Conference on Catalyzing System Change Through Healthcare Simulation: Systems, Competency, and Outcomes, held on May 16, 2017, in Orlando, Florida. Although simulation-based training is a quality and safety imperative in other high-reliability professions such as aviation, nuclear power, and the military, health care professions still lag behind in applying simulation more broadly. This is likely a result of a number of factors, including cost, assessment challenges, and resistance to change. This consensus subgroup focused on identifying current gaps in knowledge and process related to the use of simulation for developing, enhancing, and maintaining individual provider competency. The resulting product is a research agenda informed by expert consensus and literature review. CONSENSUS PROCESS The Advisory Board and Planning Committee for the 2017 Academic Emergency Medicine sponsored Consensus Conference (CC) on Catalyzing System Change Through Healthcare Simulation: Systems, Competency, and Outcomes identified the acquisition and maintenance of individual competency as a critical component of effective emergency care systems. They recommended the formation of a consensus group that would focus on the use of simulation to ensure individual competency across systems and levels of practitioners. This group conducted an extensive review of the literature across multiple disciplines and specialties, identifying knowledge and methodologic gaps that directly impact simulation-based medical education (SBME) applications in emergency medicine (EM). The consensus process led to identification of 1) proposition statements supported by current literature and 2) high-value research targets that would markedly advance the development, achievement, and maintenance of individual competency. That work was presented to a larger consensus conference breakout group focusing on simulation-based education to ensure provider competency within the healthcare From the Department of Emergency Medicine, Drexel University College of Medicine (SG, VG), Philadelphia, PA; the Department of Emergency Medicine, University of Connecticut School of Medicine (AF), Hartford, CT; the Foundation for Advancement of International Medical Education and Research (JB), Philadelphia, PA; the Department of Emergency Medicine, Hofstra Northwell School of Medicine (TM), Hempstead, NY; the Department of Emergency Medicine, University of Washington School of Medicine (DF), Seattle, WA; the Department of Emergency Medicine and Pediatrics, Yale University School of Medicine (MA), New Haven, CT; the Department of Emergency Medicine, Hennepin County Medical Center (DH), St. Paul, MN; the Department of Emergency Medicine, Kaiser Permanente Los Angeles Medical Center (JSH), Los Angeles, CA; and the MGH Learning Laboratory and Division of Medical Simulation, Department of Emergency Medicine, Massachusetts General Hospital, and the Gilbert Program in Medical Simulation, Harvard Medical School (JAG), Boston, MA. Received July 29, 2017; revision received September 5, 2017; accepted September 6, The authors have no relevant financial information or potential conflicts to disclose. Supervising Editor: Rosemarie Fernandez, MD. Address for correspondence and reprints: Sharon Griswold; sg495@drexel.edu ACADEMIC EMERGENCY MEDICINE 2018;25: ISSN by the Society for Academic Emergency Medicine doi: /acem.13322

2 ACADEMIC EMERGENCY MEDICINE February 2018, Vol. 25, No system. During this breakout session, research targets were presented along with supporting data from literature reviews. Attendees provided input, and any missing topics were solicited. Participants and attendees contributed to a final recommended research agenda that represents areas of highest priority for EM. In this article, we present the final product of the consensus process and summarize future high-priority research areas. INTRODUCTION Medical training and continuous professional development curricula are designed to ensure that clinicians are competent in all essential domains 1. Yet, the definition and measurement of proficiency as part of competency-based medical education (CBME) has sparked substantial conversation in the health care education community. Within the consensus group itself, two distinct yet complementary approaches to ensuring competency emerged: 1) the systemwide adoption of simulation-based training methods to improve patient care and/or outcome(s) and 2) the systemwide adoption of simulation-based performance measures designed to assess and benchmark provider performance. In this summary of deliberations, we explore both concepts, the former in the context of translational outcomes research and the latter in the context of the science of performance assessment. Competency-based medical education as described by Holmboe et al. 2 requires careful attention to assessment processes that are relatively continuous and frequent, criterion-based, and developmental; that use assessment methods and tools that meet minimum requirements for quality; that employ both quantitative and qualitative measures and methods; and involve collective judgments about developmental progress. The use of simulation to support CBME allows health care providers the opportunity to practice a full range of clinical skills without any risk to patients, in ways that can be customized for individualized learning and developmental progression and in settings that can be standardized and studied over time. Simulation-based educational platforms to train and assess clinical providers are particularly relevant in dynamic systems such as emergency departments (EDs), where tasks are often infrequent and complex and the consequences of sub-standard performance can be lifethreatening. 3 Yet, as noted by Ziv et al., 4 health care has lagged behind other high-reliability professions, such as aviation, nuclear power, and the military, in the use of simulation to maximize training, enhance safety, and minimize risk. Although the use of simulation-based applications for training and assessment has grown, its full application may not yet be attained for a number of reasons, including cost, resistance to change, and measurement challenges. The purpose of the consensus group was to identify critical gaps in knowledge and process that limit our ability to fully leverage SBME as part of the national shift toward CBME. Proposition 1: Simulation-based medical research and CBME should be informed by translational science research framework (Table 1) Several researchers and organizations have proposed and adopted a translational research model of SBME Briefly, simulation-based translational research considers research outcomes across three levels: performance (or learning) in the simulated environment (T 1 ); patient care processes in the clinical environment (T 2 ); and patient, system, or populationbased outcomes (T 3 ). Demonstration of impact across all levels requires the development of a rigorous research platform that both defines a criterion standard or reasonable benchmark of quality performance and also categorizes meaningful outcome measures to assess the impact of that performance 11 (Table 1). Table 1 Classification of TSR Simulation-based Education Outcomes Simulation Education Impact on T 1, T 2, and T 3 Simulation-based education T 1 T 2 T 3 Increased or improved Confidence, knowledge, skill, attitudes, and professionalism Safe patient care practices (process of care) Patient safety outcomes Target Individuals and teams Individuals, teams, systems Individuals, systems, and public health Setting Simulation lab Real patients and providers Systems and populations Data from McGaghie et al. 7 TSR = translational science research.

3 170 Griswold et al SIMULATION-BASED EDUCATION TO ENSURE PROVIDER COMPETENCY WITHIN THE HEALTH CARE SYSTEM As an example, Barsuk et al and others present a body of work that evaluates central venous catheter (CVC) training across multiple levels, thus providing examples of the steps necessary to establish translational evidence supporting simulation-based research. This work presents clear evidence linking training to clinical outcomes and serves as an important model for other work. It is important to note that there are several characteristics of CVC insertion make it a good exemplar for simulation-based translational outcomes research. First, CVC placement is a procedure that can be described in a stepwise fashion, with clear branching points and well-defined actions that characterize best practices. Second, in almost every situation an identifiable practitioner performs CVC placement. While guidelines support the involvement of nursing to ensure safety, the primary steps of the procedure are executed by a single provider, which inherently simplifies the training and assessment protocol. Finally, CVC placement is related to patient outcomes that can be clearly defined and are often collected as part of institution-wide safety and quality metrics. In EM, there are multiple procedure-based tasks that fit the description of complicated procedure (s) 31 akin to CVC insertion, for which the process of a translational research approach could be adopted. However, EM and critical care are dynamic, multifactorial practices in which cognitive, procedural, and communication skills are simultaneously applied in a team environment. Proper training of individuals to a high level of performance and the accurate assessment of performance within such a dynamic setting is challenging. Training and assessment of individual performance in such a multidimensional setting focuses not on an isolated element, but on complex care process for which evaluation metrics are presently less well-defined (e.g., the system and environmental impact while managing the altered patient, leading a team resuscitation, or navigating challenging patient family communication). In replicating a more complex practice environment, simulation educators and researchers who study individual performance attempt to isolate the performance of a single provider who is inherently subject to multiple dynamic variables. When teaching and evaluating more multifaceted skills such as resuscitation leadership in the clinical environment, researchers must consider the target of training (e.g., individual, team), the complexity of the patient, the skills and performance of team members, and the immediate ED environment. Issues related to level(s) of analysis for individuals, teams, and systems persist within the literature and present a particular challenge for simulation-based translational research The identification of appropriate outcome measures for effective provider communication in EM, for example, is challenging not only because the construct itself is subject to interpretation, but also because commonly collected proxy measures like patient satisfaction scores often cannot reliably represent the communication skill of any one individual provider. For these more complex processes, the development of reliable and valid measures that define competency, the ability to extract performance data of the individual within the clinical environment, and the identification of meaningful patient-level outcomes are necessary. Research focus 1: What domains of performance among emergency providers have the greatest impact on high-value translational outcomes, and how can they be reliably identified and studied? When training and evaluating complex skill performance, research is limited by the need to identify a criterion standard or reasonable benchmark of quality performance then translate defined standards into robust measurement systems; reliably and accurately capture individual performance in the clinical environment; and reliably link performance with relevant outcomes within a multidimensional system of emergency care. Much of the research published to date at the T 1 level has been conducted on novice learners, and many early studies compared SBME to no training at all. 34 An important factor to consider in the leap from T 1 to higher-level translational outcomes is the multifaceted complexity of care that blends procedural, cognitive, and communication skill among individuals and teams. For instance, is the clinical outcome of interest in a cardiac resuscitation case attributed to the excellent teamwork and communication or to the individual knowledge and leadership skill? In this complex clinical setting, how do varying levels of team dynamics, cognitive load impact, or medical decision-making processes impact individual performance and outcome(s)? Understanding the effect of such variability on individual performance in complex environments is an important priority in next-generation simulation research.

4 ACADEMIC EMERGENCY MEDICINE February 2018, Vol. 25, No Proposition 2: T 1 Simulation-based outcome research remains critical in understanding the potential of SBME; advanced patient outcomes analysis must be synergistic with meaningful basic science in medical education research Simulation offers the opportunity to assess performance at a highly granular level that is simply not possible in the current clinical environment. While the model of translational science is essential in clarifying the value proposition for SBME, the simulation lab, T 1 environment, remains a key investigational setting. Study protocols can be staged to replicate the live clinical environment. Such work is in fact a necessary precursor to advanced translational outcomes study, particularly as investigators explore the varied impact that cognitive, motivational, and affective elements may have on provider competency. In related fields like patient safety, experts agree that study designs targeting key process measures as marker of quality care are useful in understanding and refining best practices as part of a comprehensive outcomes research agenda. 35 Research focus 2: What strategies in translating SBME to improved patient outcomes are most effective in creating a sustainable cycle of lab-to-bedside impact? Prior to 2007, there was limited evidence that SBME had higher-level T 2 and T 3 translational outcomes. 36 It is now largely accepted that SBME has begun to produce downstream outcomes at the T 2 and T 3 translational science research (TSR) levels (see Figure 1). 12,14,20,37 58 In addition to the success of procedural domains like CVC 12,14,19,20,23,28 30, studies have begun to show improvement in other procedures 38,43 45,47,50, survival after pediatric cardiopulmonary arrest, 37,46,54,55 improved perinatal outcomes in labor and delivery, 41,42,51 improved trauma care, 40,48,49,53 and improved patient care handoffs/transitions of care. 39,52,57,58 However, the methodology of SBME studies at the T 2 and T 3 levels is pointedly more complex and few studies have published methodology that can be easily replicated and reproduced at different health care systems, thus limiting the generalizability of these studies. 59 Transparent reporting of research allows readers to clearly identify and understand what was planned, what was done, what was found, and what conclusions were drawn. 60 Recently simulation educators and leaders have published guidelines for SBME studies. These guidelines propose that researchers clearly elucidate methodology within SBME and CBME studies so that understanding, replication of work, and generalizability may be improved. Further work should include investigation to understand what strategies have been successful and effective in SBME that lead to improved clinical care. Research focus 3: What specific skills, practices, or clinical situations are priorities for simulation-based competency development, maintenance, and assessment in EM? A significant focus of the Accreditation Council of Graduate Medical Education (ACGME) has been to ensure competency as a public trust. A fundamental tenet of CBME holds that learning should be individually tailored to match inherent variation in learner progression. On a systems level, the challenge of determining competency across a diverse population of practitioners is significant and raises fundamental research questions about how to define and improve upon marginal performance. 61,62 In fact, the identification and remediation of marginal performance is a significant issue in CBME and is worthy of future study. Useful strategies to ensure that learners across the continuum reach acceptable levels of competency and proficiency include simulation-based deliberate practice and mastery learning. Given that SBME is often considered a more resource-intensive educational approach, careful consideration should be given to what kind of training and assessment could have the highest impact on patient care. Should every emergency physician be trained on the newest difficult airway adjuncts with expert feedback in the simulation lab? At what frequency and under whose supervision? How do you effectively remediate marginal performance? More broadly, of the full range of practice elements in EM, which should be routinely assessed and/or refreshed? An opportunity exists for simulation educators to collaborate with patient safety experts to identify high-priority training targets for EM. Proposition 3: Despite guidelines and conceptual models of validity testing in simulation, defining and deploying consensus measurement frameworks is challenging and remains a significant barrier to simulation-based research and competency assessment Much of the EM work on assessment work has focused on the nature of the assessment instrument

5 172 Griswold et al SIMULATION-BASED EDUCATION TO ENSURE PROVIDER COMPETENCY WITHIN THE HEALTH CARE SYSTEM CVC LP AIRWAY SKILLS COMMUNICATION/ TEAM TRAINING ACLS PEDS RESUS TRAUMA RESUS OB RESUS FUNDED ARTICLE POPULATION SPECIALTY T LEVEL OUTCOMES KSA OBSERVED Andreatta Residents Radiology T2, 3 PICC line insertion performance and success Andreatta Residents PEDS T2,3 Improved pediatric cardiac arrest outcomes Auerbach Residents PEDS T2,3 Increased infant LP success rate Barsuk Residents IM/EM T2,3 CRBSI Barsuk Residents IM/EM T2,3 CRBSI Bigham Peds physicians and nurses Pediatrics T2,3 Capella Trauma Teams Trauma T2,3 Decrease in hand off related failures Improved trauma Crofts Draycott Interprofessional labor and delivery personnel Interprofessional labor and delivery personnel OBGYN T2,3 OBGYN T2,3 Evans Residents IM T2,3 Decreased brachial plexus injury, improved delivery maneuvers Improved delivery management and neonatal outcomes Cannulation and success rate Kessler Residents PEDS T1*, 2,3 Self reported LP success Kessler Residents PEDS T2*,3* LP success rate* Kessler Residents PEDS T2*,3* LP success rate* Khouli Residents IM T1,2,3 CRBSI Knight Code Teams PEDS T2,3 Code team performance, survival of patient to discharge Lubin Nurse/EMS EMS T2*,3* Intubation success rate* AF AF Marr Trauma Teams Trauma T2 Decreased intubation time, adherence to ATLS protocols Miller Trauma Teams Trauma T2 Improved teamwork and communication Improvement in intubation Nishisaki 50, 2010 Residents PEDS T2*,3* success rate*; junior trainees intubated without increase in complication Improved protocol Peltan Residents IM T2,3* adherance, overall success rate* Riley OBGYN Teams OBGYN T2,3 Perinatal morbidity Sekiguchi Residents IM T2,3 Starmer Peds physicians and nurses Pediatrics T2,3 Steinemann Trauma Teams Trauma T2,3 Theilen Pediatric Teams PEDS T2,3 Theilen Pediatric Teams PEDS T2,3 Wayne Residents IM T2 Reduced complication rates Decrease in hand off related failures Improved teamwork, time Improved recognition of deteriorating patients Decreased PICU mortality, reduced PICU length of stay and cost savings Adherence to ACLS protocol Figure 1. SBME studies relevant to the practice of EM that report clinical translational outcomes at the T 2 or T 3 level. ACLS = Advanced Cardiac Life Support; AF = study author supported; CRBSI = catheter-related blood stream infection; CVC = central venous catheter; EM = emergency medicine; EMS = Emergency Medical Services; IM = internal medicine; KSA = knowledge skill or attitude; LP = lumbar puncture; OB = obstetrics; OBGYN = obstetrics and gynecology; PEDS = pediatrics; Resus = resuscitation; = Not reported; SBME = simulation-based medical education; T1 = indicates impact limited to the simulation laboratory; T2 = patient care processes or practice; T3 = patient outcomes, collateral effects such as cost or other value; = study reported no funding was received. *outcome without improvement.

6 ACADEMIC EMERGENCY MEDICINE February 2018, Vol. 25, No itself. Initial validity data are often limited to expert consensus and/or supported by simple comparisons of the performances of more- and less-experienced practitioners. Newer literature has focused on frameworks, such as that developed by Kane 63 and others, 64 can be used to characterize and inform the collection of validity evidence. Validity in assessment requires that inferences based on the assessment scores are reasonable. In Kane s model, these inferences, which are not focused specifically on the qualities of the assessment instrument, can be supported by evidence related to scoring (e.g., standardization of the conditions for assessment administration, well-constructed scoring instruments); generalization (e.g., reliability, precision of ability estimates); extrapolation (e.g., does performance in the simulation environment translate to realworld performance or do more advanced practitioners obtain higher score?); and implications and decisions (e.g., are competency decisions supported or is there a positive consequential impact of administering the assessment?). 65 Gathering evidence to support the validity argument is a continuous process. As more evidence is gathered, we can have more faith that assessment scores reflect the ability, or construct, of interest. It is also important to note that validity is not a property of the assessment. Rather, it is a property of the inferences that we make based on the scores. Therefore, validity evidence that is gathered in one setting, or with one group of practitioners, may not be applicable to other settings or groups of providers. As we work to define appropriate constructs or criterion standards that can be measured as part of SBME, adopting an argument-based approach to collecting validity evidence will allow for the identification of areas (e.g., extrapolation evidence) where additional research is necessary. Research focus 4: What are the next steps to improve the iterative development of assessment programs in EM that balance rigor with usability? How can the EM community enhance collaborative approaches to assessment that avoid reinventing the wheel with a new tool or assessment approach by each group? How can we develop a standardized consensus process to develop assessment tools that are informed by experience in other fields (such as aviation)? More than ever, there are rich opportunities for medical educators, researchers, safety and quality experts, and health care system leaders to work more closely together to improve health care. The adaptation of a learning health care system model as defined by the Institute of Medicine 66 would enhance collaborative efforts to share data and insights across boundaries and help drive better, more efficient medical practice and patient care. CONCLUSIONS Medical error has recently been described in the literature as the third leading cause of death in the United States. 67 Although this statement is controversial and medical error is a multifactorial process, individual provider competency is an important component of health care system based safety. As described here, a significant body of translational research has begun to show how simulation-based medical education and competency-based medical education can improve patient care and outcomes. The core lesson from early efforts to implement competency-based medical education is the realization that it is not, and should not be, a uniform or static ideology. Rather, it is an amalgam of principles and approaches that must constantly evolve to meet a primary aim: to achieve better health and health care for all through more effective medical education. 68 Simulation-based medical education is a powerful educational modality to implement competency-based medical education strategies, and focused research in the field will continue to advance systemwide safety and efficiency in health care. The authors acknowledge Michele L. Spotts for her review and graphic design of figure and Sara Hock, Tom Terndrup, Jeff Siegelman, Nicole Elliott, Michael C. Nguyen, Damon Dagnone, Charles Lei, Douglas Ander, William Bond, Paul Phrampus, Julianna Jung, Alan Cherney, Matthew Tews, Linda Papa, Megan Leo, and Christopher McCoy for their preconference contributions to this consensus conference submission. Additionally, the authors thank the following individuals for their contributions during the consensus conference: Kenneth Palm, MD, Srikar Adhikari, MD, Sara Y. Baker, MD, Trent Reed, DO, Patrick Hughes, DO, Timothy Palmieri MD, Braden McIntosh, MD, David Ruby, MD, Jessica Hernandez, MD, Ernesto J. Romo, MD, Christopher Kiefer MD, Jillian McGrath, MD, Martin Pusic, MD, Timothy Schaefer, MD, Paul Ishimine, MD, Emily Binstadt, MD MPH, Jason Langenfeld, MD, Kimberly J. Won, PharmD, Aga De Castro, MD, MPH, C. Eric McCoy, MD, MPH, Paul Ishimine, MD, Alan Cherney, MD MSc, Amanda Crichlow MD, Sara M Hock, MD, Deepika Mohan, MD, Ernesto J. Romo MD, Aga De Castro, MD, MPH, Tina H. Chen, MD, Christopher Kiefer, MD, Timothy Koboldt, MD, Jennifer A. Frey, PhD, Ryan T. McKenna, DO, Jeffrey N Siegelman, MD, Amanda Young, MD.

7 174 Griswold et al SIMULATION-BASED EDUCATION TO ENSURE PROVIDER COMPETENCY WITHIN THE HEALTH CARE SYSTEM References 1. Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach 2010;32: Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach 2010;32: AHRQ Issue Brief: Health Care Simulation To Advance Safety: Responding to Ebola and Other Threats Available at: cations/files/simulation-brief.pdf. Accessed Jul 7, Ziv A, Wolpe PR, Small SD, Glick S. Simulation-based medical education: an ethical imperative. Acad Med 2003;78: McGaghie WC. Medical education research as translational science. Sci Transl Med 2010;2:19cm8. 6. McGaghie WC. Implementation science: addressing complexity in medical education. Med Teach 2011;33: McGaghie WC, Draycott TJ, Dunn WF, Lopez CM, Stefanidis D. Evaluating the impact of simulation on translational patient outcomes. Simul Healthc 2011;6 Suppl: S McGaghie WC, Issenberg SB, Barsuk JH, Wayne DB. A critical review of simulation-based mastery learning with translational outcomes. Med Educ 2014;48: McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. Translational educational research: a necessity for effective health-care improvement. Chest 2012;142: Park CS. Simulation and quality improvement in anesthesiology. Anesthesiol Clin 2011;29: Boulet JR, Jeffries PR, Hatala RA, Korndorffer JR Jr, Feinstein DM, Roche JP. Research regarding methods of assessing learning outcomes. Simul Healthc 2011;6:S Barsuk JH, Cohen ER, Feinglass J, McGaghie WC, Wayne DB. Use of simulation-based education to reduce catheter-related bloodstream infections. Arch Intern Med 2009;169: Barsuk JH, Cohen ER, McGaghie WC, Wayne DB. Long-term retention of central venous catheter insertion skills after simulation-based mastery learning. Acad Med 2010;85:S Barsuk JH, Cohen ER, Potts S, et al. Dissemination of a simulation-based mastery learning intervention reduces central line-associated bloodstream infections. BMJ Qual Saf 2014;23: Barsuk JH, McGaghie WC, Cohen ER, Balachandran JS, Wayne DB. Use of simulation-based mastery learning to improve the quality of central venous catheter placement in a medical intensive care unit. J Hosp Med 2009;4: Barsuk JH, McGaghie WC, Cohen ER, O Leary KJ, Wayne DB. Simulation-based mastery learning reduces complications during central venous catheter insertion in a medical intensive care unit. Crit Care Med 2009;37: Barsuk JH, Cohen ER, Mikolajczak A, Seburn S, Slade M, Wayne DB. Simulation-based mastery learning improves central line maintenance skills of ICU nurses. J Nurs Admin 2015;45: Cohen ER, Feinglass J, Barsuk JH, et al. Cost savings from reduced catheter-related bloodstream infection after simulation-based education for residents in a medical intensive care unit. Simul Healthc 2010;5: Evans LV, Dodge KL, Shah TD, et al. Simulation training in central venous catheter insertion: improved performance in clinical practice. Acad Med 2010;85: Andreatta P, Chen Y, Marsh M, Cho K. Simulationbased training improves applied clinical placement of ultrasound-guided PICCs. Support Care Cancer 2011; 19: Dong Y, Suri HS, Cook DA, Kashani KB, Mullon JJ, Enders FT. Simulation-based objective assessment discerns clinical proficiency in central line placement: a construct validation. Chest 2010;137: Hoskote S, Khouli H, Lanoix R, et al. Simulation-based training for emergency medicine residents in sterile technique during central venous catheterization: impact on performance, policy, and outcomes. Acad Emerg Med 2015;22: Khouli H, Jahnes K, Shapiro J, et al. Performance of medical residents in sterile techniques during central vein catheterization randomized trial of efficacy of simulationbased training. Chest 2011;139: Ma IW, Brindle ME, Ronksley PE, Lorenzetti DL, Sauve RS, Ghali WA. Use of simulation-based education to improve outcomes of central venous catheterization: a systematic review and meta-analysis. Acad Med 2011;86: Millington SJ, Wong RY, Kassen BO, Roberts JM, Ma IW. Improving internal medicine residents performance, knowledge, and confidence in central venous catheterization using simulators. J Hosp Med 2009;4: Reid JD, Vestrup JA. Use of a simulation to teach central venous access. J Med Educ 1988;63: Smith CC, Huang GC, Newman LR, et al. Simulation training and its effect on long-term resident performance in central venous catheterization. Simul Healthc 2010;5: Peltan ID, Shiga T, Gordon JA, Currier PF. Simulation improves procedural protocol adherence during central venous catheter placement a randomized controlled trial. Simul Healthc 2015;10: Khouli H, Jahnes K, Shapiro J, et al. Performance of medical residents in sterile techniques during central vein catheterization: randomized trial of efficacy of simulationbased training. Chest 2011;139:80 7.

8 ACADEMIC EMERGENCY MEDICINE February 2018, Vol. 25, No Sekiguchi H, Tokita JE, Minami T, Eisen LA, Mayo PH, Narasimhan M. A prerotational, simulation-based workshop improves the safety of central venous catheter insertion: results of a successful internal medicine house staff training program. Chest 2011;140: Amaral LA, Uzzi B. Complex systems a new paradigm for the integrative study of management, physical, and technological systems. Manage Sci 2007;53: Klein KJ, Kozlowski SWJ. From micro to meso: critical steps in conceptualizing and conducting multilevel research. Organizational Research Methods 2000;3: Gooty J, Serban A, Thomas JS, Gavin MB, Yammarino FJ. Use and misuse of levels of analysis in leadership research: an illustrative review of leader member exchange. Leadership Q 2012;23: Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA 2011;306: Lilford RJ, Mohammed MA, Braunholtz D, Hofer TP. The measurement of active errors: methodological issues. Qual Saf Health Care 2003;12:ii Nishisaki A, Keren R, Nadkarni V. Does simulation improve patient safety? Self-efficacy, competence, operational performance, and patient safety. Anesthesiol Clin 2007;25: Andreatta P, Saxton E, Thompson M, Annich G. Simulation-based mock codes significantly correlate with improved pediatric patient cardiopulmonary arrest survival rates. Pediatr Crit Care Med 2011;12: Auerbach M, Fein DM, Chang TP, et al. The correlation of workplace simulation-based assessments with interns infant lumbar puncture success a prospective, multicenter, observational study. Simul Healthc 2016;11: Bigham MT, Logsdon TR, Manicone PE, et al. Decreasing handoff-related care failures in children hospitals. Pediatrics 2014;134:e Capella J, Smith S, Philp A, et al. Teamwork training improves the clinical care of trauma patients. J Surg Educ 2010;67: Crofts JF, Lenguerrand E, Bentham GL, et al. Prevention of brachial plexus injury-12 years of shoulder dystocia training: an interrupted time-series study. BJOG 2016; 123: Draycott TJ, Crofts JF, Ash JP, et al. Improving neonatal outcome through practical shoulder dystocia training. Obstetr Gynecol 2008;112: Kessler DO, Auerbach M, Pusic M, Tunik MG, Foltin JC. A randomized trial of simulation-based deliberate practice for infant lumbar puncture skills. Simul Healthc 2011;6: Kessler DO, Arteaga G, Ching K, et al. Interns success with clinical procedures in infants after simulation training. Pediatrics 2013;131:e Kessler DO, Pusic M, Chang TP, et al. Impact of just-intime and just-in-place simulation on intern success with infant lumbar puncture. Pediatrics 2015;135:e Knight LJ, Gabhart JM, Earnest KS, Leong KM, Anglemyer A, Franzon D. Improving code team performance and survival outcomes: Implementation of pediatric resuscitation team training. Crit Care Med 2014;42: Lubin J, Carter R. The feasibility of daily mannequin practice to improve intubation success. Air Med J 2009;28: Marr M, Hemmert K, Nguyen AH, et al. Team play in surgical education: a simulation-based study. J Surg Educ 2012;69: Miller D, Crandall C, Washington C, McLaughlin S. Improving teamwork and communication in trauma care through in situ simulations. Acad Emerg Med 2012;19: Nishisaki A, Donoghue AJ, Colborn S, et al. Effect of just-in-time simulation training on tracheal intubation procedure safety in the pediatric intensive care unit. Anesthesiology 2010;113: Riley W, Davis S, Miller K, Hansen H, Sainfort F, Sweet R. Didactic and simulation nontechnical skills team training to improve perinatal patient outcomes in a community hospital. Jt Comm J Qual Patient Saf 2011;37: Starmer AJ, Spector ND, Srivastava R, et al. Changes in medical errors after implementation of a handoff program. N Engl J Med 2014;371: Steinemann S, Berg B, Skinner A, et al. In situ, multidisciplinary, simulation-based teamwork training improves early trauma care. J Surg Educ 2011;68: Theilen U, Fraser L, Jones P, Leonard P, Simpson D. Regular in-situ simulation training of paediatric medical emergency team leads to sustained improvements in hospital response to deteriorating patients, improved outcomes in intensive care and financial savings. Resuscitation 2017;115: Theilen U, Leonard P, Jones P, et al. Regular in situ simulation training of paediatric medical emergency team improves hospital response to deteriorating patients. Resuscitation 2013;84: Wayne DB, Didwania A, Feinglass J, Fudala MJ, Barsuk JH, McGaghie WC. Simulation-based education improves quality of care during cardiac arrest team responses at an academic teaching hospital: a case-control study. Chest 2008;133: Weinger MB, Slagle JM, Kuntz AH, et al. A multimodal intervention improves postanesthesia care unit handovers. Anesth Analg 2015;121: Starmer AJ, Spector ND, West DC, Srivastava R, Sectish TC, Landrigan CP. Integrating research, quality improvement, and medical education for better handoffs and safer care: disseminating, adapting, and implementing the I-PASS program. Jt Comm J Qual Patient Saf 2017;43:

9 176 Griswold et al SIMULATION-BASED EDUCATION TO ENSURE PROVIDER COMPETENCY WITHIN THE HEALTH CARE SYSTEM 59. Cheng A, Kessler D, Mackinnon R, et al. Reporting guidelines for health care simulation research extensions to the CONSORT and STROBE statements. Simul Healthc 2016;11: von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ 2007;335: Turner JA, Fitzsimons MG, Pardo MC, et al. Effect of performance deficiencies on graduation and board certification rates: a 10-yr multicenter study of anesthesiology residents. Anesthesiology 2016;125: Rosen MA, Salas E, Wilson KA, et al. Measuring team performance in simulation-based training: adopting best practices for healthcare. Simul Healthc 2008;3: Kane MT. Validating the interpretations and uses of test scores. J Educ Measure 2013;50: American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, & Joint Committee on Standards for Educational and Psychological Testing. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association, Cook DA, Hatala R. Validation of educational assessments: a primer for simulation and beyond. Adv Simul 2016;1: The Learning Healthcare System: Workshop Summary. In: Olsen LA, Aisner D, McGinnis JM, editors. Institute of Medicine Roundtable on Evidence-Based Medicine. Washington (DC): National Academies Press, Makary MA, Daniel M. Medical error-the third leading cause of death in the US. BMJ 2016;353:i Holmboe ES, Sherbino J, Englander R, Snell L, Frank JR. A call to action: the controversy of and rationale for competencybased medical education. Med Teach 2017;39: