Vol. 209, No. 3, September 2009 Chipman and Schmitz Evaluation of Simulation Curriculum 365 Association of Program Directors in Surgery National Simul

Transcription

1 Using Objective Structured Assessment of Technical Skills to Evaluate a Basic Skills Simulation Curriculum for First-Year Surgical Residents Jeffrey G Chipman, MD, FACS, Constance C Schmitz, PhD BACKGROUND: STUDY DESIGN: RESULTS: CONCLUSIONS: In response to new Accreditation Council for Graduate Medical Education requirements about simulation skill laboratories, programs are incorporating simulation into residents training. Despite substantial research on simulators, few data exist to support the effectiveness of simulation skills curricula. We report on an Objective Structured Assessment of Technical Skills (OSATS) used to assess residents needs and evaluate a curriculum designed to increase proficiency. The five-session (10-week) curriculum covered asepsis, skin preparation, gowning, gloving, knot-tying, suturing, and excision. Performance on a 20-minute OSATS station was measured by unblinded raters using a task-specific checklist and seven global rating scales. Interns prepost improvement was assessed using paired t-tests. PGY2 and PGY3 residents were used as nonequivalent controls; their scores set a benchmark for PGY1 residents postcurriculum. Percentage of possible points earned was compared with a 75% needs criterion; ANOVA was used to assess group differences at the p 0.05 level. Seven PGY2 and 6 PGY3 residents took the OSATS; 24 of 25 PGY1s completed both the baseline and postcurriculum OSATS. At baseline, PGY1 mean percent correct total score was 49%; they performed considerably below PGY2 (68%) and PGY3 (74%) residents. PGY1 scores improved significantly after 10 weeks (p 0.001). When their postcurriculum scores were compared with PGY2 and PGY3 resident benchmarks, there were no significant differences in checklist (p 0.38), global item (p 0.29), or total scores (p 0.45). Our results suggest that the simulation curriculum helped PGY1 residents attain basic surgical skills at levels consistent with PGY2 and PGY3 residents as measured by an OSATS. Only PGY3 residents performed at the 75% criterion. (J Am Coll Surg 2009;209: by the American College of Surgeons) Disclosure Information: Nothing to disclose. Received January 12, 2009; Revised May 8, 2009; Accepted May 11, From the Department of Surgery, University of Minnesota, Minneapolis, MN. Correspondence address: Jeffrey G Chipman, MD, FACS, Department of Surgery, University of Minnesota, Phillips-Wangensteen Bldg, 420 Delaware St SE, MMC 195, Minneapolis, MN chipm001@ umn.edu Use of simulation for training surgical residents has grown considerably in recent years, with establishment of the American College of Surgeons (ACS) Accredited Education Institutes in and the release of skills curricula by the ACS and Association of Program Directors in Surgery national task force in In 2008, nearly 50 surgeons responded to a call for members from the Association of Surgical Education for a new simulation committee (November 2, 2008 private communication from committee chairdbjones to Dr Schmitz). Such interest can only be expected to grow, given the Surgery Residency Review Committee s requirement that program resources should include simulation and skills laboratories. 3 Despite the existence of a substantial amount of research on particular simulators, research on the effectiveness of skills curricula using simulation remain sparse. 4,5 (A curriculum can be defined as a planned course of instruction involving goals, objectives, prescribed learning activities of specific scope and sequence, learner assessment, and evaluation.) Notable contributions include the work of Lentz and colleagues, 6 Boehler and colleagues, 7 Anastakis and colleagues 8, and Fried and colleagues 4 in the validation of the fundamentals of laparoscopic surgery. Although training programs now have a wonderful resource in the ACS/ 2009 by the American College of Surgeons ISSN /09/$36.00 Published by Elsevier Inc. 364 doi: /j.jamcollsurg

2 Vol. 209, No. 3, September 2009 Chipman and Schmitz Evaluation of Simulation Curriculum 365 Association of Program Directors in Surgery National Simulation Skills Curriculum 2 to turn to, it is imperative that programs evaluate, in situ, the impact of simulation curricula on learning. This is true even if curricula have been adopted from existing, reputable sources. This is because the quality of implementation, not just the quality of the curricular plan, influences the learner s experience. Relatively few departments are currently able to launch rigorous research on simulation training programs. All will be required to demonstrate learner achievement of skills training objectives. Use of a carefully tailored Objective Structured Assessment of Technical Skills (OSATS) can be instrumental in not only assessing learners, but also evaluating a specific curriculum. 9,10 This article describes the development and use of an OSATS to evaluate a five-module unit of instruction (Unit 1) within a year-long basic skills course for PGY1 residents. After examining the internal consistency reliability and construct validity of OSATS scores at baseline, our evaluation plan posited two questions: 1. What is the effect of the Unit 1 curriculum on PGY1 residents skills, as measured in the simulation laboratory? 2. Do we need to teach (Unit 1) basic skills to all incoming residents? Our primary hypothesis was that simulation training would shorten the amount of time it takes for PGY1 residents to acquire proficiency in a subset of very basic skills. We defined proficient as performing at a level equivalent to PGY2 and PGY3 residents who were enrolled in our program before the simulation curriculum became available and had learned these skills on the job. We also defined need for training as scoring 75% on the OSATS at baseline. METHODS The basic skills course was developed by faculty at the University of Minnesota and taught in the Simulation Perioperative Resource for Teaching and Learning, which is an ACS Level I accredited education institute. In 2007, all 25 PGY1 residents were enrolled in the course. Six were categorical general surgery residents, the others had matched into orthopaedic, otolaryngology, neurosurgery, or urology programs, or were of nondesignated preliminary status. Curriculum development and implementation The basic skills course consisted of 14 modules, each 1¾ hours in duration. The first five modules (Unit 1), which were delivered during a 10-week period early in the year, covered asepsis and skin preparation, gowning and gloving, knot-tying, suturing, and excision techniques. The stepwise sequence of the modules deliberately involved repetition of skills taught from previous modules. Each module was formatted into a 20- to 30-minute period of interactive didactic instruction with demonstration in a large group setting, followed by approximately 1¼ hours of individual or small-group practice in separate breakout rooms and stations (additional practice outside of class was not tracked). Actual materials (eg, gowns, gloves) and low-fidelity models, such as knot boards and pig s feet, were used. Instructional equipment needs were modest: a laptop computer with projector, an overhead camera, and moveable tables and chairs were all that was required. We enlisted nine instructors to implement Unit 1; for any given module, the resident-to-instructor ratio was 6:1. To support implementation fidelity, we required lead faculty to engage in a dress rehearsal the week before the session, which lasted about 1 hour. Additionally, we provided a 1-hour just in time training to all other instructors on the day of the session. Along with simulation for center staff, both authors provided on-site coordination and support during each session. OSATS station development and implementation We created a 20-minute station incorporating skills from all five modules in Unit 1. The station involved an adult patient with a benign skin lesion. Working unassisted, residents had 20 minutes to excise the lesion and close the wound. Residents were instructed that the patient (pig s foot) was already anesthetized and in the room. Residents were evaluated on how well they prepped the site, opened the instrument tray, gowned and gloved, maintained a sterile field, used universal precautions, and excised the lesion with.5-mm margins. In addition, residents were evaluated on whether they could identify the instruments by name and handle them correctly. Residents were required to close the wound with three or more vertical mattress sutures, to demonstrate one- and two-handed ties as well as an instrument tie, and to complete six square knots per stitch. The OSATS rating instrument was created after a review of each module s learning objectives and assessment checklists. The OSATS instrument, as seen in the Appendix (available online), used a combination of task-specific checklists and global rating scales adapted from Reznick and colleagues. 9 To standardize administration, all residents received a scripted orientation from Dr Schmitz. The simulation center staff ensured that individual stations were identical in setup. Four stations in separate rooms were run concurrently under timed conditions. Five raters were trained in observation and scoring techniques. Raters were not blinded to the training level of residents.

3 366 Chipman and Schmitz Evaluation of Simulation Curriculum J Am Coll Surg Evaluation design We used a pre-post evaluation design in which interns improvement over time was measured, and PGY2 and PGY3 residents were used as nonequivalent controls; their scores (obtained at the precurriculum baseline) set a benchmark for PGY1 residents at postcurriculum. Baseline OSATS was completed by 38 residents (25 PGY1, 7 PGY2, and 6 PGY3) during 2 days. Postcurriculum OSATS was given to 24 of 25 PGY1 residents on separate days approximately 1 to 2 weeks after the end of Unit 1. To assess the reproducibility of the curriculum effects with next year s class, we administered the same OSATS in 2008 to 25 PGY1 residents who completed Unit 1. We then compared their postcurriculum OSATS scores with those of previous residents. Measures and statistics All data from the OSATS were entered into a database for statistical analysis using SPSS 14.0 software (SPSS, Inc). Four measures were derived from the OSATS data: time to complete, checklist item score (p. 1 of the OSATS instrument), mean global item score (p. 2 of the OSATS instrument), and total test score. Total test score represents the sum of a resident s checklist scores and global item scale scores. Before implementing Unit 1, we examined the internal consistency reliability of the scores collected at baseline from our three groups (ie, PGY1, PGY2, and PGY3). First, we calculated reliability coefficients (Cronbach s ) based on the checklist items alone. Five checklist subscores were used for these analyses: They represented the sum of correctly performed activities within the categories of asepsis and skin preparation, gowning and gloving, excision of a skin lesion, suturing, and knot-tying (see Appendix online). Then we calculated reliability coefficients based on the 7 global items alone and then on all 12 items constituting the total score. Next, we examined item-total correlation statistics to assess the contribution of checklist and global rating items to total score. To assess construct validity, we used one-way ANOVA to compare the mean scores for all four measures collected from our three groups. To address evaluation Question 1 about the effect of Unit 1 on interns skills, we computed paired t-test statistics (two-tailed, n 24) to determine the extent to which pre-post changes in their OSATS scores were significant. We used one-way ANOVA to compare PGY1 residents postcurriculum scores with those collected from PGY2 and PGY3 residents at baseline. We set p 0.05 as the critical level for all tests of significance. To address evaluation Question 2 about the need for training, we converted PGY1 residents scores from baseline OSATS to percent of possible checklist items performed correctly and the proportion of potential global rating scores achieved, and compared the results with our 75% critical level. A total score of 60 represents 75% of the total possible points earned on the OSATS. We set no threshold for completion time. RESULTS Reliability and validity of scores at baseline Standardized internal consistency reliability coefficient (Cronbach s ) for scores generated from all 12 items in the OSATS rating tool was r This was essentially equivalent to the coefficient generated from the seven global items alone (r 0.878), and considerably higher than the coefficient generated by the five checklist items alone (r 0.637). Although the global items measuring a resident s motion and flow, tissue handling, knot-tying, and excision techniques related most clearly to his or her total score (based on the item-total score correlations), deleting any of the items in the rating tool would lower its overall reliability (Table 1). For this reason, all items were retained in the instrument. Mean baseline scores for PGY1 residents, as shown in Table 2, were significantly below those of PGY2 and PGY3 residents on three of the four measures (checklist total, mean global score, and total score). This finding indicates that the abilities being measured by the OSATS varied with training year in expected ways (ie, lower for novices, higher for more experienced residents) which constitutes evidence of construct validity. There were no group differences in completion time by training level (p 0.290). 1. What is the effect of the Unit 1 curriculum on PGY1 residents skills? Completion time for PGY1 residents did not change from baseline to postcurriculum, but checklist, mean global item, and total scores improved significantly (Table 3) as examined in paired t-tests (p 0.001). On average, residents gained nearly 7 points (1.63 SE) on their checklist score, nearly a full point on their mean global rating score (0.17 SE), and 13 points on their total score (2.60 SE). When the PGY1 postcurriculum scores were compared with the benchmarks set by 2nd- and 3rd-year residents, there were no longer any significant differences in checklist (p 0.70), mean global item (p 0.29), or total score (p 0.45) (see Table 4). 2. Do we need to teach (Unit 1) basic skills to residents? The percent correct data shown on Table 5 suggest that in 2007 none of the PGY1 residents scored near the ceiling of the OSATS. Only the PGY3 residents performance, as a group, matched the 75% criterion set for need. Based on their baseline scores and the 75% criterion, only one PGY1 resident could have placed out of the curriculum. In 2008, the PGY1 residents group

4 Vol. 209, No. 3, September 2009 Chipman and Schmitz Evaluation of Simulation Curriculum 367 Table 1. Test Item Statistics for an OSATS Rating Tool for Excision of Skin Lesion Score range* Item-total correlation Cronbach s if item deleted Items 1. Checklist asepsis Checklist gown glove Checklist excision Checklist suture Checklist knot Global sterile field Global instruments Global excision Global suture Global knots Global tissue Global motion flow Total OSATS score 0 81 Results are based on 38 residents (25 PGY1, 7 PGY2, 6 PGY3). Total OSATS score is the sum of all five checklist and seven global item scores. *Score range is based on the number of checkboxes within each of the five checklist-item categories, and the number of intervals in the scale used for global items. The item-total correlation column presents the correlation between each item and the Total OSATS Score. Low correlations signal an item that is less consistently related to the underlying competency being assessed. Indicates what the reliability coefficient for the instrument would be if the item were to be deleted. OSATS, Objective Structured Assessment of Technical Skills. average substantially met the criterion (74% total score) after completing the curriculum. The 2007 PGY1 residents, as shown in Table 6, performed weakest on the asepsis and skin-preparation checklist items (31%), and on the global item for maintaining a sterile field (40%). They performed best on the gowning and gloving (66%), suturing (56%), and knot-tying checklists (56%). DISCUSSION This article describes an OSATS that was developed to measure resident performance in excising a benign skin lesion and used to evaluate the first five modules of a PGY1 basic skills course curriculum. The seven global items in the nationally disseminated OSATS rating tool, as structured by Reznick and colleagues, 9 make for a malleable and robust instrument. Reliability and validity are not properties Table 2. Variation of Baseline Scores by Postgraduate Training Level in 2007 PGY1 baseline (n 25) PGY2 benchmark (n 7) PGY3 benchmark (n 6) p Value Measure Time to complete (min) 0.29 Mean 0:16 0:14 0:14 SD 0:03 0:01 0:01 Checklist item total Mean SD Mean global item score Mean SD Total OSATS score Mean SD Mean differences were analyzed using ANOVA. The preliminary test for homogeneity of variances showed no significant differences between PGY groups. Post-hoc pair-wise comparisons (least significant difference) showed that PGY1 residents scores were significantly lower than PGY2 and PGY3 residents on checklist, mean global, and total OSATS scores. OSATS, Objective Structured Assessment of Technical Skills. of any instrument, per se, but of scores generated by an examination. Reliability and validity estimates are influenced not just by the rating tool, but also by the quality of rater training, the uniformity of the stations, the degree of standardization in examinee instructions, and the population being tested. 11 If residency training programs decide to amend some anchors on the original OSATS global rating scales to suit a particular need, or create new scales, or develop companion checklists to itemize behaviors for a specific procedure (all of which we did), they need to reexamine the psychometric properties of the OSATS. Our examination of this OSATS suggests the examination produced reliable (r 0.896) and valid scores because they measured a type of competence that one expects to increase with experience and by training level. High-stakes examinations typically require reliabilities of 0.90; summative examinations related to coursework should be Although the checklist items did not contribute as much as the global rating scales did to the overall test score, their removal would diminish overall reliability. Previous researchers have argued for the superiority of global Table 3. PGY1 Performance from Baseline to Postcurriculum in 2007 Baseline mean (n 25) Postcurriculum mean (n 24) Mean change SEM Paired t-test Outcomes measure Checklist score t (24) p Mean global item score t (24) p Total score t (24) p 0.001

5 368 Chipman and Schmitz Evaluation of Simulation Curriculum J Am Coll Surg Table 4. Postcurriculum OSATS Scores by PGY1 Residents Compared with Benchmark Scores Set by PGY2 and PGY3 Residents in 2007 PGY1 postcurriculum (n 24) PGY2 benchmark (n 7) PGY3 benchmark (n 6) p Value Measure Time to complete (min) 0.25 Mean 0:16 0:14 0:14 SD 0:03 0:01 0:01 Checklist item total 0.70 Mean SD Mean global item score 0.29 Mean SD Total OSATS score 0.45 Mean SD Mean differences were analyzed using ANOVA. The preliminary test for homogeneity of variances showed no significant differences between PGY groups. Results suggest that after finishing Unit 1, PGY1 residents in performed similarly to that of PGY2 and PGY3 residents in three of the four measures. OSATS, Objective Structured Assessment of Technical Skills. rating scales over checklists, 13 others have found equal or superior reliability for checklists. 14,15 Our experience suggests the merits of using both. Checklists are useful for providing learners with specific, concrete feedback. In training raters, we have also found that checklists help ground their observations, much as anchors do to global rating scales. Although checklists document whether or not discrete steps or behaviors occurred, global ratings communicate how well those behaviors were executed. Both types of data are helpful. An OSATS can be instrumental in evaluating curricula in addition to assessing learners. In this evaluation, we used a pre-post design in which PGY1 residents performance over time was measured, and PGY2 and PGY3 residents were used as nonequivalent controls; their scores (obtained at the precurriculum baseline) set a benchmark for the PGY1 residents postcurriculum. A similar design was used by Boehler and colleagues. 7 Its primary virtue is that it can serve local evaluation purposes when randomization and a concurrent control group are not feasible. This design is subject to a number of limitations. Our residents postcurriculum scores were conceivably influenced by events taking place outside of simulation class during the 3-month time period, such as elective individual practice in the skills laboratory and operative experience gained on rotation. Because the PGY2 and PGY3 residents were not tested before the baseline period, we do not know at what point they reached their demonstrated performance level. It is possible they attained their levels earlier than the second year. Small sample sizes also might have limited our ability to detect statistically significant, pairwise differences in postcurriculum results. A more important potential limitation is the use of single raters and the potential for rater bias, because several of the raters were also instructors for the curriculum, and all knew the training levels of the residents. Interestingly, data on the difference between blinded and unblinded raters in a wellstructured OSATS by Goff and colleagues 16 suggested that unblinded raters tend to score subjects consistently higher than blinded raters, but the difference was neither statistically nor educationally significant. Vogt and colleagues 17 reported that knowing the identity of the subject did affect scoring, but not in a consistent direction. Depending on Table 5. Proportion of Checklist, Global, and Total OSATS Scores Obtained by Residents in 2007 and PGY1 baseline (n 25) PGY1 postcurriculum (n 24) PGY2 benchmark (n 7) PGY3 benchmark (n 6) PGY1 postcurriculum (n 25) Measure Checklist total % Correct Mean Global total % Correct Mean Total OSATS % Correct Mean Results from three performance measures were converted to percent of possible points earned. In 2007, only the PGY3 residents matched the 75% criterion. In 2008, PGY1 residents met the criterion postcurriculum. OSATS, Objective Structured Assessment of Technical Skills.

6 Vol. 209, No. 3, September 2009 Chipman and Schmitz Evaluation of Simulation Curriculum 369 Table 6. Item Results for PGY1 Residents at Baseline in 2007 (n 25) Total possible score Mean score (SD) Mean % correct Checklist items Asepsis skin preparation (1.17) 31 Gowning gloving (1.29) 66 Excision (2.96) 49 Suturing (2.31) 56 Knot tying (3.79) 56 Global items Maintaining sterile field (1.10) 40 Instrument handling identification (1.06) 48 Quality of excision (0.81) 46 Quality of suturing (0.83) 44 Quality of knots (0.90) 43 Respect for tissue (0.88) 51 Economy of motion, flow (0.85) 43 To assess areas of strength and need, checklist and global item scores were converted to percent of possible points earned. In no area did PGY1 residents score 75% criterion. the examiner, scores were either higher or lower when they knew their subjects. Despite these limitations, our results provide evidence that the five-session Unit distributed over 10 weeks helped bring PGY1 residents skills up to levels commensurate with residents 2 to 3 years their senior, who had learned the skills on the job. Faculty members involved with testing became convinced of the need for skills training outside of the operating room. Support for the course gained strength in the second year, as positive results from 2007 were replicated by the 2008 interns. On a practical level, data generated by this evaluation were useful for revising elements of the curriculum and for garnering additional resources for simulation. Our interest in establishing the need for this training was high, in part because some faculty had questioned it. Additionally, efficient use of faculty resources is always important. If incoming residents can presumably master very basic skills during medical school, then some (if not all) residents could be excused from additional training. Other investigators have reported on the potential efficiency of skills training outside of the operating room. For example, Boehler and colleagues 7 compared the OSATS performance of 4th-year medical students who completed an intensive 20-day skills curriculum with that of PGY1 and PGY2 residents. Students mean scores on checklists increased considerably (pre- to post-) for all four skill sets taught, ie, chest tube insertion, bowel anastomosis, skin excision, and laparoscopic cholecystectomy. Posttraining, students checklist scores were comparable with those of residents. Residents remained notably superior to students in global ratings for economy of time and motion with chest tube insertion and laparoscopic cholecystectomy. In our study, global item scores for motion and flow improved considerably for PGY1 residents pre- to postcurriculum, but at postcurriculum their mean score was below that of PGY2 and PGY3 residents (PGY1 mean 3.04, PGY2 mean 3.57, PGY3 mean 4.00; p 0.017). It appears this one item is especially sensitive to maturing skills. Other evaluations of curricula longer than our 10-week unit and involving followup testing have reported mixed results. Anastakis and colleagues 8 reported that an OSATS given to PGY3 residents approximately 6 months after the conclusion of a 2-year skills curriculum yielded scores that were indistinguishable from those of PGY3 residents who began the program before the advent of simulation skills training and served as historic controls. The authors speculated that one-time training on a core procedure was unlikely to have unique, observable effects up to 2 years later, especially given the amount of operative experience gained in the interim. Lentz and colleagues 6 reported results from a lagged evaluation design in which PGY3 and PGY4 obstetric and gynecology residents with 4 years of surgical laboratory skills training were compared with residents at the same training levels who started earlier in the program and had the benefit of only 2 years of the skills curriculum. Although residents who received all 4 years of the curriculum performed considerably better on individual surgical bench tasks, they scored no better than residents with 2 years of training on a comprehensive OSATS involving multiple surgical tasks. Bench tasks were scored on 5-point scales. The OSATS was scored with a combination of checklist and global ratings. The OSATS tasks were endoscopic and open abdominal in nature and varied from year to year (so residents did not know which would be tested). In a multisite study using the same 4-year skills curriculum, the same group reported considerable positive differences between residents who had participated in the skills curriculum in one institution and those from different institutions who had not. 17 We continued, as a result of our evaluation, to require all PGY1 residents to attend the basic skills course. Additionally, because the results suggested room for improved skills at the PGY2 as well as PGY1 level, we introduced advanced knot-tying and advanced suturing modules into our 2008 PGY2 curriculum. In conclusion, as simulation training becomes mandatory, programs will need to adopt evaluation practices that provide better evidence of curriculum efficacy than anec-

7 370 Chipman and Schmitz Evaluation of Simulation Curriculum J Am Coll Surg dote and opinion. Curriculum effectiveness is as much a product of local faculty and program resources and coordination as it is of any written curriculum document. Although randomized controlled trials provide optimal data for substantiating claims of causation and generalizability, they are costly to obtain. The OSATS approach, as described here, is helpful for understanding learner needs and supporting local decisions about training. The OSATS global rating scales can be easily downloaded from the Accreditation Council for Graduate Medical Education Web site toolbox. Programs, as with any instrument, should pilot the OSATS to ensure the scores are reasonably reliable and valid before using it to evaluate a curriculum. We believe this general approach, if not the particular OSATS, can be adopted by other programs with modest resource requirements. The evaluation described here does not require costly simulators, an accredited skills laboratory, or sophisticated statistical software. What it does require is faculty commitment, along with guidance from individuals with education and simulation expertise. Author Contributions Study conception and design: Chipman, Schmitz Acquisition of data: Chipman, Schmitz Analysis and interpretation of data: Chipman, Schmitz Drafting of manuscript: Chipman, Schmitz Critical revision: Chipman, Schmitz Acknowledgment: We wish to acknowledge the work of Troy Reihsen, University of Minnesota, Simulation Perioperative Resource for Teaching and Learning, for his participation in simulation module development and implementation. REFERENCES 1. Sachdeva AK, Pelligrini CA, Johnson KA. Support for simulation-based surgical education through American College of Surgeons Accredited Education Institutes. World J Surg 2008;32: Scott DJ, Dunnington GL. The new ACS/APDS skills curriculum: moving the learning curve out of the operating room. J Gastrointest Surg 2008;12: ACGME Program Requirements for Graduate Medical Education in Surgery. Chicago: Accreditation Council of Graduate Medical Education; 2008 January. Available from: acgme.org/acwebsite/navpages/nav_440.asp. Accessed June 4, Fried GM, Feldman LS, Vassiliou MC, et al. Proving the value of simulation in laparoscopic surgery. Ann Surg 2004;240: Aucar JA, Groch NR, Troxel SA, Eubanks SW. A review of surgical simulation with attention to validation methodology. Surg Laparosc Endosc Percutan Tech 2005;15: Lentz GM, Mandel LS, Goff BA. A six-year study of surgical teaching and skills evaluation for obstetric/gynecologic residents in porcine and inanimate surgical models. Am J Obstet Gynecol 2005;193: Boehler ML, Schwind CJ, Rogers DA, et al. A theory-based curriculum for enhancing surgical skillfulness. J Am Coll Surg 2007;205: Anastakis D, Wanzel KR, Brown MH, et al. Evaluating the effectiveness of a 2-year curriculum in a surgical skills center. Am J Surg 2003;185: Reznick RK, Regehr G, MacRae H, et al. Testing technical skills via an innovative Bench Station examination. Am J Surg 1997; 180: Reznick RK, MacRae H. Teaching surgical skills changes in the wind. N Engl J Med 2006;355: Schmitz CC. Your intergalactic decoder ring has arrived: reliability and validity defined. American College of Surgeons Residency Assist Page May 24 [cited 2008 December 30, 2008]. Available from: schmitz0506.html. Accessed June 4, Downing S. Reliability: on the reproducibility of assessment data. Med Educ 2004;38: Regehr G, MacRae H, Reznick RK, Szalay D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med 1998;73: Goff BA, Lentz GM, Lee DL, et al. Development of an objective structured assessment of technical skills (OSATS) for obstetric and gynecology residents. Obstet Gynecol 2000;96: Goff B, Mandel L, Lentz G, et al. Assessment of resident surgical skills: is testing feasible? Am J Obstet Gynecol 2005;192: Goff BA, Nielsen PE, Lentz GM, et al. Surgical skills assessment: a blinded examination of obstetrics and gynecology residents. Am J Obstet Gynecol 2002;186: Vogt VY, Givens VM, Keathley CA, et al. Is a resident s score on a videotaped objective structured assessment of technical skills affected by revealing the resident s identity? Am J Obstet Gynecol 2003;189:

8 Vol. 209, No. 3, September 2009 Chipman and Schmitz Evaluation of Simulation Curriculum 370.e1 Appendix: Excision of a Skin Lesion Proctor Score Sheet Proctor Name: Actual Start Time: Resident Name: Actual Stop Time: Date: Scheduled Time: Running Time: Checklist Asepsis and Skin Prep Score: e Assesses site before skin prep is started e Applies chloraprep ( 30 sec, back forth) e Opens instrument tray with sterile technique e Opens blade package and sterilely places scalpel blade on tray e Selects nylon suture, NOT Vicryl e Sterilely places suture on tray Gowning & Gloving Score: e Opens gown pack sterilely e Opens gloves on gown or next to it sterilely e Appropriately gowns using sterile technique e Dons gloves over sterile gown cuffs with hands inside the gown sleeves Excision of Lesion Score: e Applies 3-4 sterile drapes around site e Assesses axis of resection (Langer s Line) e Holds knife perpendicular to tissue plane (incision is perpendicular to skin without flaps or skiving) e Creates elliptical incision e Incision is perpendicularly completed to the level of the fascia e Creates flaps to facilitate wound closure e Makes flaps with minimal tissue handling (minimal grasping, regrasping, tissue trauma, etc.) e Excision dimensions allow closure without puckering (length 4 width) The removed tissue: e Has half mm margins e Leaves lesion (black dot) in center intact e Contains subcutaneous fat e Is placed in specimen container Checklist Score: Suturing Score: e Positions needle in driver appropriately (1/2 to 2/3 distance from tip) e Places suture w/needle perpendicular to skin e Places suture following curve of needle e Passes needle through tissue with supination: pronates wrist to regrasp needle e Mattress sutures are made correctly (1st stitch deep, 2nd stitch shallow) e Stitches are placed at appropriate distance from wound edge (closest approx. 1 mm of wound edge) e Stitches are at same level in epidermis e Closes appropriately to evert wound edges e Epidermis is apposed without gaps Knot Tying* Score: e Starts instrument tie with square throw e Subsequent throws are square to previous e Crosses hands with each throw to place (secure) them square e Ties knot without tissue strangulation (appropriate skin tension) e Throws 6 knots e Starts one-handed tie with square throw e Subsequent throws are square to previous e Crosses hands with each throw to place (secure) them square e Ties knot without tissue strangulation (appropriate skin tension) e Throws 6 knots e Starts two-handed tie with square throw e Subsequent throws are square to the previous e Crosses hands with each throw to place (secure) them square e Ties knot without tissue strangulation (appropriate skin tension) e Throws 6 knots (*Note: surgeon s knot may be used) Global Rating Scale of Operative Performance Please circle the number for each category, irrespective of the trainee s PG level. Maintaining a Sterile Field Many instances where sterile field was compromised Occasional instances when sterile field was comprised No instances where the sterile sterile field was field was compromised Instrument ID and Handling Could not name instruments, selected wrong instrument(s); handled instruments inappropriately Quality of Excision Poor technique, lesion compromised margins insufficient Could name some, not all instruments; hesitated or changed mind in selecting instruments; handled them appropriately most of the time Named all instruments; easily selected correct instruments; used them appropriately all of the time Moderately good technique lesion intact acceptable margins Excellent technique lesion intactexcellent margins

9 370.e2 Chipman and Schmitz Evaluation of Simulation Curriculum J Am Coll Surg Quality of Suturing Poor technique, poor manual dexterity, Moderately good technique, moderate dexterity, acceptable Excellent technique, excellent dexterity, excellent closure problems with closure closure Quality of Knots Poor technique, couldn t do all 3 ties, insecure knots Moderately good technique, some ties were done better than others, mostly secure knots Excellent technique, excellent execution of all 3 ties, very secure knots Respect for Tissue Frequently used unnecessary force, or caused damage on tissue Careful handling of tissue but occasionally caused damage Very careful handling of tissues with minimal or no damage Motion and Flow Many unnecessary moves, frequent stops starts, frequently grasped, regrasped tissue Mean Global Rating Score: sum individual item scores/7 Total Score Some unnecessary moves reasonably efficient, smooth progression, occasional regrasping tissue Clear economy of movement, easy flow/rhythm throughout, minimal regrasping of tissue