Surgical skills assessment: an ongoing debate

Transcription

1 BJU International (2001), 88, 655±660 REVIEW Surgical skills assessment: an ongoing debate J. SHAH*² and A. DARZI* *Academic Surgical Unit and ²Department of Urology, Imperial College School of Medicine, St Marys Hospital, London, UK Introduction What makes a good surgeon? There are many attributes required of a surgeon, e.g. core knowledge of basic sciences, clinical decision-making skills, communication skills, operative theory and the ability to deal with intra- and postoperative problems, the skill of providing safe postoperative care, and the ability to both think and work in stressful environments. Only 25% of a skilfully performed operation is thought to be attributed to manual dexterity, yet traditionally a surgeons skill has been measured by assessing the clinical outcome of his or her patient [1]. Changes in patient expectations and a few high-pro le surgical cases suggesting that poor clinical outcome was the result of individual failure to conduct an adequate operation have resulted in surgeons having to prove their surgical competence. This, together with political pressure, changes in training and technological advances, have set the scene for a more formal measurement of surgical skills, and the setting and maintaining of surgical standards. Assessing the performance of surgeons is not necessarily bad. Feedback serves to differentiate training from practice and thus assessment will aid surgical training. Formal assessment of a surgeon will enable the Royal Colleges to provide a license for technically sound surgeons and help in the revalidation process. Finally, it will serve to restore public con dence in the surgical profession. This review discusses the requirement of a good assessment process and the mechanisms that are currently in existence. Requirements of an assessment process No one has as yet developed a valid, sensitive and reliable system to measure surgical skills that could be easily administered and that allows for the evaluation of a trainee throughout his or her training. Any method that is used to evaluate surgical skills must have a few basic components: Accepted for publication 12 July 2001 Reliability; if the test were to be administered to the same individual on separate days, in the absence of any learning, the results obtained should be similar. Feasible; the test must be practical and straightforward to administer. Fair; the results from any assessment process should be reproducible. Objective. Valid; a valid test refers to whether the test is actually measuring what it is thought to be measuring [2]. This can be further divided into predictive validity (the ability of the test to predict future performance), concurrent validity (the extent to which the results of the new assessment process correlate with the accepted reference standard that is known to measure the same domain), face validity (the extent to which the assessment resembles the real situation) and construct validity (the extent to which the test differentiates between differing levels of experience) [2]. Methods available for assessment Perhaps the rst step in assessing technical skills is to establish which skills to evaluate. To be a competent surgeon requires a multitude of attributes; should each attribute be assessed individually? For example, teamwork is a key component of the surgeons repertoire but this is rarely assessed. In this respect, anaesthetists are ahead of surgeons in that they have developed the use of a simulated operating theatre and a training programme (`Anaesthesia Crisis Resource Management) [3]. This has served to teach and to assess the performance of anaesthetists during stressful situations, and to evaluate their ability to work within a team [4]. There are many methods available for assessing surgical skills, although they are mostly informal and lack structure. The `gut feeling Skilled surgeons rely heavily on `gut feelings for in-training assessment of their trainees, and subjective # 2001 BJU International 655

2 656 J. SHAH and A. DARZI comments such as `a good pair of hands are frequently encountered in reports. These comments are usually made at the end of a rotation and rely on recollection. Many surgeons consider that they are capable of judging the technical skills of a trainee, although this judgement is made with no speci c criteria. It is also possible that the trainer may not be a good teacher and a clash of personalities will lead to eventual bias in any such assessment. This has been con rmed by studies that show that in the absence of structured criteria, two surgeons rating the same trainee will differ immensely [2]. Assessment processes that are valid, reliable and most importantly objective should be developed. Professional examinations Various Royal Colleges have examinations that assess the knowledge of a surgeon. These can consist of multiple-choice questions, short questions, and essays, together with a clinical and viva component. Poor inter-rater reliability is a common objection to the sole use of oral or viva examinations, as studies have shown that raters ranked good performances more consistently than poor performances, with only 67% agreement when rating `poor performances. Therefore, more than one examiner is required to con rm a poor performance during an examination [5]. More recently, the clinical component of these examinations have incorporated objective structured clinical examinations (OSCEs) [6]. These allow an assessment of a trainees decision-making and communication skills, using a standardized form. This form of assessment ensures that a certain minimum level of knowledge and clinical ability is attained, but provides only a `snapshot view of the trainee. However, it does not allow a determination of whether these skills are sustained or whether there is indeed improvement of their skills. Surgical-skills courses The surgical Royal colleges have now introduced mandatory intercollegiate basic surgical skills courses in the UK. In the past such courses required mere attendance, and it is only recently that instruction and practice on these skills courses is combined with assessment at the end. This serves two purposes; for the accredited surgeon it provides a forum whereby the statutory requirement for Continuing Medical Education is ful lled, and for the trainee it allows feedback on the performance of basic skills that are necessary before proceeding to the next stage of training. Audit One commonly used method of assessing a surgeon is by the clinical outcome of his or her operative procedure, as noted. This form of data collection is usually unreliable, and attributes the outcome of the patient solely to the operation and the technical skill of that surgeon. What of the postoperative care and factors such as preoperative morbidity, case selection and local facilities [7]? Audit also fails to take the disease process into account and is unable to produce qualitative data. Another problem for trainees is that operative outcome is usually attributed to the name of the consultant and therefore simple numbers may not re ect the trainee surgeons true ability. Logbooks One of the most common methods of assessment is maintaining a log of all procedures that have been carried out. This is not a new concept for the trainee, who has traditionally maintained such a record, together with a description of the level of supervision and the surgical outcome. More recently, it is likely that accredited surgeons will also need to keep such records as part of the revalidation process. This type of record, which is cheap and easy to evaluate, is purely numerical and contains little information about the quality of the technical performance. It also assumes that the trainee is competent after a designated number of procedures. Although practice and repetition are essential to learning surgical skills, in the absence of feedback the trainee may learn to be consistently wrong [8]. Thus, this assumption that mere practice will make surgeons competent is erroneous. Many logbooks are also completed retrospectively and therefore any data on surgical outcome are likely to be unreliable and subjective. Direct observation When a trainee is observed with no explicit criteria, any judgement made on that trainees skills by a more experienced trainer suffers from personal bias and is subject to a wide range of opinions. However, if directly observed using a checklist, then this method of assessment is highly reliable and valid [9]. The use of operation-speci c checklists is not a new concept; Kopta used checklists in 1971 to evaluate orthopaedic trainees and found high inter-rater reliability [10]. It has also been shown that when a group of faculty members co-operatively identify the important items for inclusion in a checklist, there is

3 ASSESSING SURGICAL SKILLS 657 greater reliability than if the checklist were to be created by one person [11]. At the University of Toronto, the Objective Structured Assessment of Technical Skills (OSATS) has been developed. This model for assessment involves direct observation of trainees performing a variety of structured operative tasks, using bench-top simulations and rotating through several stations. Performance is scored by surgeons using an operation-speci c checklist that divides the procedure into its smallest components, and a detailed overall scoring system [12]. The overall scoring system has also been developed at the University of Toronto and includes items that assess aspects of operative skill on a ve-point scale with behavioural descriptors. The authors found high correlations between the checklist and overall scoring systems, suggesting that the two methods are measuring the same quality [13]. Although it is possible to conduct assessments using checklists on real patients or volunteer human subjects, as is commonly encountered in the OSCE setting, this has the obvious problem of the lack of standardization, which makes the assessment of technical skills somewhat dif cult. An alternative is to observe the trainee carrying out surgical procedures on animals or animal tissue. The problem in the UK is that the Cruelty to Animals Act of 1976 prohibits the use of animals, e.g. an anaesthetized pig, for surgical training. However, there is no restriction to the use of animal parts, e.g. small bowel or pigs feet, and it has been shown that bench models are as good as animal models for evaluating surgical skills [12]. This is useful evidence, as bench models are cheaper, more portable and more readily available than animals, because of the growth in the commercial production of such models [14]. The result from the available data suggests that direct observation with the use of checklists for assessment appears to be both reliable and valid. such that the trainees face is not visible and the procedure is performed wearing surgical gloves. The lack of any sound would help to preserve anonymity and the tapes can be viewed at the facultys convenience. Whilst this form of surgical assessment is highly reliable and invaluable to the trainee if feedback is provided, it can also be expensive because of the extra cost of tapes and equipment; it is also a time-consuming procedure. Motion analysis More recently, work has concentrated on instrumentrelated factors to assess surgeons. One study de ned the ideal suture as one in which the needle is advanced along its curvature together with correct placement of the suture; this was then used as a form of assessment of the technical skill involved in suturing [16]. The Imperial College Surgical Assessment Device uses motion analysis as a measure of surgical dexterity. It uses three-dimensional positional data, produced by tracking devices that are attached to the surgeons hands when they move within an electromagnetic eld. This does not restrict the surgeons degrees of freedom and studies have shown that the measures such as the number of movements produced, the distance travelled, the speed of movement and the total time taken to complete the task are valid re ections of surgical skill [17]. The same technology has been used to assess surgical dexterity during an entire surgical procedure carried out on a laboratory model, and the data produced were shown to be a valid re ection of dexterity; importantly, this is an objective test [18]. Motion analysis appears to show promise as a tool of the future for surgical skills assessment, based on the theory that as a motor task is learnt, the movements made by the trainee will eventually become more ef cient [19]. Video analysis The lack of physical space, cost and restrictions in the availability of suf cient numbers of surgeons to carry out live direct observations with checklists is a problem that limits such a form of assessment in the UK. At the University of Toronto, they were able to evaluate 48 residents in a single 2-h examination with the use of bench models and 48 faculty members [15]. This results in 18 man-hours per six candidates. Clearly this is very labour intensive and therefore one way around this problem is the use of videotapes. A trainee performing a standard task could be videotaped and this could be later analysed using checklists by trained surgeons. The videotape could be set Virtual reality Another technological development has been the advent of virtual reality, or computer-based surgical simulations. This has allowed surgical training to move away from teaching within the operating theatre to other facilities. This system for training and assessment has existed for several decades in the aviation industry, but is relatively new to the surgical sphere. Virtual reality allows objective measurements of the trainee as manipulations are undertaken, and will be of great value in providing feedback and a record of progress of the trainees surgical skills. The Minimally Invasive Surgical Trainer (MIST-VR) is a system that simulates a variety of laparoscopic tasks,

4 658 J. SHAH and A. DARZI e.g. grasping, transferring from one hand to the other, and diathermy. It is a personal computer-based system that has two laparoscopic instruments attached to a frame with movements that are detected by potentiometers. MIST-VR has been validated as a tool that can distinguish between differing grades of surgeons, using variables such as the number of movements made, the time taken, the number of errors made and nally, the economy of movement [20]. Assessment modules also exist on the program and a key study using MIST-VR assessed the effect of sleep deprivation on the performance of junior doctors [21]. More recently, virtual reality simulators, e.g. for exible ureteroscopy, exible sigmoidoscopy and colonoscopy, have been developed, and these systems provide an objective assessment of the procedure with facilities that allow playback for feedback and self-learning. As these systems are further developed, it is likely that they will be incorporated into the assessment of surgeons, both those in training and those who are accredited. Laparoscopic skills Laparoscopic training has become an integral part of the general surgery curriculum, and is certainly gaining momentum again in urology. Laparoscopic surgery requires a set of skills that centre around instrumentation, optics and depth perception. However, the optimal methods for assessing laparoscopic skills remain debatable and in laparoscopic surgery, as with conventional surgery, the assessment of technical skill has lagged behind [22,23]. Both the general public and the profession are now demanding that doctors learning new skills provide objective evidence of the acquisition of the necessary skills, regardless of their previous accomplishments [24,25]. So what methods are currently available for assessing laparoscopic skills? The use of a procedure-speci c checklist, as discussed earlier, for laparoscopic procedures has been shown to be a valid and reliable method of skills assessment [26]. The McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS) has shown that basic laparoscopic skills can be objectively assessed and scored using a series of structured tasks [27]. The problem with this system is that the equipment is expensive and cumbersome, and therefore dif cult to transport. These problems led to the development of a simpli ed mirrored-box simulator that provides quantitative data for evaluating tasks such as transferring, cutting, clip and divide, looping and suturing [28]. The Advanced Dundee Endoscopic Pyschomotor Tester (ADEPT) is a system that uses computing power for the objective evaluation of laparoscopic dexterity whilst using laboratory-based tasks, but real laparoscopic instruments [29]. Performance on ADEPT correlated well with independent blind assessment of clinical competence, and it was able to identify innate abilities, i.e. aspects of performance that did not improve with practice. This system therefore has the potential advantage of being useful as a test of aptitude and thus an aid in selecting trainees. The future Growing pressures both from inside and outside the surgical profession have resulted in the need for surgeons to show that they can operate and operate well. This has been a neglected eld until recently, and very few training programmes have objective tests of surgical skill as part of their evaluation methods. Indeed, there is no doubt that the evaluation of skills in general is subject to debate [30]. Given that the attributes required to be a competent surgeon are multifaceted, it is not surprising that questions remain about what needs to be assessed. Recently, people have concentrated on manual dexterity as the measure of a good surgeon. This too is debatable, as there was no apparent difference in manual dexterity between surgeons and physicians [31]. Moreover, there was no correlation between clinical pro ciency and psychomotor skills [32]. Examinations and other cognitive tests provide an assessment of a surgeons knowledge and represent a valid measure of retention of facts, but this too should not be used as the sole basis for assessing clinical competence in a training programme [33]. We believe that objective evaluations of surgical dexterity should be used in conjunction with subjective evaluations of the trainee and cognitive tests. Once it has been decided what to evaluate there remains the problem of which method should be used to carry out the evaluation. Currently available systems include surgical audit, the use of logbooks, observation, either direct using patients or models, or indirect with videotape analysis. The use of criteria checklists, as with OSCEs or OSATS, appears to remove the subjective component of an assessment and has been shown to have high inter-rater reliability. New developments, which are currently research tools, include laparoscopic assessment simulators such as ADEPT or MISTELS, and more recently MIST-VR and virtual reality simulators. Whatever method is used must be feasible, repeatable, valid and reliable.

5 ASSESSING SURGICAL SKILLS 659 Although surgical teaching has traditionally been carried out in the operating theatre, if surgical skills are to be evaluated the tasks must be made as standardized as possible, therefore removing the element of patient variability that exists within the operating environment. The exclusive use of this venue for assessment has several limitations and therefore evaluating skills in a laboratory setting allows standardization of the tasks, with obvious ethical and nancial advantages. Conclusion At present, the assessment of surgical skills is underdeveloped. That it is necessary to assess surgeons is not part of the debate, but in urology, as with the other surgical specialities, there is a need to decide what skills to assess, who should assess them and importantly how to assess them, so that there are objective variables upon which actions can be based accordingly. References 1 Spencer F. Teaching and measuring surgical techniques: the technical evaluation of competence. Bull Am Coll Surg 1978; 63: 9±12 2 Reznick RK. Teaching and testing technical skills. Am J Surg 1993; 165: 358±61 3 Howard S, Gaba D, Fish K et al. Anaesthesia crisis resource management training: teaching anaesthesiologists to handle critical incidents. Aviation, Space Environ Med 1992; 63: 763±70 4 McLellan BA. Early experience with simulated trauma resuscitation. Can J Surg 1999; 42: 205±10 5 Burchard KW, Powland-Morin PA, Coe NP et al. A surgery oral examination: interrater agreement and the in uence of rater characteristics. Acad Med 1995; 70: 1044±6 6 Harden RM, Stevenson M, Downie W et al. Assessment of clinical competence using an objective structured examination. Br Med J 1975; 1: 447±51 7 Poloniecki J, Valencia O, Littlejohns P. Cumulative risk adjusted mortality chart for detecting changes in death rate: observational study of heart surgery. Br Med J 1998; 316: 1697±700 8 Raufman HH, Wiegand RL, Tunick RH. Teaching surgeons to operate ± principles of psychomotor skills training. Acta Neurochir 1987; 87: 1±7 9 Watts J, Feldman WB. Assessment of technical skills. In Nuefeld VR, Norman GR eds, Assessing Clinical Competence. New York: Springer, 1985: 259±74 10 Kopta JA. An approach to the evaluation of operative skills. Surgery 1971; 70: 297± Valentino J, Donnelly MB, Sloan DA et al. The reliability of six faculty members in identifying important objective structured clinical examinations items. Acad Med 1998; 73: 204±5 12 Martin JA, Regehr G, Reznick R et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997; 84: 273±8 13 Winkle C, Reznick RK, Cohen R et al. Reliability and construct validity of a structured technical skills assessment form. Am J Surg 1994; 167: 423±7 14 Hamdorf JM, Hall JC. Acquiring surgical skills. Br J Surg 2000; 87: 28±37 15 Reznick R, Regehr G, MacRae H et al. Testing technical skill via an innovative `bench station examination. Am J Surg 1997; 173: 226±30 16 Seki S, Iwamoto H, Osaki H et al. The surgeons technical skill in suturing ± an analysis of the actual suture. Surg Today 1993; 23: 800±6 17 Taf nder NJ, Smith S, Mair J et al. Can a computer measure surgical precision? Reliability, validity and feasibility of the ICSAD. Surg Endosc 1999; 13 (Suppl 1): Smith S, Torkington J, Darzi A. A computerized assessment of surgical dexterity during perfused cadaveric porcine laparoscopic cholecystectomy. Br J Surg 1999; 86 (Suppl 1): Rosenbaum DA. Reaching and grasping. In Human Motor Control. San Diego: Academic Press Inc, 1992: 197± Taf nder N, Russell R, McManus I et al. An objective assessment of surgeons psychomotor skills: validation of the MIST-VR laparoscopic simulator. Br J Surg 1998; 85 (Suppl 1): Taf nder NJ, McManus IC, Gul Y et al. Effect of sleep deprivation on surgeons dexterity on laparoscopy simulator. Lancet 1998; 352: Royston CM, Lansdown MR, Brough WA. Teaching laparoscopic surgery: the need for guidelines. Br Med J 1994; 308: Dent TL. Training credentialing, and granting of clinical privileges for laparoscopic general surgery. Am J Surg 1991; 161: 399± Forde KA. Minimal access surgery: which path to competence? Surg Endosc 1989; 8: 1047±8 25 Cushieri A. Re ections on surgical training. Surg Endosc 1993; 7: Eubanks TR, Clements R, Pohl D et al. A valid and reliable scoring system for assessing technical skills during laparoscopic cholecystectomy. Surg Endosc 1999; 13 (Suppl): S32 27 Derossis AM, Fried GM, Abrahamowicz M et al. Development of a model for training and evaluation of laparoscopic skills. Am J Surg 1998; 175: 482±7 28 Keyser EJ, Derossis AM, Antoniuk M et al. A simpli ed simulator for the training and evaluation of laparoscopic skills. Surg Endosc 2000; 14: 149±53 29 Hanna GB, Drew T, Cushieri A. Technology for psychomotor skills testing in endoscopic surgery. Semin Laparosc Surg 1997; 4: 120±4 30 Watson DCT, Matthews HR. Manual skills of trainee surgeons. J R Coll Surg Edinb 1987; 32: 74±45 31 Schueneman AL, Pickleman J, Hesselein R et al. Neurological predictors of operative skill among general surgery residents. Surgery 1984; 96: 288±95

6 660 J. SHAH and A. DARZI 32 Lazar HL, DeLand EC, Tompkins RK. Clinical performance versus in-training examinations as measures of surgical competence. Surgery 1989; 87: 357±62 33 Barnes RW. Surgical handicraft: teaching and learning surgical skills. Am J Surg 1987; 153: 422±7 Authors J. Shah, Clinical Research Fellow. A. Darzi, Professor of Surgery. Correspondence: Miss Jyoti Shah, Department of Urology, St Marys Hospital, Praed Street, London W2 1NY, UK. jyoti.shah@ic.ac.uk Abbreviations: OSCE, objective structured clinical examination; OSATS, Objective Structured Assessment of Technical Skills; MIST-VR, Minimally Invasive Surgical Trainer; MISTELS, McGill Inanimate System for Training and Evaluation of Laparoscopic Skills; ADEPT, Advanced Dundee Endoscopic Pyschomotor Tester.