POLICY ROYAL AUSTRALASIAN COLLEGE OF SURGEONS Division: Education Development & Assessment Ref. No. EDA-EXA-004 Department: Examinations Title: Conduct of the SET Clinical Examination SCHEDULE 1 ROYAL AUSTRALASIAN COLLEGE OF SURGEONS CLINICAL EXAMINATION INSTRUCTIONS FOR CANDIDATES
TABLE OF CONTENTS 1. BACKGROUND OF THE CLINICAL EXAMINATION... 1 2. DATES AND VENUES... 1 2.1. February Centralised (Australia only)... 1 2.2. June Centralised (Australia & New Zealand)... 1 2.3. Examination venues... 1 2.4. Examination sessions... 1 3. STRUCTURE OF THE CLINICAL EXAMINATION... 2 3.1. Information relevant to all stations... 2 3.2. Charts... 2 3.3. Physical Examination... 2 3.4. Non-technical Skills... 2 3.5. History... 3 3.6. Procedure... 3 4. ON EXAMINATION DAY... 4 4.1. Arriving at the Exam... 4 4.2. Examination waiting area... 4 4.3. Stethoscope and other equipment... 4 4.4. Dress code... 4 4.5. Briefing... 4 4.6. Storage of belongings... 4 4.7. Catering... 4 4.8. Starting the examination... 4 4.9. Between examination stations... 4 4.10. During each examination station... 5 4.11. Rest Stations... 5 4.12. End of the examination... 5 5. MARKING SYSTEM FOR THE CLINICAL EXAMINATION... 6 5.1. Checklist score... 6 5.2. Non-technical skills score... 6 5.3. Process score... 6 5.4. Global competency score... 6 5.5. Determining the station scores and passing standard... 6 5.6. Special Consideration... 6 6. FEEDBACK FOR CANDIDATES... 8 6.1. Examiner feedback... 8 6.2. Unsuccessful candidates... 8 7. RECENT RESULTS... 8
1. BACKGROUND OF THE CLINICAL EXAMINATION The Clinical Examination is designed to assess early SET trainees in the clinical application of the basic sciences. It tests applied knowledge of clinical conditions and scenarios expected to be encountered by junior doctors working in a hospital environment. It also evaluates cognitive and psychomotor skills required for successful surgical practice. A key component of the Clinical Examination is reproduction of conditions closely aligned to the hospital work environment, under which trainees are expected to practice on a day-to-day basis, with similar pressures and challenges. This feature of the examination contributes to the assessment of technical and non-technical competencies, such as situational awareness, assimilation of incomplete information, decision-making, and formulation of management plans within a limited amount of time. The conditions under which the examination runs are therefore as important to its validity, as the actual questions and scenarios that comprise it. 2. DATES AND VENUES The Clinical Examination is held in February and June each year. All examination dates and details of how to apply for the examination can be found on the College website. 2.1. February Centralised (Australia only) The February examination is centralised to one Australian city, meaning that all candidates who wish to sit the examination must travel to that city. 2.2. June Centralised (Australia & New Zealand) The June examination is centralised to one Australian city and one New Zealand city, meaning that all candidates who wish to sit the examination must travel to one of these cities. 2.3. Examination venues Venues in Australia will be one of the following cities, with the February and June examinations usually being held in different cities: Adelaide Brisbane Melbourne Sydney The New Zealand venue will usually be Auckland, but is subject to change at the discretion of the Clinical Examination Sub Committee. 2.4. Examination sessions In both February and June, the examination in Australia takes place in three sessions: Saturday morning Saturday afternoon Sunday morning. Candidates will sit their exam in one of these three sessions and must select their preference when registering for the exam. Please note that each session is subject to minimum (10 candidates) and maximum (20 candidates) capacities. A venue notice will be sent to each candidate confirming the session he or she has been allocated to which cannot be altered. Candidates who sit the examination on Saturday afternoon will be required to arrive early and be quarantined prior to the conclusion of the morning exam. The Sunday examination will be comprised of a different set of stations so no quarantine period is applicable. In June the examination in New Zealand is usually held in one session on Saturday morning only. Additional sessions may be added if required due to the number of candidates. Page 1
3. STRUCTURE OF THE CLINICAL EXAMINATION The Clinical Examination is an Objective Structured Clinical Examination (OSCE) consisting of 16 stations. There are four stations of each of the four station types: physical examination, non-technical skills, history-taking and procedural skills. 3.1. Information relevant to all stations 3.1.1. Each station is of 5 minutes duration, with 3 minutes between stations. A bell is rung at the start and end of each station. The first bell signals the start of the station. The second bell will be rung 5 minutes later to signal the end of the station. Warning bells prior to the end of stations are not used. 3.1.2. Each station has instructions for the candidate outside the station that can be read during the 3 minutes between stations. In addition, the instructions are also available inside the station. 3.1.3. Most stations will have a surrogate patient, although some procedure stations do not have a surrogate patient and some stations will have a surrogate playing a role other than a patient (e.g. a patient s relative, ambulance officer, nurse, medical colleague). You should extend the normal courtesy to the surrogate by introducing yourself and where appropriate explain what you need to do. 3.1.4. Ensure that you wash your hands or use alcohol gel in view of the examiner prior to examining or performing a procedure on the surrogate patient. 3.1.5. Some questions will set clear limits for the candidate (e.g. Do not examine for... or Assume that.. ). No marks will be awarded for performance outside the stated boundaries of the question. 3.1.6. If you are required to answer one or two specific questions from the examiner after completing the task, you will be told this in the instructions to candidate. This will be a standard question(s) which all candidates are asked. 3.1.7. Photographs may be used to convey additional information about the situation and the surrogate patient may be marked to indicate the site of pathology. 3.1.8. Any candidate who is a threat to patient safety, endangers themselves or others, will fail the station. 3.2. Charts Several standardized charts have been developed and may be used in some stations. Please see Appendix One for example versions of these charts. 3.3. Physical Examination "Perform a relevant physical examination..." 3.3.1. Candidates should perform a physical examination specific to the area of interest. Usually the region will be defined by the question. For example: "Examine the right hip and other relevant features of the right lower limb". The qualifier here serves to indicate that in addition to a hip examination, leg length and a brief neurovascular examination should be performed. 3.3.2. Occasionally you may be asked about the examination for a specific condition which may involve many regions. You will need to explain to the examiner what you are doing and why as you proceed. 3.3.3. You will NOT be required to examine genital, anus/rectum or female breast regions but if you think it could be relevant to the condition, you should tell the examiner that you would examine the region without doing so. 3.3.4. You are NOT required to take a clinical history and ongoing dialogue with the surrogate should be confined to the physical examination (e.g. "does it hurt", "please open your mouth"). 3.4. Non-technical Skills There are five categories in this type of station. 3.4.1. Counselling The candidate will be given information about a specific condition or the result of an investigation and will be expected to communicate this information to the surrogate patient in terms that the patient would understand. They may also be asked to discuss options available to the patient or implications of the information. Page 2
3.4.2. Obtain informed consent Candidates will be told the specific procedure the patient is to undergo. They should carry out an appropriate informed consent. 3.4.3. Breaking bad news The candidate will be supplied with unpleasant news to convey to the surrogate patient and is expected to demonstrate not only knowledge of the subject, but also an ability to communicate in an appropriate manner with empathy. 3.4.4. Working in a team The candidate will be expected to undertake a task with the surrogate who is acting as a member of the team (e.g. nurse, paramedic or colleague) providing appropriate information, undertake a checklist or trouble-shoot for the clinical scenario presented. 3.4.5. Post-operative information 3.5. History Information about the postoperative status (a complication, or new scenario) of a patient will be provided and the candidate asked to explain the problem and its implications to the surrogate patient. Simple terms should be used to convey the information and, where appropriate, pen and paper may be used to provide a diagram for the patient. It is also important to allow the patient an opportunity to ask questions to make sure they understand. The surrogate may have specific questions to ask during the station. Take a history 3.5.1. The candidate will be required to interview a surrogate patient. 3.5.2. You will be given basic information about the presentation of a clinical problem and are required to take a formal structured and relevant clinical history from the surrogate patient who has been provided with basic responses to the appropriate questions. 3.5.3. Appropriate questions asked by the candidate will be marked on the examiners checklist. The marks awarded are NOT dependant on the answers given by the surrogate. 3.6. Procedure There are three categories of procedure you may be required to demonstrate. a) Generic surgical skills such as gowning & gloving, suture technique and safe handling of instruments. b) Diagnostic procedures such as FNA, biopsy, excision of lesion. c) Emergency Procedures such as, airway management, IV access procedures. 3.6.1. The examiner at these stations will expect the candidate to demonstrate the method of performing the specified procedure on a surrogate patient, a mannequin or a model. A running commentary is expected from the candidate describing the anatomical landmarks, the presence or absence of obvious complicating factors (e.g. previous surgical scar), and a complete step by step description of the procedure using the equipment provided. 3.6.2. The equipment may be physically present or a photograph may be used to identify pieces of equipment to be used. If equipment you need is not on display you should indicate to the examiner what you would use, how you would use it and why. 3.6.3. Invasive procedure should not be performed on surrogate patients but should usually be completed on mannequins or models. As well as a description of your actions to the examiner, some explanation to and interaction with the surrogate is expected. In some questions you may be asked to do a number of short simple procedures. Page 3
4. ON EXAMINATION DAY 4.1. Arriving at the Exam 4.1.1. All candidates must arrive at the examination venue at least 30 minutes prior to the start of the examination for registration and a briefing. The requested arrival time will be specified on your venue notice. 4.1.2. When you register on arrival you will be required to provide photographic identification, such as a driver s licence, passport or hospital ID. 4.1.3. When you register you will also be required to hand in all mobile phones, watches (both digital and analogue) and any other electronic devices. Please see section 3.9 in the Conduct of the SET Clinical Examination for full details on prohibited items. 4.1.4. You will be provided with a name badge and a QR code which you will need as identification for the examination stations. 4.2. Examination waiting area Once you have registered at the examination venue and entered the waiting area you will not be permitted to leave the area. If you need to use the bathroom please ask the staff member or assistant present which bathrooms are available for candidates to use. 4.3. Stethoscope and other equipment All candidates are required to bring their own stethoscope to the examination. There will be no stethoscopes provided at the examination venue. All other equipment required for the examination stations will be provided. 4.4. Dress code 4.5. Briefing Candidates should wear formal business attire for the examination. Candidates will be given a briefing by the Examination Coordinator prior to the start of the examination. If you have any questions, please ask the Coordinator or another assistant. 4.6. Storage of belongings Prior to the start of the exam, staff and/or assistants will instruct you on where you can securely leave your belongings (bags, coats etc.) during the examination. 4.7. Catering Please note that no catering is provided for candidates and refreshments may not be available for purchase at most examination venues. 4.8. Starting the examination 4.8.1. Shortly before the examination begins you will be directed to the examination rooms within the clinic and you will be asked to wait outside the station number at which you are starting. You will be given instructions on the day as to which station number you are starting at. 4.8.2. You may proceed to read the candidate instructions on the outside of the examination door. 4.8.3. When the bell rings you are permitted to enter the examination station. 4.9. Between examination stations 4.9.1. A second bell will ring to indicate the end of the station. 4.9.2. When you finish a station proceed to the next station in sequence. Staff & assistants will be on hand to guide you if you unsure where to go. 4.9.3. If you finish the station early you may leave the room. If you do so, you must wait outside the room you have just left until the bell rings to signal the end of that station. You may not move on to read the instructions of your next exam station until this bell rings. 4.9.4. You will have 3 minutes between each examination station. This time should be used to read the candidate instructions which are posted outside the examination room. Read each question carefully and identify key words in it. 4.9.5. Candidates are not permitted to make notes or take photos outside an examination station. Any candidate seen doing so will have their notes confiscated by staff and will be reported to the Clinical Examination Committee. Page 4
4.10. During each examination station 4.10.1. When the bell rings to signal the start of the examination station you may knock and enter the room. When you enter each station place your QR code on the desk so the examiner can scan it and check your name badge. 4.10.2. Introduce yourself to the examiner and the surrogate patient (if one is present). 4.10.3. Proceed with the station as per the instructions that you have just read outside the examination room. Please ask the examiner if you need to see the candidate instructions again, as a copy is also provided in the exam room. 4.10.4. If photographic material or patient charts are provided, they should be used as part of the information available to you. 4.10.5. Aim to perform the required task in a logical order. 4.10.6. You may be asked a standard question or questions by the examiner after a certain number of minutes. You should try and answer this. The questions are only worth a minority of the marks. 4.10.7. When the bell rings, you must exit the station. 4.11. Rest Stations Depending on the number of candidates sitting the examination at your examination venue, you may have one or more rest stations during the examination. If this is the case, you will stay at this station for the full 8 minutes. If you need to use the bathroom whist at a rest station you must first obtain permission from a staff member, or assistant. 4.12. End of the examination At the end of the examination you must hand in your name badge and QR code to staff or assistants. You will be directed out of the clinic, where you can collect your belongings including your mobile phone, watches and other items which were handed in to staff on registration. Page 5
5. MARKING SYSTEM FOR THE CLINICAL EXAMINATION 5.1. Checklist score The performance of each candidate at each station is assessed in part using a checklist score. This is a list of items that the candidate must perform satisfactorily in order to achieve a score for that item. The items vary for each station. If the candidate attempts an item, but it is not performed satisfactorily, no score will be given. The items are weighted according to importance. Please see Appendix Two for an example checklist. 5.2. Non-technical skills score For each station (except procedure stations without a surrogate patient), the examiner will award a score for the performance of the candidate for non-technical skills. Each station type has a different guide with descriptors to aid the examiner in assigning a non-technical skills score. Please see Appendix Three for non-technical skills score assessment criteria. 5.3. Process score For each station (except non-technical skills stations), the examiner will award a score for the performance of the candidate for their process. Each station type has a different guide with descriptors to aid the examiner in assigning a process score. Please see Appendix Four for process score assessment criteria. 5.4. Global competency score The overall competence of the candidate in each station is assessed on a scale of 1 to 6. This score does not contribute to the station score for the candidate. Please see Appendix Five for the global competency score scale. 5.5. Threat to Patient Safety Any candidate who is a threat to patient safety, endangers themselves or others, will fail the station. 5.6. Determining the station scores and passing standard 5.6.1. The score for each station is calculated as follows. The checklist score contributes 75% to the station score. For stations with both a non-technical skills and process score (physical examination and history stations and procedure stations with a surrogate patient), the nontechnical skills and process scores each contribute 12.5% to the station score. For nontechnical skills stations, the non-technical skills score contributes 25% to the station score. For procedure stations with no surrogate patient, the process score contributes 25% to the station score. 5.6.2. The mark required to pass each station is determined by the borderline regression method. This is performed by constructing a regression of the station score against the global competency score for all candidates and setting the cut score at the station score that corresponds to the Borderline Pass score. 5.6.3. The minimum passing score for the whole examination is the sum of the 16 station pass marks plus one standard error of measurement. 5.6.4. In order to pass the exam, candidates must pass at least two stations of each type (Physical Examination, Non-technical skills, History and Procedure) as well as achieving the minimum passing score for the whole exam. 5.6.5. A candidate whose aggregate score lies within one standard error of measurement of the cut-off mark and who passes at least two stations of each type (Examination, Non-technical skills, History and Procedure) in the same examination on each of two attempts will be deemed to have passed. 5.7. Special Consideration 5.7.1. Special consideration may be granted to a candidate who has undertaken or is about to undertake the examination where illness, bereavement or other serious matter beyond the control of the candidate has had an adverse effect, or has the potential to have an adverse effect on his/her result. 5.7.2. Special consideration may also be granted in relation to an incident that occurred during an examination. Details of the incident must be provided and the incident must also have been Page 6
brought to the attention of the Examiner or College staff member at the examination venue at the time it occurred. 5.7.3. An application form must be submitted to the Examinations Department within 72 hours of completion of the examination. No late applications will be accepted. 5.7.4. Full details and the application form can be found on the College website. Page 7
6. FEEDBACK FOR CANDIDATES 6.1. Examiner feedback For any candidate with a global competency score borderline pass or less, the examiner is required to select from a series of descriptors of deficiencies in performance. There is also an opportunity for the examiner to provide free text feedback on any candidate. These forms of feedback may be used to assist in the process of the supervisor discussing performance with failing candidates. 6.2. Unsuccessful candidates All unsuccessful candidates will be provided with a feedback report within 5 working days of the release of results. This report will also be sent to the candidate s current surgical supervisor, who will be asked to arrange a counselling meeting within 4 weeks of receiving the report. This meeting should help to provide the candidate with some insight into the reasons why they were unsuccessful and help them to prepare for their next attempt. 7. RECENT RESULTS The proportion of candidates passing the Clinical Examination in recent sittings is shown in the table below. Examination date Pass rate 18 October 2008 88% 14 February 2009 89% 6 June 2009 81% 13 February 2010 87% 29 May 2010 89% 12 February 2011 91% 28 May 2011 91% 18 February 2012 84% 16 June 2012 86% 9 &10 February 2013 80% and 85% 15 June 2013 95% 8 & 9 February 2014 86% and 90% 31 May 2014 93% 7 & 8 February 2015 93% and 96% June 2015 93% 6 & 7 February 2016 80% and 75% June 2016 88% 25 & 26 February 2017 79% and 76% 24 & 25 June 2017 76% and 93% Page 8
APPENDIX ONE CHARTS Page 9
Page 10
Page 11
Page 12
Page 13
APPENDIX TWO EXAMPLE CHECKLISTS Page 14
APPENDIX THREE NON-TECHNICAL SKILLS SCORE ASSESSMENT CRITERIA Non-technical skills Please enter a non-technical skills score Score Category Example of non-technical skills competence assessment criteria Page 15
APPENDIX FOUR PROCESS SCORE ASSESSMENT CRITERIA Process Score - Examination Station Score Category Example of examination competence assessment criteria Process Score - History Station Score Category Example of history competence assessment criteria Page 16
Process Score Procedure Station Score Category Example of procedure competence assessment criteria Page 17
APPENDIX FIVE GLOBAL COMPETENCY SCORE SCALE Page 18