School of Medicine. OSCE operating procedure

Transcription

1 School of Medicine OSCE operating procedure October 2012

2 Contents 1. Introduction 3 2. Overview of tasks and responsibilities 4 3. Prior to the exam 5 4. On the day of the exam 6 5. After the exams 7 6. Year specific OSCE information 8 7. Standard setting 10 C:\Users\Setup\Desktop\policies\word\OSCE SOP doc 2

3 1. Introduction This document lays down the operating procedure for the conduct of OSCEs in years 3, 4 and 5 of the Keele University undergraduate medical curriculum. It will ensure that the OSCEs function in a co-ordinated manner across the curriculum and that all of the tasks required for an OSCE to take place are undertaken consistently and in a timely manner. Most tasks are common to all of the OSCEs and will be described first. Variations relating to individual OSCEs will be covered in sections relevant to each year. Particular issues, which have been encountered over the last few years, are as follows: Poor attendance at emendation meetings OSCE stations being considered at emendation meetings even though considerable work is still required. This causes emendation meetings to overrun and discourages attendance. Problems with patient recruitment (where real patients are required) due to unrealistic expectations of authors about which patients can readily be identified and late notice to the team running recruitment. Late changes to stations (or at least queries about stations) only days or even hours before the exam is due to run. Whilst if a significant error is picked up it will need to be corrected, this is rarely if ever the case and causes unnecessary work for the assessment team close to the exam. Inconsistent formatting of stations. Too much material being sent out to examiners. To assist year leads, the assessment team and the skills team a timeline for the tasks required will be agreed by the beginning of each academic year to ensure that tasks can be completed in a timely manner thus reducing stress on those involved and in particular to avoid errors from occurring. It should also allow for some piloting of new stations, thus minimising the risk of using potential problem stations in an exam. The main problem tends to be too much content; whilst it is unlikely that it will be possible to pilot stations with students, and this is probably undesirable anyway, it is feasible for members of faculty to walk through a new station to identify any potential problems. C:\Users\Setup\Desktop\policies\word\OSCE SOP doc 3

4 2. Overview of tasks and responsibilities Task Responsibility Decide/blueprint stations Year leads/director of Assessments Review existing/banked stations Year leads/director of Assessments Identify authors for new stations Years leads Write new stations Those identified by the year leads Determine and communicate real Year leads and Assessment team patient requirements Recruit SPs Skills team Recruit real patients Patient Volunteer co-ordinator Pre-emendation station review Year leads/director of Assessments/Assessment team administrator/skills Team representative Finalise marking schemes Year leads/director of Assessments Emendation Year leads/unit leads/director of Assessments Send to externals Assessment team Incorporate/respond to externals Year leads/assessment team comments Examiner recruitment Assessment team Examiner training Director of Assessment, year leads, Skills and Staff Development teams Send out examiner information Assessment team Examiner briefing Director of Assessment/year leads Standardisation of simulated patients Skills team for two-site OSCEs Y4 and 5; Y3 examiners of same station Exam Assessment team/skills team/year leads C:\Users\Setup\Desktop\policies\word\OSCE SOP doc 4

5 3. Prior to the exam Emendation: The purpose of the emendation meeting is to confirm that the stations can be understood and will perform on the day of the exam. In addition the process acts as a final check that the material being assessed is suitable and that there is sufficient breadth of coverage. All basic work such as formatting must be completed prior to the emendation meeting to ensure that the meeting will last for a reasonable length of time. Emendation meetings will, wherever possible, be chaired by the Director of Assessments with the year leads also in attendance. As well as representation from across the year at least one member of the skills team must be present. After emendation stations will be sent to external examiners for review. After incorporation of their comments the stations will be considered final. At this point no further alteration will be made except in the unlikely event that there is an error which is fundamental to the functioning of the station. This is to avoid the practice of large numbers of phone calls taking place in the 48 hours prior to the exam requesting changes to stations. Where concerns do need to be raised this should be with year leads only, who will have the final say as to whether any alteration will be made. Examiner recruitment: All new examiners must have attended generic OSCE training prior to examining. All examiners must have attended training specific to the exam so that they are updated about any changes for that year and have also attended a calibration session. In year 5 all examiners must have had previous experience. Examiner information: Examiners will be sent information about their station during the week prior to the exam. The information will consist of: Time/date/location of session(s) A single copy of the student information A single copy of the questions that they will ask the candidate (if any) A copy of the SP script (if applicable) A copy of the marking scheme A copy of the relevant global score descriptors C:\Users\Setup\Desktop\policies\word\OSCE SOP doc 5

6 4. On the day of the exam Examiners will arrive at the designated time for their session and sign in. Once all examiners are present a standardised briefing will be undertaken which summarises all of the points that have been covered in the examiner training such as content of the year, purpose of the exam, tasks in the station, scoring system and feedback. Once the briefing has been completed examiners will go to their stations and familiarise themselves with the station. Where an SP is present standardisation will take place, and where real patients are being used examiners will check the history and/or physical signs for themselves to reach their own conclusions about the level of difficulty. ipads will generally be used for scoring and feedback and all examiners must have undergone training beforehand. A paper backup will be available should there be a technical failure. At the end of the session the invigilator will conduct a debrief with all of the examiners, SPs and patients to gather immediate feedback about how stations have functioned. It is vital that, if any alterations are made to how a station runs in the first session of an exam, this is communicated to subsequent examiners to ensure consistency. C:\Users\Setup\Desktop\policies\word\OSCE SOP doc 6

7 5. After the exams A formal process of incorporating feedback from examiners, others staff, SPs, patients and students will be undertaken. Generally it is expected that this will take the form of a meeting between the assessment team and year leads. The timing will depend on the year or the course, but meetings should take place before any work on the exams for the next academic year to ensure that any necessary alterations have been incorporated (or at least considered) before the next set of exams are put together. C:\Users\Setup\Desktop\policies\word\OSCE SOP doc 7

8 6. Year specific OSCE information Year 3 The OSCE takes place during the Easter assessment period, between the end of the blocks and the commencement of the SSC and CCS period. There are 12 stations of 9 minutes in length. Students spend 30 seconds reading the instructions outside the station and 8minutes and 30 seconds completing the task. Students then have 30 seconds to move between stations. The OSCE is intended to assess suitability of students to progress into Module 4. Stations will test consultation skills, examination skills and procedural skills that have been learnt during the third year. As such, simulated patients and student volunteers will be employed, but there will be no real patients in the exam. The re-sit OSCE takes place in the resit assessment period at the end of July/beginning of August. Year 4 The OSCE takes place towards the end of June after all five units have been completed and before students commence the SSC period. There are 15 stations of 9 minutes in length with 30 seconds to move between stations and 30 seconds for candidates to read the instructions outside stations. The stations are assessed at FY1 standard for the skills that have been learnt in year 4. However, not all of the skills relevant to FY1 are eligible for inclusion as some are not covered in year 4. Real patient examination stations will be included in the exam, with an expectation of a minimum of 3 real patient stations. The re-sit OSCE takes place in the resit assessment period at the end of July/beginning of August. Year 5 The OSCE takes place in April after completion of the two 15 week blocks in primary and secondary care. There are 14 stations of 10 minutes in length with 30 seconds to move between stations and 30 seconds for candidates to read the instructions outside stations. C:\Users\Setup\Desktop\policies\word\OSCE SOP doc 8

9 The stations are assessed at FY1 standard. Obstetrics will not be included in this OSCE as it is not revisited in year 5. Up to two stations can involve procedural skills, and of the remainder half of the stations should involve consultation skills and half should involve examination of real patients. The re-sit takes place after the elective period in June. Students who will be required to re-sit the exam will be informed before they go on elective to allow them to arrange remediation. C:\Users\Setup\Desktop\policies\word\OSCE SOP doc 9

10 7. Standard setting: Across the whole curriculum the Borderline regression method will be used. A pass mark is calculated for each station and students must pass a given number of the stations to pass the exam. In Year 3, students must pass a minimum of 8 from 12 stations, in Year 4 a minimum of 12 of 15 stations and in year 5 a minimum 11 of 14 stations. There will also be a safety net whereby a student s total score across the exam must be greater than or equal to the summed pass mark across all of the stations. In resits it is not possible to use the BR method reliably as there will generally be too few students and they are by definition a pre-defined borderline group. Therefore a combination of approaches will be used: In stations with known performance the known pass mark will be used In new stations an Angoff will be undertaken Opinions of the examiners will be sought on the day The borderline regression data will also be gathered C:\Users\Setup\Desktop\policies\word\OSCE SOP doc 10