How to Use the Draft Blueprint for the Pre-Registration Examinations (PRES) Level 3

How to Use the Draft Blueprint for the Pre-Registration Examinations (PRES) Level 3 Prepared at the request of the Medical Council by Dr Helen Hynes, Dr Catherine Sweeney, Dr Nora McCarthy and Dr Siun O Flynn Medical Education Unit, School of Medicine, University College Cork. May 2015 1

The authors would like to acknowledge the assistance of Dr Paul Kavanagh of the Medical Council, and Prof Carl Vaughan and Ms Geraldine McNamee of the Intern South Network 2

Table of Contents Page 1 Introduction 4 2 Aims 6 3 How to use the Blueprint 7 4 Summary 17 5 References 18 6 Bibliography 18 7 Appendices 22 3

1. Introduction This Blueprint for the PRE-Registration Examination System (PRES) was prepared, at the invitation of the Medical Council, by a team of Medical Education experts and practicing clinicians, based in the Medical Education Unit of University College Cork, with the support of the Intern South Network. A test Blueprint is a grid, which allows examiners to generate content-valid exams by linking the required subject content and competencies to the items appearing on the test. 1 PRES is the assessment for International Medical Graduates, who wish to become licensed to practice in Ireland but who do not otherwise qualify under the EU/EEA/Swiss agreements. PRES Level 1 is a submission of documents to determine eligibility. In its current form, the PRES Level 2 is a multiple-choice test of knowledge. PRES Level 3 is a clinical assessment of communication skills, practical skills and data interpretation. An Objective Structured Clinical Examination (OSCE) assesses communication and practical skills, and Data Interpretation is assessed using a written test. The standard expected at the PRES examination is that of a medical graduate on completion of his/ her intern year. In preparing this blueprint, we performed a thorough review of documents relating to standards, and training in the Irish context including: PRES Handbook, 2015 National Intern Curriculum, 2012 Intern Handbook, Information and Logbook for Interns, 2008. The National Intern Education and Training agreement, 2014-2015 Your Training Counts, Results of the National Trainee Experience Survey. Medical Council, 2014 Guidelines for Medical Education and Training for Interns. Medical Council, 2011 4

Talking about good professional Practice. Medical Council, 2014 Medial Education, Training and Practice in Ireland 2008-2013. Medical Council, 2014 Eight Domains of Good Professional Practice. Medical Council, 2010 Guide to Professional Conduct and Ethics for Registered Medical Practitioners. Medical Council, 2009 Professional. Guidelines for Doctors. Medical Council, 2011. Holmboe E and Zaborowski MA. Recommendations for the examinations for specialists and generalized divisions. 2012 We also performed an extensive review of licensing criteria, curricula and blueprints from other jurisdictions, namely the UK, USA, Canada, Australia and New Zealand, and reviewed blueprints for postgraduate examinations. A full bibliography is presented in Appendix A. 5

2. Aims A blueprint is valuable strategy for enhancing and defending the validity of an examination. 2 It allows educators to define the sample of items and competencies to be included in the test. The main aim of this Blueprint is to provide a usable and clear document, which will allow the Medical Council, or any institution or organisation hosting the PRES Level 3, to plan an examination, which widely samples the knowledge, and skills required of a doctor who wishes to practice medicine in Ireland. The Blueprint does not aim to be an exhaustive list of all clinical presentations, but rather a workable list of common or important issues, which we would expect a medical practitioner to manage appropriately. This includes issues of professionalism and cultural competence, which are difficult to assess. The Blueprint contains content for assessment of skills and application of knowledge (PRES Level 3, OSCE and Data Interpretation) and also takes into consideration core competencies required of a doctor, and types of assessment. The Blueprint takes account of the current National Intern Curriculum. We are aware that the National Intern Curriculum may be reviewed and revised in the near future; however the Blueprint should be viewed as a template, which can be updated as required to take this, and any other changes in medical licensing in Ireland into account. 6

3. How to use the Blueprint The Blueprint consists of 2 sheets: the first contains multiple menus of content, mapped to domains and competencies, and the second consists of a series of grids for the 3 components of PRES 3, into which PRES coordinators can insert all the material being tested, to ensure that wide sampling of content areas, domains and competencies has been built in to the examination. Figure 1: Sheet 1 of the Blueprint The Blueprint is set out as a 2 dimensional grid with the Medical Council s 8 Domains of Practice and core content on 2 columns of the Y-axis, as seen below in Figure 2. 7

: Figure 2: Y-Axis of Sheet 1 (2 linked columns) A drop-down menu lists common or important topics related to each theme as shown in Figure 3 below. The drop-down menu is accessed by clicking on the small triangles on the right hand side of the boxes on the grid. Not every box contains a drop-down menu. For example, there is no menu at the Ethics / Practical skills intersection, as it is felt that ethics is more appropriately assessed as a communication station. The boxes on the grid that do not contain drop-down menus are coloured in dark grey. 8

Figure 3: The clinical topic drop-down menu associated with the Cardiovascular / Vascular system (Sheet 1) The X-axis lists the assessment modalities (MCQ / Data Interpretation / Communication Skills and Practical Skills) as shown in figure 4 below. Figure 4: The assessment modalities on the X-axis (Sheet 1) 9

Multiple drop-down lists of potential items and competencies can be viewed by clicking on the small triangles on the right hand side of the boxes on the grid as shown in Figure 5 below, which illustrates the drop-down menu associated with the intersection of Patient Safety and Quality Care with Practical and Procedures. Figure 5: Drop down lists of competencies for any given clinical topic and assessment modality. A printable version of Sheet 1 is found in Appendix 1, but for optimal use and interactivity, it is recommended that the Excel version be used. 10

To use the Blueprint, items should be sampled widely from all content, competency and assessment headings. The second sheet may be used to apply the Blueprint to the examination. The example shown below in Figure 6 is the grid that would be used for the Communication Skills part of the PRES Level 3 OSCE. Figure 6: The grid (Sheet 2) for the Communication Skills component of the PRES Level 3 OSCE. 11

The examination coordinator should insert the numbers of the stations, and the competency to be tested in each station, and then map out which of the 8 Domains of Practice and which areas of clinical content are covered for each station. An example of how the Blueprint was trialed at a recent PRES Level 3 examination is shown below in Figure 7 a, b and c. Figure 7a: Sample Blueprint for Communication Stations (Sheet 2) 12

Figure 7b: Sample blueprint for Practical Stations (Sheet 2) 13

Figure 7c: Sample Blueprint for Data Interpretation Stations (Sheet 3) 14

To use the blueprint, the examination coordinator / committee should first choose a wide sample of content and competencies from Sheet 1 (Figure 1). The next step is to insert these into Sheet 2 to come up with a Grid of the examination. Each potential examination station / question, should then be reviewed to see which of the 8 Domains of Practice, and main areas of scholarship are covered. For example an OSCE Station where the task is to Explain to a parent how to use a spacer device for managing her child s of asthma would appear on the grid as being a Communication Skills Station and covering Patient Safety and Quality care, Relating to patients, Communication and Interpersonal skills, Management, Respiratory, Paediatrics and General Practice. This is shown below in Figure 8. Figure 8: Mapping an OSCE station to the domains of practice and content areas on the grid (Sheet 2)

This is done for all Stations and written questions, which form part of the examination. The examination coordinator must then review the grid as a whole to ensure that content has been sampled as widely as possibly. The image below shows the entire grid for the 12 OSCE stations and 6 data interpretation questions, used in a recent PRES Level 3 examination. Figure 9: Overview of examination content for recent PRES Level 3 (Sheet 2) 16

4. Summary This Guide relates to a draft Pre Registration Examination System (PRES) Blueprint, which was developed at the request of the Medical Council to map out the competencies, content and domains tested in the PRES examinations. The Draft Blueprint consists of 2 sheets: the first contains multiple menus of content, mapped to domains and competencies, and is used for choosing content. The second sheet is a grid for use when mapping PRES 3 examinations, in which PRES coordinators can insert all the material being tested, to ensure that wide sampling of competencies, content and domains has been built in to the examination. Printable versions of the Blueprint are available in the Appendices of this document, but these lack the interactive features, which have been built into the Excel version of the Blueprint. There is also a web-based version of the Blueprint available on Google Docs, which can be access from this link: https://docs.google.com/spreadsheets/d/1w_ogdmg- IU_9dhfp8kP0PIwLO010--upPJvnSm5H22E/edit?usp=sharing 17

5. References 1. Bridge PD, Musial J, Frank R, Roe T and Sawilowsky S. Measurement practices: Methods for developing content-valid student examinations. Medical Teacher 2003;25:414 421. 2. Newble D. Techniques for measuring clinical competence: objective structured clinical examinations. Medical Education 2004;38:199-203 6. Bibliography American Board of Internal Medicine. Blueprints. http://www.abim.org/about/examinfo/blueprint.aspx Australian Medical Council Clinical Examination Specifications Booklet, 2014. http://www.amc.org.au/files/f71a8c3dc8760292bfb6e4378b9ed2199a6 740d8_original.pdf Blueprint for the General Medical Council s Professional and Linguistics Assessment Board examination. Bordage G, Brailovsky C, Carretier H, Page G. Content validation of key features on a national examination of clinical decision-making skills. Academic Medicine 2005;70:276 281 Coderre S, Woloschuk W, Mclaughlin K. Twelve tips for blueprinting. Medical Teacher 2009; 31(4):322 324 Dauphinee WD, Blackmore DE, Smee SM, Rothman AI, Reznick RK. Using the judgments of physician examiners in setting the standards for a national multi-center high stakes OSCE. Adv Health Sci Education: Theory Prac 1997;2:201-11 Eight Domains of Good Professional Practice as devised by Medical Council, 2010 Friedlich M, Smee S, and Wood T. Is it feasible to include a technical skill station on a national licensing examination? The American Journal of Surgery 2007;198:86-89 18

General Medical Council 2014. Review of the PLAB test: final report. Code: GMC/PLABR/0914 General Medical Council PLAB Test Blueprint, 2014. http://www.gmcuk.org/doctors/plab/blueprint.asp General Medical Council. The state of medical education and practice in the UK report: 2013. GMC, 2013.. www.gmcuk.org/state_of_medicine_final_web.pdf_44213427.pdf Good Medical Practice in Seeking Informed Consent to Treatment. Medical Council 2008. https://www.medicalcouncil.ie/news-and- Publications/Publications/Information-for-Doctors/Good-Medical- Practice-in-Seeking-Informed-Consent-to-Treatment-pdf.pdf Good Medical Practice, 2013 GMC. www.gmcuk.org/guidance/good_medical_practice.asp Guide to Professional Conduct and Ethics for Registered Medical Practitioners, Medical Council 2009. https://www.medicalcouncil.ie/news-and- Publications/Publications/Information-for-Doctors/Guide-to- Professional-Conduct-and-Ethics-for-Registered-Medical- Practitioners.pdf Guidelines for Medical Education and Training for Interns, Medical Council, 2011 Guttormsena S, Beyelera C, Bonvinb R, Fellera S, Schirloc C, Schnabela K, Schurtera T and Berendonka C. The new licensing examination for human medicine: from concept to implementation. Swiss licensing examination for human medicine. Swiss Med Wkly 2013;143:w13897 Holmboe E and Zaborowski MA. Recommendations for the examinations for specialists and generalized divisions. 2012 Intern Coordinator and Tutor Network Project Report, Medical Council 2006. https://www.medicalcouncil.ie/news-and- Publications/Publications/Information-for-Doctors/Intern-Coordinatorand-Tutor-Network-Project-Report-pdf.pdf Intern Handbook / Information and Logbook for Interns, Medical Council, 2008. http://www.medicalcouncil.ie/news-and- Publications/Publications/Archive-Publications/Intern-Handbook- Information-and-Logbook-for-Interns-2008.pdf Kovacs E, Schmidt E, Szocska G, Busse R, McKeeM and Legido- 19

Quigley H. Licensing procedures and registration of medical doctors in the European Union. Clinical Medicine 2014;14(3):229 38 Lillis S, Roblin H. Progress of successful New Zealand Registration Examination (NZREX Clinical) candidates during their first year of supervised clinical practice in New Zealand. N Z Med J. 2014;127(1399):36-42 Medial Education, Training and Practice in Ireland 2008-2013. A Progress Report, Medical Council. https://www.medicalcouncil.ie/news-and- Publications/Publications/Education-Training/Medical-Education- Training-and-Practice-in-Ireland-A-Progress-Repropt-.pdf Medical Council of Canada Qualifying Examination. http://mcc.ca/examinations/mccqe-part-ii/application-information/ Medical Council of Canada, Draft Blueprint Project. 2013. http://mcc.ca/wp-content/uploads/blueprint-report-draft.pdf Medical Deans of Australia and New Zealand Competencies Reports, 2014. http://www.medicaldeans.org.au/projects-activities/completedprojects/competencies Mookherjee S, Chang A, Boscardin CK, Hauer KE. How to develop a competency-based examination blueprint for longitudinal standardized patient clinical skills assessments. Med Teach. 2013 Nov;35(11):883-90 National Intern Curriculum, 2012 National Intern Education and Training Agreement, 2014-2015 NZREX Clinical Handbook for Candidates, Medical Council of New Zealand, Wellington, 2014. https://www.mcnz.org.nz/assets/news-and- Publications/Booklets/NZREX-Candidate-Handbook.pdf PRES Handbook, Medical Council, 2015. http://www.medicalcouncil.ie/information-for-doctors/examinations- /PRES-Handbook-Jan-2015.pdf Professional and Linguistics Assessment Board, General Medical Council, UK. http://www.gmc-uk.org/doctors/plab.asp Professional. Guidelines for Doctors. The Medical Council, Dublin, 2011. https://www.medicalcouncil.ie/information-for- Doctors/Professional-/Professional-- Guidelines.pdf 20

Pugh D and Smee S (2013). Guidelines for the development of Objective Structured Clinical Examination (OSCE) Cases. Ottowa: Medical Council of Canada Sonderen MJ, Denessen E, Cate, OT, Splinter T, and Postma CT. The clinical skills assessment for international medical graduates in The Netherlands. Medical Teacher 2009;31(11):e533-e538 Talking about good professional Practice. Medical Council, 2014. https://www.medicalcouncil.ie/news-and- Publications/Publications/Annual-Reports-Statistics-/Talking-aboutgood-professional-practice.pdf The state of medical education and practice in the UK. GMC. 2011. www.gmc-uk.org/publications/10586.asp The Trainee Doctor, GMC, 2011. www.gmcuk.org/trainee_doctor.pdf_39274940.pdf Tomorrow s Doctors, GMC 2009. www.gmcuk.org/education/undergraduate/tomorrows_doctors.asp UK Foundation Programme Curriculum, GMC 2014. www.foundationprogramme.nhs.uk/index.asp?page=home/keydocs United States Medical Licensing Examination. www.usmle.org/ USMLE Step 2 Clincial Skills. Content Description and General Information. 2014. www.usmle.org/pdfs/step-2- ck/2014_step2ck_content.pdf WFME Global Standars for Quality Improvement in Medical Education. European Specifications for Basic and Postgraduate Medical Education and Continuing Professional Development. Denmark, 2007 WHO Patient Safety Curriculum Guide for Medical Schools, 2009. www.who.int/patientsafety/education/curriculum/en/ Your Training Counts, Results of the National Trainee Experience Survey, Medical Council, 2014. https://www.medicalcouncil.ie/education/career-stage- Postgraduate/Your-Training-Counts/Your-Training-Counts.pdf 21

Appendix A: Printable Version of Sheet 1 of PRES Blueprint Theme Patient Safety / Quality Care Relating to patients Communication/ Interpersonal skills Communication / Teamwork Management Professionalism Clinical skills Scholarship General Medicine / Surgery Paediatrics Obstetrics / Gynaecology Psychiatry / Behavioural General Practice Cardiovascular and Peripheral Vascular Respiratory Gastrointestinal Renal Neurological Musculoskeletal Endocrine Dermatology Haematology / Oncology Infectious diseases Breast Genitourinary Gerontology Surgical practice Emergency Medicine Ophthalmology / ENT PRES Level 3 Topics Data Interpretation Communication Skills Practical Skills Choose content from columns below

Appendix B (1): Printable Version of Sheet 2 of Blueprint: Data Interpretation (PRES Level 3) Data Interpretation: PRES Level 3 Station Number Station # Station # Station # Station # Station # Station # Theme Patient Safety / Quality Care Relating to patients Communication/ Interpersonal skills Communication / Teamwork Management Professionalism Clinical skills Scholarship General Medicine / Surgery Paediatrics Obstetrics / Gynaecology Psychiatry / Behavioural General Practice Cardiovascular and Peripheral Vascular Respiratory Gastrointestinal Renal Neurological Musculoskeletal Endocrine Dermatology Haematology / Oncology Infectious diseases Breast Genitourinary Gerontology Surgical practice Emergency Medicine Ophthalmology / ENT 23

Appendix B (2): Printable Version of Sheet 2 of Blueprint: Communication Skills (PRES Level 3) Communication Skills: PRES Level 3 Station Number Station # Station # Station # Station # Station # Station # Theme Patient Safety / Quality Care Relating to patients Communication/ Interpersonal skills Communication / Teamwork Management Professionalism Clinical skills Scholarship General Medicine / Surgery Paediatrics Obstetrics / Gynaecology Psychiatry / Behavioural General Practice Cardiovascular and Peripheral Vascular Respiratory Gastrointestinal Renal Neurological Musculoskeletal Endocrine Dermatology Haematology / Oncology Infectious diseases Breast Genitourinary Gerontology Surgical practice Emergency Medicine Ophthalmology / ENT 24

Appendix B (3): Printable Version of Sheet 2 of Blueprint: Practical Skills (PRES Level 3) Practical Skills: PRES Level 3 Question Number 1 2 3 4 5 6 Theme Patient Safety / Quality Care Relating to patients Communication/ Interpersonal skills Communication / Teamwork Management Professionalism Clinical skills Scholarship General Medicine / Surgery Paediatrics Obstetrics / Gynaecology Psychiatry / Behavioural General Practice Cardiovascular and Peripheral Vascular Respiratory Gastrointestinal Renal Neurological Musculoskeletal Endocrine Dermatology Haematology / Oncology Infectious diseases Breast Genitourinary Gerontology Surgical practice Emergency Medicine Ophthalmology / ENT 25