Remedial education Assessor feedback IPAC Conference 2013 Dr Steven Lillis Medical Adviser, MCNZ Nikita Takai Professional Standards Coordinator, MCNZ
Background Results of remediation
New Zealand 13,500 active medical practitioners All are covered by the same legislation (Health Practitioners Competency Assurance Act 2003) All required to have Annual Practicing Certificate issues by the responsible authority (The Medical Council of New Zealand)
The Act (HPCAA 2003) Requires the authority to review the competence of a health practitioner when concerns have been raised about competence Requires the authority to decide if the practitioner meets or does not meet the required standard on the basis of the review.
The Act (HPCAA 2003) Permits the authority (MCNZ) to order Competence program Recertification program
Competence/recertification program Pass any examination or assessment or both Complete a period of practical training Undertake a course of instruction Undertake a period of supervised practice Allow another health practitioner to examine clinical notes.
Who gets notified? H.D.C 47% Other 24% MCNZ Public 13% Health practitioner 16%
Process... Notification (200) Complaints triage Considered by Council (70) Assessment ordered (40)
Structuring the assessment Council orders assessment with areas of concern Converted into Terms of Reference that guide the assessment.
Performance Assessment Team 3 people, 2 peers and a lay person over 1 2 days. At the place of work of the doctor
Performance Assessment Team Always includes: Interview Notes review Observation of consultations Case based oral Closing interview
Performance Assessment Team Always includes: Interview Notes review Observation of consultations Case based oral Closing interview
Performance Assessment Team Depending on type of concerns, may include: Observation of procedures Prescribing addictive medications review Professional interactions assessment Complementary and Alternative Medicine
Outcome RECOMMENDATIONS: Dr XXX needs remediation in communication skills and in understanding how to record accurate clinical notes.
Failed assessments...the results of 24 consecutive cases of failed assessments.
Distribution Category 1 (no concerns) Category 2 (concerns in some areas of practice) 14 Category 3 (widespread concerns) - 10
Age
Age Variable This study % National data % Age (p < 0.0001) < 30 0 11 30-39 4 23 40-49 30 27 50-59 25 26 60-69 33 11 70 + 8 2
Gender
Gender Variable This study % National data % Gender (p = 0.0001) Male 79 60 Female 21 40
Scope of practice Variable This study % National data % General practice 25 24 Psychiatry 8 5 General Surgery 8 2 Internal medicine 4 7 Radiology 4 3 Not vocationally registered 50 32
Nature of concerns (top 6) Nature % of doctors Clinical records 65 Communication 52 Organisational 48 Clinical knowledge 43 Clinical reasoning 43 Prescribing 39
Progress 24 failed assessment 19 entered remediation 5 did not engage in remediation
12 months Education program Detailed remediation plan Monthly education supervisor meetings and reports A range of possible tasks: Audit Communication skills course Joining peer groups Notes reviews
Education supervisors Experienced clinicians Same or very similar scope Experience in delivering education Attend training with MCNZ Supported by case managers and advisers Remedial education is fundamentally different
Progress 24 failed assessment 5 did not engage in remediation 19 entered remediation 13 had satisfactory monthly reports 6 had formal 2 nd assessment
Same format as 1 st 2 nd Assessment One member of previous team included
Progress 24 failed assessment 5 did not engage in remediation 19 entered remediation 13 had satisfactory monthly reports 6 had formal 2 nd assessment 5 were found not to be competent
Doctor 1st assessment 2nd Assessment A Category 2 Category 3 B Category 2 Category 1 C Category 3 Category 2 D Category 3 Category 2 E Category 3 Category 3 F Category 3 Category 3
Doctor 1st assessment 2nd Assessment A Category 2 Category 3 B Category 2 Category 1 C Category 3 Category 2 D Category 3 Category 2 E Category 3 Category 3 F Category 3 Category 3
Doctor 1st assessment 2nd Assessment A Category 2 Category 3 B Category 2 Category 1 C Category 3 Category 2 D Category 3 Category 2 E Category 3 Category 3 F Category 3 Category 3
Doctor 1st assessment 2nd Assessment A Category 2 Category 3 B Category 2 Category 1 C Category 3 Category 2 D Category 3 Category 2 E Category 3 Category 3 F Category 3 Category 3
Conclusion Assessment and remediation dependent on legislation Workplace based assessment is feasible and can be robust Multiple modes of assessment in workplace based assessment Remediation requires good structure and process
Assessors Dr Steven Lillis Medical Adviser, MCNZ Nikita Takai Professional Standards Coordinator, MCNZ
Why? Assessors are contractors to MCNZ Users of methods of assessment Many are College assessors Lay assessors have unique perspective Independent and informed opinion on assessment methods
Process of the assessment Variable Appropriate Reasonable Thorough Unobtrusive Decision Likert scale 5 point scale used
Process Feedback survey requested of all 37% response rate
Assessment tools Tools General assessment Retrospective notes review Case based oral Prescribing Professional interactions Communication skills Peer ratings Observation of consults Practice systems Variable Appropriate Instructions Apply Reliability
Assessment tools Tools General assessment Observation of consults Retrospective notes review Case based oral Prescribing Professional interactions Communication skills Peer ratings Practice systems Variable Appropriate Instructions Apply Reliability
General assessment Opening interview Observation of consultations Retrospective notes review Case based oral Peer ratings Closing interview.
General assessment
Record review 20 sets of clinical notes (letters to GPs and to other referrers for hospital doctors) Randomly chosen Last 3 months
Record review
Lasts 1 2 hours Case based oral Based on any clinical material uncovered in the retrospective notes review or observation of consultations Uses principles of Script concordance testing
Script concordance testing You saw a 48 year old male with 3 days of cough, temperature of 37.5, heart rate of 92, respiratory rate of 16 and no respiratory findings. If the temperature had been 38.5 and the respiratory rate 24, how would this have changed your management?
Case based oral
Prescribing Analysis of prescribing data Modified case based oral Role plays The newly diagnosed hypertensive man aged 45. Etc
Prescribing
Communication skills Explorative semi-structured interview Covers areas of provision of information to patients patient consent power sharing patient rights privacy issues cultural sensitivity interpersonal boundaries.
Easy to game Communication skills Useful for International Medical Graduates (IMG)
Communication skills
Communication skills Not commonly used IMG with English as second language
Peer ratings
Peer ratings Removed due to concerns from assessors Difficult to administer Replaced by standard Multi-source feedback
Observation of consultations
Observation of consultations Standard Mini-CEX form Applicable to ward round, out-patient clinic, General practitioner type consult
Practice systems The objective of this assessment tool is to assess the medical practitioner s systems essential to patient safety with regard to: Premises Equipment and its safe use and maintenance Access and availability Record systems.
Practice systems - example Criteria for medical assessors met part met The practice has and uses correctly either a steam steriliser or autoclave for sterilisation of reusable instruments and materials (or makes appropriate alternative arrangements) Sterile or surgically clean instruments are stored in a manner that ensures maintenance of their sterility Lighting is adequate for the procedures performed not met n/a
Practice systems
Conclusion Limited by respondent numbers Confidence in process Confidence in the methods Believe the outcome is both fair and helpful