Jocelyn Lockyer PhD Senior Associate Dean, Education Professor, Department of Community Health Sciences University of Calgary 1
No financial conflicts of interest to report College of Physicians and Surgeons of Alberta Physician Achievement Review Program Since 1996 Instrument development, testing, and assessment 2
Collaborators CPSA John Swiniarski, Bryan Ward (deceased), Trevor Theman Physician Performance Committee University of Calgary Research Team..Claudio Violato, Herta Fidler Physician Advisor..Ray Lewkonia Pivotal Research Inc Steve Dennis 3
Describe the Alberta Physician Achievement Review (PAR) Program Discuss the evidence supporting PAR as a tool for screening and quality improvement Describe dissemination and uptake by others within and outside Canada Provide an opportunity to answer questions about PAR 4
Multisource feedback program (aka 360 degree evaluation) Questionnaire based assessment Feedback about observable behaviors is obtained every 5 years from 8 Physician colleagues (peers, referring MDs, referral MDs) 8 Co-workers (e.g., nurses, pharmacists, dieticians) 25 Patients Self 5
MSF Medical colleagues Self-assessment Co-workers Patients 6
First level screen which might Provides physicians with data about their practices to guide improvement Identify need for further assessments Source of unique and useful data that can inform many of the CanMEDs roles, particularly Communicator Collaborator Professional Medical expert Relatively inexpensive Adaptable Repeat administration (5 years) 7
General/family physicians (revised 2010) Surgical specialists Medical specialists Pediatrics Psychiatry All other medical specialists Anesthesiologists Episodic care providers Diagnostic Imaging Laboratory Medicine 8
Diagram 1 The College s PAR Program PAR Survey Response Feedback Self-Analysis PAR Committee Review Advice to Physician Individualized Assessment Office Peer Review Competence Assessment 9
Good assessment should offer 1. Validity or coherence 2. Reproducibility or consistency (reliability) 3. Equivalence 4. Feasibility 5. Educational effect 6. Catalytic effect 7. Acceptability Norcini et al, Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 Conference, Medical Teacher 2011 10
Criteria Validity Reliability Equivalence Acceptability Evidence Developed by physicians with input from physicians, health care professionals and patients. Factor analyses showing items correlate in intended ways to form scales. Positive but low correlation between sources. 22-25 patients, 8 colleagues and 8 co-workers for reliability suitable for formative (lower stakes) assessment based on G- and D-studies. Comparisons in Nova Scotia and Alberta. Comparisons by different schools of graduation. High response rates from all sources. >70% of physicians report obtaining valuable feedback. 11
Criteria Educational effect Catalytic effect Evidence 49% report making a change in practice as a direct result of PAR. Most changes on patient care, communication with patients, colleagues, and co-workers and stress management Improvement in scores when tested 5 years apart. Not studied. 12
Total 9816 Family medicine 4355 Surgeons 1161 Anesthesiologists 472 Medicine (and subspecialties) 1506 Pediatricians 442 Psychiatrists 617 Episodic care 842 Diagnostic imaging 271 Laboratory medicine 150 13
Dissemination >25 publications in peer review journals Adopted Nova Scotia (NS-PAR) Manitoba (M-PAR) Being Tested British Columbia Council of Teaching Hospitals of Ontario Royal College of Physicians and Surgeons of Canada Practice eligible route Tested Medical Council of Ireland Netherlands (Internal Medicine) Singapore (Emergency Medicine) 14
Questionnaire based assessment in which feedback about observable behaviors is provided by other physicians, co-workers (e.g., nurses, social workers), and patients Program that meets criteria for a good assessment Initiative that can be adopted and used by others but attention needs to be paid to Purpose Quality improvement/formative assessment How it functions in different settings with special reference to how physicians use the data to inform practice change 15
16