TO DEVELOP AND VALIDATE A LEARNING AND ASSESSMENT INSTRUMENT

Size: px

Start display at page:

Download "TO DEVELOP AND VALIDATE A LEARNING AND ASSESSMENT INSTRUMENT"

Harry Leonard
6 years ago
Views:

1 TO DEVELOP AND VALIDATE A LEARNING AND ASSESSMENT INSTRUMENT A Delphi-Study Overview Purpose Method & Design Research Protocol Phase 1: Pre-Design Phase 2: Delphi Phase 3: End-Design Expert-panel Results Conclusion Discussion Midwifery Learning and Assessment Instrument 1

Background Embo s Model Embo Mieke. 2015. Integrating workplace learning, assessment and supervision in health care education. http://pub.maastrichtuniversity.

2 Background Embo s Model Embo Mieke Integrating workplace learning, assessment and supervision in health care education. -b660-48cf-ad32-2a9f4effff Midwifery Learning and Assessment Instrument Background Embo s Model Implementation in Midwifery Schools Flanders Midwifery Competency Framework (11 competencies) Midwifery Learning and Assessment Instrument 2

Background Research team: OIT Education Innovation trajectory Digitalising workplace learning Joint effort: Midwifery + Speech Pathology 5 13-10-2016 Midwifery Learning and Assessment Instrument

3 Background Research team: OIT Education Innovation trajectory Digitalising workplace learning Joint effort: Midwifery + Speech Pathology Midwifery Learning and Assessment Instrument Purpose validation study Workplace learning: learning & assessment Postnatal care competencies in the Maternity Ward Valdidation by educational and professional experts Determining students readiness to graduate Evidence-based + uniform way training and assessing students Ultimate Goal = QUALITATIVE EDUCATION + MIDWIFERY CARE Midwifery Learning and Assessment Instrument 3

4 Method & Design Research Protocol Flanders Midwifery Competency Framework 2 EVALUATION CRITERIA: Relevance en Feasibility Curriculum Learning Outcomes Inventory of Medical Antwerp School Pre-design ROUND 1 ROUND 2 ROUND 3 End-design Midwifery Assessment instruments DELPHI-STUDY PHASE 1 PHASE 2 PHASE Midwifery Learning and Assessment Instrument Method & Design Research Protocol Flanders Midwifery Competency Framework Curriculum Learning Outcomes Inventory of Medical Antwerp School Midwifery Assessment instruments Pre-design 11 Competencies 2 EVALUATION CRITERIA: relevance en haalbaarheid 1. Physiology, care and guidance ROUND 2. 1 Risk detection ROUND and 2 Risk Selection ROUND 3 3. Pathology, care and guidance 4. Psycho-social context 5. Ethics and legislation 6. Health promotion 7. Collaboration and communication DELPHI-STUDY 8. Coaching 9. Personal and professional development 10. Evidence-based care 11. Quality and care End-design PHASE 1 PHASE 2 PHASE Midwifery Learning and Assessment Instrument 4

Method & Design Research Protocol Flanders Midwifery Competency Framework Behavioural Cues Curriculum Learning Outcomes Inventory of Medical Antwerp School Midwifery Assessment instruments Pre-design

5 Method & Design Research Protocol Flanders Midwifery Competency Framework Behavioural Cues Curriculum Learning Outcomes Inventory of Medical Antwerp School Midwifery Assessment instruments Pre-design Knowledge, skills and attitudes 11 Competencies 2 EVALUATION CRITERIA: relevance en haalbaarheid 1. Physiology, care and guidance ROUND 2. 1 Risk detection ROUND and 2 Risk Selection ROUND 3 3. Pathology, care and guidance 4. Psycho-social context 5. Ethics and legislation 6. Health promotion 7. Collaboration and communication DELPHI-STUDY 8. Coaching 9. Personal and professional development 10. Evidence-based care 11. Quality and care End-design 89 ASSESSMENT CRITERIA PHASE 1 PHASE 2 PHASE Midwifery Learning and Assessment Instrument Method & Design Research Protocol 2 evaluation criteria Flanders Midwifery Competency Framework Curriculum 1. RELEVANCE Learning Outcomes Inventory Validity of Medical Antwerp Reliability School Sensitivity Midwifery 2. FEASIBILITY Assessment instruments Measurable Acceptable Pre-design ROUND Appropriate 1 for ROUND assessing 2 observable ROUND 3 behaviour? Sufficient concrete for reliable assessment? Formulated without overlap? Measurable in practice? Acceptable for students/ assessors? DELPHI-STUDY End-design 6-point LIKERTSCALE PHASE 1 PHASE 2 PHASE Midwifery Learning and Assessment Instrument 5

6 Methode en Design Onderzoeksprotocol Analysis Beoordelingslijst Artevelde 1. Descriptive statistics Voorontwerp Beoordelingslijst Median Artsen scores Level of consensus (25 th Overige beoordelingslijsten and 75 th tertile) Mann-Whitney U-test 2. Thematic content-analysis Comments FASE 1 Medianscores 4 AND Consensus Ronde 1 Ronde 2 Ronde strongly strongly disagree agree Eindontwerp Midwifery Learning and Assessment Instrument Method & Design Expert-panel University non-university GYN - PED Expert-panel N = 90 Workplace experts N = 60 Educational Experts N = Maternity Wards Minimum 3-years Experience 12 Midwifery Schools Drop-out percentage 70%: 30 experts Doctors N = 20 Midwives N = 20 Teachers N = 24 Experts in competency-based education N = Midwifery Learning and Assessment Instrument 6

Results DELPHI-STUDY Beoordelingslijst Artevelde Respons rate: 67% - 71% - 58% (n = 32/34/28) Beoordelingslijst Artsen Characteristics experts: Overige

1 Ronde 2 Ronde 3 Eindontwerp 41% Workplace n=12/11/10 44% DELPHI-STUDIE 20 midwives 19 teachers 6 experts competency-based education FASE 1 FASE 2 FASE

and feasible. All assessment criteria were rated as relevant. Consensus was obtained by all but one assessment criteria.

7 Results DELPHI-STUDY Beoordelingslijst Artevelde Respons rate: 67% - 71% - 58% (n = 32/34/28) Beoordelingslijst Artsen Characteristics experts: Overige beoordelingslijsten +/- 54 % > 10 years Maternity Ward +/- 51 % > 10 years assessing students Voorontwerp 2% 13% 1 pediatrician + 2 gynaecologists Ronde 1 Ronde 2 Ronde 3 Eindontwerp 41% Workplace n=12/11/10 44% DELPHI-STUDIE 20 midwives 19 teachers 6 experts competency-based education FASE 1 FASE 2 FASE 3 Education n=20/23/ Midwifery Learning and Assessment Instrument Results Descriptive statistics: All competencies were rated as relevant and feasible. All assessment criteria were rated as relevant. Consensus was obtained by all but one assessment criteria. A Mann-Whitney U-test pointed out little differences between groups. Non-acceptance of criteria was due to lack of feasibility Midwifery Learning and Assessment Instrument 7

8 Results Thematic content-analysis of the comments: Many remarks were made! (n= 331/106/31) Poor applicability for assessment Limited or infrequent learning opportunities Irrelevant criteria for maternity ward Impossible to perform autonomously by students The lack of concreteness of the formulation The lack of transparency because of overlap Midwifery Learning and Assessment Instrument Results Thematic content-analysis of the comments: Suggestions by the experts: Combining or splitting certain criteria Changing terminology (e.g. caretaker in stead of patient) Adding additional criteria Inclusion of attitudes after competencies Adding a list with clarification of the criteria Midwifery Learning and Assessment Instrument 8

9 Results DELPHI-STUDY 17 criteria represented 10 criteria represented 1 st ROUND 2 nd ROUND 3 th ROUND Competency-based structure remained Amount of assessment criteria from 89 over 53 to Midwifery Learning and Assessment Instrument Midwifery Learning and Assessment Instrument 9

10 Discussion Michels (2012) found comparable results. Reinforcement of the Flemish Midwifery Competency Framework, but also a revisions. Collaboration & communication with caretakers & caregivers. Assessment of students in competency-based education: The assessment is also competency-based Competency-based framework needs translation into readily observable assessment criteria. Performance-based criteria are more descriptive and are directly linked to observable behaviour ( What student does ) Delicate balance between developing a practical tool AND assessing the necessary behavioural cues Midwifery Learning and Assessment Instrument Discussion Workplace challenge: Dealing with limited or infrequent learning opportunities. Autonomy by students in the workplace. simulation-based learning Research challenge: Difficult to sensitize the workplace Midwifery Learning and Assessment Instrument 10

11 Discussion Methodological considerations Little consensus on Delphi-research 3 features (Dalkey, 1969): Anonimity (dealing with social pressure) Controlled feedback (suggestive?) Statistical group response (reducing group pressure for conformity) Drop-out in Delphi (70% is common) Midwifery Learning and Assessment Instrument Future A complete set of instruments (Shumway & Harden, 2003) Valid Reliable (effective and accurate trust!) Impact on learner and educational programme Practical (user-friendly/ uniformity) Uniform / generic assessment criteria International partners E-portfolio (Medbook) Continuous professional development Midwifery Learning and Assessment Instrument 11

23 13-10-2016 Midwifery Learning and Assessment Instrument Embo, M.P. (2015). Integrating workplace learning, assessment and supervision in health care education.

12 Midwifery Learning and Assessment Instrument Embo, M.P. (2015). Integrating workplace learning, assessment and supervision in health care education. Maastricht University Press; Embo, M. P., Driessen, E., Valcke, M., & van der Vleuten, C. (2014). A Framework To Facilitate Self-Directed Learning and Supervision In Midwifery Practice: A Qualitative Study of Supervisors' Perceptions. Nurse Education in Practice. doi: /j.nepr Embo, M. P., Driessen, E. W., Valcke, M., & Van der Vleuten, C. P. (2010). Assessment and feedback to facilitate self-directed learning in clinical practice of Midwifery students. Med Teach, 32(7), e doi: / X Dalkey, N. C., The Delphi Method: an experimental study of group opinion, Santa Monica, USA: RAND. Shumway, J.M.; Harden & R.M. (2003). AMEE Guide No. 25: The assessment of learning outcomes for the competent and reflective physician. Medical Teacher, Vol. 25, No. 6, 2003, pp Michels, N.R.M., Driessen, E.W., Denekens, J., Van Gaal, L.F., Bossaert, L.L., & De Winter, B.Y. (2012). A Delphi study to construct a CanMEDS competence-based inventory applicable for workplace assessment. BMC Medical Education 12(86). VLOR, Midwifery education competency framework, Brussels: The Flemish Education Council Midwifery Learning and Assessment Instrument 12

Faculty perceptions of entrustable professional activities-based resident evaluations in Obstetrics and Gynecology residency

Faculty perceptions of entrustable professional activities-based resident evaluations in Obstetrics and Gynecology residency Lord MG, Lawrence AA, Murchison AB, Johnson IM Background 2014: ACGME Milestones