TO DEVELOP AND VALIDATE A LEARNING AND ASSESSMENT INSTRUMENT A Delphi-Study Overview Purpose Method & Design Research Protocol Phase 1: Pre-Design Phase 2: Delphi Phase 3: End-Design Expert-panel Results Conclusion Discussion 2 13-10-2016 Midwifery Learning and Assessment Instrument 1
Background Embo s Model Embo Mieke. 2015. Integrating workplace learning, assessment and supervision in health care education. http://pub.maastrichtuniversity.nl/700fdd2c -b660-48cf-ad32-2a9f4effff95 3 13-10-2016 Midwifery Learning and Assessment Instrument Background Embo s Model Implementation in Midwifery Schools Flanders Midwifery Competency Framework (11 competencies) 4 13-10-2016 Midwifery Learning and Assessment Instrument 2
Background Research team: OIT Education Innovation trajectory Digitalising workplace learning Joint effort: Midwifery + Speech Pathology 5 13-10-2016 Midwifery Learning and Assessment Instrument Purpose validation study Workplace learning: learning & assessment Postnatal care competencies in the Maternity Ward Valdidation by educational and professional experts Determining students readiness to graduate Evidence-based + uniform way training and assessing students Ultimate Goal = QUALITATIVE EDUCATION + MIDWIFERY CARE 6 13-10-2016 Midwifery Learning and Assessment Instrument 3
Method & Design Research Protocol Flanders Midwifery Competency Framework 2 EVALUATION CRITERIA: Relevance en Feasibility Curriculum Learning Outcomes Inventory of Medical Antwerp School Pre-design ROUND 1 ROUND 2 ROUND 3 End-design Midwifery Assessment instruments DELPHI-STUDY PHASE 1 PHASE 2 PHASE 3 7 13-10-2016 Midwifery Learning and Assessment Instrument Method & Design Research Protocol Flanders Midwifery Competency Framework Curriculum Learning Outcomes Inventory of Medical Antwerp School Midwifery Assessment instruments Pre-design 11 Competencies 2 EVALUATION CRITERIA: relevance en haalbaarheid 1. Physiology, care and guidance ROUND 2. 1 Risk detection ROUND and 2 Risk Selection ROUND 3 3. Pathology, care and guidance 4. Psycho-social context 5. Ethics and legislation 6. Health promotion 7. Collaboration and communication DELPHI-STUDY 8. Coaching 9. Personal and professional development 10. Evidence-based care 11. Quality and care End-design PHASE 1 PHASE 2 PHASE 3 8 13-10-2016 Midwifery Learning and Assessment Instrument 4
Method & Design Research Protocol Flanders Midwifery Competency Framework Behavioural Cues Curriculum Learning Outcomes Inventory of Medical Antwerp School Midwifery Assessment instruments Pre-design Knowledge, skills and attitudes 11 Competencies 2 EVALUATION CRITERIA: relevance en haalbaarheid 1. Physiology, care and guidance ROUND 2. 1 Risk detection ROUND and 2 Risk Selection ROUND 3 3. Pathology, care and guidance 4. Psycho-social context 5. Ethics and legislation 6. Health promotion 7. Collaboration and communication DELPHI-STUDY 8. Coaching 9. Personal and professional development 10. Evidence-based care 11. Quality and care End-design 89 ASSESSMENT CRITERIA PHASE 1 PHASE 2 PHASE 3 9 13-10-2016 Midwifery Learning and Assessment Instrument Method & Design Research Protocol 2 evaluation criteria Flanders Midwifery Competency Framework Curriculum 1. RELEVANCE Learning Outcomes Inventory Validity of Medical Antwerp Reliability School Sensitivity Midwifery 2. FEASIBILITY Assessment instruments Measurable Acceptable Pre-design ROUND Appropriate 1 for ROUND assessing 2 observable ROUND 3 behaviour? Sufficient concrete for reliable assessment? Formulated without overlap? Measurable in practice? Acceptable for students/ assessors? DELPHI-STUDY End-design 6-point LIKERTSCALE PHASE 1 PHASE 2 PHASE 3 10 13-10-2016 Midwifery Learning and Assessment Instrument 5
Methode en Design Onderzoeksprotocol Analysis Beoordelingslijst Artevelde 1. Descriptive statistics Voorontwerp Beoordelingslijst Median Artsen scores Level of consensus (25 th Overige beoordelingslijsten and 75 th tertile) Mann-Whitney U-test 2. Thematic content-analysis Comments FASE 1 Medianscores 4 AND Consensus Ronde 1 Ronde 2 Ronde 3 1 2 3 4 5 6 strongly strongly disagree agree Eindontwerp 11 13-10-2016 Midwifery Learning and Assessment Instrument Method & Design Expert-panel University non-university GYN - PED Expert-panel N = 90 Workplace experts N = 60 Educational Experts N = 30 15 Maternity Wards Minimum 3-years Experience 12 Midwifery Schools Drop-out percentage 70%: 30 experts Doctors N = 20 Midwives N = 20 Teachers N = 24 Experts in competency-based education N = 6 12 13-10-2016 Midwifery Learning and Assessment Instrument 6
Results DELPHI-STUDY Beoordelingslijst Artevelde Respons rate: 67% - 71% - 58% (n = 32/34/28) Beoordelingslijst Artsen Characteristics experts: Overige beoordelingslijsten +/- 54 % > 10 years Maternity Ward +/- 51 % > 10 years assessing students Voorontwerp 2% 13% 1 pediatrician + 2 gynaecologists Ronde 1 Ronde 2 Ronde 3 Eindontwerp 41% Workplace n=12/11/10 44% DELPHI-STUDIE 20 midwives 19 teachers 6 experts competency-based education FASE 1 FASE 2 FASE 3 Education n=20/23/18 13 13-10-2016 Midwifery Learning and Assessment Instrument Results Descriptive statistics: All competencies were rated as relevant and feasible. All assessment criteria were rated as relevant. Consensus was obtained by all but one assessment criteria. A Mann-Whitney U-test pointed out little differences between groups. Non-acceptance of criteria was due to lack of feasibility. 14 13-10-2016 Midwifery Learning and Assessment Instrument 7
Results Thematic content-analysis of the comments: Many remarks were made! (n= 331/106/31) Poor applicability for assessment Limited or infrequent learning opportunities Irrelevant criteria for maternity ward Impossible to perform autonomously by students The lack of concreteness of the formulation The lack of transparency because of overlap 15 13-10-2016 Midwifery Learning and Assessment Instrument Results Thematic content-analysis of the comments: Suggestions by the experts: Combining or splitting certain criteria Changing terminology (e.g. caretaker in stead of patient) Adding additional criteria Inclusion of attitudes after competencies Adding a list with clarification of the criteria 16 13-10-2016 Midwifery Learning and Assessment Instrument 8
Results DELPHI-STUDY 17 criteria represented 10 criteria represented 1 st ROUND 2 nd ROUND 3 th ROUND 89 53 56 Competency-based structure remained Amount of assessment criteria from 89 over 53 to 56 17 13-10-2016 Midwifery Learning and Assessment Instrument 18 13-10-2016 Midwifery Learning and Assessment Instrument 9
Discussion Michels (2012) found comparable results. Reinforcement of the Flemish Midwifery Competency Framework, but also a revisions. Collaboration & communication with caretakers & caregivers. Assessment of students in competency-based education: The assessment is also competency-based Competency-based framework needs translation into readily observable assessment criteria. Performance-based criteria are more descriptive and are directly linked to observable behaviour ( What student does ) Delicate balance between developing a practical tool AND assessing the necessary behavioural cues. 19 13-10-2016 Midwifery Learning and Assessment Instrument Discussion Workplace challenge: Dealing with limited or infrequent learning opportunities. Autonomy by students in the workplace. simulation-based learning Research challenge: Difficult to sensitize the workplace 20 13-10-2016 Midwifery Learning and Assessment Instrument 10
Discussion Methodological considerations Little consensus on Delphi-research 3 features (Dalkey, 1969): Anonimity (dealing with social pressure) Controlled feedback (suggestive?) Statistical group response (reducing group pressure for conformity) Drop-out in Delphi (70% is common) 21 13-10-2016 Midwifery Learning and Assessment Instrument Future A complete set of instruments (Shumway & Harden, 2003) Valid Reliable (effective and accurate trust!) Impact on learner and educational programme Practical (user-friendly/ uniformity) Uniform / generic assessment criteria International partners E-portfolio (Medbook) Continuous professional development 22 13-10-2016 Midwifery Learning and Assessment Instrument 11
23 13-10-2016 Midwifery Learning and Assessment Instrument Embo, M.P. (2015). Integrating workplace learning, assessment and supervision in health care education. Maastricht University Press; http://pub.maastrichtuniversity.nl/700fdd2c-b660-48cf-ad32-2a9f4effff95 Embo, M. P., Driessen, E., Valcke, M., & van der Vleuten, C. (2014). A Framework To Facilitate Self-Directed Learning and Supervision In Midwifery Practice: A Qualitative Study of Supervisors' Perceptions. Nurse Education in Practice. doi: 10.1016/j.nepr.2014.01.015 Embo, M. P., Driessen, E. W., Valcke, M., & Van der Vleuten, C. P. (2010). Assessment and feedback to facilitate self-directed learning in clinical practice of Midwifery students. Med Teach, 32(7), e263-269. doi: 10.3109/0142159X.2010.490281 Dalkey, N. C., 1969. The Delphi Method: an experimental study of group opinion, Santa Monica, USA: RAND. Shumway, J.M.; Harden & R.M. (2003). AMEE Guide No. 25: The assessment of learning outcomes for the competent and reflective physician. Medical Teacher, Vol. 25, No. 6, 2003, pp. 569-584 Michels, N.R.M., Driessen, E.W., Denekens, J., Van Gaal, L.F., Bossaert, L.L., & De Winter, B.Y. (2012). A Delphi study to construct a CanMEDS competence-based inventory applicable for workplace assessment. BMC Medical Education 12(86). VLOR, 2014. Midwifery education competency framework, Brussels: The Flemish Education Council. 24 13-10-2016 Midwifery Learning and Assessment Instrument 12