Training Evaluation Framework and Tools (TEFT) www.go2itech.org/resources/teft MEPI Webinar PRIME-K presentation December 2012
Focus on Outcomes Human Resources for Health TWG: to date PEPFAR has supported an estimated 3.7 million training and retraining encounters for health care workers. Despite this great achievement, there has been little to no evaluation of these training interventions. Framework Practical guidance Tools Case examples
ENVIRONMENT HEALTH SYSTEM / POPULATION ORGANIZATION INDIVIDUAL Organization systems improvements Population level systems improvements Training/ Educational Intervention Individual knowledge, attitude, skill Individual Individual Organization - Population level Organization (impacts) Population Outcomes (impacts) TRAINING EVALUATION FRAMEWORK
Training/ Educational Intervention Individual knowledge, attitude, skill Individual Individual TRAINING EVALUATION FRAMEWORK
ORGANIZATION INDIVIDUAL Organization systems improvements Training/ Educational Intervention Individual knowledge, attitude, skill Individual Individual Organization Organization (impacts) TRAINING EVALUATION FRAMEWORK
HEALTH SYSTEM / POPULATION ORGANIZATION INDIVIDUAL Organization systems improvements Population level systems improvements Training/ Educational Intervention Individual knowledge, attitude, skill Individual Individual Organization Organization (impacts) Population level Population (impacts) TRAINING EVALUATION FRAMEWORK
Situational Factors Trainee background knowledge Trainee experience Intrinsic motivation Patient education level Patient, trainee family demands Management support HR staffing levels, salaries, burnout Available drugs, supplies, equipment and infrastructure Facility systems appointments, records, flow, referrals Patient needs National, regional, community systems labs, supply chain National, regional policies Partner programs Available health workforce, including informal private, attrition issues Pre-service program Retention factors, e.g., pay scales Political instability Prevalent disease Natural disasters Food availability Seasonal changes Patient access to food, transportation Available community support resources TRAINING EVALUATION FRAMEWORK
ENVIRONMENT HEALTH SYSTEM / POPULATION ORGANIZATION INDIVIDUAL Organization systems improvements Population level systems improvements Training/ Educational intervention Individual knowledge, attitude, skill Individual Individual Organization Population level Organization (impacts) Population Outcomes (impacts) SITUATIONAL FACTORS INDIVIDUAL Trainee background, knowledge, experience, education Intrinsic motivation Family demands ORGANIZATION Management support HR staffing levels, salaries, burnout Available drugs, supplies, equipment and infrastructure Facility systems appointments, records, flow, referrals Patient needs HEALTH SYSTEM/POPULATION National, regional, community systems labs, supply chain National, regional policies Partner programs Available health workforce, including informal private, attrition issues Pre-service program Retention factors, e.g. pay scales ENVIRONMENT Political instability Prevalent disease Natural disasters Food availability Seasonal changes Patient access to food, transportation Available community support resources TRAINING EVALUATION FRAMEWORK
Training on HIV Staging HCW can correctly stage patients with HIV HCW correctly initiates ARV for eligible patients more often Patients tx by trained HCWs Increased CD4 TRAINING EVALUATION FRAMEWORK
ORGANIZATION INDIVIDUAL Facility-level Systems Improvements: Staging checklist used Training on HIV Staging HCW can correctly stage patients with HIV HCW correctly initiates ARV for eligible patients more often Patients tx by trained HCWs Increased CD4 Facility increase in correctly initiated eligible patients Facility-level: Increased CD4 TRAINING EVALUATION FRAMEWORK
ENVIRONMENT HEALTH SYSTEM / POPULATION ORGANIZATION INDIVIDUAL Facility-level Systems Improvements: Staging checklist used Populationlevel Systems Improvements: All clinics use new checklist Training on HIV Staging HCW can correctly stage patients with HIV HCW correctly initiates ARV for eligible patients more often Patients tx by trained HCWs Increased CD4 Facility increase in correctly initiated eligible points Populationlevel increase in correctly initiated Facility-level: Increased CD4 Populationlevel: Reduced HIV morbidity TRAINING EVALUATION FRAMEWORK
Situational Factors Background knowledge, experience Intrinsic motivation Patient education level Family demands Management support HR staffing levels, salaries, burnout Available drugs, supplies, equipment and infrastructure Facility systems appointments, records, flow, referrals Patient needs National, regional, community systems labs, supply chain National, regional policies Partner programs Available health workforce, including informal private, attrition issues Pre-service program Retention factors, Political instability Prevalent disease Natural disasters Food availability Seasonal changes Patient access to food, transportation Available community support resources e.g., pay scales TRAINING EVALUATION FRAMEWORK
6 Planning Steps Which training to evaluate Step 1 Anticipated Outcomes Step 2 Situational Factors Step 3 Evaluation Considerations How to evaluate your training Step 4 Define Evaluation Questions and Indicators Step 5 Evaluation Design and Methods Step 6 Evaluation Plan
Situational Factors Worksheet Factors that might affect the evaluation of effectiveness of the training intervention Individual Level Trainee background: Knowledge, experience, education Trainee life circumstances: Family demands, health status, attitude, motivation Other: (List) What is the issue and what can you do about it in your evaluation? Issue: What to do: Issue: What to do: Issue: What to do:
6 Planning Steps Which training to evaluate Step 1 Anticipated Outcomes Step 2 Situational Factors Step 3 Evaluation Considerations How to evaluate your training Step 4 Define Evaluation Questions and Indicators Step 5 Evaluation Design and Methods Step 6 Evaluation Plan
Questions and Indicators Template (Maternal and child health training) General Evaluation Questions Did the maternal and child health training result in trainee capacity to provide quality health care services (i.e. according to national guidelines)? (Individual level) More Specific Evaluation Questions Did the trained health care workers ( trainees ) demonstrate competency in key curriculum areas? (knowledge/skills) Were trainees able to provide quality care at worksites? (Performance) Very Specific Evaluation Questions Were trainees able to achieve a passing score on the knowledge test? Did trainees demonstrate passing skill score in skills lab (i.e. controlled environment)? What percentage of trainees received satisfactory ratings for their clinical practice by their supervisors/ mentors. Anticipated Outcomes Health care workers have knowledge and skills to treat most common concerns of MCH patients. Trainees provide quality care to MCH patients. Outcome Indicators % of trainees achieving passing scores on knowledge tests. % of trainees achieving passing scores on skills test. % of trainees who are observed to correctly perform 80% of the key competencies.
Core development team members: Assistance from: Acknowledgements Francie Petracca, Ph.D. Tom Perdue, MPH Vivian Bertman, MSW; Shay Bleumer-Miroite, MPH; Michelle Desmond, MSW, MPH; Marrianne Holec; Ellen MacLachlan, PhD; Stacey Lissit, MPH, MS; Hope O Brien, MPA, MPH; and Vickery Prongay, MPA Feedback from staff at: OGAC; USAID; Prime-K; University Research Corporation; IntraHealth; University of California, San Francisco; I-TECH country offices Funding and support provided by the Health Resources and Services Administration (HRSA) and the PEPFAR Human Resources for Health Technical Working Group.
The I-TECH Training Evaluation Framework and Tools (TEFT) Download the framework and tools at www.go2itech.org/resources/teft