Program Design: Integrating Research and Evaluation Session A7: 9:30-10:45 Abiyou Kiflie Abiy Seifu, Gareth Parry
Faculty Gareth Parry, PhD Senior Scientist, IHI gparry@ihi.org @GJParry03 Abiyou Kiflie, MD, MPH Deputy Country Director, IHI Ethiopia Akiflie@ihi.org Abiy Seifu, MPH Lecturer School of Public Health, Addis Ababa University seifu9@gmail.com @AbiySe
Success!
A population-based, multifaceted strategy to implement antenatal corticosteroid treatment versus standard care for the reduction of neonatal mortality due to preterm birth in low-income and middleincome countries: the ACT cluster randomized trial Fernando Althabe et al The Lancet. 2015 Feb 20;385(9968):629-39 Fernando Althabe Interpretation Despite increased use of antenatal corticosteroids in low-birthweight infants in the intervention groups, neonatal mortality did not decrease in this group, and increased in the population overall. For every 1000 women exposed to this strategy, an excess of 3 5 neonatal deaths occurred, and the risk of maternal infection seems to have been increased Conclusions: Implementation of surgical safety checklists in Ontario, Canada, was not associated with a significant reductions in operative mortality or complications.
..described in the 1980s by American program evaluator Peter Rossi as the Iron Law of arguing that as a new model is implemented widely across a broad range of settings, the effect will tend toward zero.
Reduction in Effectiveness from Applying Same Fixed-Protocol Program in Different Contexts Innovation sample Evaluation sample Immediate wide-scale implementation Parry GJ, et al (2013).
Where Can a Model Be Amended to Work? Innovation sample Identify contexts in which it can be amended to work as we move from Innovation to Prototype to Test and Spread Parry GJ, et al (2013).
Implementation science at the crossroads Richard J Lilford BMJ Qual Saf. 2017 Nov 28:bmjqs-2017 Fernando Althabe What we need to come up with is interventions that will do more good than harm, without requiring a degree of specification that would be hard to replicate and/or that only works when implemented by a research team committed to the cause Increasing the generalisability of improvement research with an improvement replication programme John Øvretveit et al Quality and Safety in Health Care. 2011 Apr 1;20(Suppl 1):i87-91 Fernando Althabe If a quality improvement is found effective in one setting, would the same effects be found elsewhere? Could the same change be implemented in another setting?
Core Concepts & Detailed Tasks Core Concepts Use a reliable method to identify deteriorating patients in real time. Detailed Tasks and Local Adaptations MEWS >=5 MEWS >=4 When a patient is deteriorating, provide the most appropriate assessment and care as soon as possible 2 Nurses 1 Physician 1 Nurse 1 Physician 1 Physician Theory Action
1) Generating the pressure (will) for ICUs to take part 2) A networked community 3) Re-framing BSIs as a social problem 4) Approaches that shaped a culture of commitment 5) Use of data as a disciplinary force 6) Hard edges (4) Milbank Quarterly, 2011
From an Improvement Perspective: Learning Learn what is takes to bring about improvement. Social Change Improvement requires social change and that people are more likely to act if they believe. Context Matters Interventions need to be amended to local settings (contexts). Initial Concepts Concepts rather than fixed protocols are a good starting point for people to test and learn whether improvement interventions can be amended to their setting. We need: Theory-driven rapid-cycle formative evaluation
Salzburg Global Seminar Session 565 M. Rashad Massoud University Research Co., LLC
Framework for learning about improvement Objective for stakeholders Theory of change Evaluation Continuum Context Pure External Evaluation Highly Embedded Evaluation What How Improvement Program Design Evaluation Design
Improvers and Evaluators as best friends Evaluability Assessment With all key stakeholders: Agree the Theory of Change Five Core Design Components Agree the evaluation design, including: Agreeing on the evaluation questions Formative and/or summative approaches Availability and Use of Data Available human and financial resources Leviton LC et al Evaluability assessment to improve public health policies, programs, and practices. Annual Review of Public Health. 2010 Apr 21;31:213-33.
Evaluation Design 15 The What The How The Context Innovation phase: Model development typically takes place in a small number of settings, and evaluation questions should focus largely on The What To what extent can all the changes be implemented? Testing Phase: The aim is to identify where a model works, or can be amended What are barriers and What facilitators is the overall impact of the to work. Hence, although refining The What will occur, developing The How and to implementing the changes? model on patient outcomes? The Context will also be important. Which elements of the model had the greatest impact on patient outcomes? Spread and Scale-up Phase: The aim is to spread or scaling up the model in contexts earlier work has indicated it is likely to Within work or what be amended settings does to work. the To what extent can all the Here, the What and the Context should be well model developed, work, and or can the be focus amended will be changes be implemented? primarily on The How. to work? What are barriers and facilitators To what extent did the to implementing the changes implementation of the model locally? vary across settings?
Approaches to assess attribution 16 http://www.academyhealth.org/evaluationguide
17 Ethiopia Health Care Quality Initiative
Ethiopia Country Background Total Population, 2017: 94,228,000 GDP per capita, 2017: US$660 Neonatal Mortality Rate, 2016: 29 per 1000 live births Maternal Mortality Ratio, 2016: 412 per 100,000 live births Facility delivery: 26% Sources: World Population Review, WHO, EDHS 2016
Ethiopian Health Care System Ministry of Health Agencies Regional Health Bureaus Zonal Health Department Woreda Health Office
Improvements in Maternal and Child Health Ambitious initiatives of the FMoH led to a two-thirds decrease in child mortality between 1990 and 2012, thus achieving Millennium Development Goal 4 three years before the target year (2015) Critical progress in access and coverage However, rates of neonatal and maternal mortality remain unacceptably high Further progress will require more system-level change Across health system levels Across the MNH continuum of care Move beyond coverage high quality, patient-centered, equitable care
Program Components Creation of Ethiopian National Health Care Quality Strategy with the Ethiopian FMOH Aligned with the Ethiopian Health Sector Transformation Plan Builds on the existing quality and equity initiatives in the country Activate a culture of continuous improvement at all levels of the healthcare system Multi-level QI capability building training activities Launch and test large-scale results-focused collaboratives in maternal and neonatal health Demonstrate impact of QI methods to accelerate change in key priority area Create scalable woreda-wide model for operationalizing QI for national scale-up
Overall Project Driver Diagram Development and Implementation of Unified Quality Strategy Institutionalized Culture of Quality Government Ownership Improved Healthcare Outcomes and Improved Quality of Health Services All Other Drivers of Quality Services (WHO Building Blocks) QI Capability Building across the Ethiopian Health Sector at All Levels Demonstration of Use of QI on One Priority Agenda (MNH-focused Collaboratives to Reduce Maternal and Newborn Mortality)
MNH Collaborative Aims Short-term Aim End of Prototype Phase Medium-term Aim End of Test-of-Scale Phase Long-term Aim End of 5 Years Improve quality of antenatal care, delivery management, and postnatal care Improve management of complications related to leading causes of maternal and neonatal death Improve demand for care services through reduced delays in seeking and reaching quality care Reduce maternal and neonatal facility-based mortality in participating sites by 30% over a period of 30 months Reduce maternal and neonatal mortality across Ethiopia by 30% over a period of 5 years Habits of Continuous Improvement Culture of Continuous Improvement 30 months 5 years
MNH Collaborative Driver Diagram Increased Health Seeking Behavior Optimize the ability of the HEW to educate the community Community Engagement for awareness creation and positive influence Use culturally acceptable strategies to improve dissemination and uptake of key health messages Create positive experiences through every health encounter Utilize the Health Development Army structure to reach the house hold Use schools as a dissemination mechanism Use multimedia for Health education activities Use each facility visit to educate/counsel mothers towards raising their health seeking behavior Improved experience at care Reduce maternal and neonatal facility-based mortality in participating sites by 30% over a period of 30 months Improved mechanisms to reach appropriate level of health care facility Improved quality of care at health institutions (safe, effective, patientcentered, timely, efficient, equitable) Improved referral network Improving transportation mechanisms (ambulance and others) for immediate response Maximizing the potential of nearby health facilities to avoid unnecessary referral Create a culture of QI and leadership Availability of skilled and respectful health personnel Improve the reliability of the supply chain management system to deliver essential commodities all the time Availability of national guidelines, clinical protocols and job aids Create structure (QI teams, committees, plan) to facilitate and execute work Improve data quality through DQA s Create a learning platform for collaboration and routine use of data for improvement Increase the skills of health professionals and health managers to use QI methods and tools Organize learning collaborative among health facilities serving the same geographic areas (full Woreda Coverage) Training in key MNH national protocols Onsite mentorship to maintain skills and address skills gaps Maximize efficiency of existing facility staff Professionals get regular updates on the management and prevention of key causes of mortality Address gaps in essential commodities as defined in baseline assessment Dissemination of existing protocols and support for local development when necessary Timely identification, prevention and management of life threatening conditions to mothers and newborns Fast tracking/triaging/follow-up mechanism Reliable implementation of labor and delivery bundle Reliable implementation of the "MNH" checklists/relevant guidelines Support for a care delivery system that ensures respectful care for patients Incorporation of compassionate and respectful care (CRC) change ideas and training in learning sessions Clean, safe, comfortable spaces for patients and staff
PHCU + Hospital Unit ( scalable unit ) 1 Collaborative includes 7-11 QI teams (depending on # of participating hospitals) X1 Primary Hospital Woreda Health Office X2 Referral Hospital WHO sends 1-2 officials to participate in LS Referral Hospitals send 2 teams: -Neonatal (5 people) -Maternal (5 people) Primary Hospitals (when present) send 1 team Health Center Health Center Health Center Health Center Health Center HC and linked HP send 1 team: -3 people from HC -1 from each HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP = Health Post
Learning Collaborative Design Identify focus area and core indicators Conduct Baseline Assessment Address gaps in clinical and QI skills and supplies (training and procurement of essential supplies) Learning Session 1 Action Period 1 Learning Session 2 Action Period 2 Learning Session 3 Action Period 3 Learning Session 4 Finalize change package, publicize & spread Intensive coaching to support teams to improve system and skills gaps (visits, phone calls, engagement of program and supervisory managers, data collation & interpretation) 12-18 months
Learning Collaborative Core Indicators Process ANC (coverage) ANC (quality) Delivery Management (coverage) Delivery Management (quality)* Delivery Management (quality)* Delivery Management (quality)* Delivery Management (quality-sick newborns) PNC (coverage) PNC (quality-sick newborns) PNC (qualitypreterm/lbw) Antenatal care coverage four visits Measure Percentage of pregnant women tested for syphilis during ANC1 Proportion of births attended by skilled health personnel Proportion of deliveries with 100% compliance to on admission bundle Proportion of deliveries with 100% compliance to before pushing bundle Proportion of deliveries with 100% compliance to soon after birth bundle Proportion of neonates treated for birth asphyxia Percentage of women who attended postnatal care 48 hrs after delivery Proportion of neonates treated for sepsis Percentage of preterm or low birth weight infants put on KMC Data Source* HMIS HMIS HMIS Chart review with checklist Chart review with checklist Chart review with checklist HMIS or Delivery register HMIS HMIS or Delivery register Delivery register
Clinical Outcome/Impact Measures Institutional and community level (when available) measures of: Maternal Mortality Neonatal Mortality Stillbirths Perinatal Mortality
Phased Design Prototype: Design and refine district-wide QI approach to catalyze change in the key priority area of maternal newborn health Gain experience in different regions Gain experience in different geographic archetypes (agrarian, urban, pastoralist) Build will for change and QI capability at all levels Produce contextualized change packages Test of Scale: Test the scalability of the prototype-designed approach by integrating more completely into the routine system Examine role of LEAD and university hospitals in scale-up design Prime the system for full-scale up and explore structures for scaleup (RHB, ZHB) and lay the plan for scale-up (phase 3)
Progress to Date: Learning Collaboratives Fogera Woreda, Amhara (pop. 264,512) Launched April 2017 Limu Bilbilu/Bekoji Woreda, Oromia (pop. 237,820) Launched April 2016 IHI MNH Initiative Regions IHI MNH Initiative Regions Tanqua Abergele Woreda, Tigray (pop. 107,081) Launched August 2016 HC + HPs in Zana Woreda, Tigray (with Last 10 Kilometers) (pop. 144,246) Launched November 2016 Amibara, Afar (pop. 94,718) Launching January 2018 Duguna Fango Woreda, SNNPR (pop. 118,051) Launched October 2016 = Prototype Phase Woreda
Prototype Phase (18 months) Key = Agrarian = Urban = Pastoral
Test-of-scale Phase (18 months) Key = Agrarian = Urban = Pastoral
Improving Maternal and Neonatal Health in Ethiopia Inputs Activities Outputs Short term Outcomes Medium term Outcomes Long term Outcomes Context Ethiopia has made great progress reducing child mortality, however neonatal mortality rate and maternal mortality ration remain high In partnership with the FMOH, we plan to use QI to accelerate improvement building on assets and strengths of Ethiopian health care system and working with partner organizations. Office Space: -Central office in Addis Staff: -Staff at central Addis Office -International and USbased staff and faculty Operations: -Registration in Ethiopia Partners: -FMOH, RHBs, ZHB s, WoHO s -L10K -Evaluation partners: CPC, IDEAS Tools QI How to Guides (codeveloped by IHI and Aurum Institute South Africa QI methodology tools harnessed from other projects National Quality Strategy -Conduct assessment of current health system with regard to: data systems: leadership and functionality; existing QI initiatives -Co-facilitate stakeholder sessions for syndication to gain buyin -Co-develop strategy with FMOH, inclusive of implementation guidelines and evaluation metrics and development of a Patient Rights Charter Improvement Collaboratives Set-Up: Define clinical bundles and select core indicators with MOH approval; create all QI coaching/clinical mentorship tools, create program monitoring and implementation tools; analysis of existing strategies; baseline data collection; identify early adopters Prototype Phase: Test promising changes with representative slice of health system via collaborative woredas in 3 regions (Oromia, Tigray, Amhara); test measurement system, leadership engagement, data system; test integrated clinical and QI mentoring Test of Scale Phase: Expand to 21 woredas in 5 regions (Oromia, Tigray, Amhara, SNNPR, Afar); continued mentoring of prototyping sites; test and further develop data systems and other infrastructure needs required for scale-up; engage testof-scale implementing partner (TBD); test model of leveraging existing health structures for QI approach; begin integration with NQS Go to Full Scale: Fully leverage existing structures and meetings, add more scalable units within each of the existing 5 regions; expand to 3 new regions in first year; expanding to the remaining 4 regions in the second year; fully integrate with NQS Capacity Building In each phase, build leadership, managerial, and point of care capacity needed for scale up to next phase via: - 9 L&F course waves for WoHO, and facility-level staff - 5 QILM courses for FMoH, RHB, and WoHO staff for each region - 1-2 waves of IHI s IA Course for coaches at the national and regional level, IHI senior project officers, M&E officers - 3 Senior Leaders QI Courses for leaders at the national level - 5 Data Quality Trainings at each prototype LS2 for facilitylevel staff working with data. Community Engagement And Education -Work with public education company to develop educational radio or TV dramas -Engage HEWs to register pregnancies, promote ANC skilled deliveries, and PNC; build data collection and QI skills -Develop client satisfaction feedback mechanism Measurement And Evaluation Internal and external evaluation with development of operational research agenda to optimize local engagement National Quality Strategy -Assessment for strategy -Co-developed NQS document Improvement Collaboratives Set-Up: Initial bundles ready for testing and core indicators selected; clear roles for stakeholders; early adopters engaged Prototype Phase: Locally developed/tested change package; 150 health staff across three regions engaged in QI teams participating in collaboratives; learning shared across groups; methods for building needed infrastructure identified; full FMoH ownership Test of Scale Phase: Standardized process for integrating initiative into existing systems; standardized materials, including manual for coaching QI team meeting, reporting template; locally adapted QI training and reference materials printed for distribution Go to Full Scale: How-to guide for implementing change package nationally; QI fully integrated into health system structures Capacity Building -20 FMoH and RHB QI coaches trained to support prototype -15-30 IAs to lead and monitor quality activities nationally -150 senior leaders with enhanced understanding of QI in healthcare -Up to 180 skilled improvement coaches to lead QI teams -Up to 150 regional, zonal, and district staff with bolstered QI leadership skills -Up to 1,500 regional, district, and point-ofcare staff with working knowledge of QI and capable of infusing it into standard review meetings -150 trained data quality experts Community Engagement And Education -Radio or TV program to spread messages related to Maternal Newborn health and/or respectful maternal care. -Client satisfaction feedback standard materials Measurement And Evaluation Baseline data, regular opportunities to reflect on progress toward aims Individual health worker: -Increase knowledge and skills in QI, testing change ideas, collecting real time data for improvement and using data for decision making. -Increase clinical knowledge and skills -Learn promising practices from peers and other change packages Team: -QI team formation -Ongoing QI team meetings and team activities e.g. testing of ideas, data collection etc. Community: -Increase careseeking behavior for preventative and curative maternal/newborn care services (preconception, ANC, delivery, and PNC) -HEWs increase tracking of data Health System: -National Quality Strategy to institutionalize sustainable QI -Integration of quality structures and quality body -Improved data quality Improve reliability of care processes for maternal health: Antenatal care - Promote early registration of pregnant mothers -Increase subsequent ANC visits - Screen, prevent and treat pregnancy-related conditions and complications e.g. APH, hypertension, HIV, Anemia, Malaria etc. Labour & delivery -Increase % of skilled deliveries -AMTSL -provide compassionate and respectful care -Screen, prevent and treat L&D conditions/complications e.g. obstructed labor, ruptured uterus, pre-eclampsia/eclampsia, PPH, PROM, Postnatal Care -Immediate breastfeeding -Early postnatal care -Routine subsequent postnatal care Improve reliability of care processes for newborn health: -Prevention of prematurity -Routine care of newborn -Screen and manage complications i.e. Pre-term care, Sepsis care, Asphyxia care etc. -Routine postnatal care including vaccinations Community engagement -postnatal follow ups of mother/baby pair in the community -compassionate and respectful care at all levels of facility-based care Referral systems -Strengthen referral and transportation system Reduce maternal and neonatal facilitybased mortality in participatin g sites by 30% over a period of 30 months. Assumptions What is necessary in order for this project to proceed and see results as planned? FMoH, Regional, and woreda-level leadership, woreda-level change agent for joint coaching with IHI PO in prototype phase, engaged woreda-level coach in TOS phase, will for improvement at health facilities External factors - What factors outside of the project may be a barrier or facilitator to reaching your desired outcomes? Turnover in health facilities, low health-seeking behaviors, low rates of facility deliveries, shifting baseline due to pastoral communities, political stability in regions of implementation
Efficient Tool Development and Use for Action Real-time coaching Reporting to inform mentorship Monitoring of program activities (M/E) Formal evaluation and implementation research
IHI Tool and Data Flow Collaborative-level reports Collaborative Dashboard IHI Program Dashboard Guiding Documents: M/E Framework (indicator definitions) Implementation Manual Tools that aggregate data Facility Workbooks Tools for data extraction and collection Program Monitoring Tool Clinical bundle collection tool Quarterly complications deep dive tool Medication and Equipment Survey Tools for onsite clinical/qi mentorship Safe Childbirth Checklist and Clinical mentorship checklists (ANC, delivery, PNC) Aim Statement Template Driver Diagram Template Fishbone Template Measures Template PDSA Template Run chart Worksheet Learning Session assessments
Routine Programmatic Data Use 36 Supervise program and support technical staff (ie, monitor program delivery, gaps, challenges, and new approaches) Develop change package and anticipate areas in need of further testing Direct QI coaching to facilities with greatest performance gap (ie, QI activity engagement, indicator performance) Document critical externalities data quality, changes in leadership and governance, massive flooding in Amhara (*can add photo to slide), cholera outbreaks
Routine Programmatic Data Use 37 Example baseline assessment revealed 0 neonatal deaths in past year in 1 district Response: Engaged district political leadership to lead discussion on underreporting pressures and create commitment and safety in honest reporting moving forward Track progress in core quantitative measures, address challenges with participants and facilitating leaders, and celebrate successes
Evaluation 38
Evaluation Aims 39 Mixed Method evaluation encompasses prototype and test of scale phases with following aims: 1. Describe the intervention as implemented in each phase and changes as approach is scaled 2. Understand the mechanisms of action of the QI approach, how individuals and teams change over time and how teams function within the QI approach, and health care worker motivation 3. Evaluate the impact of the intervention on facility level MNH quality of care and outcomes during the prototype and test of scale phases and compare findings 4. Assess the cost-effectiveness of the intervention
Leverage Partnerships 40 FMoH lead implementer and primary end-user of results IHI co-lead implementer, overseeing overall evaluation design and coordination Addis Ababa University key local research partner to support design, data collection, and analysis University of North Carolina key international partner with experience in complex QI program evaluation (mixed methods) London School of Hygiene and Tropical Medicine (IDEAS) - key international partner with experience in assessing the how and why of QI (mixed methods)
Four Evaluation Components Different institutions lead sub components, with IHI overseeing all, and AAU supporting all: 1. Quantitative Impact Analyses: to determine whether the intervention is leading to improved MNH processes and outcomes to understand what program and facility level factors lead to improved program implementation and quality of care 2. Qualitative assessment of maternal Perception of Care and utilization of Services (experiential quality) 3. Mixed-methods assessment of Individual Change and Team Functioning: the change in knowledge and attitudes among LS participants (quantitative) the functioning of QI teams (quantitative) health worker motivation as it relates to QOC (quantitative) how QI leads to change (qualitative) 4. Cost-Effectiveness Analysis
Relevance 42 Results will help determine: ultimate degree of institutionalization of the approach appropriateness/readiness for full scale-up locally and globally contribute to the limited literature on the mechanism of action of QI and program effectiveness at scale
Operational Research Agenda Evaluation linked with Research Capability Building program, to ensure local implementers and leadership are deeply involved in evaluation and publication operational research agenda supports development of local implementation scientists and data use for decision-making Ministry partners in program design and implementation also prioritized understanding impact of initiative Development of dissemination plan in advance with open dialogue
Example Dissemination Plan 44
Conclusions 45 Integration of evaluation into program allows for the most intentional high quality data collection that can serve: Program implementation needs Routine internal evaluation (M/E) to adapt implementation in realtime to meet the program aims and patient needs by rigorously documenting process Formal evaluation (generalizable learning, local and global publications and policy implications) Less is more Death by documentation can lead to poor quality data think intentionally about every single data element to be collected (purpose program only, program + internal evaluation, program + internal evaluation + formal evaluation) Implementation science and research requires rigorous implementation and rigorous evaluation married together in harmony
Recommendations 46 Complex programs require complex evaluation designs Create a rigorous design and plan, build in times for data/experience review and course correction Optimistic realism to the realities of program implementation Be intentional and document adaptations as you go to allow for the most rigorous analysis and data interpretation Maximize partnerships and have open dialogue on roles and expectations for research outputs (paper list, authorship teams) Allows for building in meaningful learning and experience
Questions? 47