Impact Evaluation Concept Note HEALTH MILLENNIUM DEVELOPMENT GOALS PROGRAM-FOR-RESULTS (P4 R) ETHIOPIA Development Impact Evaluation Initiative Innovating in Design: Evidence for Impact in Health Cape Town, South Africa December 10-14, 2012
Intervention Summary Program-for-Results ($120 million) The first P4R program in the health sector, will focus on MCH and general health strengthening Will disburse according to progress on 9 Development-Linked Indicators: 5 MCH indicators, such as rate of skilled-delivery or immunization, and 4 system indicators such as PHC rate of prompt HMIS reporting From start will be national program, but will pilot select sub-components such as Facility and district (Woreda) Balanced Score Card to assess quality of service and management practice
Evaluation Questions The Evaluation Questions are as follows: What is the impact of P4R on the priority health outcomes (as represented by the DLIs)? How does P4R influence the health system development? Does the BSC approach at the Woreda/facility level improve health services and health outcomes? Does BSC + monetary incentives result in further gains as compared to the BSC only approach?
Evaluation Design Given the national scope of P4R, not all questions can be answered with IE Questions 1 and 2 will employ alternative approaches to generate information and evidence The pilot nature of the BSC activity will enable a traditional IE
What is the impact of P4R on priority health outcomes? Primary and secondary data will track outcomes over the study period Can use the time series of indicators from before and during implementation to conduct differential interrupted time series analysis This involves contrasting change in DLI-linked indicators with change in health indicators (such as, perhaps, adult curative care) not likely to be influenced by P4R The key assumption: indicators not influenced by P4R can be identified and measured Exploratory analysis will look at sensitivity of all health indicators to public-sector expenditures and activities to identify possible comparator indicators
Indicator Level Illustration of differential interrupted time series DLI indicator Estimated impact of P4R at time T Unaffected indicator Start of P4R Time
How does P4R influence health system development? Process evaluation: a mixed-methods high frequency monitoring and analysis plan will: Record implementation pace of P4R and all inputs, activities, and outputs conducted in accordance with P4R planning Will investigate understanding and influence of P4R among key health system actors at federal, regional, zonal and woreda level, including both system managers and providers. Will investigate influence of P4R on goal setting, planning, and plan implementation to identify improvements and areas of concern. Provide immediate feedback to program owners for consideration of reforms Conducted every six months
What is the impact of the BSC and BSC + incentives on health outcomes? The pilot nature of BSC activities enables a large-scale randomized cluster trial BSC is a management tool to help improve health system performance a checklist that covers facility or Woreda management practices, conditions of service, quality of care, and hold facilities accountable for delivering care targets Govt. will purposive select 3 regions (urban, agrarian, pastoralist) BSC will be tailored to facilities and Woreda Health Bureau P4R will pilot BSC in 20-30 Woreda per region An additional 20-30 Woreda per region will pilot BSC + incentives Remaining Woreda in region will serve as control Assignment of Woreda determined by stratified randomization
IE Design summary
THE TEAM
2. Results Chain: Maternal health Inputs Activities Outputs Outcomes Long-term Outcomes Management of health system inputs incentivized by P4R Supplying equipment/ ambulances Midwife and health officer training Basic OB care Functional ambulance services Trained health officers Increased availability of commodities Increase in maternal health service utilization and quality Reduction in maternal mortality rate and maternal morbidity 11
2. Results Chain: Child health Inputs Activities Outputs Outcomes Long-term Outcomes Management of health system inputs incentivized by P4R Strengthening of cold chain systems Supplying of vaccines Immunization campaigns Supplying bed nets Health officer training Functional cold chain equipment Outreach campaigns held Insecticidal nets distributed Primary health care staff trained Increased availability of key commodities Increased immunization coverage Increased utilization of child health services Reduction in childhood mortality and morbidity 12
2. Results Chain: Health systems Inputs Activities Outputs Outcomes Long-term Outcomes Management of health system inputs incentivized by P4R Constructing Health Centers Supplying medical products and equipment Validating HMIS semiannually Undertaking surveys and studies Health centers built Health facilities report HMIS information on time Annual Facility Readiness Assessment undertaken Finalized BSC assessment tool Improved HMIS Roll out of balanced score card and institutional performance incentives Improvement in facility readiness score Increase in information- and evidence-based decision making Improved health system performance Improved general health of population 13
Time line Study Phases Time Baseline data collection 2013 Program begins June 2013 Midline data collection June 2015 End line data collection June 2017