Synthesis Report. Essential Services for Health In Ethiopia. Health Systems Performance Improvement End-line Survey. Contract 663-C

Essential Services for Health In Ethiopia Health Systems Performance Improvement End-line Survey Synthesis Report Contract 663-C-00-04-00403-00 September 2008 Addis Ababa

Cover Photo: Health facility staff describes how health center monitors progress using HMIS data. Essential Services for Health in Ethiopia is implemented by John Snow, Inc. in collaboration with Abt Associates, Academy for Educational Development, and Initiatives, Inc. This report was made possible by the support of the American people through the United States Agency for International Development (USAID) under Contract 663- C-00-00403-00. The contents are the sole responsibility of John Snow, Inc. and do not necessarily reflect the views of USAID or the United States Government.

Table of Contents 1. Executive Summary... 1 2. Background and Evaluation Objective... 2 1.1. Background... 2 2.2. Objective... 2 3. Performance Improvement Intervention Description... 3 4. Methods of Evaluation... 3 4.1. Design... 3 4.1.1 Sample... 3 4.1.2 Performance Elements Reviewed... 4 4.1.3 Data Collection Tools... 4 4.1.4 Field Testing... 5 4.1.5 Team Composition and Training... 5 4.2. Survey... 5 4.3. Data Analysis... 6 5. Evaluation of Findings... 6 5.1. Staffing and Staff Capacity Building... 6 5.1.1 Management Staffing... 6 5.1.2 Technical Staff... 7 5.1.3 Staff Orientation... 8 5.1.4 In-Service Training Planning and Management... 9 5.1.5 Availability of Trained Staff... 10 5.1.6 Staff Retention... 11 5.2. Standards... 11 5.2.1 Service Delivery Standards... 11 5.2.2 Use of Service Delivery Standards... 12 5.2.3 Management Standards... 13 5.2.4 Management Committees... 14 5.3. Supervision... 15 5.3.1 Supportive Supervision... 15 5.3.2 Supportive Supervision Tools... 15 5.3.3 Supervisory Roles and Responsibilities... 16 5.3.4 Supervision Planning... 16 5.3.5 Supervision Frequency... 17 5.3.5 Self Assessment... 19 5.3.5 Sharing of Supervisory Findings... 20 5.4. Health Management Information Systems... 20 5.4.1 HMIS... 20 5.4.2 HMIS Committees... 20 5.4.3 HMIS Analysis and Data-Driven Decision-Making... 22 5.4.4 HMIS Wall Charts... 22 5.5 Review Meetings... 24 5.5.1 Plan and Frequency of Review Meetings... 24 5.5.2 Review Meeting Agendas... 26 6. Recommendations... 26 i

List of Tables Table 1: End-line Assessment Sample...4 Table 2: Average Number of Management Posts Designated Per Office...6 (By Region)...6 Table 3: Management Positions at RHB, ZHDs and WorHOs (%)...7 Table 4: Availability of Technical Staff at Health Centers (By Region)...8 Table 5: Staff Orientation (Average All Regions) (%)...8 Table 6: In-Service Training Inventory and Training Needs Identified (Average All Regions) (%)...9 Table 7: Woredas with At Least One Trained Staff (%)...10 Table 8: Health Centers with At Least One Trained Staff (%)...10 Table 9: Availability of Service Delivery Standards at Health Centers (%)...12 Table 10: Use of Service Delivery Standards at Health Centers (%)...12 Table 11: Availability and Use of Management Standards (Average All Regions) (%)...13 Table 12: Use of Management Standards RHB (Average All Regions) (%)...13 Table 13: Frequency of Management Committee Meetings (Average All Regions) (%)...15 Table 14: Offices with Documented Supervision Plans (By Region) (%)...17 Table 15: Reported Average Number of Supervision Visits per Year...17 Table 16: Supervision Visits completed Based on Available Documents (%) 19 Table 17: Offices and Facilities That Use ISCLs...19 for Self Assessment (By Region) (%)...19 Table 18: Improvement of Supervision Feedback Systems (By Region) (%).20 Table 19: Offices and Facilities with Active HMIS Committees (%)...22 Table 20: Facilities That Use HMIS Data for Planning (%)...22 Table 21: Availability of HMIS Wall Charts (Average All Regions) (%)...24 Table 22: Planning and Implementation of Review Meetings (By Region)...24 Table 23: Health Centers That Attended Review Meetings (%)...26 Table 24: Subjects Covered in Review Meetings (Average All Regions) (%) 26 ii

Abbreviations AIDS Acquired Immunodeficiency Syndrome ARI Acute Respiratory Infection EC Ethiopian Calendar ENA Essential Nutrition Actions ESHE Essential Services for Health in Ethiopia Project FMoH Federal Ministry of Health GC Gregorian Calendar HC Health Centre HF Health Facility HIV Human Immunodeficiency Virus HMIS Health Management Information System HP Health Post HRM Human Resources Management IMNCI Integrated Management of Newborn and Childhood Illnesses ISCL Integrated Supervisory Checklist NGO Non-Governmental Organization PI Performance Improvement PMTCT Prevention of Mother to Child Transmission Q Quarter RHB Regional Health Bureau ROPPA Results-Oriented Performance Planning and Appraisal SNNP Southern Nations Nationalities and Peoples (Region) TB Tuberculosis USAID United States Agency for International Development VCT Voluntary Counseling and Testing VVM Vaccine Vial Monitor WorHO Woreda Health Office ZHD Zonal Health Department iii

1. Executive Summary The USAID-funded Essential Services for Health in Ethiopia (ESHE) Project is a fiveyear program aimed at improving child health in Ethiopia. ESHE provides technical support in child health, healthcare financing, and performance improvement to the Amhara, Oromia and Southern Nations Nationalities and Peoples (SNNP) Regions. The objective of the 2008 performance improvement end-line assessment was to evaluate the effectiveness of ESHE performance improvement activities against 2004 baseline data. The performance improvement end-line assessment included a comprehensive selection of ESHE focus zones, woredas and their respective health centers. Four data collection tools were used to guide group interviews, facility observation, and record reviews. Four three-person teams of ESHE staff and partners carried out the survey in each region. Up-to-date staffing wall charts exist at 76 percent of zonal health departments (ZHDs), 91 percent of woreda health offices (WorHOs), and 93 percent of health centers across the regions, and facilities are able to justify staffing needs based on documented gaps between staffing standards and actual staffing levels. The number of management posts and the percentage of these posts filled have improved. SNNP meets national staffing standard of 13 technical staff for health centers. Average numbers of technical staff available in Amhara and Oromia are 10 and 11, respectively. Across regions, midwives and environmental health workers are in shortest supply. At zonal, woreda, and facility levels, the practice of documenting training and using training records to help select staff in need of training is taking place. However, practices have not yet reached the level of systematically assessing staff training needs and using this information to drive decisions by higher levels about what training programs are needed. Availability and use of standards has improved. Availability of clinical standards at facilities improved across regions from 59 percent to 100 percent and management standards from 0 percent to between 67 percent and 91 percent. Management standards are used and management committees exist, are active, and are addressing critical management issues. Of the 83 health centers visited, 89 percent reported having management committees that meet regularly. Reviews of management committee meeting books show committees are discussing critical issues such as human resource management, health care financing, service expansion, performance evaluation, and logistics. Over the last four years, supervision has transitioned from an auditing approach to integrated supportive supervision. Staff feel the current supervision approach is constructive and is helping them improve performance. While the frequency of supervision still needs to be improved, a steady increase has occurred with the average number of visits per year estimated by supervisees surpassing 50 percent of the minimum standard in most cases. Document review reflected a lower achievement of between 17 percent and 19 percent. Oromia had the highest achievement of supervision visits with 54 percent of regional health bureau (RHB) 1

supervision to zones completed and 33 percent of WorHO supervision of health centers completed. Health Management Information Systems (HMIS) committees are active in 85 percent of ZHDs, 99 percent of WorHOs, and 88 percent of health facilities. 87 percent of health facilities use HMIS data for planning, as compared with 49 percent at the time of the baseline. Staff at all levels review action plans and compare actual performance to targets. Personnel at woreda and facility levels analyze performance and feel empowered and responsible for improving performance. Ninety percent of health centers reported participating in at least one review meeting in the last six months as compared with 40 percent at the baseline. Meeting minute books show critical performance issues are discussed including performance and action plans, strategies for improving poor performing indicators, and quality improvement. Budget constraints appear to pose the greatest obstacle to the realization of review meetings. Oromia and SNNP Regions show the greatest gains in performance improvement activities in use of management and service delivery standards, supervision, and review meetings. Amhara is doing well in HMIS management, but is generally weaker in other performance improvement areas. High staff turnover at the Amhara RHB and change in leadership at the ESHE regional office may explain the weaker performance in Amhara. Oromia RHB demonstrated very strong support for performance improvement contributing to stronger results in this Region. 2. Background and Evaluation Objective 1.1. Background ESHE is a five-year program aimed at improving child health in Ethiopia. ESHE provides technical support in child health, health care financing, and performance improvement to the Amhara, Oromia and Southern Nations Nationalities and Peoples Regions. It aims to increase the use and community acceptance of and support for high-impact child health, family planning and nutrition services, products and practices including increasing exclusive breastfeeding, immunization coverage and vitamin A, iron and folate supplementation, and use of insecticide-treated bed nets. ESHE also is tasked with improving health system resources by strengthening financial planning and management at regional, zonal and woreda levels, and with improving health sector systems by establishing management and service delivery standards, improving supportive supervision, and strengthening HMIS management systems. 2.2. Objective The objective of the performance improvement end-line assessment, conducted in 2008, was to evaluate ESHE performance improvement activities by measuring ESHE inputs against 2004 baseline data for each of the three ESHE focus regions: Oromia, Amhara and SNNP. The key performance improvement inputs assessed included staff management and capacity building systems, standards, supervision, 2

HMIS and review meetings. This summary report assesses the ESHE performance improvement component s achievements by presenting the average results of all three regions against baseline averages and comparing results in the three regions to identify and explore regional variations in performance improvement achievements. 3. Performance Improvement Intervention Description The baseline assessment of performance improvement systems in Oromia, SNNP, and Amhara examined various elements affecting staff and health system performance including performance reviews, availability of equipment and transportation, communication systems, availability of supplies, staffing levels and human resources management, supervision and HMIS. Baseline data indicated that a number of these systems were weak. ESHE chose to focus on four main components of performance improvement: staff management and capacity building, standards and guidelines, integrated supportive supervision, and HMIS. Other key elements of performance management were not undertaken by ESHE as they were to be covered by other projects (such as drug supply), they were being addressed on a national scale by the Federal Ministry of Health (FMoH) (such as staff performance assessments), or they were beyond the scope of the Project (such as transportation and communication systems). The ESHE Performance Improvement team developed a number of tools and training curricula to build systems and capacity for performance improvement. These tools included a set of management standards for the health system to list and guide management responsibilities and tasks at every level of the health system, from the region to health posts. To introduce supportive supervision systems, ESHE developed a set of supportive supervision guidelines and checklists along with a scoring system to help supervisors identify priority areas and monitor performance change over time. To support this system, ESHE provided training to personnel in all regions and at all levels on implementing the supportive supervision system. Finally, an HMIS data-analysis and decision-making curriculum was developed and staff were trained to better manage and use HMIS data. Results of these interventions are documented in the findings. 4. Methods of Evaluation 4.1. Design Sample The end-line assessment included a comprehensive selection of ESHE focus zones and woredas in the three regions. Three RHBs, 13 ZHD, 70 WorHOs, and 83 health centers were included in the assessment. The Ethiopian health system is in a period of rapid expansion. Newly restructured urban health offices, new WorHOs, and new health centers were excluded from the study as they have only been operational for several months and would not yield results on performance improvement interventions. Health posts, while included in the baseline assessment, were also 3

excluded from the end-line survey because ESHE performance improvement interventions were concentrated at RHBs, ZHDs, WorHOs, and health centers. Table 1: End-line Assessment Sample Region RHB ZHD WorHO Health Center Oromia 1 4 21 20 Amhara 1 4 22 28 SNNP 1 5 27 35 TOTAL 3 13 70 83 Performance Elements Reviewed While a broad range of factors influence individual and facility performance, ESHE designed the end-line survey to capture performance of focus areas: Number and distribution of management and service delivery staff at health facilities and WorHOs; Availability and use of management standards; Procedures for determining needs for additional staff, to whom these needs are communicated and/or with whom staffing needs are negotiated; Availability and content of orientation for new staff members to the responsibilities of their position; Availability and use of service delivery standards that guide service delivery, supervision, and quality monitoring; Availability of supervisory systems, the regular provision of integrated supportive supervision, and how supervisory findings are shared at various levels of health system; Documentation of HMIS data and whether HMIS data are used for informed decision-making; Regularity and content of performance review meetings. 4.1.3 Data Collection Tools Four data collection tools designed as group interviews with facility observation and record review components were used to gather data. A tools was designed for each level: 1. Regional Health Bureau Questionnaire 2. Zonal Health Department Questionnaire 3. Woreda Health Office Questionnaire 4. Health Center Questionnaire To facilitate a rapid and user-friendly data collection process while ensuring the collection of robust data, the team employed a combination of data collection methods. The questionnaires were designed not only to document the availability of materials such as service delivery standards, plans and wall charts and procedures, but also to gather information about the content of plans, the subjects covered in meetings, and staff views on how useful performance improvement interventions have been for their jobs. The questionnaires required the data collection teams to interview managers and staff, review meeting books and 4

supervision records, and observe facility wall charts. In addition, the team kept daily notes of issues or problems that emerged, but were not adequately captured by the data collection questionnaires. Field Testing The data collection team field tested the instruments in Amhara Region. Field testing outside the ESHE focus zones and woredas was necessary because of the comprehensive survey sample. Based on findings of the field test, the team amended the tools prior to conducting the data collection. Revised data collection instruments were then used in all three regions. 4.1.5 Team Composition and Training Four three-person teams executed the survey in each region. The data collection teams were composed of one ESHE Performance Improvement staff as team leader and two government health workers. As ESHE has worked hard to forge partnerships with RHBs of the three regions, government staff stationed at ZHDs, WorHOs, and health centers were invited to participate. Eight government health workers were selected by each RHB. Efforts were made to ensure government staff were assigned to teams that would evaluate program activities outside of the zone in which they were stationed. Data collectors were trained for two days in Amhara and three days in Oromia and SNNP. The training included a detailed review of the data collection instruments, practice interviewing sessions with group observation and feedback, and field testing (Amhara) or field exercise (Oromia and SNNP). Data collectors were trained to wait for unprompted responses, unless specifically noted otherwise, and to correctly complete the data collection questionnaires. In addition, while in the field, team leaders observed government partners leading interviews to ensure they could perform to a high standard. No data collector conducted an interview alone unless the team leader was confident of interviewer s capabilities (based on observations). 4.2. Survey The teams developed detailed travel and interview schedules for each region before departure to the field. These schedules were reviewed with ESHE Regional Performance Improvement Specialists, data collectors from respective zones, and drivers, to verify the schedule and the routes chosen were feasible. Because of the distance between sites, most interviews were conducted by each team as a group. Where possible, some teams did split to speed up the survey process and ensure all WorHOs and health centers could be covered in the five days allotted for data collection. To ensure the most accurate information possible, interviews were conducted as focus group discussions with representatives of management and technical staff at offices and health centers present. This was particularly important since high staff turnover has meant knowledge about systems, procedures, and practices is not equally shared by all staff. 5

4.3. Data Analysis A Microsoft-Access database and code sheet was developed to facilitate data entry. ESHE hired a database expert to conduct the data entry for all three regions. Responses to open-ended questions were text coded and entered into the database. When complete, the database expert submitted the data to ESHE for analysis. Simple data analysis processes were used, focusing on frequencies and proportions. End-line data were matched against data from the baseline assessment to arrive at the comparisons presented in this report. 5. Evaluation of Findings 5.1. Staffing and Staff Capacity Building Management Staffing The data suggest a great improvement in both the number of management posts designated for the different levels and the percentage of these posts actually filled. The baseline and end-line survey teams asked staff at RHBs, ZHDs, and WorHOs to obtain information on the number of management posts designated for their office and the number of these posts that are filled. Table 2 shows increases in the average number of management posts at RHBs, ZHDs, and WorHO. It illustrates regional diversity, with Oromia designating a greater number of management posts at each level, but also indicates an overall trend to increasing management staff. The data indicate the percentage of posts filled has improved. The average percentage of posts filled in 2004 was 83 percent for the RHB, 68 percent for the ZHDs, and 50 percent for WorHOs as compared 92 percent for RHB, 85percent for ZHDs, and 68 percent for WorHOs in 2008. Part of this change may be attributed to better human resources management. At the time of the baseline, offices and facilities were aware that there were gaps in personnel, but they were not systematically keeping track of the personnel at their sites, comparing that information with staffing standards, and regularly submitting requests for additional staff to higher levels using staffing data. The management standards developed by ESHE in partnership with RHBs clearly outline responsibilities for each office and facility with regard to human resources management including maintaining staffing data and submitting staffing reports. The survey team observed up-to-date staffing wall charts in almost all ZHDs, WorHOs, health centers (see Table 21), suggesting that a more systematized approach to human resources management is taking place. Table 2: Average Number of Management Posts Designated Per Office (By Region) Baseline Number of Posts End-line Number of Posts Office Oromia Amhara SNNP Oromia Amhara SNNP Total: RHB 13 12 23 19 16 ND Total: ZHDs 3-4 3-5 4 7-9 5-8 5 Total: WorHOs 4 3-4 6 5 5-9 5-6 6

Table 3: Management Positions at RHB, ZHDs and WorHOs (%) Baseline Posts Filled End-line Posts Filled Office Oromia Amhara SNNP Avera Oromi Amhar SNNP Avera ge a a ge Total: RHB 62 100 87 83 95 88 ND 92 Total: ZHDs 46 82 75 68 55 100 100 85 Total: WorHOs 25 84 42 50 56 83 64 68 5.1.2 Technical Staff To assess staffing levels of technical posts, the survey team documented numbers and types of technical staff at health centers and compared this with national staffing standards for health centers. The data suggest that neither Oromia nor Amhara Regions have met the national staffing standard for health centers. In fact, the increase in the percentage of management posts filled contrasts with a decrease in the number of technical posts filled, suggesting the Regions achievements in improving management staffing may be made at the expense of providing technical staff to health facilities. SNNP, on the other hand, has demonstrated both an improvement in the percentage of management posts filled and the average number of technical posts per health facility and is the only Region to meet the national staffing standard of 13 per health facility. Like the baseline, a wide variation in the number of staff available at different sites existed, with a low of just 1 technical staff and a high of 24 technical staff. The largest gap exists for midwives with health centers having an average of only 0.2 to 0.35 1 midwives compared with a standard of 3 midwives per site. Oromia and Amhara also show a gap in the number of nurses available, with only 5 and 5.3 nurses per site against a standard of 6, while the average number of nurses per site in SNNP is 9. Environmental health workers is another category of staff universally in short supply, with regions having an average of only 0.5 to 0.7 environmental health workers per site as compared with a standard of 1 per site. 1 The end-line survey did not specifically ask for the number of midwives. The average calculated in this report is based on the number of health centers that mentioned midwives when asked what other technical staff work at the facility. It, therefore, may not be an accurate count of the number of midwives at the sites. 7

Table 4: Availability of Technical Staff at Health Centers (By Region) Category of Staff Nationa l Standa rd Categories of Staff Assessed Health Center Average Oromia Health Center Average Amhara Health Center Average SNNP Public Health Officer 1 Public Health Officer 1.4 1.4 0.9 Senior Public Health 1 Nurse 2 5 5.3 9 Nurse Senior Clinical Nurse 1 Junior Public Health 1 Nurse Junior Clinical Nurse 3 Junior Environmental 1 Environmental 0.7 0.5 0.6 Health Worker Health Worker Junior Laboratory 1 Laboratory 1.7 1.8 1 Technician Technician Junior Pharmacy 1 Pharmacy 1.1 0.6 1 Technician Technician Junior Midwife 1 Other: 0.35 0.13 0.2 Primary Midwife 2 Midwives Total End-line 13 11 10 13 Baseline 13 11 7 5.1.3 Staff Orientation When new staff members begin work they are unfamiliar with the working practices at the office or health facility they join. They need to be oriented, briefed on their specific responsibilities, and provided guidance and support for performance expectations. A clear and comprehensive orientation to these issues provides a new staff member with confidence and helps them to become effective team members as rapidly as possible. ESHE recognized from the baseline assessment that orientation of new staff was not done in an effective manner and usually consisted of simply introducing the new staff member to his or her colleagues. The end-line survey showed that almost all offices and health centers report providing a briefing by the head to new staff. As shown in Table 5, 92 percent of ZHDs, 95 percent of WorHO, and 98 percent of health centers provided orientation to new staff. Briefing by the head of the office remains the main mode of orientation, although 70 percent of ZHDs and 53 percent of health centers say they practice attaching new staff to an experienced staff member for a period of time, which is a positive development. While the data suggest that staff are oriented to their responsibilities to a greater degree than in the baseline, the fact that staff turnover remains an obstacle to the performance of key management and supervision tasks suggests that the content of the orientation still does not adequately address these key issues. Table 5: Staff Orientation (Average All Regions) (%) ZHD WorHO Health Center Criteria (N=13) (N=70) (N=83) Offices/facilities with 92 95 98 2 The end-line survey asked for the number of nurses at the facility. Therefore some health centers only mentioned junior nurses in the other category. The averages represented here are aggregates of the number of nurses mentioned in both sections. 8

orientation for new staff Briefing by a higher level 18 Briefing by the Head of the office/facility 76 87 92 Attachment to an experienced staff member 70 40 53 Special training 30 Other 0 14 7 5.1.4 In-Service Training Planning and Management To be effective, training should be based on assessed knowledge or skill gaps; should be well planned, executed and evaluated; and should be adequately documented to reduce duplication and enable effective monitoring. The ESHE performance improvement component encouraged managers to assess staff training needs, send staff to trainings where they will acquire skills they need to do their jobs rather than simply send them to training as a perk, regardless of the relevance of the training to their responsibilities and to reduce the number of staff who receive multiple trainings in the same topic. Data gathered during the baseline assessment showed that no offices or facilities planned or coordinated in-service training; rather training was driven by invitations from partners, such as donor-funded projects. To address this issue and develop a more systematic approach to in-service training management, the management standards developed by ESHE in collaboration with RHBs set clear guidelines and lines of responsibility for assessing training needs, documenting training provided, and evaluating training activities. The survey team collected data from ZHDs, WorHOs, and health centers to assess understandings of the roles and practices of each level for in-service training. The data show an improvement over the baseline where few inventories were found, with 77 percent of ZHDs, 79 percent of WorHO, and 65 percent of health centers currently having in-service training inventories. In the case of health centers only 76 percent of the 55 health centers that had in-service training inventories kept them upto date and only 16 percent kept notes on which staff require refresher training. Table 6: In-Service Training Inventory and Training Needs Identified (Average All Regions) (%) ZHD (N=13) WORHO (N=70) Health Center (N=84) End-line End-line End-line Have in-service training inventory 77 69 65 In-service training inventories are up-to-date 76 Have documented a need for refresher training 16 ESHE has made significant strides in documenting standards for in-service training planning and coordination. At the zonal, woreda and facility levels, documenting training and using training records to help select staff in need of training and reduce the number of staff receiving multiple trainings in the same topics is becoming standard practice. However, practices have not yet reached the level of 9

systematically assessing staff training needs and using this information to drive decisions by higher levels about what training programs are needed. It is not surprising that this fairly sophisticated management practice has not yet taken root in any of the regions, but future health systems strengthening interventions should reinforce current training management practices and assist in furthering training assessment and planning systems. While training is important for improving staff capabilities, it also draws away critical human resources, particularly long-term upgrading training of several months to more than a year. At the time of the assessment an average of 3.4 staff members per woreda were away from their post on long-term training with some woredas having as many as ten staff members on long-term leave for training purposes. This underscores the importance of having solid training management systems to ensure staff receive the training they need and desire and health service delivery is not affected by staff shortages. 5.1.5 Availability of Trained Staff ESHE trained staff in clinical issues related to child health and performance improvement methods such as supportive supervision and HMIS management. Table 7 & 8 show both WorHO and health centers have staff trained in EPI, IMNCI, ENA, supportive supervision, and HMIS. In addition, responses in the other category showed that many WorHOs and health facilities have staff trained in health care financing (by ESHE), voluntary counseling and testing (VCT), prevention of mother to child transmission (PMTCT), malaria, post abortion care, TB, and drug management. Training results are fairly even across the regions but are strongest in Oromia. It is not evident if this achievement is due to the proximity of Oromia to Addis Ababa (enabling frequent visits), to lower staff turnover in the Region, or to better performance by Oromia cluster teams. Table 7: Woredas with At Least One Trained Staff (%) Training Area Oromia Amhara SNNP Average All Regions EPI 100 100 82 94 IMNCI case management and supervision 71 73 74 73 Essential Nutrition Actions 86 91 74 84 Supportive supervision 100 9 96 97 HMIS 100 100 74 91 Other 100 77 82 86 Table 8: Health Centers with At Least One Trained Staff (%) Training Area Oromia Amhara SNNP Average All Regions EPI 90 96 74 87 IMNCI case management and supervision 95 96 97 96 Essential Nutrition Action (ENA) 85 89 80 85 10

Supportive supervision 40 3 68 3 14 41 HMIS 85 89 34 69 Other 95 78 100 91 5.1.6 Staff Retention Although the baseline assessment did not assess staff retention, the management standards developed by ESHE covered human resources management and the need to have clear strategies for retaining staff, including strategies for providing incentives to staff. In Ethiopia, as in many African countries, staff turnover has a significant impact on health system performance. The assessment development team, therefore, decided to include questions on staff retention in the RHB and ZHD questionnaires. All three RHBs reported having strategies for staff retention, including facilitating staff transfers to desired locations, offering upgrades for staff, in-service training, summer training programs, and providing some financial incentives. Likewise, many zones have staff retention programs that including facilitating transfers, offering upgrade training, and in-service training, but usually no financial incentives. These efforts are important, but persistent problems in staff allocation and distribution suggest that they may not be enough to adequately address staff shortages, particularly in rural areas. Now that a system for documenting staff retention strategies is in place, regions, zones and WorHOs may need some assistance to develop more effective staff retention strategies, including financial incentives as well as staff recognition and reward. 5.2. Standards 5.2.1 Service Delivery Standards No complete packets of service delivery standards were available in 2004 at facilities in Oromia, Amhara, or SNNP. The standards that were available at health facilities included: managing acute respiratory infection (ARI), diarrhea and malaria, the VVM (Vaccine Vial Monitor) standards associated with polio eradication efforts, and a guide for classification of diseases. ESHE adapted and developed service delivery standards for child health issues. To ensure that the standards stay at the facility, rather than being appropriated by staff for individual use, the standards were packaged in a binder. A review of availability of standards conducted in the survey revealed a great improvement in the availability and use of standards. Whereas an average of only 59 percent of health centers had any service delivery standards on site in the baseline review, 100 percent of facilities had some service delivery standards in the end-line 3 ESHE provided its supportive supervision training mainly to RHBs, ZHDs, and WorHOs staff. Health center heads were included in the supportive supervision training, but not general staff. In addition, the high turnover of staff has meant that many trained heads have moved on to other facilities or other jobs altogether, hence the relatively low percentage reported here. 11

survey and 87 percent had the complete package of service delivery standards provided by ESHE and the FMoH available. Table 9: Availability of Service Delivery Standards at Health Centers (%) Assessmen t Standards Oromia Amhara SNNP Average All Regions 59 Baseline Any service delivery standards available 60 (N=15) 67 (N=15) 50 (N=15) End-line Any service delivery 100 100 100 standards available (N= 20) (N = 28) (N=35) End-line EPI Modules (1-5) 70 100 91 87 End-line IMNCI Chart Booklet 90 100 97 96 End-line ENA Counselor s Guide 70 82 86 79 End-line End-line Malaria Diagnosis and Treatment Guideline Other (ART, Leprosy, VCT, HMIS) 5.2.2 Use of Service Delivery Standards 100 N = 83 50 79 77 69 70 79 77 75 To have a positive impact on service delivery quality, standards must be applied. In addition to providing service delivery standards, ESHE trained staff to use the standards for varied activities such as service delivery and supervision, annual action plan preparation, individual performance plan preparation, and epidemic preparedness and response. According to provider and manager reports, use of the standards has improved substantially. The end-line survey showed that 84 percent of health centers use some or all service delivery standards in providing services to individual patients, 75 percent use some or all to guide supervision of the quality of service delivery, 59 percent use some or all standards in preparing the annual action plan, 42 percent use some or all in preparing individual performance plans, and 57 percent use some or all standards in epidemic preparedness response (Table 10). Use of service delivery standards was weakest in Amhara, where only 61 percent of health centers reported using some or all of the service delivery standards in providing services to individual clients, as compared with 95 percent and 97 percent in Oromia and SNNP, respectively. Table 10: Use of Service Delivery Standards at Health Centers (%) Use 4 Oromia Amhara SNNP End-line (N= 20) End-line (N= 28) End-line (N= 35) Average All Regions End-line (N= 83) In providing services to individual patients 95 61 97 84 To guide supervision of the quality of service delivery 80 57 89 75 In preparing the annual plan of action 65 39 74 59 In preparing the individual result- 45 32 49 42 4 The end-line assessment questionnaire asked about use of all, some, or none of the questionnaires for the purposes listed in the first column under use. The data represented under the end-line assessment column indicate that at least some of the standards listed in Table 9 were used in the manner specified. 12

based performance plans. In epidemic preparedness and response 55 50 66 57 5.2.3 Management Standards No comprehensive management standards were available in the regions in 2004. ESHE developed management standards for each office and service delivery level, trained staff and managers in the content and use of the standards, and disseminated the standards to all levels. The management standards detail each office s responsibilities with regard to management oversight; planning, monitoring and evaluation; technical support/oversight of quality service delivery; human resources management; logistics and engineering support; epidemic preparedness support; and financial management. Table 11: Availability and Use of Management Standards (Average All Regions) (%) RHB ZHD Woreda Health Center Baseline N=3 N=12 N=47 N=63 Availability of management standards 0 0 0 0 End-line N=3 N=13 N=70 N=83 Availability of management standards 67 83 91 82 Percent of offices/facilities that have management standards that say they use them 33 89 83 68 The end-line survey found the reported use of the management standards to be quite good across regions (Table 11). The existence of regular management committee meetings, staff retention systems, and availability of performance management plans provides evidence for their use. Standards are most often being used for the development of annual action plans, but also for supportive supervision and monthly HMIS action planning (Table 12). It would be desirable to see increased use in ensuring that management systems are in place for epidemic preparedness plans, financial management and individual performance planning, all of which remain relatively undeveloped. Table 12: Use of Management Standards RHB (Average All Regions) (%) Uses RHB 5 ZHD WorHO Health Center In preparing annual plans of action 100 62 68 72 In preparing individual results based performance plans 0 22 33 43 In conducting supportive supervision 100 78 67 53 To guide monthly management meetings 0 70 69 69 To guide quarterly review meetings 0 57 52 59 6 To guide monthly HMIS reviews and action planning 100 62 ND 54 In epidemic preparedness and response 0 37 14 27 To guide financial management 0 45 44 32 Other 0 13 19 12 5 Data available for Oromia only. 6 Based on Oromia and SNNP data. Data were not available for Amhara. 13

5.2.4 Management Committees Availability and reported use of management standards alone is not enough to ensure they are used and applied appropriately. The end-line survey also asked each level about availability of management committees and reviewed committee minute books to establish the date of the last meeting and issues being discussed by the committees. The data suggest that management committees exist, are active, and are addressing critical management issues. Management committee meetings are regular at most levels, with all RHBs stating they have regular management committee meetings and 89 percent of health centers saying their management committees meet regularly. The management standards for each region stipulate different standards for meeting regularity at various levels with meetings, ranging from weekly for Oromia RHB to monthly for SNNP WorHOs. A great majority of RHBs, ZHDs and WorHOs are meeting this standard (Table 13). On the other hand, only 44 percent of health centers are meeting the standard which for all regions is every two weeks although 84 percent of health centers are meeting on a monthly basis. Because of this, regions may want to consider revising the standard for health centers to monthly rather than bi-weekly meetings. Most encouraging were the issues documented in the management committee minute book: across regions, health center management committees discussed human resource issues such as training, work schedule and timeliness, and incentives. They also discussed health care financing, logistics issues, service quality and supervision, all key elements of the management standards. 14

Table 13: Frequency of Management Committee Meetings (Average All Regions) (%) Criteria RHB (N=3) ZHD (N=13) WorHO (N=70) Health Centers (N=83) Management committee meets regularly 100 92 91 89 Percent of offices/facilities whose management committee meets according to the standard 100 7 93 83 44 Percent of offices/facilities whose management committee meets at least monthly 5.3. Supervision 5.3.1 Supportive Supervision 100 100 95 84 The ESHE baseline assessment in 2004 found all woredas used supervisory checklists but these checklists were divided by program area and tended to be used for assessing facility records. In all regions, the process was conducted in an audit fashion, by a variety of technical supervisors who worked more or less independently, and provided no clear guidance for performance improvement. One of ESHE s key interventions was the introduction of an integrated supportive supervision process and tools. The purpose was to improve quality and results of supervision such that it is better coordinated, focuses on the whole package of services offered, is constructive and supportive to help motivate staff, and results in concrete action plans to guide and monitor performance improvement activities. Unlike the previous tools, the integrated supportive supervision checklists (ISCLs) were to be used by an integrated team, stressed discussion and observation, in addition to record review, and emphasized the review of supervision findings with staff and development of action plans. 5.3.2 Supportive Supervision Tools The survey revealed that across regions, these supportive supervision tools are widely available, are in use, and are appreciated by staff. To improve staff and supervisors capacity to address strengths and weaknesses and identify priority areas for remedial interventions, ESHE integrated a scoring system into the ISCLs. This system was field-tested in Oromia in October 2006. After the field test, the revised and finalized tools have been in use in the Oromia since December 2006 and SNNP since mid-2007. Amhara has a copy of the scored checklists but has not yet implemented the scoring system. In Oromia and SNNP the scoring system has been received well and, although appearing complicated, staff use it with ease and enthusiasm. Across regions and offices, staff have said they prefer the integrated supportive supervision checklists to the previous supervision format. Of greatest importance to them is the tool s focus on a supportive approach, including active problem solving. Managers stated the tools have helped them make supervision more collaborative 7 Based on Amhara and SNNP data, no data available for Oromia. 15

and more facilitative. They also felt the process has resulted in improved performance. Perhaps more importantly, service providers at facility level expressed enthusiasm for the process, saying they felt it was more constructive and helped them to improve their work. These sentiments were expressed repeatedly across the regions. One challenge of supportive supervision checklists, specifically those with the scoring system, is that they must be revised with care. The scoring system requires that all scored questions be applicable and high performance always score a 2 and that low performance is represented by a 0. Both Oromia and SNNP have made changes to the ISCLs to reflect their respective regional priorities. These revisions have already demonstrated that the restrictions of the scoring system are hard to follow when adapted by people who do not know it well and that the scoring system can become compromised. Another key to the success of the supportive supervision checklists is that they remain manageable. Care should be taken when revising the tools on a regional level so that they do not become too long and ineffective. 5.3.3 Supervisory Roles and Responsibilities A thorough understanding of the supervisory roles and responsibilities among different levels of the health system is a prerequisite to effective supervision. The baseline assessment revealed considerable role confusion, with every level asserting its responsibility to supervise every level under it, rather than focusing on the level directly below. This made supervision unfeasible, duplicative and confusing. The Project sought to address this issue by working with RHBs to include supervision responsibilities into the management standards. The management standards clearly lay out responsibilities, stating that the RHB is primarily responsible for supervising ZHD, ZHD for WorHOs, WorHOs for health centers, and health centers for health posts. While RHBs, ZHDs or WorHOs can still choose to do periodic checks of other lower levels, performance is measured based on their fulfillment of supervision responsibilities for the level directly below. Data collected during the end-line survey suggest that RHBs, ZHDs and WorHOs are providing supervision to the levels directly below them, although each continues to do spot checks of other offices and facilities. This makes clear that there is a positive change in understanding of supervisory roles and responsibilities by the RHB and its lower structures. However, supportive supervision remain irregular, suggesting that offices may want to focus more energy on meeting their obligations for supervising the level directly below them and put off supervision of other levels until primary supervisory responsibilities have been met. 5.3.4 Supervision Planning At baseline assessment, 50 percent of ZHDs in Oromia, 40 percent in Amhara and 60 percent in SNNP had plans to supervise WorHOs on a quarterly basis. Likewise, 81 percent of WorHOs in Oromia, 80 percent in Amhara, and 80 percent in SNNP had plans for quarterly supervision visits to health centers. As demonstrated in Table 14, the end-line assessment shows improvement in supervision planning across regions, almost all ZHDs and WorHO having documented plans for quarterly supervision. 16

Table 14: Offices with Documented Supervision Plans (By Region) (%) Region ZHD ZHD WorHO WorHO Baseline End-line Baseline End-line Oromia 50 100 81 90 Amhara 40 100 80 86 SNNP 60 80 80 85 5.3.5 Supervision Frequency Supervision visits need to adhere to schedules in order to be effective. Documenting the frequency of supportive supervision proved challenging for the assessment team; staff perspectives are important, but are not verifiable. When supervision is complete, supervising levels are supposed to leave a copy of the ISCL so that staff in the office or facility have a record of supervisory findings and a copy of the action plan to guide improvements. The assessment team tried to verify staff assessments of supervision frequency by reviewing available completed checklists at the office or health facility under review. Unfortunately, many offices and facilities do not appear to keep copies of the supervisory checklists for very long, so data on number of supervision visits conducted remains weak. That said, availability of checklists and interviews with staff suggest that while the frequency of supervision visits has improved, supervising levels have rarely been able to conduct quarterly supervision visits due to staffing, time and/or budget constraints. Table 15 shows WorHOs provided the greatest number of supervision visits. This is important since it has the most direct effect on service delivery Table 15: Reported Average Number of Supervision Visits per Year Minimum Standard Criteria Supervision to ZHD by RHB Supervision to WorHO By ZHD Supervision to Health Centers By WorHO Oromia Two visits/year by the RHB to the ZHDs Four visits/year to WorHOs and Health 1.5 2.7 2.8 8 Centers Amhara Four visits/year 0.13 1.6 2.6 SNNP Two visits/year by the RHB to the ZHDs Four visits/year to WorHOs and Health Centers 1.4 2.7 3.4 Average All Regions 1 2.3 2.9 Achievement All Regions (%) 38 58 73 Keeping the limitations of record keeping in mind, the review of supervision documents suggest that while the frequency of supervision fell far from meeting the standard of the regular quarterly visits for WorHOs and health centers, supervision frequency did improve over the course of the Project. Table 16 shows the increase in percentage of supervision visits achieved, assuming quarterly supervision to each 8 Calculated based on the number of health facilities that reported an average number of times the WorHO visited. 17

office or health center, from 2005 through 2008. Supervision of WorHOs by ZHDs was strongest with an average of 18 percent of visits being conducted in 2007-2008. WorHO supervision achievement for health centers followed closely at 17 percent. Oromia demonstrated the strongest achievement in completing supervision visits with 33 percent of supervision visits by WorHOs to heath centers achieved as compared with 6 percent and 13 percent in Amhara and SNNP, respectively. Oromia s achievement is likely linked to the great interest shown and leadership provided by the Oromia RHB for supportive supervision. 18