External Evaluation of Effectiveness of UNICEF Nutrition Accelerated ed Reduction of Child and Maternal Under-Nutrition in Seven Districts of Sierra

Transcription

1 External Evaluation of Effectiveness of UNICEF Nutrition Accelerated ed Reduction of Child and Maternal Under-Nutrition in Seven Districts of Sierra Leone Final Report January 2017

2 Report Presented by Global Health Liaisons, LLC 2009 Lansdowne Way Silver Spring, MD Evaluators Christina Blanchard-Horan, PhD Magda Rodriguez Gonzalez, PsyD Elisabetta Aurino, PhD Muthoni Njage Ibrahim Fofanah, Prof 1 P a g e

3 ACKNOWLEDGEMENTS This report was made possible thanks to the significant time, effort, and contributions of many people, both inside and outside UNICEF. The evaluation team would like to recognize individuals who made this evaluation possible. We would like to acknowledge Kajali Paintal, UNICEF Nutrition Manager, Alison Jenkins, UNICEF Chief Child Survival, and the entire UNICEF Nutrition team. Thanks also to Mr. Kshitij Joshi, UNICEF Director of Communication Development and M&E; Kebir Hassen, UNICEF BCC Community Health Specialist; Ms. Charlotte d Elloy at Irish Aid, and Fortune Maduma at the World Food Programme. We would also like to express our appreciation to Aminata Shamit Koroma of the Ministry of Health and Sanitation Director of Food & Nutrition and the National SUN Technical Focal Point. The Implementing Partners (IPs) were also critical to this assessment, these included Alfred Swarary, Director at SILPa in Kenema, Patrick Sannoh and Emmanuel Monina at PACE in Tonkolili, Abdul Sankoh and Anita Sankoh at CAWeC in Kambia, Hawa Saffa and Felix Kpang at DIP in Port Loko, Saidu Kanu and Brima Samura at WHI in Bombali, and Joseph Bangura and Thomas Mansaray at CAUSE Canada in Moyamba. They provided tremendous input, cooperation, and collaboration during the evaluation process. We would like to acknowledge the Keyboard Social Science Research Institute Ibrahim Fofanah, Akin Thomas, Alusine Bahasa, and Moses Sorba -- for their tireless effort, knowledge sharing team spirit, and professionalism with data collection under difficult conditions and tight time constraints. The evaluation team included Christina Blanchard-Horan, team lead, Jasmine Fledderjohann, Elisabetta Aurino, and Sam Dickson, biostatisticians, Tahrima Choudhury, and Magda Rodriguez Gonzalez, behaviour change, Muthoni Njage, policy, and Gakenia Wamuyu Maina, nutritionist specialist. Of course, we would like to thank all participants in this study, including the District Medical Officers Sesay Foday, Sesay Tom, Kamara Sayo, Jimisa Augustsinen, M. Vandy, David Bome, and A.J. Moosa. We thank the community health and facility workers, caregivers, and beneficiaries. Thank you all for your time and valuable insights. We are especially thankful for your taking the time from the important work you do to respond to our questions. Your contributions ensure a more robust evaluation. 2 P a g e

4 TABLE OF CONTENTS Acknowledgements... 2 Table of Contents... 3 List of Tables... 0 List of Figures... 0 Abbreviations and Acronyms... 1 Executive Summary... 2 Chapter 1 Background Policy & Advocacy UNICEF Sierra Leone Programme Objectives Ebola Programme Resources Objective of the Evaluation Methodology Chapter 2 Findings of the Evaluation Overall Relevance, Appropriateness and Coherence Equity and Coverage Efficiency & Effectiveness of Nutrition Supply Management Efficient and Effective SAM Management Health Outcomes Efficiency & Effectiveness of Social Mobilization and IYCF Adequacy of M&E Framework Sustainability Chapter 3 Conclusions Chapter 4 Lessons Learned & Recommendations Recommendations References Appendices Appendix I: Sierra Leone Sociodemographics Appendix II: Supplementary Tables Appendix III Financials PHOTO 1. YOUNG BENEFICIARY, MAKENI, SIERRA LEONE PHOTO 2. TEAM CROSSING RIVER TO REACH PHU PHOTO 3. CAWEC, PUJEHUN PHOTO 4. RUTF SUPPLY, KAMBIA PHOTO 5. SIERRA LEONE PHU PHOTO 6. NUTRITION EDUCATION SESSION, PUJEHUN PHU PHOTO 7. COMMUNITY HEALTH WORKER ANNEXES (CAPTURED IN A SEPARATE DOCUMENT) ANNEX I, TOR ANNEX II, LIST OF PERSONS INTERVIEWED AND SITES VISITED ANNEX III, LIST OF DOCUMENTS CONSULTED ANNEX IV, EVALUATORS BIO DATA AND/OR JUSTIFICATION OF TEAM COMPOSITION ANNEX V, EVALUATION MATRIX ANNEX VI, INFORMED CONSENTS ANNEX VII DATA COLLECTION TOOLS 3 P a g e

5 List of Tables TABLE 1. PRIMARY DATA COLLECTION TABLE 2. RESULTS AREAS DEFINED TABLE 3. IPS IN EACH DISTRICT TABLE 4. CASELOAD AND STOCKING, BY YEAR TABLE 5. STOCKOUTS MAY 2014 JAN 2015 IN SEVEN DISTRICTS (FHS DATA) TABLE 6. NUMBER OF FACILITIES 2013, 2014, 2015, TABLE 7. TYPES OF NUTRITION-RELATED ADMISSIONS REQUIREMENTS MENTIONED TABLE 8. FACILITY STAFF S YEAR OF LAST TRAINING PRIMARY DATA TABLE 9. CHANGE IN ADMISSIONS PRE/POST EBOLA TABLE 10. OEDEMA, RED MUAC, SEVERE WASTING REDUCTION 2013, 2014 AND 2015 IN THE SEVEN DISTRICTS TABLE 11. MONTHLY ADMISSIONS FOR OEDEMA IN SEVEN DISTRICTS TABLE 12. MSG MEMBER TRAINING - PRIMARY DATA OCT TABLE 13. DENSITY QUESTIONNAIRE CHILD S AGE TABLE 14. CAREGIVERS OF CHILDREN PARTICIPATING IN VARIETY INTERVIEW - PRIMARY DATA TABLE 15. MEASURES FOR EFFECTIVENESS OF BEHAVIOUR CHANGE TABLE 16, LESSONS LEARNED SUMMARY TABLE 17, OVERALL FINDINGS & CONCLUSIONS TABLE 18, TRAINING FOR MSGS TABLE 19, FACILITIES PROVIDING AND NOT PROVIDING OTP SERVICES IN THE SURVEY TABLE 20, DISTRICT POPULATION <5, <1, AND NEONATES TABLE 21, RATES FOR STUNTED, SEVERELY STUNTED, WASTING, UNDERWEIGHT BY DISTRICT TABLE 22, CURE, DEATH, AND DEFAULT RATES 2013, 2014, AND TABLE 23, 2013, 2014, TOTAL ADMISSIONS BY DISTRICT BY YEAR TABLE 24, AVERAGE NUMBER OF CHILDREN ADMITTED IN OTP % CHANGE PRE- POST EBOLA 2014/ TABLE 25, BUDGET ACCELERATED REDUCTION OF CHILD AND MATERNAL UNDERNUTRITION TABLE 26, 2014 EBOLA OUTBREAK RESPONSE PROPOSAL BUDGET TABLE 27, BUDGET , REDUCING CHILD AND MATERNAL UNDERNUTRITION IN SELECTED DISTRICTS TABLE 28, TOTAL OF FUNDS ($) List of Figures FIGURE 1. THE SEVEN EVALUATION DISTRICTS... 3 FIGURE 2. BUDGETING OF FUNDS AVAILABLE FIGURE 3. TIMELINE OF DOCUMENTS REVIEWED FOR UNICEF NUTRITION PROGRAMME EVALUATION FIGURE 4. PERCENTAGE OF MSGS WITH GROUP FACILITATION SKILLS, UNMARRIED TEENS AND MALE COUNSELLORS BY DISTRICT (IP REPORTED) FIGURE 5. CHALLENGES FACED BY PHUS DISTRIBUTING SUPPLIES FIGURE 6. AVERAGE PERCENTAGE OF RUTF STOCKOUTS AT OTPS OCT 2014 JAN FIGURE 7. FACTORS THAT HELPED WITH NUTRITION AWARENESS ACTIVITIES, ACCORDING TO FACILITY STAFF FIGURE 8. ADMISSIONS BY DISTRICT OVER TIME FIGURE 9. ADMISSIONS BY CAUSE/PROPORTION FIGURE 10. PREVALENCE OF SAM & MAM BASED ON VARIOUS DATA SOURCES FIGURE 11. MEAN MONTHLY ADMISSIONS WITH OEDEMA FIGURE 12. CURE RATES IN THE SEVEN DISTRICTS FIGURE 13. DEFAULT RATES IN THE SEVEN DISTRICTS FIGURE 14. DEATH RATES IN THE SEVEN DISTRICTS FIGURE 15. NUMBER OF CHILDREN WHO ATTENDED GROWTH-MONITORING PROGRAMME AND PERCENT CHANGE FROM PRE-EBOLA TO DURING EBOLA CRISIS FIGURE 16. IP-REPORTED MSGS IYCF TRAINED JAN 2014 MAR FIGURE 17. ADVANTAGES TO FOOD VARIETY, CAREGIVER DOERS AND NON-DOERS FIGURE 18. FOOD DIVERSITY 6 24 MONTHS - PRIMARY DATA OCT

6 Abbreviations and Acronyms ACTS BCC BPEHS CPAP DFN DHS EVD FHCI FGD FSN FSNA HFA HCP HF TAG IEC IMAM IP IPF IYCF KII KIIs MAM MICs MNP MoHS MSG NFNSP NGO OOP ORS OTPs PHU ReSoMal RUTF RUIF SAM SILPA SLMS SLNNS UBC UNICEF WFP WHO Action for Community Transformation and Sponsorship Behaviour Change Communication Basic Package of Essential Health Services Country Programme Action Plan Director of Food & Nutrition Demographic and Health Survey Ebola Virus Disease Free Health Care Initiative Focus Group Discussions Food Security Network Food Security Nutrition Assessment Health Facility Assessment Healthcare Providers Home Fortification Technical Advisory Group Information, Education and Communication Integrated Management of Acute Malnutrition Implementing partner In-patient facility Infant and Young Child Feeding Key Informant Interview Key Informant Interview surveys Moderate Acute Malnutrition Multiple Indicator Cluster Survey Micronutrient Powders Ministry of Health and Sanitation Mother Support Group National Food and Nutrition Security Policy Non-Governmental Organization Out of pocket Oral Rehydration Salts Outpatient Therapeutic Programmes Peripheral health unit Oral rehydration solution for severely malnourished children Ready to use therapeutic food Ready to use infant formula Severe Acute Malnutrition Sierra Leone Poverty Alleviation Agency Sierra Leone Micronutrient Survey Sierra Leone National Nutrition Study University of British Columbia United Nations International Children's Fund World Food Programme World Health Organization 1 P a g e

7 External Evaluation of Effectiveness of UNICEF Nutrition Programmes in Seven Districts of Sierra Leone EXECUTIVE SUMMARY The UNICEF Nutrition Programme has worked in partnership with the Sierra Leone Ministry of Health and Sanitation (MoHS) Director of Food & Nutrition (DFN) since 2013 to reduce malnutrition by a third by To achieve this drastic reduction, UNICEF utilised a multi-pronged strategy, including the following: a) support upstream policy reforms, advocacy, institution building and decentralised planning and budgeting, b) influence policy and planning to improve nutrition, c) build health worker capacity to deliver quality treatment of severe acute malnutrition (SAM), d) advocate and contribute to the development of the integrated management of acute malnutrition (IMAM) and promotion of proper infant and young child feeding (IYCF) protocols, e) strengthen and expand mother support groups (MSGs) and promote behavioural and social change in households, f) procure and distribute nutritional supplies, g) strengthen systems for real-time monitoring and evaluation (decentralised monitoring) and h) build capacity and emergency coordination. Programme objectives were to improve the quality of treatment, ready-to-use therapeutic food (RUTF) supply chain, feeding practices at the community level and coordination with nutrition partners. This report breaks these UNICEF strategies into five basic categories, which are in line with the UNICEF log frame and designed to address the programme objectives. The categories are 1) Advocacy and Policy Change (nutrition policy framework), 2) Improved Data Systems and Monitoring and Evaluation (M&E) Reporting, 3) SAM Treatment, 4) Supplemental Food Supply Management and 5) Social Mobilisation - IYCF counselling and community engagement to change behaviour. The programme is a continuation of the collaboration between UNICEF Sierra Leone and Irish Aid, established in This evaluation is part of a commitment made to Irish Aid to evaluate the effectiveness and impact of UNICEF s work in Irish Aid-supported districts. Purpose of the Evaluation In April 2016, Global Health Liaisons LLC was contracted by UNICEF/Sierra Leone to evaluate the Irish Aidfunded Nutrition Programme. The purpose of the evaluation, as stated in the scope of work, was to understand how the Nutrition Programme in the seven Irish Aid-supported districts 1 (Figure 1) worked and determine what worked well, where, why and under what circumstances. It also sought to generate knowledge as well as identify best practices and lessons learned. Objectives of the Evaluation 1. Obtain an unbiased assessment of whether the planned activities with inputs invested have led and/or contributed to the achievement of the expected results (outputs, outcomes and impact). 2. Identify the reasons the achievements have (or have not) been made. 3. Examine any unintended positive or negative consequences of the programmes. Methodology The analytical framework for this evaluation was based on the 17 research questions outlined in the Evaluation Framework and written into the results (Annex V in attached document). The evaluation used six collection methods: a literature review, 20 key informant interviews (KIIs), 1,920 semi-structured 1 Districts included Bombali, Kambia, Kenema, Moyamba, Port Loko, Pujehun and Tonkolili. 2 P a g e

8 interviews, seven focus group discussions (FGDs) and secondary data from surveys and reports. These came from multiple sources: the Health Facility Assessment (HFA), the Sierra Leone National Nutrition Study (SLNNS) and the Sierra Leone Micronutrient Survey (SLMS). Data files from these sources and the decentralised data were provided by UNICEF. Implementing partners (IPs) provided screening and training data. The literature review generated tentative findings for each evaluation question and guided further investigation. The review included primary documents with three UNICEF proposals and seven progress reports, which were submitted over the programme evaluation period. Reports generated from decentralised data were also reviewed as were the M&E Handbook, IMAM protocol and Evaluation of IMAM & IYCF Programmes (2013). The team also reviewed additional literature provided by UNICEF, including findings from various nutrition studies, such as the Micronutrient Feasibility Study, and countrylevel reported data. A complete list of publications reviewed can be found in Annex III. The analysis of data from KIIs, semi-structured interviews and secondary data sources contributed to the understanding of programme impact on health over time and validation of reported results and findings from the literature review. Data collection and reporting were grouped into the primary themes outlined by the Terms of Reference (TOR): 1) Relevance, Appropriateness and Coherence, 2) Effectiveness and Efficiency and 3) Scale-up and Sustainability. Main Findings Relevance, Appropriateness and Coherence of the Programme Assess the relevance, appropriateness and coherence of the Nutrition Programme in the seven Irish Aid-supported districts in relation to global, regional and country programme strategies concerning the specific context of Sierra Leone. UNICEF has been a major advocate for child health and contributor to the development of health policy that supports nutrition improvement around the globe. UNICEF Sierra Leone collaborated with other UN partners to establish strategies while appropriately leveraging existing relationships with the Sierra Leone MoHS and DFN. This contributed to nutrition policy change and the identification of barriers and bottlenecks to improving IYCF practices. They built capacity at government facilities to address these by expanding the availability of IYCF counselling through health facilities, community health workers (CHWs) and MSGs in the community. Figure 1. The seven evaluation districts Evaluation District UNICEF has made numerous contributions to the development of knowledge around the problem of malnutrition in Sierra Leone. At a national level, the Food Security Network (FNS) Policy Implementation Plan of was developed by the MoHS, with support from UNICEF. Together with the World Health Organisation (WHO) and the World Food Programme (WFP), UNICEF has been a key partner in the treatment of acute malnutrition in Sierra Leone. 3 P a g e

9 UNICEF supported knowledge building with the implementation of the Multiple Indicator Cluster Survey (MICs) survey; they conducted a situation analysis in 2013 and supported a feasibility study for micronutrients in 2013, which was followed up with a pilot in Targets for micronutrients were not achieved in The target was 50% of the target facilities reporting postpartum vitamin A supplementation, while UNICEF achieved 45%. Of the 50% targeted for reporting on vitamin A and deworming for children under 5, 31.3% of the target was achieved for micronutrient powders (MNPs) for children. Also in 2015, UNICEF supported the post-ebola Comprehensive Food Security and Vulnerability Analysis (CFSVA). Following the Ebola crisis, they supported a facility assessment. UNICEF supported the WHO to design the guides for the IMAM. UNICEF partnered with the WFP to roll out the IMAM guidelines and associated training for the use of RUTF and, for Ebola, ready-to-use infant formula (RUIF) to treat severely malnourished infants (see Figure 3 for a timeline.) The IYCF counselling guide, the National Food and Nutrition Security Plan (NFNSP) of 2013 as well as decentralised monitoring results indicators were designed and aligned with global standards (see Section 2.7). Most recently (2016), the M&E handbook for IYCF and community-based management of acute malnutrition (CMAM) was developed in a partnership that included UNICEF, the WHO and the WFP to strengthen IYCF practices in Sierra Leone. IYCF promotion would emphasise exclusive breastfeeding (EBF) up to 6 months and complementary feeding with continued breastfeeding until 2 years of age. The Nutrition Programme has been widely appreciated at multiple levels (e.g. national, district, peripheral healthcare unit (PHU), community) for its role in nutrition advocacy, planning and support. UNICEF used its unique position to provide guidance and advocacy to the government. UNICEF practiced relevant advocacy to develop national guidelines and programmes in partnership with the MoHS, local and international non-government organisations (NGOs) and other UN agencies. The Country Programme Action Plan (CPAP) was an agreement between the government of Sierra Leone (GoSL) and UNICEF that contained a logical framework for their collaboration. The framework has been the basis for UNICEF s work in Sierra Leone for the past five years. (However, UNICEF did not share its methods or documents used for reporting between the two entities.) The UNICEF and DFN MoHS collaboration has been critical to the successful integration of nutrition services into the existing health system. The following is a summary of the findings related to relevance, appropriateness and coherence: UNICEF appropriately engaged local NGOs as IPs, building local capacity to support programme monitoring for quality assurance, training and supply management at the district and facility levels as well as providing support for the community. The IMAM strategies to scale up facilities and train healthcare staff have been appropriate to the region. Although some of these efforts were delayed during the Ebola outbreak, UNICEF and MoHS have made strides towards increasing the number of facilities and training and retraining staff to provide quality nutrition services. Efforts towards collaborative multi-sectoral initiatives that provide treatment and address food security were not evident. Although this was discussed during national-level interviews, it was not evident at the facility level. Community training through MSGs was an appropriate and efficient method of reaching hard-toreach communities with IYCF counselling and was consistent with UNICEF s global strategies to address malnutrition. However, there was a need to ensure for more effective messages and identify champions to diffuse appropriate IYCF knowledge to communities. 4 P a g e

10 The reporting framework for the programme was captured in log frames housed in proposals and progress reports; these were lacking details and indicators on supply management, Child Health Days, nutrition screening and referrals and training. The coherence of programme strategies and planned results was evidenced by the connection not only at national levels but also between communities and facilities. Facilities valued community counselling and referrals and coordination with district-level IPs. MSG members, MSG leaders, CHWs and Maternal and Child Health aids (MCH Aids) indicated that they had a strong connection with the facility, some going daily and/or weekly to the facility. However, coherence was not evidenced in supply provision. The connection between district-level implementers and facilities was also demonstrated during interviews with staff at outpatient therapeutic programme (OTP) sites, when they stressed the importance of training and interaction with IPs. Effectiveness and Efficiency Assess the effectiveness and efficiency of the Nutrition Programme in the seven Irish Aid-supported districts in terms of delivery of services to infants and young children and promoting appropriate IYCF behaviours/practices. Evidence of the effectiveness and efficiency of SAM management programme services was determined by the improvement of cure rates, default rates and M&E reporting over the evaluation period. It was noted that UNICEF and the DFN had recognised challenges in the area of supply management, responding with the construct of an accountability matrix. The matrix was not provided to evaluators for investigation. Donor respondents underscored ongoing concerns around supply chain management accountability. Based on FGDs, the rollout of IYCF in facilities has done well in changing people s perspectives. They understand their ability to make a difference in their children s health. Perceptions such as these ultimately tend to lead to more successful and sustained behavioural changes in health practices. While the effort to improve IYCF practices has improved the knowledge and beliefs of Sierra Leoneans, focus group data show that self-regulation skills and abilities, such as the ability to track child growth, are poor in the seven districts. Child growth monitoring needs to be emphasised as an element of the nutritional campaign. Based on the review of the secondary data available (e.g and 2013 Demographic and Health Surveys, 2010 Multiple Indicators Cluster Survey, 2014 Sierra Leone Nutrition Survey), severe stunting and wasting were declining in some districts, prior to the 2014 Ebola Virus Disease (EVD) epidemic, which may be attributed to a combination of factors that includes the Nutrition Programme. The lack of data from 2015 onwards constitutes a gap in the evaluation of programme impact during and after the Ebola outbreak. For the promotion of appropriate IYCF behaviour, systems to improve IYCF practices have been established. CMAM has been integrated into the overall national nutrition system, from policy improvements that support efforts to identify and treat SAM cases to the training of health workers to implement the protocol. These resulted in increased screening, and the improvement in cure rates demonstrates the efficacy of these efforts. Discussions to discern behaviour change during FGDs revealed a need to ramp up IYCF counselling and dispel misunderstandings about EBF, food density (such as the relationship between thick porridge and constipation), food variety and access to vitamin-rich foods. There was a need to continue efforts to 5 P a g e

11 increase EBF for children under 6 months, to improve complementary feeding and to work with other agencies in a multi-sectoral approach towards improving access to an improved variety of foods. The following is a summary of the findings related to efficiency and effectiveness: Although the framework did not detail nutrition screening, per data collected from IPs, close to 2 million (1,935,266) screenings were reported by IPs in six districts (covering December 2013 August 2016). Global Acute Malnutrition (GAM) and Moderate Acute Malnutrition (MAM) were reduced in the seven districts Secondary data and the project End Report demonstrate that GAM and Moderate Acute Malnutrition (MAM) rates were declining. However, these data need to be assessed cautiously, and further data are needed to prove causality, as other factors are also likely influencing these outcomes. Cure, non-recovery and default rates improved over the three years between January 2013 and December 2015, suggesting IMAM training was impactful. M&E reporting contributed to the efficiency and improvement of nutrition programming and was in line with core Sphere standards, the Humanitarian Charter and Minimum Standards in Humanitarian Response. 2 Decentralised monitoring for reporting resulted in improved tracking systems for performance. However, as stated above, data were unreliable, and there was a lack of secondary data from 2015 onwards. Although UNICEF and the MoHS made strides in the development of monitoring systems, there was a need to strengthen these further: for instance, the evaluator faced months of missing data from some districts (which was most likely related to the EVD epidemic). This could also speak to the capability of decentralised entities to keep track of performance measures in a timely/constant manner during emergencies. The evaluator considered these factors when analysing existing datasets combined with primary data collected during the evaluation. The desk review and national- and district-level interviews also revealed concerns about the ability of the programme to effectively meet the supply demands. UNICEF was one of the key players supporting the government in supply management, but the responsibility should not lie with UNICEF alone. Reports conveyed systemic and structural challenges related to supply chain management. UNICEF worked with the DFN to design a supply chain accountability matrix to address concerns about supply management. It was rolled out at the national level in 2015 and to the districts in Whether the matrix was used in a timely or regular way was not investigated, and results from this investigation suggest there remain issues around supply chain management at the district and facility levels. UNICEF Sierra Leone remains flexible within the environment in which they are working, adapting as needed. UNICEF diverted funds to conduct a facility assessment after the Ebola crisis, which was needed to understand Ebola s impact on health services. UNICEF experienced challenges with the rollout of the IMAM protocol changes. Training of MSGs and health facility staff was also delayed and re-established post-ebola. Rollout of the IMAM guidelines was completed in late 2015 and in the first quarter of Sustainability and Scale-up Assess the extent to which the programme is sustainable in addressing undernutrition in young children with attention to the less-reached, disadvantaged and vulnerable groups. In summary, numerous programme activities demonstrated integration into the national system and support for sustainable policy and interventions. UNICEF, in collaboration with GoSL, supported the 2 The Sphere Project Humanitarian Charter and Minimum Standards in Humanitarian Response, P a g e

12 NFNSP and National Technical Committee for Nutrition Surveys. In collaboration with the WHO, UNICEF supported the MoHS to develop and distribute IMAM guidelines and train staff in its use. The technical assistance and financial support provided by UNICEF to the MoHS was essential for the development and production of the June 2014 guidelines. Assess leadership, guidance and technical support at all levels and the adequacy/institutional capacity to respond to the lead role UNICEF is expected to play at the field level for contributing to the sustainable and equitable reduction of child undernutrition. Since the start of the evaluation period in December 2013, UNICEF has had several changes in the Nutrition Programme staff, including changes in leadership and virtually all team members. These shifts in leadership and team members occurred during and after the Ebola outbreak. Changes involved all but one staff member that was at UNICEF during the interviews had been involved in activities before Therefore, leadership, guidance and technical support at all levels have shifted over time. Pre-Ebola, when the programme clearly had some momentum in changing policy and working with government, they had worked with other UN agencies to design appropriate guidelines and strategic plans and roll out the integration of the Nutrition Programme into the health system and had started to build a multi-sectoral effort to address food security. UNICEF advocacy with the GoSL led to the MoHS joining the international Scaling Up Nutrition (SUN) movement. They established the Directorate of Food and Nutrition (DFN) at the MoHS and housed her in an office adjacent to the UNICEF Nutrition Programme. The current leadership appears to be experienced in addressing hard-to-reach communities and is building the UNICEF team to further the mission of the UNICEF office in Sierra Leone. The evaluation team noted a level of disorganisation when it came to requests for information, especially regarding raw data. Files were disorganised, and half of the districts under evaluation had missing data from the main report that were captured in other reports from January 2015, which had not been incorporated into the combined files. Given that UNICEF s greatest financial commitment is the procurement and provision of nutrition supplies, the future should see UNICEF taking a leading role, in partnership with the DFN, to ensure supply management accountability. Building local capacity by strengthening local IPs will contribute to sustainability. UNICEF s support for local NGOs as IPs should ensure they are properly trained to aid in the efforts to improve programme data reporting and supply distribution and provide additional support for community outreach. At the district level, it could improve programme performance data collection and training of healthcare staff and community educators and screeners. Two years into the project, UNICEF expressed concern about the capacity of the SUN Secretariat to develop a multi-sectoral plan and coordinate related activities. UNICEF should take a leading role to encourage multi-sectoral approaches to food security and provide coordination support to move this forward. UNICEF could strengthen efforts with other UN agencies, like the WFP and the Food and Agriculture Organisation (FAO), to connect communities with opportunities to address food shortages and increase food diversity. This would be in line with UNICEF s Multi- Sectoral Approach to Nutrition Services (2016). 3 Conclusions The UNICEF Nutrition Programme design follows the core Sphere standards, a framework supported by a wide range of international humanitarian agencies to improve the quality of humanitarian assistance and 3 UNICEF Multi-Sectoral Approaches to Nutrition - #4 7 P a g e

13 accountability. UNICEF did not document the application of any Theory of Change. To do so requires a programmatic review of results and strategies and documentation of how the Theory of Change is used, and it should be built into the design of the programme. The following conclusions are organised by programme objectives. Objective 1: Improve the quality of the treatment of SAM by increasing adherence to national IMAM protocols at OTPs and In-Patient Facilities (IPFs). UNICEF improved the quality of the treatment of SAM. The Nutrition Programme in Sierra Leone is showing signs of impact on the health of children. Mortality rates have decreased, as have health indicators. IMAM treatment protocol training and the expansion of sites offering OTPs for nutrition demonstrated an increase in access to SAM treatment. Increases in training and opening of new sites have influenced SAM outcomes. Indicators show gradual improvement in cure rates over the years. Default rates have also plummeted over that same period. Deaths were reduced in most districts. Further analysis is needed to assess programme effectiveness during emergencies. The recommendation would be to create a system that works collaboratively with existing emergency procedures. Objective 2: Strengthen the supply chain management of RUTFs and improve communities adhesion and participation in IMAM activities. UNICEF and the MoHS recognised the issues around poor supply chain management and designed a plan to address them. Together they developed an accountability matrix. Consultants did not observe improvement in supply chain management, likely because the initiative was recently rolled out. UNICEF reached tens of thousands of communities through MSGs, CHWs and facilities. This led to large-scale SAM screening for participation in and adhesion to IMAM activities, as was demonstrated by the high number of screenings and reduction in default rates over the years. Objective 3: Improve IYCF practices at the community level through community-based structures and PHUs. The programmes were found to be relevant to the Sierra Leone context, in that UNICEF identified challenges in the target community and designed systems to address them. However, work remains to be done. There was no evidence of growth monitoring and tracking after the Ebola epidemic. Other than IPs reports of almost 2 million screenings of children, there was no community-level screening documentation outside of UNICEF reports to verify these numbers. Staff and health community training was effective, in that implementers understood the strategies that were being used to change behaviour and applied this knowledge to curing children with SAM. Facility staff members were trained in the IMAM guidelines and data collection, yet more work is needed to improve the data-collection aspect of their work and to continue to improve the quality of IYCF counselling intended to change behaviour. Behaviour change communication (BCC) was strengthened by being mindful of cultural practices, using the local language and using illustrations of foods grown in the country. However, access to food/drugs and the ability to identify with MSGs and medical personnel still prove to be challenging. Objective 4: Maintain adequate coordination of nutrition partners and activities from decentralised to central level. The role of UNICEF as nutrition sector lead partner with government and its technical leadership in pushing forward the programme policies and guidelines demonstrates a strong relationship with the MoHS. This has positioned UNICEF to guide the integration of nutrition services into the Basic Package of Essential Health Services (BPEHS), critical to the transition from emergency response to countryled and -owned services. With the support of UNICEF, the MoHS has led the initiative that integrated nutrition into public health systems in hundreds of communities. 8 P a g e

14 CHAPTER 1 BACKGROUND As of 2015, according to UN census data Sierra Leone had a population of 6,592,000**,. 4 According to UNICEF, the UN calculates children under 5 years at 16.5% of the population, under 1 year at 4% and neonates at 3.5%, making the total population under 5 years of age 1,087, In the seven districts under evaluation, the total population is 3,368,222 and that under 5 years is 555,756 (Appendix, Table 21). Per the Sierra Leone Multiple Indicator Cluster Survey (MICS) conducted in 2010, Sierra Leone has the highest child mortality rate in the world at 217 deaths per 1,000 live births. Mortality for children under 5 is close to 15%. One of the leading causes of morbidity and mortality is malnutrition. 6 Severe Acute Malnutrition (SAM) is defined globally by the z-score of nutrition indexes of the World Health Organisation (WHO) growth standards. SAM is defined by a very low weight for height (below -3z scores], or below 70% of the median of the National Centre for Health Statistics standard, and by the presence of nutritional oedema. At the start of 2012, malnutrition was widespread, with acute malnutrition demonstrated by wasting and underweight at 6.9% and 18.7%, respectively, and stunting at 42.2%. Across the seven evaluation districts, there were some positive changes in child malnutrition between 2010 and 2014, with a decrease of -6.7% and -3% in the prevalence of stunting and wasting, respectively. However, the absolute levels of malnutrition are still alarmingly high, with 35% and 5.6% prevalence of stunting and wasting, respectively, and 1.3% of children under 5 that are severely wasted (see Table 19 in Appendix II for Supplementary Tables). UNICEF has advocated policy change and worked with local government to put the changes in place with a goal of reducing malnutrition. UNICEF identified local non-government organisation (NGO) partners to implement programme strategy outputs that sought to impact communities. 7 Programmes were directed at changing demand, by educating mothers to bring potentially malnourished children to centres (outputs) where they could receive supplemental food. This involved training community health workers (CHWs) and mother support group (MSG) members and leaders using implementing partners (IPs). Since 2009, UNICEF has supported the Ministry of Health and Sanitation (MoHS) in Sierra Leone to scaleup high-impact nutrition interventions in the country. 8 UNICEF has been advocating for policy change, providing technical support and provision of supplies countrywide. The MoHS/UNICEF Nutrition Programme sought to increase capacities of government and community institutions to deliver highquality and equitable nutrition services in line with the Basic Package of Essential Health Services (BPEHS). Nutrition services included screening for oedema and prevention strategies for malnutrition. Integrating nutrition into the basic services package, UNICEF sought to improve the quality of treatment services. They would increase skills of health workers, increase community counselling skills, increase micronutrient fortification, offer supplementation for children under 5 years, strengthen multi-sectoral and national strategy implementation coordination and develop and implement communication strategies that reduce teen pregnancy. 4 UN Population data, retrieved 8 Jan 2017, which are produced by United Nations Statistics and Population Divisions (6,592,000). ** Population census data were released after analysis. The total population was now 7,092,113 at the time the report was released. 5 CSD population projection provided by UNICEF 6 Demographic and Health Survey (2013) 7 SILPA in Kenema, PACE in Tonkolili, CAWeC in Kambia, DIP in Port Loko, WHI and CAUSE in Canada 8 Evaluation report IMAM and IYCF P a g e

15 The UNICEF/MoHS Sierra Leone Nutrition Programme was designed to align with global priorities to address malnutrition. Contributions included the development of integrated management of acute malnutrition (IMAM) guidelines consistent with global goals and the alignment of national priorities with global priorities (e.g. Sphere standards in IMAM and infant and young child feeding (IYCF) measures). The national nutrition implementation plan, in which UNICEF was involved, is aligned with the World Health Assembly Global Nutrition global targets to reduce stunting by 40%, anaemia by 50% and low birthweight by 30% and to reduce wasting to less than 5%. In addition, the goals include increasing exclusive breastfeeding (EBF) to 50% by The MoHS annual work plan lays out a plan for the various Nutrition Programme implementers. UNICEF, UK AID, Irish Aid, the European Union, the Japanese Agency for International Cooperation, the International Rescue Committee, Save the Children, CARE International, the World Bank, the Canadian International Development Agency, the Netherland s government, the UN Population Fund, the WHO, the World Food Programme (WFP), the Food and Agriculture Organisation (FAO) and others are working with the MoHS to bring about change, as evidenced by the Child Survival and Development (CSD) annual work plan. The CSD plan contributes to the vision of the health sector, which is to ensure a functional national health system delivering efficient, high-quality healthcare services that are accessible, equitable and affordable for everybody in Sierra Leone. These agencies support the national target of reducing mortality and morbidity in the population under 5 years, with a target of reducing rates by one-third by The UNICEF/MoHS Nutrition Programme sought to address the underlying causes of malnutrition 10 : (i) insufficient access to food, (ii) inadequate maternal and childcare practices, (iii) poor water sanitation and (iv) inadequate health services. UNICEF supported the establishment and strengthening of integrated health systems that would address inadequate maternal and childcare practices and inadequate health services. In so doing, UNICEF, MoHS and partners designed systems to transfer knowledge to communities and women of childbearing age and children under 5 years so that they could access and utilise quality essential nutrition as well as high-impact prevention and care services. UNICEF improved access to facility-based services through expansion of the interventions in seven districts. To effect change and create an enabling environment, the UNICEF advocacy and partnership strategy engaged the Sierra Leone MoHS to implement nutrition policies and strategies, with UNICEF providing capacity development, job support and nutrition supplies aimed at improving the Photo 1. Young beneficiary, Makeni, Sierra Leone nutritional status of children, building local capacity through training health workers and designing methods to monitor and evaluate services. UNICEF sought to provide outreach strategies to generate increased demand for services through initiatives such as support for MSGs with behaviour change messages designed to address IYCF practices and training of health staff to properly implement the IMAM protocol. 9 WHO 2025 Global Targets global nutrition report 10 P a g e

16 1.1 Policy & Advocacy To what extent has UNICEF s work on advocacy, coordination and policy influencing helped to strengthen programme implementation? Several UNICEF advocacy efforts have influenced government priorities towards improving the treatment services for acute malnutrition. Policy advocacy and influence successes are tracked in UNICEF s log frames in reports. Advocacy and coordination are included in the UNICEF log frame as a nutrition policy framework, and strategies for children and women of childbearing age are in place and operational. The targets to finalise the National Food and Nutrition Security Plan (NFNSP) and create an action plan were achieved in Coordination mechanisms - UNICEF met the target to establish coordination mechanisms. Quarterly meetings were held with the National Scaling Up Nutrition (SUN) Secretariat, although not regularly attended by all; technical support was provided by the United Nations Renewed Efforts Against Child Hunger and Undernutrition (REACH) Senior Programme Specialist under the umbrella of the UN System; there was a need to improve the understanding of roles and responsibilities of the SUN Secretariat as well as to support further district coordination. District coordination was evidenced at several levels; District Nutritionists (DNs) provided updates on training. IPs were involved in the rollout and implementation of decentralised monitoring, which involves quarterly visits to sites as well as providing support to district and facility staff. At the local level, of the 110 staff members interviewed at facilities, 98% said they hold regular monthly meetings with CHWs and MSGs. IPs and DNs held training exercises with facility staff on IMAM guidelines and CHWs on IYCF. Although a written communication strategy was not available, user-friendly materials were developed, distributed and verified. In addition to supporting the implementation of IMAM and IYCF guidelines and tools, UNICEF provided support to the effort for the development and finalisation of a Teenage Pregnancy Reduction Communication Strategy and Implementation plan. The NFNSP resulted in part from this advocacy. This was also evidenced by priority area 4 of the policy, which focused on the reduction of malnutrition, the under-5 prevalence of moderate and severe acute malnutrition, referred to as MAM and SAM, respectively. To reduce malnutrition, the NFNSP identified needed improvements for optimum IYCF and breastfeeding practices, micronutrient deficiency prevention, diarrhoea and parasite control and acute malnutrition (priority #6), among others. 11 UNICEF advocacy led to the prioritisation of nutrition. In 2010, the government created the BPEHS and introduced the Free Healthcare Initiative (FHCI), which greatly improved access to healthcare for all pregnant women, lactating mothers and children under 5 years old. The FHCI targeted pregnant women, breastfeeding mothers and children under 5. It was implemented at a cost of US $35,840,173, most of which (86.5%) was provided by international partners, such as UNICEF. The initiative abolished user fees and ostensibly provided drugs and treatment free of charge in every public health facility. The FHCI also refurbished hospitals and paid for the supply of drugs and health worker wages. The initiative has already recorded a positive impact, contributing to improvements in the availability of essential drugs for the management of childhood illnesses in health facilities and reduced healthcare costs for women and children, per the Sierra Leone Health Facility Survey The MoHS and other ministries with direct impacts on nutrition and food security, such as agriculture, have developed and implemented policies that provide clear direction to their various sectors. In 2015, 11 Ibid 5 11 P a g e

17 the MoHS solicited the EU to support the newly released Social Services Delivery Pact, which emphasises delivering four major priority sectors: 1) health, 2) education, 3) social protection and 4) sustaining service delivery through expanding government s fiscal space and revamping the private sector. 12 UNICEF was involved in the Sierra Leone Food and Nutrition Security Policy Implementation Plan ( ), which was developed to supplement the initial food security policy and recognised that malnutrition in the country could only be alleviated by establishing strong linkages with all the relevant sectors and partners. However, the plan was not reviewed at mid-stages, as originally planned. UNICEF advocacy has also been evidenced by their work to engage the government of Sierra Leone (GoSL) to join the SUN global movement in This was due to efforts dating back to 2008, when the heads of the four Initiating Partners (the FAO, UNICEF, the WFP and the WHO) committed to Renewed Efforts Against Child Hunger and Undernutrition, giving rise to an inter-agency partnership referred to as REACH. UNICEF also worked with the MoHS to strengthen the Food and Nutrition Programme, by setting up a directorate to handle food and nutrition issues within the MoHS, as well as developing a five-year implementation plan for the period for the NFNSP. Due in part to UNICEF advocacy and policy support, as well as that of other UN agencies such as the WFP and USAID, nutrition improvement became a priority within the GoSL five-year Poverty Reduction Strategic Plan for the period To prevent and treat undernutrition in infants and young children, UNICEF in partnership with the MoHS and others are implementing varied facility- and community-based programmes and interventions. Some of the facility-based programmes include the IMAM, which provides treatment to children under 5 suffering from SAM in MoHS Peripheral Health Units (PHUs) as well as capacity building and support to outpatient treatment programmes (OTPs) and inpatient facilities (IPFs). At the community level, outreach efforts such as the biannual Mother and Child Health Weeks (MCHWs) have been conducted for micronutrient supplementation and deworming. In addition, UNICEF has supported the establishment of trained MSGs, established in communities for the screening, IYCF counselling, referral and treatment of children with SAM. 1.2 UNICEF Sierra Leone Programme Objectives The programme under evaluation is a continuation of the collaboration between UNICEF Sierra Leone and Irish Aid that was established in The programme objectives have been as follows: 1. To improve the quality of the treatment of SAM by increasing adherence to national IMAM protocols at OTPs and IPFs 2. To strengthen the supply chain management of ready-to-use therapeutic foods (RUTFs) and improve communities adhesion and participation in IMAM activities 3. To improve IYCF practices at the community level through community-based structures and PHUs 4. To maintain adequate coordination of nutrition partners and activities from decentralised to central level 1.3 Ebola During the Ebola epidemic, Sierra Leone had more cases than any other country, totalling 14, The first cases of Ebola were reported in Sierra Leone in mid-march It spread quickly. By mid-june, an explosive outbreak was clearly underway in Kenema, after a well-known traditional healer died of Ebola (WHO, 2014). More than 300 Ebola cases link back to one funeral. The number of cases in Sierra Leone WHO 12 P a g e

18 increased from 50 in March to 400 the first week in October. The WHO report further indicated that The government hospital in Kenema could no longer cope. Several nurses working there were quickly infected and 12 of them died. Reported cases continued at around 600 per week until December of 2014, when there were a reported 400 cases per week. The first week in January 2015 was the first week since October 2014 that the number of weekly reports of Ebola cases was below 400. Two hundred cases a week were reported until the first week of March in Of the seven districts under evaluation, Kenema, Port Loko and Bombali were among the hardest hit, with between 500 and 4,000 cases reported over the course of the disease in Sierra Leone (WHO Map of Total Cases 2016). The other districts, Pujehun, Kambia, Moyamba and Tonkolili, each reported as many as 500 cases. 1.4 Programme Resources Irish Aid awarded UNICEF three grants for the Nutrition Programme evaluation period. The first UNICEF proposal submitted to Irish Aid was in 2012 and focused on the treatment of acute malnutrition, which encompassed the procurement and distribution of RUTFs and other nutritional supplies, OTP training for facility staff and the provision of job aids to improve service quality. Because the award began in August of 2012 and went to May 2015, it overlapped with the evaluation period, complicating the evaluation of how programme resources were utilised. Total funds available from the first award in 2012 were $963,464 for programmes to December 2014, overlapping with only one year of the evaluation timeframe, December 2013 to December In the second proposal to Irish Aid, which came during the Ebola crisis in October 2014, UNICEF s focus had expanded to include social mobilisation and the procurement of additional food supplements. The funds available for social mobilisation in the proposal amounted to $146,556, equivalent to 17% of the total funds available (Figure 2). The largest portion of the budget of $851,539 in 2013 was devoted to the procurement and distribution of nutritional supplies (about 69% of the total budget). In the third proposal, the scope of the programme expanded further by encompassing more areas in addition to those that had already been applied, such as IYCF, policy support and decentralised monitoring. The only budget information was provided in various progress reports. Unfortunately, UNICEF reporting on these three awards was not straightforward. Aside from overlapping evaluation dates in the seven progress reports, grant , Accelerated Reduction of Child and Maternal Undernutrition 01/01/ /02/2015, was ultimately combined with results from SM This made it impossible to tease out the various initiative costs over time. Grant , Ebola Response, was submitted separately. Budgets that were provided are captured in Appendix IV, Financial Tables. This expansion of the project scope was reflected in a change in the relative budgetary importance of the different project areas: procurement and distribution of nutrition supplies only made up about 10% of the budget in the third proposal (total funds available in 2014: $524,264 over a total of $5,012,667). IYCF in the latest proposal made up about 27% of the budget, policy and strategy about 15%, monitoring less than 1% and project and support costs about 10% 14 Ibid P a g e

19 Figure 2. Budgeting of funds available % 23% Share of total funds available budget (3rd Proposal) 16% 17% 43% SAM Treatment IYCF Policy and strategy Monitoring Project support costs Funds utilised under the first proposal ( ) were not provided to evaluators (Appendix II, Financial Tables). The total expenditures for funds between 2013 and 2015 were $3,990,922. Of the budget for the Ebola response, unused funds came to $88,368. UNICEF overspent on RUTF by $54,411 (83% of the budget) and underspent by -$93,787 on ready-touse infant formula (RUIF) and -$43,455 on BP100 (a nutrient-fortified wheat-and-oat bar designed to provide a similar nutritional profile to F-100 by the WHO). 15 More than 2 million dollars ( $2,043,401) went unused. From approximately October 2104 to December 2015, the total of unused funds was $2,131,769. This underspending was literally in the middle of the Ebola outbreak and was likely due to decreased access to facilities during this emergency. 1.5 Objective of the Evaluation The purpose of this evaluation was to understand how the Nutrition Programme in the seven Irish Aidsupported districts worked and determine what worked well, where, why and under what circumstances. This section presents a review of relevant literature and results from primary and secondary data on the implementation of the Nutrition Programme. It also provides some notions on the status of UNICEF/MoHS Nutrition Programmes. In the broader context, the evaluation generates knowledge and identifies some best practices and lessons learned that can help to strengthen national programmes and inform nutrition advocacy activities. The intention is to influence the national government, support scale-up, increase the budget and build a commitment to sustainability through ownership and overall leadership Rationale for the Evaluation The present evaluation is part of a commitment made to Irish Aid (the prime donor) to evaluate the effectiveness, efficiency and impact of UNICEF s work that has been implemented in seven Irish Aidsupported districts since 2013, which include Bombali, Kambia, Kenema, Moyamba, Tonkolili, Port Loko and Pujehun. The evaluation will generate knowledge and identify best practices and lessons learned that can be transferred to other future programmes. The evaluation findings will also be used for advocacy activities with the intention of influencing central government to support scale-up and increase budget allocation, commitment and overall leadership for the Nutrition Programme Scope This evaluation covered the three-year period from December 2013 to June However, data were collected from facilities, caregivers and policy implementers during 2016 from April to September due to the various issues mentioned in the next section. The evaluation does not cover the implementation of 15 Solade Pyne-Bailey, Ministry of Health and Sanitation, Government of Sierra Leone, Nutritional Support for Ebola Survivors in Sierra Leone, WHO 14 P a g e

20 the micronutrient feasibility study and pilot, although some components may be referenced in the evaluation to ensure a full picture of the programme is provided Limitations The primary challenge faced by the evaluation team was the availability and complexity of the data. Although the secondary data facilitated a much more valid and robust programme evaluation (by allowing a quasi-experimental pre- versus post-implementation design), most types of health indictor data were only available up to 2014, covering only one year of the evaluation period and previous years. However, the performance data were available up to 2015, although these were partial datasets, which also limited the analysis. The UNICEF Sierra Leone team indicated they had not collected baseline data specifically for the programme. They had used the 2010 MICs for baseline data and the 2014 Sierra Leone National Nutrition Study (SLNNS) as midline data. The next coverage survey would not be completed until after this evaluation. The decentralised monitoring data that had been requested at the start of the evaluation, which would allow the careful examination of programme performance, were sent to the consultant after the draft report phase had begun. The detailed longitudinal analysis of the implementation of PHU-specific programmatic changes and their impact on child health outcomes was limited to retrospective accounts from qualitative interviews. While we used secondary survey data to track progress in child health indicators at the district level, children in these national surveys could not be matched to their respective PHUs. As Photo 2. Team crossing river to reach PHU with all ecological/observational studies, there was a risk that child health improvements at the district level could not be causally attributed to programmatic actions, as there may have been other programmes/factors linked to child health enhancements. Variation in programme implementation from site to site, which would have been highly informative for tracking programme efficacy in this regard, was not possible given secondary data limitations. Additionally, because there was no baseline data, we used reasonable best-guess estimates in calculating the sample size for the primary data collection. However, both the inaccuracy of these estimates and environmental constraints experienced during data collection impacted the sample, as did the difficulty in making statistical comparisons between districts. This was due to the long distances between randomised sites and the physical barriers encountered by the team, which had been underestimated. These combined factors led to a considerably smaller sample, despite doubling the number of data collectors to reach a randomised sample, as UNICEF requested. 15 P a g e

21 The consultant s ability to thoroughly assess how resources impacted results was also limited. Firstly, we were not provided access to financial data beyond progress reports. We were not able to obtain the cost per child treated, as we were not provided the necessary financial information. The number of supplemental food portions given to each child was an unknown and was not documented. Additionally, distribution often varies with each situation (e.g. distance from facility, time of year and number of portions in a carton of food supplements). If the RUTF is cartons of Plumpy nut, there should be 150 portions per carton. Despite being requested, details about supplies, such as F-75 (a therapeutic milk product that contains 75 kcal and 0.9 g protein per 100 ml), were not provided. As soon as the child is stabilised on F-75, F-100 is used as a catch-up formula to rebuild wasted tissues (contains 100 kcal and 2.9g protein per 100 ml). In addition, we were not provided other costs of personnel distributing supplies to communities, as these were funded through the MoHS. The combination of these factors led to our inability to investigate the impact of funding on results Evaluation Questions To determine if programme objectives were met, the specific evaluation objectives formed the theme for which information was sought. The Evaluation Framework outlines these major evaluation questions and the evaluation strategy for addressing the evaluation questions. The 17 evaluation questions with subquestions are detailed in the Evaluation Framework (Annex V, Evaluation matrix). 1.6 Methodology Evaluation of the programme focused on three broad criteria: (a) relevance, appropriateness and coherence; (b) efficiency and effectiveness; and (c) sustainability and scale-up. Data for the first set of questions came primarily from decentralised monitoring and primary data collection in the districts and at health facilities. The second set of questions relied mainly on a quantitative analysis of secondary crosssectional nutrition and health surveys and UNICEF/IP data. The final set of questions drew mainly on primary data and UNICEF/partner information. Employing a mixed-methods design, the methodology outlined below responds to research questions within this broad area. Specific research questions are outlined above. Research questions are outlined in each of the primary focus areas as mentioned above (e.g. relevance and appropriateness) and outlined in the Research Framework. The consultants completed a desk review (documents reviewed during the desk review can be found in Annex II) and conducted interviews with UNICEF. They then designed an inception report that was used to guide the evaluation. The timeline shown in Figure 3 below showcases the documents reviewed and is organised by the duration of various UNICEF nutrition programmes completed by the time of publication or implementation Sampling Originally, the sample size was determined by considering the following: staff, original population size targeted by UNICEF (including both PHUs and households) and distribution of PHUs within clusters, desired margin of error and degree of confidence. A sample of 70 PHUs using clustering methods was suggested by the consultant, given time and resources. However, discussions with UNICEF revealed a desire for a random sample and an expansion of the number of assessed PHUs to include non-otp sites. A comprehensive list of all PHUs, stabilisation centres (SCs) and IPFs was provided by UNICEF from the Health Facility Assessment (HFA) survey. The source of the PHU data was the 2015 facilities survey that was conducted post-ebola Virus Disease (EVD). 16 P a g e

22 Figure 3. Timeline of documents reviewed for UNICEF Nutrition Programme evaluation 1/1/10 1/1/11 1/1/12 12/31/12 12/31/13 12/31/14 12/31/15 12/30/16 12/30/17 Sierra Leone Multiple Cluster Indicator Survey 2010 Evaluation of Management of Acute Malnutrition The State of Food Security and Nutrition in Sierra Leone 2011 Nutritional Status of Sierra Leone Survey using SMART Methodology. SLEAC and SQUEAC Evaluation of the Sierra Leone National Community- National Food and Nutrition Security Implementation Plan Implementing Partners Mapping End report: Accelerated Reduction of Child and Maternal Under-Nutrition National Food and Nutrition Security Implementation Plan Demographic Health Survey preliminary report Decentralised Monitoring Results for the Unicef/MoHS Nutrition Programme Evaluation of Integrated Management of Acute Malnutrition and Infant and An Analysis of Developing State capacity to prevent malnutrition in Sierra Sierra Leone Micronutrient Survey 2013 Reducing Child and Maternal Under-nutrition in Selected Districts of Sierra GoSl/Unicef Annual Work Plan 2014 SUN Movement Annual Progress Report: Sierra Leone Sierra Leone National Nutrition Survey 2014 Sierra Leone Health Facility Survey 2014 Final progress report: Ebola Outbreak Nutrition Response Sierra Leone Country programme document GoSL/UNICEF Annual Work Plan CPD Summary Results Matrix Country Programe Action Plan Between the Government of Sierra Leone and Sierra Leone Health Facility Assessment 2015: Impact of the EVD Outbreak State of Food Security in Sierra Leone Post Ebola 2015: Comprehensive Food Researching livelihoods and services affected by conflict: Understanding Sierra Leone Household Food Security Survey in Rural Areas 17 P a g e

23 At that time, no decentralised data were available to the consultant. Following inputs from UNICEF, the original plan to purposively select CHC, maternal child health posts (MCHPs) and maternity hospitals/ipfs with OTP services from each district was modified. A random selection of facilities was chosen from a list of facilities from the HFA survey (1,175), including both OTP and non-otp sites, per UNICEF s request. Of the 490 PHUs and 685 CHPs provided in the UNICEF list, 360 were randomly selected. The random selection was mapped and targeted by the data-collection team. Table 20 shows the OTP participation breakdown Data Collection Due to the request for changes to the methodology mentioned above (see Section Limitations), a team of seven enumerators was doubled to 15 (six women and nine men), and two supervisors were also added, for a total of three. All enumerators and supervisors were trained over a period of three days in October in Freetown by evaluation and nutrition specialists. The training included the proper use of the survey tools and an overview of malnutrition in Sierra Leone, including passive screening, MAM treatment, OTP staff overview and nutrition monitoring and evaluation. Table 1. Primary data collection District DLO, DN, DMO 20 Facility In-charge & nutritionist 209 Level Completed Data Method National 9 Total Key Informant Interviews CHWs & MSG Leaders 222 Density screening (caregivers) 394 Diversity screening (caregivers) 395 Breastfeeding screening (caregivers) Semi-structured survey with open-ended questions Semi-structured survey with open-ended questions Semi-structured survey with open-ended questions Semi-structured survey with open-ended questions Semi-structured survey with open-ended questions 680 Semi-structured survey with open-ended questions Total surveys 1,920 Semi-structured surveys MSGs and Caregivers 7 Total Focus Group Discussions After the training sessions, a subset of enumerators and all supervisors tested the six electronic survey tools and the focus group guide. The electronic tool was then updated with modifications found during testing and rolled out to all enumerators prior to data collection. Supervisors and enumerators conducted interviews in the seven districts. Where possible, women interviewed women to increase data validity. Enumerators travelled together from one district to the next, starting with Kambia. Table 1 shows the level, type and number of each data-collection method. Consultants conducted key informant interviews (KIIs) with 21 district-level implementers (District Medical Officer (DMOs), IPs and DNs), and 210 semi-structured interviews were conducted with facility staff. Of the random sample of facilities, a little more than half (n=106) were OTP sites (Table 20 in the Appendix). There were 221 semi-structured interviews conducted with MSG leaders and 1,163 with caregivers. In addition to interviews, FSNA-level data were provided in Excel files. Fourteen FGDs were held with caregivers of children 2 years and younger and MSG and community leaders Analysis A cross-sectional analysis of the data for effectiveness and efficiency of the Sierra Leone Nutrition Programme was conducted using primary data (KII, interviews, surveys and FGDs) and secondary data to triangulate and confirm findings from the desk review, reported performance indicators and decentralised monitoring data. DHS data from 2008 and 2013, MICS data from 2010 (which was used as a baseline for 18 P a g e

24 Irish Aid funding 16 ) and SLNNS data from 2014 were used to calculate health indicators for the seven districts using the STATA and R software packages. Working within data constraints, we tracked progressions in child undernutrition and mortality as well as successful implementation of the programme in accordance with guidelines. In addition to the survey data (i.e. Sierra Leone Micronutrient Survey (SLMS) and SLNNS), the requested secondary data referred to here included MSG, district and national worker training records, decentralised monitoring data, PHU under-5 population and OTP records and district store and PHU procurement records. However, only decentralised monitoring data were provided. Evaluators checked the consistency of findings generated by both qualitative and quantitative data. Often, points where data diverged offered the greatest interest and additional insights. Triangulation involved examining the consistency between different data sources collected using the same method (e.g. at different moments in time) or comparing people with different viewpoints. UNICEF did not have a clear plan for behaviour change. Therefore, to analyse behaviour change, the evaluation team used recommended measures from the Food Security Network (FSN). 17 Their measures were designed to explore barriers affecting behaviour. To determine contact with MSGs and impact on behaviour change, the consultant modified the FSN tools to include questions about access to MSGs and IYCF counselling. Tracking change over time is essential, as districts are likely to differ systematically on sociodemographic indicators. It is important to recognise that the cross-sectional prevalence of malnutrition could not be adequately distinguished between programme effects and other elements influencing nutrition outcomes. Therefore, results are largely descriptive with statistical significance indicated where possible. 16 Irish Aid Proposal for project/programme: Accelerated Reduction of Child and Maternal Undernutrition in Sierra Leone 17 Little, Bonnie L., FSN Network Practical Guide to Conducting a Barrier Analysis 19 P a g e

25 CHAPTER 2 FINDINGS OF THE EVALUATION The results section of this evaluation is shaped around the key results areas: relevance, appropriateness, coherence, efficiency, effectiveness, sustainability and scale-up. Table 1 provides definitions of research areas as defined in the Terms of Reference (TOR). The four UNICEF focus areas 1) Policy and Advocacy, 2) IYCF and SAM Treatment, 3) Promotion & Social Mobilisation and 4) Micronutrient Supplementation are woven within the framework. It begins with an overview of findings from the analysis of the secondary data from 2008 to 2014 for the seven districts. Definitions of the results areas outlined in the TOR are provided in Table 1 below. Table 2. Results areas defined Results Area Definitions Appropriateness Suitable or fitting for a purpose, person, occasion etc. Good Practice A method or technique that has shown better results compared with those achieved with other means; should be effective, relevant, monitored & documented and replicable Coherence Interrelated programmes woven together to produce improved outcomes Coverage Reaching both urban and rural populations - equity Efficiency The accomplishment of or ability to accomplish a job with a minimum expenditure of time and effort Equity Coverage and access Relevance Applicable to the issue, conducive to the proof of a pertinent hypothesis (a pertinent hypothesis being one that, if sustained, would logically influence the issue) Sustainability Results that continue over time (reduced malnutrition) Programming ability to continue performance Continual ownership Scale-up Activity increase in size or number 2.1 Overall Relevance, Appropriateness and Coherence How relevant, appropriate and coherent are the programme strategies and planned results to reduce child undernutrition within the country? The UNICEF/MoHS Nutrition Programme strategies and planned results were designed to align with Sphere standards. Both the design of the programme and the results areas are considered when reporting results. In the following, we consider the programme design and programme planned results Programme Design The UNICEF Nutrition Programme in the seven districts under evaluation is part of a larger effort to improve nutrition in Sierra Leone. UNICEF appropriately combined support for nutrition governance and policy development through partnership and coordination efforts with the MoHS to build capacity and scale up high-impact nutrition interventions, improve knowledge management and SAM treatment and promote IYCF and micronutrient supplementation. Through advocacy actions, policy has been refined to support nutrition efforts. Social mobilisation efforts have included developing the capacity of MSGs and CHWs. These have shaped the results section of this evaluation. The UNICEF/MoHS Nutrition Programme strategies were founded on evidence-based research into barriers and bottlenecks, programme feasibility and programme coverage. Programme strategies and priorities have shifted over time, in part due to events beyond the control of UNICEF, such as the 2014 Ebola crisis, where services were interrupted and some staffing was impacted. Events such as the rollout of the IMAM protocol 20 P a g e

26 changes and the training of MSGs and health facility staff were delayed and re-established post-ebola. Priorities shifted and new initiatives arose due to the changing environment. For example, post-ebola, UNICEF and MoHS conducted a facility assessment that assessed the status of health services in Sierra Leone and found a 3% reduction in SAM admissions and 28% reduction in children attending the growth-monitoring programme as well as 7,536 additional under-5 deaths during the EVD outbreak, a 20% increase over the baseline of Neonatal deaths were 7% higher (780). Facilities surveyed had moved from having a 4% to a 2% closure in facilities by the end of Core Sphere Standards To what extent is the programme being implemented in accordance with recommended practices? The intentions of the programme are well laid out in the proposals that have been submitted to Irish Aid and are in line with Sphere standards for humanitarian response. 18 The UNICEF log frame provides the structure and logic for Irish Aid programmes. Although the structure needs refinement, as previously mentioned, it forms the basic structure and logic behind the programme design. In addition to measuring Sphere targets, the design aligns with core Sphere standards, outlined as follows. Core Standard 1: People-centered humanitarian response - Design the programme to meet needs that cannot or will not be met by the state or the affected people. In a 2014 report conducted by Secure Livelihoods Research Consortium, Developing State Capacity to Prevent Malnutrition in Sierra Leone, UNICEF was recognised as one of the most critical contributors for partnership. The nature of the programme to serve vulnerable populations in hard-to-reach communities aligns with Core Standard 1. Core Standard 2: Coordination and collaboration activities offer a forum for development - stakeholders to share information. UNICEF practiced advocacy, developed national guidelines for IMAM, IYCF and M&E, for example, implemented programmes in partnership with local NGOs, developed the capacity of the government system and monitored the programme for quality assurance. These approaches were used with a goal to address deficiencies in understanding and policies to improve malnutrition. Core Standard 3: The third core standard addresses the need for the systematic assessment of needs. Decisions about interventions were driven by studies supported by UNICEF. Two such studies were the Simplified Lot Quality Assurance Sampling Evaluation of Access and Coverage (SLEAC) and Semi-Quantitative Evaluation of Access and Coverage (SQEAC) surveys in 2011 used to evaluate the IMAM programme. Three barriers were identified that drove the focus of the initial proposal and set the coverage baseline at 12%. 19 These barriers were key to programme development from that point forward. The identified barriers were lack of awareness and understanding of the programme by caregivers/mothers and the community at large; deficiencies in active case finding by community-based workers or volunteers; and inconsistent supply of RUTF. It also determined that coverage was low in all but two of the seven districts (i.e. below 20%). Only three of the seven surveyed districts had moderate coverage (between 20% and 50%). Those with moderate coverage included Bo, Pujehun and Kenema. Other examples of data that were collected to understand nutrition in Sierra Leone were (a) the 2014 SLNNS, which assessed the major contextual factors contributing to malnutrition, such as IYCF practices, food security, Water, Sanitation, and Hygiene (WaSH) and a health situation analysis to define programme planning priorities, and (b) a Facility Health Survey (FHS) that was conducted (2015) to determine the status of health facilities post-ebola. 20 Results provided indications of the level of facility staffing, services and nutrition services offered; (c) the Food and Nutrition Directorate s (FND) first Sierra Leone micronutrient deficiency survey was conducted in November 2013; (d) a micronutrient assessment was conducted in 2014 and determined the feasibility of micronutrient messages. Although results were delayed, due to the Ebola 18 Sphere Handbook, SLEAC SQUEAC Sierra Leone Impact of the EVD outbreak on Sierra Leone s Primary Health Care System (March 2015) 21 P a g e

27 crisis, findings were disseminated and plans implemented for scale-up in 2016, which then reached 64,200 children with micronutrient powders (MNPs). Core Standard 4: Design and response should be appropriate. The agency s response should be based on an impartial evaluation of needs and address unmet needs in relation to the context. 21 UNICEF worked with the MoHS to design programmes based on an assessed needs and risks. They also considered the context in which the programme would take place. The design of the programme was based on existing data, such as the situational analysis and subsequent SQEAC and Lot Quality Assurance Sampling (LQAS) studies, which were used to enrich the knowledge base about the risks and context of improving nutrition in Sierra Leone. Core Standard 5: Performance transparency and learning standards are achieved when agencies continually examine the effectiveness, quality and appropriateness of their response. Agencies adapt their strategies in accordance with monitoring information and feedback from impacted groups. Although systems for internal tracking were found to need improvement, overall, UNICEF approached the programme design to ensure knowledge sharing across stakeholders. A further explanation of how this was achieved is captured within the outcome area increasing knowledge and understanding and impact monitoring, which involves knowledge generation, analysis and report dissemination. Core Standard 6: Aid worker performance Agencies should employ aid workers with the appropriate knowledge, skills, behaviours and attitudes to deliver an effective humanitarian response. Based on interviews with the UNICEF staff, they were found to be well trained and knowledgeable about nutrition in Sierra Leone. They had the appropriate skills, behaviours and attitudes to deliver a humanitarian response. However, there has been staff turnover over the years. There may have been insufficient human resources to meet the demands of the programme during and after the Ebola crisis. At the time of the evaluation, a nutrition manager, the chief of CSD and several other staff members were relatively new to UNICEF Sierra Leone, most there less than 6 months. However, there was a nutritionist who had been with the programme for several years. Theory of Change To what extent are the programme results and strategies underpinned by a clear Theory of Change? Although UNICEF staff had indicated their strategy was founded in Theories of Change during evaluation discussions, there was no evidence that strategies were planned or documented using a Theory of Change model. 22 The framework included expected results, indicator targets and achievements but was lacking external factors and assumptions that would drive an evaluation Alignment with Global and Regional Initiatives and Priorities To what extent are the programme strategies aligned with the global and regional initiatives as well as the priorities of the national government? UNICEF goals were in alignment with international and national policy, and log frame targets were consistent with those outlined in national and international strategies to address undernutrition. UNICEF supported the MoHS in the development of the Sierra Leone nutrition policy, IMAM guidelines and national plans for IYCF. The UNICEF targets and approaches were coordinated through the MoHS to implement throughout Sierra Leone. The GoSL has demonstrated its commitment to their engagement at the local, district and national levels, as is outlined in the background section of this report. Country Programme Action Plan (CPAP) ministries were committed to working closely with UNICEF on nutrition programmes. The MoHS was responsible for CSD programmes, which led and facilitated planning meetings and programme reviews with support from UNICEF. 23 Goals of UNICEF and the GoSL were in alignment as was reflected in the CPAP results and Resources Framework. There were congruent policies agreed upon by UNICEF and the GoSL UNICEF Country Programme. 21 Ibid 7 22 UNICEF Supplementary Programme Note on Theory of Change 23 CPAP P a g e

28 UNICEF worked with the MoHS to establish the CPAP and the NFNSP with the GoSL. In addition, as a member state of SUN, the GoSL demonstrated its commitment to achieving the six global nutrition targets, as were set by the World Health Assembly: 40% reduction of the global number of children under 5 who are stunted 50% reduction of anaemia in women of reproductive age 30% reduction of low birthweight No increase in childhood overweight Increase the rate of EBF in the first six months up to at least 50% Reduce and maintain childhood wasting to less than 5% A situational analysis, on the status of children in Sierra Leone, based on MICS and DHS 2008 served as the foundation of the Nutrition Programme, which was funded by Irish Aid. The analysis concluded that IYCF, acute malnutrition and micronutrient deficiencies among children under 2 years, adolescents and women of childbearing age were key nutrition issues in Sierra Leone. Based on this and the global guidelines (2012 World Health Assembly), the strategic plan emerged. Activities and interventions were aligned with these priorities to reduce stunting and anaemia, increase quality and scale-up of IMAM, prevent and treat micronutrient deficiencies and provide institutional support to combat SAM and MAM. The age group of focus is the first 1,000 days, from conception to the second year of age. The Sierra Leone IYCF strategy document, completed in June 2016, adhered to WHO recommendations. Screening and Referral - The strategy that was developed, with support from UNICEF and input from programme partners, provided a framework focused on feeding practices for children under 24 months and was aimed at preventing and reducing stunting and wasting. The screening through MUAC was based on the WHO revised growth standards that included mid-upper arm circumference (MUAC) of <11.5 mm, bilateral pitting oedema and weight/height z-scores. As per the national guidelines, CHWs were to identify and refer children found to be at risk using anthropometric measurements (e.g. MUAC) or where oedema was evident. Screening has been typically conducted at two levels: in communities and at the health facilities. Active case finding was integrated into the BPEHS through training and the provision of supplies. UNICEF is currently supporting the development of another protocol for community-based IYCF operational guidelines; the strategy is intended to facilitate the implementation of a comprehensive community- and facility-based IYCF programme. The four primary areas outlined in the UNICEF project work plan are clearly rooted in the national food and nutrition security interventions: UNICEF Key Result Areas Management of SAM Promotion of optimal IYCF practice Micronutrient supplementation Support to the GoSL and SUN secretariat to establish and strengthen multi-sectoral coordination on food and nutrition security Sierra Leone FNP Interventions endorsed by GoSL #1 - Treat acute malnutrition #2 - Improve breastfeeding and complementary feeding #3 - Increase micronutrient intake #4 - Improve household food security Treatment of SAM The UNICEF Nutrition Programme in the seven districts was appropriate, in that it utilised measures set forth by Sphere and aligned with national and international guidelines. Following the EVD outbreak in West Africa, UNICEF worked with the MoHS to revise the national guidelines for the management of SAM to address the situation. UNICEF spearheaded the printing and distribution of 61,400 copies of the revised protocol to PHUs (423) and hospitals (20) and an abridged version for OTPs across the country. Together with the Directorate 23 P a g e

29 of Food and Nutrition (DFN), UNICEF engaged IPs to conduct on-the-job training for health workers using the revised IMAM guidelines. All 55 nutritionists involved in SAM treatment underwent a five-day training programme on the revised IMAM protocol. The aim was to improve the quality of services at OTPs and IPFs. The number of trained health workers increased. However, the number of health workers trained in each district is unclear, as data for only two districts were provided (Section 2.6) Breastfeeding and Complementary Feeding To promote appropriate and adequate complementary feeding, the Sierra Leone government worked with UNICEF to plan, develop and implement an IYCF strategy document. The DFN created the national IYCF strategy with support from UNICEF; the protocol provided a framework for IYCF implementation nationwide. An accompanying pictorial booklet, showing locally available foods, was developed and utilised by IPs at PHUs during MSG discussions, food demonstrations and training sessions conducted by CHWs. The importance of access to basic and skilled support for mothers and caregivers of children was mandatory for optimal nutritional outcomes. The UNICEF programme integrated and supported the training of both skilled and lay workers required to enable the successful rollout of IYCF activities. MSGs were targeted by the government for the implementation and scale-up of the IYCF strategy to provide counselling and food demonstrations at the community level. The process was initiated by training CHWs who in turn trained 20,583 members of MSGs. The purpose of an evaluation, which was utilised a monitoring tool, developed and rolled out in March 2016 by UNICEF was to assess training effectiveness. The evaluation revealed the need to strengthen MSG training and increase support for community IYCF counselling. 24 Although the report lacked generalisability and validity, UNICEF responded to address the identified gaps in MSG Training and Supervision Guidelines, which clearly outlined MSG roles and responsibilities (see Section 2.6 Social Mobilisation) Micronutrient Deficiencies Severe micronutrient deficiencies are easier to diagnose and treat on a case-by-case basis. However, in populations where micronutrient deficiencies are prevalent, diagnosis is difficult and treatment may need to be tackled through population-wide interventions, such as food fortification. The secondary data analysis highlighted that micronutrient deficiencies are a public health problem in Sierra Leone, with about 80% of children found to be anaemic in both 2008 and 2013 (see Annex II). Although the plan created by the MoHS to increase micronutrient intake through the introduction of MNPs was delayed due to the EVD outbreak, a 60-day pilot project was eventually launched in Kono and Pujehun in November Following the successful uptake of MNPs, the IYCF programme implementation was scaled up, with support from UNICEF, allowing the supply of MNPs in the two districts as of June The MNP effort was one of six interventions targeted by the DFN to address the high prevalence of micronutrient deficiencies among women of childbearing age and children under 5. Additional data on anaemia prevalence will allow an additional investigation of programme impact. This evaluation did not include micronutrient activities for women of childbearing age Household Food Security Funds were also used to improve household food security and build institutional support to the GoSL. UNICEF, through UN REACH, supported the scaling up of a multi-sectoral approach to food security. Through REACH, UNICEF supported the development and launch of the NFNSP ( ). In the plan, UNICEF was to provide technical support and resource mobilisation to the national nutrition effort. Throughout the evaluation period, UNICEF worked to provide support for the SUN Secretariat at the MoHS. The overall role of UNICEF has been taking the lead in the coordinating of stakeholders and ensuring logistical 24 End Report, Accelerated Reduction of Child and Maternal Undernutrition Project in Sierra Leone (Grant: SC120582) 24 P a g e

30 support in emergency situations. UNICEF reported that they provided support for the regular national and district Multi-Sectoral Coordination Committee meetings between December 2014 and May Additionally, UNICEF provided support for four government officers, including the SUN Secretariat coordinator, the Deputy Chief Medical Officer and the DFN. In a mid-2015 progress report, UNICEF expressed concerns about the capacity of the SUN Secretariat to develop a multi-sectoral plan and coordinate related activities. Nonetheless, in November 2015, UNICEF reported the completion of printing, and the plan had been launched by the vice-president, government representatives, paramount chiefs, development partners, UN agencies, IPs and civil society organisations. The launch was rolled out in Bombali for the North and Bo for the East and South. In mid-2015, district-level dissemination was completed for three districts, including two under this evaluation, Kambia, Port Loko and Koinadugu. 25 By the end of the first quarter of 2016, the remaining districts had been completed. The assumption is that the issues with the Secretariat of concern in mid-2015 had been addressed CMAM/IMAM Approach Community-based management of acute malnutrition (CMAM) is the preferred approach to delivering therapeutic care. International approaches to CMAM agree that SAM should be addressed utilising three treatment modalities 26 : 1. Community mobilisation that includes effective communication, the findings of active cases, referrals and follow-up; 2. Outpatient treatment for SAM without medical complications; 3. Inpatient treatment for SAM with medical complications or in young infants. UNICEF and its IPs, based in all project districts, have played a crucial role in delivering services at the community and facility levels and have engaged in programme activities including case screening and verification, RUTF distribution, data collection and reporting and training of MSGs. UNICEF supports local NGOs in the role of IP, which strengthens local capacity to roll out programme activities and guidelines in a culturally sensitive manner. In the process, UNICEF established and re-established Table 3. IPs in each district Photo 3. CAWeC, Pujehun District Pujehun Kambia Moyamba Kenema Bombali Port Loko Tonkolili Partner details SAVE the Children CAWeC CAUSE SILPA WHI DIP ACTS agreements with IPs: Community Action for the Welfare of Children (CAWeC), Sierra Leone Poverty Alleviation Agency (SILPA), Action for Community Transformation and Sponsorship (ACTS), Christian Aid for Under-Assisted Societies Everywhere (CAUSE) Sierra Leone and the Development Initiative Programme (DIP) in Port Loko. Of the IPs, all but one are local NGOs that roll out UNICEF programmes from district to facility and community. They are supported in implementing community IYCF and IMAM programmes and working with 25 Accelerated Reduction of Child and Maternal Undernutrition project in Sierra Leone report 1 June 2014 to 30 June P a g e

31 district-level officials to implement nutrition programmes. Through UNICEF and MoHS efforts and with support provided by Table 4, IPs in each district these IPs, national-, district- and community-level meetings were held to discuss nutrition-related issues. IPs supported supply management and delivery, trained health facility staff and community health providers and ensured programme surveillance. The IPs were essential to the implementation of the Nutrition Programme. 2.2 Equity and Coverage With regards to equity of coverage, the CPAP between the GoSL and UNICEF outlines the commitment to targeting vulnerable populations. They targeted children under 5 years of age for SAM screening, unwed teen mothers were targeted to involve them in MSGs and health days for women and children aimed to improve growth monitoring and micronutrient distribution. Moreover, UNICEF focused on ensuring pregnant and lactating women received micronutrients in one of the districts during the pilot of a micronutrient intervention. The emphasis of this report will be on the MSG and growth monitoring, as there was no data from UNICEF on health days for women and children or the micronutrient programme to consider for this assessment. Efforts to increase the reach of the Nutrition Programme to serve vulnerable populations resulted in a third proposal to Irish Aid, submitted in UNICEF requested resources to increase the geographical coverage of MSGs from 17.2% to 50% of communities in the country. This was captured under Social Mobilisation in the proposal (see Section 2.6). The United Nations Population Fund (UNFPA, 2015) had estimated that 28% of girls aged years were pregnant or had already given birth at least once. Men and community leaders have been shown to influence and empower women. 27 UNICEF and the GoSL incorporated a strategy to engage men, teen mothers and community leaders in MSGs. UNICEF sought to strengthen the community IYCF counselling programme, with an increased focus on teenage mothers in addition to the screening of children for SAM. According to reporting IPs, the number of MSGs with at least one unmarried teenage mother as a member was 2,706. Reporting IPs were in the following districts: Kenema, which had only three of 2,485 (<1%) IYCFtrained MSGs, Bombali (25/3,566 or 7%), Tonkolili (300/452 or 63%), Kambia (944/9,144 or 10%) and Moyamba (1434/11,392 or 13%). The Save the Children IPs, the IPs in Pujehun, did not provide data on the scale-up of MSGs with teenage mothers. Thus, information is needed to determine their performance on this indicator. IPs were asked about the number of MSG groups that had members with facilitation skills, how many had trained male counsellors and how many included unmarried teens. The percentages generated below are from the six districts reported, representing 20,583 MSGs trained in IYCF over the past three years. 28 Figure 4 below shows that Port Loko, Moyamba, Kambia and Kenema compose most of the MSGs with facilitation skills and male counsellors. Engagement of teenage unwed mothers is lowest in Kenema, Tonkolili and Bombali. Pujehun did not report. During the EVD outbreak, there was a move to target orphans and children 6 59 months with survival packages. Although this was mentioned as a strategy, it was not tracked in progress reports. The IYCF counselling component did include aspects on the prevention of the mother-to-child transmission of HIV. Again, tracking of progress in this area was not evident in reports, nor was it captured in progress reports. The treatment of SAM also included children living with HIV as part of the target group. Again, however, these data were not included in reports. 27 SUN Empowering Women and Girls to Improve Nutrition BRIEF-6-EMPOWERING-WOMEN-AND-GIRLS-TO-IMPROVE-NUTITION-BUILDING-A-SISTERHOOD-OF-SUCCESS.pdf 28 IP data from 2014, 2015 and P a g e

32 Figure 4. Percentage of MSGs with group facilitation skills, unmarried teens and male counsellors by district (IP reported) MSGs, facilitation, -IP Reported 8.0% 7.0% 6.0% 5.0% 4.0% 3.0% 2.0% 1.0% 0.0% 6.8% 6.5% 6.8% 4.8% 7.0% 7.0% 4.4% 4.6% 4.6% 5.8% 0.0% 3.4% 1.1% 1.5% 0.0% 0.0% 0.1% 0.1% Port Loko Moyamba Kambia Kenema Tonkolili Bombali %MSGs Facilitation %MSG unmarried Teens % Male Counselors Coverage for marginalised and less-accessible groups, like HIV-positive children, could not be assessed due to the lack of data. However, it is reasonable to say, based on the findings from this evaluation, that hard-toreach areas and groups in targeted areas were contacted through MSG members and CHW referrals. IPs in five of the seven districts reported they had trained almost 5,000 CHWs (Kenema, Bombali, Moyamba, Kambia and Port Loko). Qualitative data at the facility and district levels suggested that CHWs and MSG leaders were an important component of the success of the programme. Caregivers and facility staff (>50%) all recognised the role of the community in the increase in referrals to hard-to-reach areas. 2.3 Efficiency & Effectiveness of Nutrition Supply Management To what extent have resources been translated to results? The performance of supply management activities was documented in progress reports and included supplies obtained and distributed. Although UNICEF tracked PHUs providing treatment for SAM, reports did not include indicators of nutrition supply management. Furthermore, reports about supply management largely covered eight or more districts. The breakdown by district was provided for the Ebola response only. The progress report log frames related to supply management were largely in textual format and included only cartons purchased and distributed to district medical supply stores. No utilisation data were provided. Decentralised monitoring IMAM stock information (stock in/stockout/stock balance information) was ostensibly given to the contractors. However, the files were missing those data (either they were not entered or they had been redacted), making it impossible to determine from those data the status of stockouts. Accounting related to supply accountability was limited to reports and findings from the surveying of facility staff about the challenges they face related to nutrition supplies. UNICEF reports indicate that annual projections of supply needs are based on the distribution plan strategy and are determined by MoHS caseload reports. Calculations are carried out annually with the DFN, UNICEF, the WHO, the WFP and ACF and are based on the annual SAM burden, caseload, SAM prevalence and population projection for children under 5 years of age in each district. RUTF is to be stored in a central warehouse before distribution to the districts. From there, the distribution goes to the DMO and DN, who is to prepare a plan for distribution. Plans should be developed during the second month of each quarter. Photo 4. RUTF supply, Kambia 27 P a g e

33 According to UNICEF, after two months, buffer stocks at the DMS are sent to the OTPs and a fresh distribution plan for the next quarter is developed. In addition to RUTF, supplies include MUAC tapes, referral slips and anthropometric measurement equipment for the management of SAM Supply Delivery Most (62%) facility staff who responded to questions about supply delivery (n=83) said delivery was quarterly (in accordance with the plan), 25% said supplies were delivered monthly (possibly facilities closer to district centre), while roughly 7% said there was no set time of delivery. Of 103 respondents, 50% indicated that scheduled deliveries had arrived late, 54% said they had run out of supplies due to problems with scheduling and receiving supplies, 40% said they had challenges storing nutrition supplies and 32% said they had difficulty distributing them. Almost 40% said they were unable to keep up with nutrition supply demand. However, this may have been due to the lack of regular and complete reports to the district stores by PHUs with OTPs, an ongoing challenge. Roughly 30% of facility staff also said there had been times when delivery and pickup were impossible due to adverse conditions. These were largely due to infrastructure issues (poor/no roads and rivers) between facilities and districts. IPs, DLs and DMOs (53%) confirmed problems with distribution related to infrastructure and poor roads. More than half of facility staff interviewed (51%) said the biggest challenge to storing supply is pest problems. Rodent infestation interferes with effective and safe storage. Limited storage space also challenges facilities (33%). Security was also mentioned as an issue by 20% of the respondents, as well as temperature control (e.g. room too hot/too cold) by 13%. The typical PHU mentioned they faced more than one challenge in storing supplies. Figure 5. Challenges faced by PHUs distributing supplies Challenges faced by PHUs in distributing supplies 60 % PHU respondents that mentioned challenge PHU health workers were also interviewed about their challenges in distributing nutrition supplies to communities, as shown in Figure 5. Misinformation and poor knowledge about nutrition and healthy feeding existed in the target communities. In addition, about half of the PHU respondents said they had difficulties reaching caregivers yet encouraged more community-level activity. Only 9% of respondents mentioned they did not face any challenges in distributing supplies; these were primarily staff at IPFs. These findings echo the concerns outlined before the start of the programme, signifying the need for more to be done to address these challenges. 28 P a g e

34 Between June and November 2015, UNICEF and IPs were informed of issues with the management of RUTF at facilities, as was detailed in an OTP verification exercise. It is unknown if pest control issues were discussed during the investigation. The report indicated there was low adherence to IMAM protocol and inappropriate management of RUTF at facilities. Subsequently, UNICEF implemented IMAM protocol re-training (discussed in Section 2.6) Stockouts Per progress reports, between May and September 2014, the average percentage of districts that had stockouts was 62%, which included the seven districts under evaluation. Between October 2014 and January 2015, the average percentage that had stockouts had dropped to 31%. Table 4. Caseload and stocking, by year * total caseload 17,464 16,541 13,714 7 districts RUTFs 17,105 4,000 13,746 F F , Note: caseloads are computed from UNICEF IMAM data: they are the sum of total new entries plus old cases. RUTFs, F75 and F100 data are from UNICEF progress reports * Data on 2015 are until May 2015 Table 4 outlines the admissions and stock for the seven districts from 2013 to 2015, based on UNICEF progress reports. Fewer RUTF cartons were provided in 2014, likely due to the Ebola outbreak. In the December 2014 May 2015 quarterly report, nutrition supplies distributed totalled 13,746 cartons of RUTF, 116 of F-75 and 129 of F-100 for IPFs. During the Ebola response, supplies included another 28,423 cartons of RUTF distributed to OTPs, and 121 cartons of F-75, 133 cartons of F100 and 43 cartons of ReSoMal (oral rehydration solution for severely malnourished children) were distributed to IPFs. In 2016, between January and June, distribution had gone down. A total of 8,183 cartons of RUTF were distributed to 603 OTPs, and 36 cartons of F-75, 80 cartons of F100 and 21 cartons of ReSoMal were distributed to 20 IPFs. Distribution by district and stockout rates were reported. The report served as a progress report for grants SM14001 and SC The Ebola outbreak nutrition response final report (SM ) did not provide specific data on the distribution of RUTF. Data were also not available on stockouts between February 2015 and December Although supplies were available at the district-level during Ebola, the outbreak delayed the delivery of stock to PHUs. The average RUTF stock balance was greatest in October 2014 in Port Loko at 135,225, which decreased to 46,711 in January Similarly, in Moyamba, the average RUTF stock balance was the greatest in October 2014 and decreased in January Similar trends followed in other districts, where stock was the highest in October 2014, except in Tonkolili where the stock was the greatest in December This end-of-quarter stock reality was likely due to the end-of-year government moratorium on public gatherings. Table 5 below provides a breakdown of the percentage of stockouts by district during the Ebola outbreak. Tonkolili had the highest percentage of stockouts (88%) between May and September 2014, with Port Loko and Kenema not far behind at 76% and 72%, respectively. In 2015, the UNICEF Supply Division procured nutritional supplies, including over 34,851 tons of RUTF, or 15,340 cartons. By the official end date of the Ebola epidemic in March 2015, these were distributed to 427 OTPs and 19 SCs. Also at that time, UNICEF redirected funds to implement an HFA to investigate the impact 29 P a g e

35 Ebola had on health services and the supplies that they provide. Table 5 outlines the findings around these data showing the impact of Ebola on stock balances and stockouts in the seven districts under evaluation. Table 5. Stockouts May 2014 Jan 2015 in seven districts (FHS data) District May Sept 2014 Oct Jan 2015 Comment (FHS, n6) Tonkolili 88% 31% Tonkolili had the highest rate of stockouts in May Sept This was also the district with the fewest number of PHUs offering OTPs. Port Loko 76% 38% Port Loko had the second highest stockout rate in May Sept. After this, the average RUTF stock balance between Oct 2014 and Jan 2015 was the greatest here at 88,304, in contrast to Moyamba where the average RUTF stock balance was 22,899. Kenema 72% 35% In December 2014, Kambia and Kenema had the highest average percentages of OTP facilities that had stockouts of RUTF at 47% and 48%, respectively. Kambia 59% 40% Kambia was at close to 60% stockouts in May Sept 2014 and in Oct Jan 2015 had the highest average percentage of OTP facilities that had stockouts of RUTF among the intervention districts. Moyamba 56% 19% At 19%, both Moyamba and Pujehun had the lowest average percentages of OTP facilities with RUTF stockouts. Pujehun 49% 19% Bombali 35% 33% Average 62% 31% Figure 6 shows stockouts by month post-ebola, with stockout increases in January 2016 in almost all districts, except Bombali and Kambia. We can speculate that calculations were based on the previous year s calculated caseload, as the calculations would have been impacted by the Ebola response. Calculated caseloads post- Ebola should have been considered. However, data were not available to conduct a thorough examination of this phenomenon. Figure 6. Average percentage of RUTF stockouts at OTPs Oct 2014 Jan 2015 Average % RUTF Stock-outs % 42.9% 42.9% Average percentage of RUTF stock-outs at OTP facilities by districts / month Oct 2014 Jan % 26.5% 44.1% 47.1% 41.2% 37.3% 35.3% 35.3% 45.1% 31.3% 37.5% 25.0% 31.3% 34.4% 32.8% 47.5% 24.6% 5.0% 22.5% 20.0% 27.5% 14.7% 17.6% 20.6% 23.5% Bombali Kambia Port Loko Tonkolili Kenema Moyamba Pujehun Districts from Oct2015-Jan2016 October November December January Accountability Matrix UNICEF and donor concerns about accountability related to supply management led to the design and implementation of an accountability matrix draft in November 2015, which was to help ensure RUTF was delivered to the PHUs. It was also designed to track shipments and provide training for calculating caseload estimates. In December of 2015, UNICEF in partnership with DFN organised four regional stakeholder meetings of approximately 400 participants to review the draft matrix. Participants included political and religious 30 P a g e

36 leaders, traditional healers, market women s associations and other civil society organisations as well as district health management teams and nutrition partners. The accountability matrix was finalised in March 2016 and rolled out countrywide by the DFN. Following implementation, reports indicated some gaps with the matrix, which were to be addressed. In the first quarter of 2016, UNICEF supported DNs to provide sessions with CHWs and MSG members to raise awareness on the importance of RUTF and to mobilise the community to address the misuse of RUTF. District and partner nutritionists were to conduct joint monthly OTP supply monitoring, with a goal of improving caseload estimation, RUTF monitoring and stockout estimation. However, despite requests, the accountability matrix was not made accessible to evaluators. Therefore, PHUs were asked about the delivery of supplies as part of the primary data collection conducted in October Caseload calculations, delivery and reports of misuse of supplies were the main issues identified around supply management. This led UNICEF and the DFN to design an accountability matrix that would address these issues. The proportion of OTPs reporting stockouts and misuse in 2015 was 80%, which was reduced to 57% by June 2016, a 23% improvement. 2.4 Efficient and Effective SAM Management Photo 5. Sierra Leone PHU How efficient and effective are OTPs in managing and treating children with SAM within the communities? The efficiency of OTPs in managing and treating children with SAM in communities was measured in terms of health services (i.e. screening, admission and default rates). 29 Inputs included scale-up of activities, such as increasing the number of facilities providing OTP services, CMAM/IMAM-trained healthcare workers and community members. There were increases in OTP sites providing service coverage and screening. Overall, we observed an increase in coverage of OTP services through the expansion of OTP sites, an enhanced number of trained personnel and increased screening (see Section 2.6). The evaluation also revealed decreases in default and admission rates (Section 2.5.1), suggesting an improvement in the quality of screening and counselling services. These factors suggest that the programme was efficient to the extent that quality services were accessible (i.e. they had trained staff and demonstrated an improvement in health outcomes). The efficiency of the OTP sites to reach and treat all SAM patients was hampered by the inability at the facility level to maintain RUTF supplies (challenges are outlined in Figure 5 above.) Six of the seven IPs reported for this evaluation that from December 2013 to August 2016, the UNICEF/MoHS programme had conducted almost 2 million screenings of children under 5 years of age for SAM (1,935,266). Their accounting included all screenings from IPFs, OTPs and MSGs. Given the under-5 population, we concluded the reports captured both new and repeat screenings. For example, children who had been screened in the community and gone to OTP facilities were screened again. Multiple screenings such as this make it impossible to estimate the percentage of the population reached. 29 Peter Hussey, Vries, H, Romley et al., A Systematic Review of Healthcare Efficiency Measures, Health Serv Res Jun; 44(3): P a g e

37 Six of the seven IPs reported. Within their communities, 87,996 SAM cases were identified (4.5%). There were 354,774 MAM cases identified across these six districts (18.3%). According to the DFN 2016 report, 93.2% of children admitted into the SAM programme were cured, exceeding the national and local targets. The most resource-intensive component of Accelerated Reduction of Child and Maternal Undernutrition was SAM management. The effectiveness of SAM management at OTPs considered healthcare worker and community training (Section 2.6) and improvements in health outcomes for those admitted. We found improvements in health outcomes for severe stunting and wasting during the first years of the programme period as well as improvements in cure rates and reductions in the number of admissions and those nonrecovered (Section 2.5) between 2013 and the end of Combined, these factors suggest that health worker capacity to treat and manage SAM improved over time and had an impact on the health of those admitted as well as improvements in screening Scale-up of Health Facilities Providing OTP The UNICEF target of scaling up health facilities to provide OTPs was set at 65% of 427. The effort was delayed in 2015, because of Ebola, but the target was met in Table 6 outlines the number of facilities across four years. To build capacity and increase coverage of CMAM and IYCF in 13 districts, in 2013, UNICEF sought to scale up to 50% of the 427 target PHUs that offered OTPs in Sierra Leone. Table 6 shows PHU scale-up data from reports and the decentralised monitoring database for the seven districts. It shows an increase in OTP from 269 in 2013 to 377 across the seven districts by the end of Table 6. Number of facilities 2013, 2014, 2015, 2016 MAY 2013 (SCALE-UP) FROM REPORT 30 DISTRICT SC OTP Total OTP/Total Facilities DEC 2015 (OTP SCALE-UP) FROM HFA / DM SC OTP Total OTP/Total Facilities MARCH 2016 (OTP SCALE-UP) SC OTP Total OTP/Total Facilities BOMBALI / / /97 KAMBIA / / /66 KENEMA / / /119 PORT LOKO / / /101 MOYAMBA / / /96 PUJEHUN / / /73 TONKOLILI / / /103 TOTAL % % % CHART ASSUMES SAME NUMBER OF PHUS IN 2016 AS 2015 (655 IN THE SEVEN DISTRICTS). DATA WERE NOT AVAILABLE BY DISTRICT FOR Bo and Tonkolili were districts targeted in late 2015 when scale-up resumed. This was the appropriate action for Tonkolili, which had the lowest coverage of facilities (18) providing OTP services and the highest SAM rates. Tonkolili was also hard hit by the EVD outbreak, with over 500 cases (Section 1.3). UNICEF reported that in the first quarter of 2016, re-training was completed and scale-up of PHUs had reached 88% of the original target of 427 PHUs (377/427). The increase included Bombali (78 OTPs/4 IPFs) and Tonkolili (72 OTPs/3 IPFs). Kenema had the greatest number of PHUs providing OTP services in 2015 and 2016 at 61 OTP sites. The districts with the lowest number of PHUs providing OTP services in 2015 were Tonkolili (42) and Moyamba (41). By 2016, Tonkolili had been scaled up to 78 OTP sites. Between 2015 and early 2016, Bombali and Tonkolili received the biggest increase in the number of OTP sites, with no other districts experiencing increases. In fact, Bombali, Port Loko and Kenema had reductions in the total number of PHUs. Overall, UNICEF met and exceeded their targets and improved and expanded OTP services in the seven districts. 30 Final Report, Evaluation of IMAM & IYCF Programmes May P a g e

38 2.4.2 Scale-up of Health Workers To what extent have the national guidelines to manage and treat children with SAM helped to deliver services more efficiently at the health facility and community levels? The Sierra Leone national guidelines for IMAM (2014) contain clear standardised treatment protocols designed to help deliver services more efficiently at health facilities and in communities. The goal of training health centre staff was to ensure adherence to national guidelines and improve the quality of nutrition services delivered at health facilities. This would be achieved if deaths and defaulters were reduced and quarterly reporting and timely calculations of caseload and supply needs were ensured. Feedback was also to be given to the districts on performance. The performance indicator was the number of health facilities with a minimum of one supportive supervision. Scale-up involved training of community and facility health workers on IYCF protocols and IMAM guidelines. It utilised various strategies, including training of trainers, development, publication and distribution of guidelines and educational materials and on-the-job supervision. By March 2015, there were 469 OTP staff members trained at 257 PHUs. By late 2015, 378 PHUs had trained staff (number of staff not available). Compared to the other districts, Kenema had the greatest number of facilities with trained staff (113). Tonkolili had the lowest number of trained staff at 23, compared to other districts. According to the June 2016 report, 548 facilities had at least one health worker trained in the national SAM protocol in Sierra Leone. Assessing the full scope of staff knowledge goes beyond the scope of this evaluation. However, an inquiry was made into knowledge of screening for programme entry. Table 7 below shows the average responses to questions about admissions. Of the 106 respondents, 96% said that admissions for OTP involved a MUAC <11.5cm, and 94% said z-score W/H <3. Half of them also mentioned anthropometric measures for OTP admissions. Responses were consistent with referral requirements and reflected the effectiveness of the training and re-training for admissions screening. Table 7. Types of nutrition-related admissions requirements mentioned Staff indicated admission Mean N=106 requirements Loss of appetite Complications Caretaker doesn t want outpatient care for child Anthropometric criteria W/L or W/H < 3 z-score MUAC < 11.5cm Staff members at OTP sites were asked about the factors that helped programme scale-up. Figure 7 below shows that of the 97 health workers that responded to this open-ended question, 89% said training, 86% said materials, 84% said community outreach and 83% said supervisions. We ascertained that among staff who responded from randomly selected OTP sites (N=106), 87.5% had been trained in SAM treatment, 71.7% had attended an IMAM refresher training session and 78.6% had specifically been trained in IYCF. On average, 90% of the staff interviewed (N=106) said their facilities provide IYCF counselling. 33 P a g e

39 Figure 7. Factors that helped with nutrition awareness activities, according to facility staff Factors that helped scaling up nutrition awareness % respondents that mentioned factor PHU Support Implementing Partners Support MSG Leader support Materials / Supplies Training Other IMAM Protocol Integration Phase wise - rollout of revised national IMAM protocol in Sierra Leone had started in late What was the IMAM protocol integration into the overall national nutrition system and programme? Between 2013 and 2016, UNICEF supported the DFN (MoHS) to continue the integration of the IMAM protocol into services provided in the seven districts. During the Ebola outbreak, UNICEF submitted another proposal, which added the delivery of RUIF for infants with Ebola-infected mothers/caregivers (see Section 2.3 on Supply Management). The request for funding also proposed a need for on-the-job training for health workers on the RUIF protocol. The effort would also cover training CHWs and MSG members on modified no-touch screening methods. A verification exercise was conducted in June 2015 at 144 OTPs in the seven districts to assess the quality of IMAM programme implementation. Results indicated a need to conduct refresher training for all OTP health staff using the revised IMAM guidelines. In 2016, UNICEF made the target of conducting refresher and onthe-job training for health workers that used the revised IMAM protocol, with a goal of improving the treatment of SAM and quality of services and ensuring the regular review of admitted cases to reduce stays. The scale-up of on-the-job training, rollout of revised guidelines and support supervisions for health workers and community members strengthened the capacity of health management teams. Working with IPs, UNICEF provided on-the-job support to health staff and printed and distributed job supervision tools during the rollout of the revised IMAM protocol. Revised tools for all 603 functional OTPs were distributed and training was carried out by the DFN (MoHS) with engagement IPs GOAL and ACF. Virtually all (99%) of the facility-based respondents (n=100) at OTP sites indicated that they had the IMAM guidelines available and that they were useful. Of these, 86% also said they had counselling cards, and evaluators visually confirmed that 83% of facilities had nutrition education posters. All respondents felt the guidelines were useful to them in delivering services. Table 8. Facility staff s year of last training primary data To determine training coverage, we inquired with staff about the training they received. Of those interviewed, 72% of health workers had specifically participated in IMAM refresher training. They said that they had received training, materials and support supervision (91.5%, n=106). Table 8 shows that most of the staff interviewed had their last training in Facility staff s year of last training Freq. Percent Cum Regular supervision and on-the-job training are essential to improving the efficiency of PHU staff and they reduce the mismanagement of nutrition supplies. (Tonkolili KII) IYCF Counselling Overall, the scale-up of training in the national guidelines for facility health workers and CHWs improved the management and treatment of children with SAM, in that it increased the number of staff and community members trained in screening and IYCF counselling. In 2014, UNICEF reported that only 25% of 34 P a g e

40 children 6 24 months of age in targeted communities received appropriate complementary feeding (SLNNS 2014). However, no new data were available after The November 2015 progress report indicated that the rate of complementary feeding was at 14% in Moyamba. (No other districts reported and no other data were available.) IYCF policy improvement efforts included the Sierra Leone Infant and Young Child Feeding Strategy, which was validated and finalised in June Efforts of UNICEF and the DFN encompassed operational guidelines for comprehensive IYCF programme rollout. In June 2016, UNICEF reported that five master trainers had been trained. SILPA in Kenema and DIP in Port Loko reported on IYCF counselling conducted between January 2014 and August 2016 with caregivers of children under 5 years of age. Bombali WHI reported counselling from October 2014 to August CAWeC in Kambia reported counselling from January 2015 to August Tonkolili ACTS reported IYCF counselling from January to August Bombali had no reported IYCF counselling when the Ebola outbreak was hitting hardest. Photo 6. Nutrition education session, Pujehun PHU Other than Bombali, IP reports indicated the districts hardest hit by EVD reported the most IYCF counselling sessions. Port Loko had 53 OTP sites and a total of 297,384 IYCF counselling sessions reported between 2014 and They had a 4-month stop in reporting between June and September, the same time it was reporting numerous cases of EVD. Bombali reported 70,730 IYCF counselling sessions between October 2014 and August 2016 and 78 as of The IP in Kenema reported that 62,534 IYCF counselling sessions were held between January 2014 and September 2016 in Kenema. Tonkolili reported that in two months of 2016 (July and August), 998 IYCF counselling sessions were held. This district also had the lowest number of sites offering OTP services, introducing new OTP sites in The Pujehun IP did not provide reports on counselling. 2.5 Health Outcomes To what extent have the programme resources/inputs been translated into results? There were improvements in the nutrition indicators over the programme evaluation period. An analysis of secondary data highlighted significant improvements in nutrition indicators in the seven districts between 2008 and 2014 (see tables in Annex II). During this period, there were marked improvements in the percentages of children under 5 years that were severely stunted (HAZ<-3sd) from 20.2% in 2008 to 11.4% in 2014 (p<0.001). Rates of wasting (BAZ <-2) also declined from 9.8% in 2008 to 5.6% in 2014 (p<0.001). In addition, severe wasting decreased from 3.7% to 1.3% during the period (p<0.001). However, discerning causal inference related to programme impact was not possible due to other factors influencing nutrition and a lack of 2015 and 2016 data. Further analysis of data disaggregated by districts shows that severe stunting was reduced by 14.79% between 2008 and 2014 in Port Loko and reduced by 19.09% in Moyamba (see tables in Appendix II). These were statistically significant changes showing the most improvement occurred during 2013 and 2014, when UNICEF programmes were scaling up. However, we cannot causally attribute these reductions to the programme alone. In the same period, there were improvements in several other factors contributing to the reduction of malnutrition. These included poverty reduction (-5% households under the poverty line) and slight improvements in maternal education, although the mean levels were quite low in 2014 (1.3 years of 31 UNICEF June 2014 June 2016 Progress Report 35 P a g e

41 education per woman). Furthermore, it is unclear how the 2014 Ebola crisis impacted these indicators, as findings from health and nutrition surveys have not yet been reported. Reports on the number of screenings provided for children under 5 were provided by IPs in each district, except Pujehun. These reports indicated that close to 2 million children under 5 participated in SAM screening (1,935,266) between December 2013 and August Of those children, 87,996 were found to have SAM (4.5%). There were 354,774 MAM cases identified across the seven districts (18.3%). Figure 10 shows a combination of datasets that demonstrate the disparity in findings on rates of SAM and MAM. Regardless of the dataset used, there were clear improvements in MAM and SAM percentages from 2013 to Note the high number of MAM cases in Kenema. Due to time and resource constraints, the cause of the disparity between Kenema and the other districts was not investigated, although the unusually high MAM figure may also be driven by incorrect reporting Admissions and SAM Burden The training of community representatives has increased the number of patients screened for malnutrition over the years by district. Admissions have been reduced from 2013 to 2015 by 22% likely due to improved food security and/or improved referrals. Figure 8 shows the reduction in admissions over the evaluation period. Although there were reductions in admissions, there was not a statistically significant change in the average number of children admitted to OTPs pre-ebola in 2013/2014 and during and after Ebola 2014/2015 in the seven districts. Figure 8. Admissions by district over time Admissions by District over Time Bombali Kambia Kenema Port_Loko Moyamba Pujehun Tonkolili However, in Kenema where the highest number of OTP sites was located during the EVD outbreak, there was a reduction of 20% in the number of children admitted. In contrast, Port Loko had an increase of 31% in the number of children admitted during that same time. Figure 9 below shows the proportion of admissions between 2013 and 2015, as reported by IPs by cause of admission. Without more detailed information, it is not possible to determine why Port Loko had an increase and Kenema had a reduction in admissions. 32 The tool used to request screening was developed for implementation in August, but the evaluation was delayed. 36 P a g e

42 Figure 9. Admissions by cause/proportion Dec 2013 Dec 2014 Dec 2015 total annual admissions 17,464 16,541 13,714 Table 9. Change in Admissions Pre/Post Ebola Change in OTP Admission Numbers from Pre-Ebola Period (Oct'13 Jan'14) to during Ebola (Oct'14 Jan'15) Decentralised monitoring data, October 2016 Province Admission in OTP services Change in Average from Oct Jan 2013/4 to Oct Jan 2014/5 Northern Bombali 14% Kambia 5% Port Loko 31% Tonkolili -18% Eastern Kenema -20% Moyamba -14% Southern Pujehun -4% Due to data limitations, to estimate the prevalence of SAM and MAM among those screened in the evaluation districts, we divided the number of children screened with SAM and MAM, minus old cases, by the total number screened. We recognise this as a rough estimation of prevalence, with the underlying hypothesis that the entire sample of screened children is representative of the underlying population. This hypothesis relies on the way in which screening was conducted: if mothers are invited to go to screening centres, there may be unobserved factors that may compel the children at the highest risk of malnutrition to attend screening more frequently compared to lower-risk children, which may lead to higher prevalence. However, if the screening is conducted such that all the children in the communities are screened, the observed rates should be closer to the true values in the population. In absence of other secondary data from 2015, the IP data (which included community screening) were the only available information offering estimates on the percentage of children screened with SAM/MAM. 37 P a g e

43 Figure 10. Prevalence of SAM & MAM based on various data sources Prevalence of SAM and MAM among screened children based on various data sources % SAM rate (estimate) SAM rate from partners SAM rates from secondary data MAM rates from partners data MAM rates from secondary data The SAM rate estimate in the seven target districts is based on total admissions from CMAM data divided by the population under 5 years (UN Population data). The population size is estimated by starting from the population under 5 years in 2015 in the seven districts and discounting it for the growth rate of 3.2% (the growth rate used in the population projections). Rates from partners are estimated by dividing the total number of children with SAM (MAM) by the total number of children screened by the partners in the seven districts. Rates from secondary data are estimated by using the DHS 2013 and SLNNS Our estimate is based on these findings over three years. Secondary and partner data captured two years Severity of Cases In addition to a reduction in admissions across the years from 2013 to December 2015, the severity of cases also shifted. We can see a slight increase in severe wasting between 2013 and 2014, probably due to decreased access to care due to Ebola (Table 10). However, by the end of 2015, severe wasting was back down by 25% from 2014 and lower than in There was a 63% decrease in the number of oedema cases between 2013 and 2015, over 800 fewer cases than the previous year (Table 10). Figure 11 below breaks these down by district and provides the monthly average of admissions with oedema by year. Although there was no statistically significant reduction in admissions, decreases in the average monthly admissions for oedema cases were likely due to the effects of increased OTP services, IYCF counselling, MSG and CHW referrals and distribution of RUTF. There were decreases in oedema regardless of increases in the number of facilities (Table 10). In case of Tonkolili, UNICEF doubled the number of sites in that district, which contributed to increased referrals/treatments. Despite this annual decrease in the monthly rates of oedema between 2013 and 2015, Tonkolili had the highest monthly rates of the seven districts every year. This makes sense, in that Tonkolili only had 16 OTP sites Kambia also had high rates compared to other districts. There was also a decrease in the monthly average of oedema cases in Kambia. All districts had a decline in monthly oedema cases except Kenema, Port Loko and Pujehun. In Kambia, there was a decrease in monthly admissions for Table 10. Oedema, red MUAC, severe wasting reduction 2013, 2014 and 2015 in the seven districts Oedema 2,324 2,198 1,394 Red MUAC 10,523 9,732 8,502 Severe wasting 3,951 4,056 3,144 Other Old cases ADMISSIONS 17,464 16,541 13,714 Source: DM data the OTP until 38 P a g e

44 oedema but not a corresponding increase in the number of facilities in the district (Table 9). There were also decreases in Bombali, Moyamba and Tonkolili. Figure 11. Mean monthly admissions with oedema Average Percentage of Monthly Admissions with Oedema (CMAM/IMAM data) Bombali Kambia Kenema Port Loko Moyamba Pujehun Tonkolili Table 11. Monthly admissions for oedema in seven districts. BOMBALI KAMBIA KENEMA PORT LOKO MOYAMBA PUJEHUN TONKOLILI TOTAL mean/sd mean/sd mean/sd mean/sd mean/sd mean/sd mean/sd mean/sd Pr = (1.45) (3.22) (0.97) (1.74) (1.09) (2.26) (2.27) (1.94) Pr = (1.52) (3.14) (0.64) (1.40) (1.19) (1.48) (2.07) (1.70) Pr = (1.31) (1.25) (0.86) (2.62) (0.88) (1.91) (1.70) (1.63) Performance Indicators Cure, death and default rates are the primary indicators of programme performance. Indicators show gradual improvement in cure rates over the years, with the highest cure rates from 90.5% in 2013 to 93.6% average in 2015 (Figure 12). Default rates have plummeted over that same period. The 2013 default rates in Tonkolili and Bombali were 10% and 11%, respectively, between 2013 and By 2015, defaulters had been reduced in Bombali and Tonkolili to 6% and 8% below Sphere standards, respectively (Figure 13). Death rates also decreased in most districts to below Sphere minimum standards. Kenema, Port Loko and Pujehun also remained below 3% (Figure 14), between 1 and 2%. Bombali, Kambia and Moyamba went from 3% to 2%. The exception was Tonkolili, which had a surge in deaths in Overall performance at sites showed improvement. 39 P a g e

45 Figure 12. Cure rates in the seven districts 100% 95% 90% Cure Rates % 80% 75% 87% 91% 93% 89% 96% 94% 93% 95% 95% 95% 96% 96% 87% 93% 92% 94% 97% 98% 88% 83% 86% Bombali Kambia Kenema Port Loko Moyamba Pujehun Tonkolili Figure 13. Default rates in the seven districts % 15% 10% Default Rates % 0% 11% 6% 4% 8% 2% 3% 5% 3% 3% 3% 3% 2% 10% 5% 6% 5% 2% 1% 10% 15% 8% Bombali Kambia Kenema Port Loko Moyamba Pujehun Tonkolili Figure 14. Death rates in the seven districts Death Rates % 6% 5% 4% 3% 2% 1% 0% 3% 3% 3% 6% 2% 2% 2% 2% 2% 1% 2% 2% 1% 2% 1% 2% 1% 1% 1% 2% 2% Bombali Kambia Kenema Port Loko Moyamba Pujehun Tonkolili Growth Monitoring Figure 12 shows that, overall, the number of children who attended the growth-monitoring programme pre- Ebola was reduced by 28% during the presence of Ebola in the seven evaluation districts. Between October and January , the average number of children attending the growth-monitoring programme fell 40 P a g e

46 by 50% in Port Loko (an EVD hotspot), 33 the greatest change in any of the seven districts. Pujehun had the smallest reduction (14%) in children attending the growth-monitoring programme during Ebola. This is no surprise, given the moratorium on gatherings during the EVD outbreak. No other tracking data on growth monitoring were available. Figure 15. Number of children who attended growth-monitoring programme and percent change from pre-ebola to during Ebola crisis Children attending monitoring growth programme, Before and during Ebola Pre-Ebola average Oct-Jan 2013/14 During Ebola average Oct-Jan 2014/ % -32% -50% -28% -25% -17% -14% -28% 1000 children Bombali Kambia Port Loko Tonkolili Kenema Moyamba Pujehun Total Average Source, DHS data, n7, by year average + change ) 2.6 Efficiency & Effectiveness of Social Mobilisation and IYCF How efficient and effective have MSGs been in improving IYCF counselling and in community outreach? In the first proposal to Irish Aid for the implementation years 2012 to 2014, UNICEF indicated social mobilisation would engage communities through MSGs. This initiative has continued throughout the evolution of the Nutrition Programme. The effectiveness of the programme was to be measured by the number of health facilities trained in IYCF and the percentage of children aged 0 24 months reporting at least two contacts with an MSG, which would be disaggregated by district and age. IYCF counselling was to be implemented at both the community and facility levels. The level of detailed data required to confirm two contacts caregivers had with MSGs was not provided. Of MSG members in the seven districts, 75% reported they specifically had exposure to IYCF counselling. In the proposal, UNICEF requested support for addressing challenges associated with low technical nutrition knowledge and capacity. They sought to increase the geographical coverage of MSGs to strengthen the skills of MSG members in IYCF counselling through training and on-the-job coaching. Indicators were added to social mobilisation. Measures reported were as follows: Percentage of women in contact with MSGs - UNICEF reported in 2015 a baseline of 85.7%, which had not changed since the 2014 report (not disaggregated by sex). MSGs trained UNICEF reported 17,344 MSG members had been established and trained by November Almost all MSG leaders who were surveyed had been trained in guidelines for identifying malnourished children. Table 12 shows the training sessions received by MSGs. Two-thirds (75%) of respondents had MSG training in proper IYCF. Most (79%) had received more than one training session. Results showed that 89% had received EBF counselling/training and 80% had been trained more than once Health Facility Survey Analysis - 4 Dec P a g e

47 Table 12. MSG member training - primary data Oct 2016 MSG Training N mean sd Respondent is female % Received any training about proper IYCF % Received any training about EBF % Received training more than once % Trained in the guidelines for identifying malnourished children % CHWs & MSGs Behaviour change communication (BCC) representatives in the Community Health Unit at UNICEF described the characteristic roles of CHWs and UNICEF s work with CHWs to implement and improve IYCF and community outreach. During interviews, they reported that CHWs were often secretaries of the MSGs, working at the community level. Men often acted as secretaries and CHWs sometimes led MSG groups. The CHWs discussed the counselling cards (booklets of proper IYCF practices and referral information created with the WHO). During the screening, if the MUAC was yellow, they linked the child and the family with an MSG. If it was red, they referred them to the nearest facility. CHWs were also responsible for following up with the number of households, working with MSG groups to cover certain households. CHWs and MSGs are an integral component of the UNICEF/MoHS Nutrition Programme, efficient at referral and community outreach. Through their support, CMAM and IYCF have increased their reach. Facility staff indicated 94% of the time that new programme admissions were referred by CHWs. They also indicated many referrals came from MSGs. IPs reported that MSGs made 13,356 referrals between October 2015 and October Most community health providers (CHWs, MSG leaders and aides) added that the breastfeeding and IYCF programme was implemented very well (78%) or somewhat well (19%). When asked how easy it is to counsel caregivers about IYCF practices, 50% said it was very easy to counsel caregivers and 32% said somewhat easy, while 14% said that it was not easy to implement. Community health providers said transportation was most often (90%) the barrier to reaching the people they wanted to reach. They also felt that lack of training was a barrier to coverage (40%). When asked what made it easy to get nutrition messages to the community, IP support was ranked highest at 86%, next to PHU support (67%). Another important factor they Photo 7. Community health worker felt helped to scale up nutrition awareness was MSG scale-up activities (87%). Training ranked high at 73% and materials at 70%. On average, community representatives had three or more of the following available: posters (76%), manuals (71%) and MUAC tapes (83%). Only 37% mentioned SAM guidelines as critical and only 44% said they had counselling cards Strengthening Behaviour Communication What has been achieved in terms of CMAM protocol in practice for the period under evaluation? The evaluation shows some strengthening of BCC, in that UNICEF exceeded their coverage of IYCF-trained MSGs in the seven districts and increased the number of trained MSGs. By March 2015, UNICEF was reporting 5,994 MSGs trained in 11,220 communities (10,020 were in the seven districts), exceeding their goal of 50% coverage, having achieved 60% coverage by March There was 42 P a g e

48 another increase in the number of trained MSG members by May 2015 for a total of 12,742. By then, 37% of mothers of children under 24 months were encountering MSGs. 34 This expansion is in alignment with the NFNSP. Further evidence indicates as many as 28,919 MSGs were trained between January 2014 and March 2016 (data were provided by IPs). Figure 16 shows the training of MSGs from January 2014 to August 2016 as reported by IPs. Moyamba had the most trained MSGs in 2014, Kambia had the most in 2015 and again Moyamba had the most in Interviews with 221 MSGs revealed 70% had attended at least one IYCF training session. Almost two-thirds (70%) of all caregivers with whom we spoke said they had attended an MSG training or counselling session on IYCF. Of the 651 caregivers of children under 6 months, 395 (61%) had met with an MSG and 256 (39%) had not. This exceeds the 50% target UNICEF set for women in contact with MSGs as a performance measure. 35 Figure 16. IP-reported MSGs IYCF trained Jan 2014 Mar 2016 IP Reported MSGs IYCF Trained Jan Mar 2016 Bombali Kambia Moyamba Portloko Tonkolili Kenema To understand the effects of programmes on behaviour change, barriers and bottlenecks, the consultants interviewed 1,163 women: 651 with children under the age of 6 months, 374 with children aged 6 to 24 months and 234 with children ages 6 to 9 months. All interviews included questions about IYCF counselling. On average, of the 651 caregivers of infants under 6 months who were interviewed, 93% said they had received counselling for breastfeeding, 62% said they had attended an MSG and 96% said they had exclusively breastfed their child in the past. The design of all three surveys aimed to gather the most accurate data about IYCF behaviour; consequently, the sample of 651 caregivers with children under 6 months of age was reduced to 59 individuals when the interviewees did not meet the inclusion criteria for further questions about EBF. Most mothers were excluded from the study when they could not remember when they first introduced semisolid foods, such as soup or porridge. At that point, the interview ended. Only 9% could remember when solids were introduced, leaving 59 participants to discuss behaviour around EBF. This unfortunately reduced the strength of the findings about breastfeeding. However, one assumption can be made about this phenomenon: women had introduced semi-solids before their children had reached 6 months of age EBF Counselling At the start of the programme evaluation period in 2013, the rate of breastfeeding for children under 6 months was understood to be at 42%. UNICEF reported that the rate of EBF was at 53% as of 30 May It was unchanged in the November 2015 report. No other data were available to assess EBF. Out of the 59 participants who remained in the study, 30 had followed the recommended EBF practices (51%), and 29 had not. Results would have been vastly different if mothers who could not recall when they had introduced complementary feeding had not been excluded. 34 Accelerated Reduction of Child & MTL Jan 2014 May 2015 SM14001 and SC Note: Caregivers were solicited at randomly selected facilities. 36 December 2014 May 2015, Irish Aid UNICEF Nutrition Progress Report 43 P a g e

49 These findings suggest that messages about the definition of EBF are inadequate. There needs to be more of an emphasis on advising mothers to exclusively breastfeed up to 6 months and educating them on the definition of EBF and the right time to add solid foods to children s diets. Since we can only verify that 5% of the total surveyed population had practiced EBF during the first 6 months, these results should be considered with care; a full assessment of EBF is needed to determine the actual number of women practicing proper EBF. Given the high prevalence of exclusion, we can infer that EBF was not widely practiced; hence, its benefits need to be communicated more effectively. However, if one considers the entire sample and assumes that caregivers introduced solids early, which is why they could not recall when solids were introduced, only 2.7% were practicing proper EBF Food Density In 2014, UNICEF proposed a need to increase the quality of CMAM and IYCF intervention services by the end of the year. Children 6 23 months who were breastfeeding and receiving solid and semi-solid foods were targeted in eight districts, including all seven districts under evaluation. By November 2015, UNICEF reported the rate of complementary feeding behaviour was at 14% in Moyamba. No other districts reported and no other data were available. Table 13. Density Questionnaire Child s Age Density Caregiver s child Average Age Oldest Youngest Mean age Age in months at time of interview 9 mo 36 mo 6 mo 9 mo To further understand knowledge about complementary feeding/food density/quality, the evaluation team interviewed a total of 286 mothers of children aged 6 24 months about food density/quality IYCF practices (their children s age breakdown is in Table 13.) We sought to determine whether they practiced the behaviour. Those who practiced the behaviour were identified as Doers. Those caregivers who did not practice the behaviour were placed in the Non-Doer category. Out of those interviewed, 70% said they had started solids at 8.6 months of age on average. Most of the participants believed that feeding thick porridge would help decrease diarrhoea (73%) and 11% said their child would sleep better when they had thick porridge. Most of the caregivers (81%) did not see a disadvantage to feeding thicker porridge. However, 42% of Non-Doers shared that a great disadvantage of feeding dense foods is that their child would become constipated. As with food diversity in the following section, a third of participants mentioned that it would be easier to feed their children denser foods if the food was cheap and available in the market (31%). Eighteen percent of participants found that it is much easier to continue feeding dense foods if the baby likes it or if it is easy to serve. Spouses were named as the most supportive of feeding the children dense foods, but 23% noted that nurses and MSGs would disapprove of feeding thick foods from the four food groups each day. This is due to either a misconception about the effects of feeding dense foods within the community (constipation) or, possibly, misunderstandings by MSGs and CHWs, all of which need further investigation. Of those interviewed, the majority felt their children were not likely to become malnourished in the next year: 63% of mothers that did not feed thick porridge (Non-Doers) to their children compared with 90% of mothers that did feed thick porridge (Doers) to their children. Forty-five percent of Non-Doers, however, found that they did not see their child becoming malnourished as a serious problem; inversely, 69% of the Doers believed it was a serious problem. A notably perceived advantage of dense foods stated by Non-Doers was child health (26%) and child growth (47%), which were part of the counselling provided to the participants surveyed Food Variety/Diversity There were 236 women interviewed about their behaviour around food variety and IYCF. The average age of their children was 16 months (Table 14). We called caregivers who followed recommended feeding practices Doers. We found Doers fed their children four or more food groups in proportional differences ranging from 23 80% (see Figure 18). Both Doers and those who were not able to practice good IYCF, Non-Doers, largely fed cereals to their children. 44 P a g e

50 As Figure 17 below shows, those caregivers who fed their children from more than four food groups were indicated by 23.6% of 236 interviewees. For these Doers, their children s diets were balanced, consisting largely of cereal and vitamin A fruits and vegetables, legumes and nuts, fresh foods and eggs between 88 and 93% of the time. They mentioned dairy (69%) and other fruits and vegetables (34%) less often. These eating practices are in line with the recommended diet for children under 5 years old, but more dairy and a broader variety of fruits and vegetables should be incorporated into their diets. The most frequently mentioned food group for those feeding on fewer than four food groups (Non-Doers) was cereal; 63% said they fed their children cereal. Only 27% said they fed their children fresh foods, 22% said legumes and nuts and 22% said dairy. Even less often, 16% of those interviewed said they gave their children vitamin A fruits and vegetables, and 11% mentioned giving their children other fruits and vegetables. Clearly, these caregivers are deficient in terms of the number of food groups they feed their children. Focus groups revealed that the low use of dairy and a variety of fruits and vegetables was largely due to a lack of access either due to financial reasons or the lack of these foods at local markets. Figure 17. Advantages to food variety, caregiver Doers and Non-doers Table 14. Caregivers of children participating in variety interview - primary data Variety Caregiver's child Average Age in 16.0 months Youngest 7 Oldest 24 Mean age 16 Age in months at time of interview Positive Consequences of Food Variety Advantages: child gets smarter Advantages: child sleeps Advantages: child gains weight & appetite Advantages: child growth Advantages: child healthy 0% 5% 2% 1% 9% 10% 23% 49% 54% 57% 0% 10% 20% 30% 40% 50% 60% Non-Doer Doer Of the 369 caregivers that responded to the diversity questionnaire, 54% had attended an MSG training session and 45% had not. Respondents indicated the following perceived positive consequences of feeding a variety of foods: 54% of Doers believed food variety makes a child healthy, while 23% of Non-Doers mentioned child health. They mentioned child growth (57%) more often than Doers (49%), and Doers mentioned child weight gain and appetite (10%) more often than Non-doers (1%). Those that attended the MSG training, regardless of whether they were actively following the recommendations, did not see a disadvantage to feeding their children from more than four food groups. Caregivers in general understood the benefits and risks associated with diverse food feeding for their children. However, not even a quarter of them were Doers, either because they lacked the resources to practice or because food was not available in the case of dairy and some fruits and vegetables. 45 P a g e

51 Figure 18. Food diversity 6 24 months - primary data Oct Factors That Impact Behaviour Participants noted that the factors making it easier to comply with the behaviours being recommended, such as feeding diverse foods, were described as the food being cheap and available in the market as well as the counselling received by MSGs in IYCF. Sixty-nine percent of the participants found that the lack of financial resources was a disadvantage and challenge to them being able to provide a variety of foods to their children. Medical personnel were said to be the most influential people in terms of child health (79%), and 61% shared that nobody disapproved of them feeding their children from more than four food groups. By contrast, 10% shared that they perceived the community disapproving of making the recommended changes. Forty-one percent of Non-Doers (those who did not feed from four or more food groups) shared that a lack of financial resources was the main barrier to being able to comply with the recommendations. Non-Doers (70%) feared the severity of the consequences of their child becoming malnourished more than Doers (50%). Caregivers in both groups largely felt that their children would not become malnourished in the next year (81%). Sixty-two percent of Non-Doers found it difficult to remember to include four food groups compared to 5% of Doers. Close to half (45%) of those surveyed noted that they would like to receive additional training in IYCF. Based on our analysis, social support, both in the community and immediate family, was strong, and there were no cultural taboos or spiritual beliefs that got in the way of complementary feeding as recommended (Bamfa was not addressed in this evaluation). Those surveyed found overall that the educational training materials that were being used helped them understand the counselling and training. However, there was a lack of cues in the form of counselling materials in the environment at home to remind caregivers to feed a variety of foods. The biggest challenge caregivers faced was the low availability of foods in the market. There was also an overall need for support to purchase foods. Furthermore, participants beliefs were mildly improved, specifically around their understanding of the severity of malnutrition. Nonetheless, misinformation about food density is an area that needs continued targeting. Additionally, many mothers who were interviewed found that supplemental foods were not well managed at the PHUs and access was problematic. 46 P a g e