Terms of Reference for End of Project Evaluation ADA and PHASE Nepal August 2018 1 - Background information PHASE Nepal, the project holder ( grantee ), is a Non Governmental Organization registered with the District Administration Office Kathmandu. PHASE works for the people in remote hilly and mountainous districts of Nepal to provide primary health care services and to improve access to education and livelihood opportunities. The Austrian Development Agency (ADA), the operational unit of Austrian Development Cooperation, concentrates its activities on key regions in Africa, Asia, South-Eastern and Eastern Europe as well as the Caribbean. Besides cooperating with the governments of priority countries, ADA also promotes initiatives of civil-society and private-sector organisations in developing countries. With funding from the Austrian Development Agency, PHASE Nepal has been implementing the Integrated Programme for the health of women and children in far Western Nepal from January 2016 in Mugu and Bajura Districts in Western Nepal. The programme originally contributed to the Framework Program 2016-2018 Health Care for Mountain Farmers in Nepal led by EcoHimal Austria, but from October 2016 it is funded by ADA as a stand-alone project implemented by PHASE Nepal. The project goal is to Improve health service, access to education and nutrition status for women and children in Far Western Nepal. As part of the agreement, the project holder is responsible for appointing an external evaluator to evaluate the project outputs and outcomes compared with those set in the project log-frame to express an independent professional opinion on the success of the project and whether the funds granted to the Project have been effectively applied for their intended purpose. 2 - Project Description Project Goal: Improving health service, access to education and nutrition status for women and children in far Western Nepal
Overall objective (outcome): Better access to health care and better health and nutrition status in the target population, particularly mothers and children under 5 years of age Outcome indicators: 1. Increased % of mothers having health care during childbirth (Attended delivery) 2. Increased % of mothers attending for antenatal care 4 times (ANC 4) 3. Decreased % of underweight children under 5 years (U5UW) Specific objectives and indicators: 1. improved access to health services 1.1. increased number of patients accessing primary care services 1.2. increased number of women receiving antenatal and postnatal care 1.3. increased number of women receiving skilled care during childbirth 1.4. increased number of young children receiving vaccinations and growth monitoring 2. improved access to nutritious foods 2.1. higher number of children eating leafy green vegetables more than twice a week 2.2. higher number of farmers producing vegetables 2.3. higher number of women rearing chickens and producing eggs 2.4. higher number of improved breed goats 3. improved access to education and health information for adult women 3.1. increased number of women able to read a simple sentence in Nepali 3.2. number of women taking part in health awareness sessions > 5,000 3.3. Increased % of women aware of key messages about child health and nutrition Expected results: 1. 16,700 people living in target area have access to good standard of primary care services, aligned with government policies 1.1. approx. 3,500 women of reproductive age have access to family planning, maternal health care including skilled birth attendance and reproductive health care 1.2. approx. 2,500 children under 5 years of age have access to better child health care and weight monitoring 2. 1200 people in target area grow more diverse and nutritious food (figure includes cofunded project) 3. 640 women in target area have attended adult literacy classes and are able to read and write
Project Summary: The overall main objective is to improve the health and livelihoods of the target population, particularly women and children. This is achieved through improving access to high quality primary healthcare services, nutritious foods and literacy and health education. Through a long term continuous presence of project staff in the area, who support and revitalise government services, people, especially women, have been be made aware of their basic right to health care services. At the same time, livelihoods have been strengthened by small scale agricultural skills building programmes which also included the provision of vegetable seeds and breeding animals. The main objective of this activity was to increase amount and variety of agricultural production, mostly to improve nutrition but to some extent this also opened opportunities for beneficiaries to sell some of their stock and produce locally. The education pillar of the intervention concentrated on women s literacy, involving the literacy groups into the livelihoods programme. Educating women at least to basic literacy level helped empower them and increases the likelihood that their children attend and finish school. PHASE has also been conducting other education activities in Bajura, specifically teacher training, which added to the effectiveness of the intervention. PHASE programmes are based on the assumption that poor access to health care, lack of education and food insecurity are all aspects of a wider complex context of poverty and that they can be addressed in an integrated way. Interventions in the three different sectors are synergistic and a combined approach where one sectoral intervention builds on the achievements of the other is much more effective than if each separate sector had been addressed on its own. Region: This project is implemented in 2 locations (formerly VDCs) in Bajura (Rugin, Bichaya) and 5 in Mugu (Dhainakot, Jima, Ruga, Natarphu, Photu). The project was phased-out from Kotdanda in 2017 and started in a new location, Jima. Project duration: 1 st January 2016 to 31 st December 2018 Project Budget: The total budget for the project is 755,633,00 for three years starting from 1 st January 2016 and ending on 31 st December 2018. (This includes the original framework project budget agreed with EcoHimal Austria) 3 Report commissioners
This evaluation report is commissioned by PHASE Nepal (Executive Director: Dr Jiban Karki) subject to approval by ADA (desk officer: Ms Sylvia Hinger). 4 - Purpose of the independent final evaluation The purpose of the evaluation is to assess the progress made towards achievement of the specific objectives of the project, its outcome and impact on the lives of beneficiaries that include the families, communities and governments. It will also measure the level of community and other stakeholder participation. It will also identify the intended and unintended outcomes, best practices, lessons learned as well as challenges arising from project execution. In addition, the evaluation will present conclusions and recommendations for learning and future intervention. The independent final evaluation report will be used to inform ADA s understanding of the grantee s performance at the project level and will support learning from the project outcomes to inform further work in the area. 5 Principles of the evaluation The Evaluation must be in keeping with the OECD (DAC) quality criteria for evaluations: Ø Independence The planning and execution of the evaluation must assure the greatest possible objectivity and impartiality. Ø Credibility The evaluation team must be able to draw on the required methodological and subject related knowledge as well associated skills. The methods applied in the evaluation and its findings must be presented clearly in the report. Ø Participation Evaluation is designed as a process: External expertise and assessment is combined with a critical discussion by the project stakeholders and target population to gain new perspectives and reach agreement on future work. Ø Transparency The subject, purpose, scope, addressees, evaluation questions, methods, schedule, qualifications of the evaluation team, reporting and coordination must be clearly defined in the terms of reference for the evaluation Ø Utility The evaluation findings should be useful for all stakeholders. Steps must be taken to ensure that they are implemented by the policy and operational decision-makers. All evaluations include the international quality criteria of relevance, efficiency, effectiveness, impact and sustainability. 6 Specific objectives of the evaluation
The evaluation has two objectives: a) To independently verify (and supplement where necessary), the grantees record of achievement as reported through Annual Reports and defined in the project logframe; b) To assess the extent to which the project was good value for money, which includes considering: How well the project aligns with ADA s goals of supporting the delivery of the SDGs. Whether the project met its objectives How well the project applied value for money principles of effectiveness, economy, efficiency in relation to delivery of its outcome What has happened because of the project that wouldn t have otherwise happened 7 - Evaluation questions The evaluation should answer the following questions: Relevance To what extent did the project support achievement towards the SDGs? To what extent did the project target and reach the poor and marginalised? To what extent did the project mainstream gender equality in the design and delivery of activities (and or other relevant excluded groups)? How well did the project respond to the evolving needs of target beneficiaries,? Efficiency To what extent did the project deliver results on time and on budget against agreed plans? To what extent did the project understand cost drivers and manage these in relation to performance requirements? Effectiveness To what extent are the results that are reported a fair and accurate record of achievement? To what extent has the project delivered results that reflect value for money? To include but not limited to: How well the project applied value for money principles of effectiveness, economy, efficiency in relation to delivery of its outcome; What has happened because of the project that wouldn t have otherwise happened To what extent has the project used learning to improve delivery?
What are the key drivers and barriers affecting the delivery of results for the project? Impact To what extent and how has the project built the capacity of civil society? How many people s lives have been positively affected by the project? such as: o farmers producing vegetables and raising chickens o women accessing safe delivery services and literacy classes o children being better fed To what extent and how has the project affected people in ways that were not originally intended? Leverage and Sustainability To what extent has the project leveraged additional resources (financial and in-kind) from other sources? What effect has this had on the scale, delivery or sustainability of activities? To what extent is there evidence that the benefits delivered by the project will be sustained after the project ends? 8 - Evaluation methods Specific details around the methods that are the most appropriate for assessing project performance will be decided together with PHASE Nepal at the start of the evaluation. Evaluation methods should be rigorous yet proportionate and appropriate to the context, data should be triangulated so that findings are as robust as possible. The research should be based on a contribution-based approach which results in a plausible account of the difference that ADA s funding has made to the impact of grantees. This approach should be informed by a wide range of evidence sources that are brought together to produce a plausible assessment of the contribution of grantees to higher level outcomes and impacts. The evaluator should apply a mixed-methods approach for assessing project performance. This would combine qualitative data to provide an explanation of why and how the project has achieved the type and scale of results that are quantitatively observed. Relevant primary and secondary research may include: Review of existing documents and review of context and poverty trends including the project proposal and other documents, annual and quarterly reports, monitoring and evaluation reports. Field observation of the targeted community Household survey using survey questionnaires
Interviews and workshops with PHASE key staff, local authorities and local groups Focus Group Discussions involving primary project participants, other social groups and key stakeholders. Report writing and debriefing with project and management teams Scoping reviews of secondary studies and sources, measuring impact where possible and proportionate through comparison groups and other quantitative methods Verifying reported data through back checking and quality control assessments. List of partners/stakeholders to be involved/consulted in the evaluation include Government bodies (DCC, Health, Agriculture, Livestock and Education offices at the municipal level) Community group s Participants PHASE Nepal staff (management as well as field staff) Materials to review will include: ADA Guide to Project Evaluations if any The original application for funding and updated documents MOU with ADA for funding Project logframes (original and revised) Monitoring data Monitoring systems Annual reports and comments provided if any Other evaluation reports undertaken by the grantee Financial information / information on resources spent Information on synergies / collaboration with other actors Published material (e.g. To demonstrate sharing of learning with others) 9 - Deliverables and timeframe We will be able to provide space in the PHASE Nepal office in Thimi (between Kathmandu and Bhaktapur) for the consultant to work. Suggested Time Frame By 30 th September 2018 Contract awarded By 30 th October 2018 desk study, research design, planning and preparation (5 days) By 15 th December 2018 Research and 15 days field visit to Far West Nepal + 9 days for travel By 31 st December 2018 Initial draft report due (15 days for data analysis and report writing) End of January 2019 Final report due (5 days for finalization and submission)
The following should be delivered: Inception report with detailed evaluation plan by 30 th October First draft report by 31 st December Final evaluation report by 31 st January 2019 Submission of hard and electronic copies of materials, data collected / analyzed and other evaluation documents. A summary power point presentation highlighting main findings and recommendations The independent final evaluation report needs to be a substantial document that (a) Answers all the elements of the Terms of Reference (ToR) (b) Provides findings and conclusions that are based on robust and transparent evidence (c) Where necessary supplements the grantee s own data with independent research. The main body of the report must be limited to 40 pages (exclude annexes). One of the annexes should consist of a table which summarises the findings according to the OECD-DAC criteria. To ensure consistency across evaluation reports, the following structure should be used for reporting: Executive Summary Introduction Purpose of the evaluation Organisation context Logic and assumptions of the evaluation Overview of ADA funded activities Evaluation Methodology Evaluation plan Proposed research methods and tools. Strengths and weaknesses of selected design and research methods Summary of problems and issues encountered Findings Overall Results Assessment of accuracy of reported results Relevance Effectiveness Efficiency Sustainability
Impact Conclusions Summary of achievements Overall impact and value for money of project activities Lessons learnt (where relevant) Project level - management, design, implementation Policy level Sector level Recommendations Annexes (such as) Independent final evaluation terms of reference Evaluation research schedule Evaluation framework Data collection tools List of people consulted List of supporting documentary information Tables, charts and photos. Details of the evaluation team Grantee management response to report findings and recommendations 10 - Required Qualifications A minimum of five years experience in programme/project evaluation in an international development context. Experience of results-based monitoring and evaluation Ability to design and plan the evaluation approaches and research methodologies, including quantitative and qualitative research methods. Qualifications in Public Health or Nutrition of primary researcher or at least one team member Relevant subject matter knowledge and experience given the aims and objectives of the project and the context in which it is being delivered Significant experience of working in Nepal Excellent writing skills Fluency in English and Nepali
11 - Terms of payment PHASE Nepal shall pay the consultant 90% of the agreed total fee upon submission of the draft report and the final 10% on submission and acceptance of the final evaluation report. 12 Tenders Tenders should include and will be assessed on Method statement description of activities to deliver requested services; 30-40% Project team s and organisation s experience and relevant expertise; 15% Project and resource plan; 15% Quality assurance and risk; 10% Price 20% - 30% We will invite the top 3 bids for presentation for selection. The budget of approx. NPR 400,000 is inclusive of all costs covering team member costs, research costs and any other costs associated with the completion of the work. Costs of travel to and from the project area will be provided by PHASE Nepal, but the local travel and other costs within Kathmandu if any should be borne by the consultant team themselves. PHASE Nepal team will offer support with the project visit to the far West. The project sites are up to 4 days walk away from the nearest road head or the airport in Eastern part of Bajura and Southern part of Mugu districts. Applications from qualified firms/individuals should be submitted by 31 st August 2017 to info@phasenepal.org