Title of Position: Evaluation Team of GFFO Project, Ukraine Location: Kramatorsk, Ukraine (Government Controlled Area) Contract term: 19 September to 26 October 2018 (including preparation, evaluation and report writing) Application Deadline: 10 August 2018 1. About ADRA Ukraine ADRA Ukraine is a local non-governmental organization (NGO) in the field of humanitarian aid and development cooperation. The organization is member of the large global ADRA network present in more than 125 countries. ADRA Ukraine has been operating in Ukraine since 1993 and used to provide activities through 4 offices Ukrainian Head Quarter in Kiev and 3 regional offices in Mariupol, Kramatorsk and Severodonetsk. Due to humanitarian response, ADRA Ukraine is providing humanitarian assistance to internally displaced people in 15 regions of Ukraine. Since the start of the military conflict in Eastern Ukraine, ADRA takes care of about 1,000,000 people affected from the man-made crisis in Ukraine. 2. About the project funded by German Federal Foreign Office (GFFO) The project includes two implementation phases. The first phase was implemented by ADRA Ukraine between June 2016 October 2017 in response to the needs of the most vulnerable population living in the government controlled area of Donetsk and Luhansk regions. The first phase was focused on Shelter/NFI, PSS and WASH sectors, including the following activities: shelter repair services, heating materials distribution, PSS assistance, heating systems installation, provision of clean drinking water, hygiene certificates provision and promotion of hygienic habits through theatrical performance. Total budget: 1,999,650.00. The second phase with a period from November 2017 to September 2019 covers Shelter/NFI, PSS, WASH and a total budget of 3,000,000 ; with focusing on shelter repair services, house insulation, winterization assistance, PSS activities, provision of clean drinking water, promotion of hygiene practice, provision of hygiene items and improvement of access for isolated residents living along the contact-line to basic needs and social services. The evaluation will focus on shelter assistance and psychosocial support which are the main components of the project. The geographical coverage includes: - 26 west-side contact-line locations of Donetsk and Luhansk oblasts for psychosocial service activities; - Slavyansk, Nikolaevka, Seleznevka, Semenovka, Andreevka, Kramatorsk, Hornyak, Sieversk, Mironovsky and Luhanskoye of Donetsk oblast, along with Troitskoye, Popasnyi and Zolotoye of Luhansk oblast for shelter activities. The exact locations of the evaluation will be selected during the Kick-off Workshop. Overall Goal: Contributing to the Humanitarian Response Plan by reducing the numbers of IDPs Project Goal: People affected by the war receive protection from the weather and cold and a strengthening of their mental health. 1
The already achieved shelter component is designed as such: Result of the Shelter Component: Provision of adequate living space that offers protection from the weather and privacy. Activity 1: Light building repairs carried out on 414 households (approx. 996 beneficiaries) Activity 2: Medium-sized building repairs carried out on 799 households (approx. 1647 beneficiaries) Activity 3: Heavy building repairs carried out on 30 households (approx. 109 beneficiaries) The already achieved PSS component is designed as such: Result of the PSS component: Critical gaps in delivery of quality essential and life-saving health services to conflict-affected population have been closed Activity 1: 6,900 beneficiaries, including those living along the line of contact, have received individual and group support in psychosocial care through mobile teams and child-friendly spaces (CFS). Activity 2: 3,750 beneficiaries have received psychosocial support, including physiotherapy sessions at ADRA community center (Kramatorsk and Sieverodonetsk) Activity 3: 50 psychologists and social workers were trained in practical methods and trauma management techniques The following progress has been achieved on PSS indicators by 31 st of May: Activity 1 1565 beneficiaries; Activity 2 894 beneficiaries. 3. Objectives The purpose of the Mid-Term Evaluation with tentative date in September 2018 is to assess the impact, effectiveness, efficiency and accountability 1 of the Shelter and PSS Components to develop lessons learnt for use in future policy work, and programme design and development. The project evaluation will be externally presented to the shelter cluster and humanitarian actors as well as to the donor German Federal Foreign Office. ADRA will also present the report to the beneficiaries and the local government. The lessons learnt / recommendations will be incorporated into our institutional learning and their outcome will be used for future projects. It is planned that the evaluator / evaluation team who will conduct the mid-term evaluation will be prioritized for consideration in the final evaluation November 2019. The following evaluation questions might be considered during the evaluation: Impact & Relevance o To what extent did the provision of assistance discourage migration and increase the number of returnees? o Was the reconstruction of shelters relevant during the present conflict? Were repaired houses destroyed again? PSS o Did the affected population develop positive coping strategies? Could the affected people strengthen their resilience? Did the assistance help to avoid or reduce practicing negative coping strategies among the project beneficiaries? o Was there other impact by the PSS component? How has the project assistance influenced the life of the community/families/beneficiaries? 1 Cf. Evaluation of Humanitarian Action Guide (ALNAP): https://www.alnap.org/system/files/content/resource/files/main/alnap-evaluation-humanitarian-action-2016.pdf 2
PSS & Shelter: o To which extent did the project contribute to the implementation of the Humanitarian Response Plan in 2016 and in 2017? Effectiveness o To which extend were beneficiaries involved in the repair work? Which key factors helped/or prevented to involve beneficiaries in the repair work? o How and to what extend are the shelters used by the beneficiaries? (Self-inhabited, rented, sold, empty, etc.) PSS & Shelter o To which extend were the activities instrumental in achieving the desired output? o To which extent were the project beneficiaries satisfied with the quality of Shelter and PSS services? Efficiency o Did the repair work go according to schedule? Was the time appropriate between request by the beneficiaries and the assistance? PSS & Shelter o Were others, more cost-efficient ways of implementation assessed? Accountability o To what extend did the shelter provided meet the recommendations by the Ukraine Shelter Cluster and the standards by the Global Shelter Cluster? PSS o To what extend did the PSS component meet the IASC Guidelines on Mental Health and Psychosocial Support in Emergency Settings? PSS & Shelter o To which degree did ADRA Ukraine target beneficiaries in a transparent and accountable way? Was the selection process transparent for the affected population? o To what extent did ADRA Ukraine respond to complaints and feedback by the beneficiaries? Coordination PSS & Shelter: o To which degree did ADRA Ukraine coordinate with other stakeholders, to enhance synergies and to avoid gaps, duplication and resources conflicts? ADRA will utilize the evaluation findings for both improving the quality of work for accountability purposes and for lesson learning, and more specifically for: Improving compliance with the Core Humanitarian Standards (CHS) and other standards relating to accountability and transparency of our work, as well as technical quality of our programmes. 3
4. Approach and Methodology of the Evaluation Develop the study design in consultation with ADRA Ukraine program staff and ADRA Germany staff. The design of data collection tools (Questionnaire, focus group questions and informed consent forms) must be validated by ADRA Germany. Literature review of relevant documents (e.g. National policies, grey literature, reports, relevant ADRA guidelines, evaluations and studies from relevant humanitarian and governmental actors including also the project internal evaluation surveys, minutes from shelter, and the Inter Agency Coordination groups) detailing shelter issues to assess the relevance & impact. Perform primary data gathering through key informant interviews and focus group discussions among key external stakeholders (government, CSOs, UN coordinators, etc.). Furthermore, focus group discussions, key informant interviews with beneficiaries and the affected population in general and as well a household survey (the exact number of interviewees will be elaborated during the workshop). The interviews, focus group discussions and household survey will capture information on impact, effectiveness, efficiency and accountability. The evaluation on shelter components and their respective results will require home visits. The evaluation on PSS components and their respective results requires home visit as well but to some extent phone calls can also be organized. Oversight of data collection and data analysis The evaluator/evaluation team will review data and visit ADRA Ukraine in the field to interview relevant stakeholders in cooperation with ADRA Ukraine Program Director, Project Manager, Monitoring & Evaluation team. Presentation and verification of key findings among key stakeholders interviewed and with ADRA Ukraine program staff before final report writing Drafting of final report and concise recommendations to guide ongoing project activities and identify how government, non-government and humanitarian actors could improve their accountability in compliance with the Core Humanitarian Standards. The recommendations will be verified with the relevant stakeholders to ensure their validity. Presentation of key findings to ADRA Ukraine programme staff, ADRA Germany, cluster representatives and delegation of the German Federal Foreign Office 5. Deliverables during the Evaluation Proposal outlining methodology and work plan (max. 4 pages) The proposal outlining the methodology of the planned evaluation and the work plan are part of the documents which needs to be submitted with the application Inception report (max. 10 pages) The inception report should set out the planned approach to meeting the evaluation aims and objectives, methodologies to be used and questions to be answered through the reviews and planned 4
interviews. It should provide a description on how data will be collected and drafts of suggested data collection tools such as questionnaires and interview guidelines. The proposal outlining methodology and work plan should be used as basis for the inception report. Draft evaluation report (for the structure refer to Final evaluation report) Tentative Deadline: 11 October 2018 Final evaluation report including a summary (max. 35 pages excluding annexes) Tentative Deadline: 21 October 2018 The report should include (but is not limited to) the following: Executive summary (max. 2-3 pages) Evaluation aims, objectives, and scope Methodology (reflection and linking to the TOR and possible constraints leading to deviations from the TOR) Findings (related to the objectives of the TOR) Conclusions Recommendations Lessons learnt Annexes (including ToR, maps, questionnaires, list of interviewees, and bibliography) Deadline: Will be agreed at the debriefing workshop Language All documents should be written in English. 6. Evaluation Duration (with suggested number of days in the field) Tender- Deadline: 10 August 2018 Recruitment of Evaluation Team: 23 July to 17 August 2018 Kick-off Workshop in Kiev: 19 September 2018 (1 day) Inception Report: 22 September 2018 (2 days) Evaluation Phase: 23 September to 3 October 2018 (11 days) Draft Report: 11 October 2018 (5 days) Debriefing with ADRA in Kiev: 12 October 2018 (1 day) Final Evaluation Report: 21 October 2018 (3 days) Final Finding Presentation to stakeholders: until 26 October 2018 (2 days day) Total: 25 days Specific Dates are subject to mutual agreement. 7. Experience and skills required The evaluator/evaluation team should have the combined following qualifications: Demonstrable experience (minimum 5 years) in evaluation of development programs related to Shelter and PSS in conflict and displacement settings 5
Experience in leading program/project evaluations (quantitative and qualitative) in international development and/or relief settings Experience in the design and implementation of multi-methodological evaluation approaches and tools (quantitative and qualitative) Russian and Ukrainian language skills Sound data analysis skills with the use of appropriate statistical software for data analysis and visualization both quantitative (e.g. SPSS) and qualitative (e.g. Atlas.ti) Familiar with the OECD-DAC evaluation criteria for Humanitarian Action, ALNAP, Sphere standards and related international quality (e.g. CHS) and evaluation standards Excellent organizational and report writing skills (demonstrated through work samples) Excellent verbal and written communication skills in English Ability to work effectively and independently in a cross-cultural environment Willingness to travel to project sites Asset: Experience of working in Ukraine and/or post-soviet region 8. Financial Resources: ADRA will be responsible for hotels & transport within Ukraine The financial proposal should include beside consultancy rate also expenses for per diem, international flight if needed and other upcoming expenses 9. Human Resources: The evaluator / evaluation team will be responsible for translation & enumeration during the field visit. The evaluator/evaluation team receives all required support in terms of organizing the field trip, security issues and access to ADRA documentation. Evaluator / Evaluation Team must adhere to humanitarian principles, ADRA Protection Policy, and ethical values of ADRA. 6
The application should include: A cover letter addressing the selection criteria above. Applicant s CV, highlighting experience relevant to this evaluation. A sample report from a prior assignment with content directly relevant to this evaluation. Technical proposal illustrating the applicant s understanding of the TOR, draft evaluation framework and plan including any logistic support required. Financial proposal: The financial proposal should provide cost estimates for services rendered including daily consultancy fees. Contact details of at least two references from among recent employers or clients. Application deadline: 10 August 2018 Applications to be submitted to: ADRA Germany René Fechner Rene.Fechner@adra.de & Lukas Driedger Lukas.Driedger@adra.de Please indicate Mid-term Evaluation Team in the subject line. Only shortlisted candidates will be contacted. Your application is collected for the purpose of recruitment and personal administration. Unless you direct otherwise (for example if you would like the application kept on file for future vacancies) the application forms (and attachments) of unsuccessful applicants will be destroyed after 6 months. It is the agency policy to protect, and keep secure, all personal data collected. All personal data is processed for the purpose of recruitment, and, in the case of successful applications, for the satisfactory administration of their employment, and for no other purpose. 7