Evaluation of the European Commission's Humanitarian and Disaster Risk Reduction Activities (DIPECHO) in Indonesia. Final Report

Transcription

1 Evaluation of the European Commission's Humanitarian and Disaster Risk Reduction Activities (DIPECHO) in Indonesia Contract number ECHO/ADM/BUD/2013/01201 Final Report 6 December 2013 Evaluators: Claude de Ville de Goyet (Team Leader) Philip Buckle John Ievers Hening Purwati Parlan Joseph Viandrito The opinions expressed in this document represent the views of the authors, which are not necessarily shared by the European Commission

2 TABLE OF CONTENTS Table of Contents... ii Figures and Tables... v List of Figures... v List of Tables... v List of Pictures... v List of Abbreviations and Acronyms... vi Executive Summary Introduction: Indonesia in the ASEAN Context The Country The Vulnerability to Hazards Disaster Risk Reduction in Indonesia and the ASEAN DG ECHO s Programme in Indonesia Humanitarian Assistance Disaster Risk Reduction ECHO office in Jakarta Purposes and scope of the evaluation Methodology and limitations Evaluation Design Conduct and Methodology Limitations Key evaluation questions: findings, specific analysis and CONCLUSIONS To what extent have DG ECHO strategic orientation and intervention logic addressed the needs of the most vulnerable communities and categories of population, also in the areas most exposed to frequent natural hazards? (EQ1 - Relevance) Findings Analysis and Specific Conclusions To what extent have ECHO (humanitarian aid) and DIPECHO-funded activities contributed to reducing vulnerabilities, enhancing the capacity to cope, and boosting the resilience of targeted at risk populations, national institutions and the wider disaster risk reduction stakeholder community to recurrent natural hazards in Indonesia? (EQ2 Effectiveness) Findings Analysis and specific conclusions To what extent have DG ECHO-funded DRR activities contributed to reinforcing subnational response capacities, including preparation and awareness of local communities? (EQ3 - Effectiveness) Final Report PARTICIP GmbH ii

3 4.3.1 Findings Analysis and Specific Conclusions To what extent do ECHO funded DRR programmes complement actions developed, endorsed and/or supported by national government, EC cooperation instruments or other external assistance, taking into account DG ECHO s mandate? (EQ4 Coordination-Complementarity and Coherence) Findings Analysis and specific conclusions To what extent have DG ECHO-funded DRR activities had an impact at national level, through replication, scaling-up, advocacy, or complementarity with national initiatives (EQ5 Effectiveness - Sustainability) Findings Analysis and specific conclusions To what extent LRRD aspects have been taken into account in the design and implementation of ECHO and DIPECHO strategies and activities? (EQ6 Effectiveness) Findings Analysis and Specific Conclusions To what extent the following cross-cutting issues have been taken into account in the design and implementation of ECHO and DIPECHO strategies and activities? (EQ7 Cross cutting Issues) Findings Analysis and Specific Conclusions Which elements in the institutional context and in the relations between the different actors have had a higher impact in the effectiveness and scaling up of the DRR interventions? (EQ8 Efficiency) Findings Analysis and Specific Conclusions To what extent have the project activities and results had lasting effects after hand over? (EQ9 Sustainability) What were the added value and the relevance of DG ECHO presence in Indonesia for the last five years? (EQ 10 Impact) Findings Analysis and Specific Conclusions Overall conclusions DRR and Development From Communities to National and Regional Levels Consortium Approach DIPECHO Exit Strategy DIPECHO Projects ECHO Office in Jakarta EU Civil Protection Role Final Report PARTICIP GmbH iii

4 6 Recommendations Critical DIPECHO Strategic Recommendations Other Strategic Recommendations Operational Recommendations: Annex A Specifications Annex B Detailed Description of the methodology implemented and the Tasks carried out by each Expert Annex C List of Agencies and Sites Visited, incl. Persons Interviewed Annex D Summary presentation of individual case study visits IoM in Garut district, West Java Building sub-national capacities from a low starting point Ciamis, West Java Linking shelter relief to rehabilitation and development Cross-cutting issue Mainstreaming disaster preparedness focusing on children with disabilities West Sumatra Enhancing Community Based Disaster Management and Capacity Building Annex E Map of Areas covered by the Operations Financed under the Action and the Sites visited by the Contractors Annex G Bibliography Annex H Proposed Revised Tasks of DG ECHO Program Officer in Jakarta Final Report PARTICIP GmbH iv

5 FIGURES AND TABLES List of Figures Figure 1- Site visit locations: Padang, Jakarta, Garut, Ciamis, Yogyakarta and Kupang from East to West List of Tables Table 1: Profile of 6 selected ASEAN countries... 6 Table 2: Natural Disasters Table 3: Top 10 Natural Disasters in Indonesia... 8 Table 4: Summary of evaluation phases List of Photos Photo 1: Mount Merapi eruption (from DG ECHO Annual Report 2010) Photo 2: Evacuation routes and facilities Photo 3: Padang - community evacuation road Photo 4: limits on small DRR projects; disposing of waste Photo 5: the Early Warning System developed by Mercy Corps Photo 6: flood level Photo 7: ASB bamboo shelter Final Report PARTICIP GmbH v

6 LIST OF ABBREVIATIONS AND ACRONYMS 3Cs AADMER ACF AHA Centre APG ASB ASEAN AusAID BNPB BPBD CBDRM CBDRR CP DEVCO DG ECHO (ECHO) DIPECHO DM DRM DRR EC EEAS EQ ERAT ERC ERT EU EWS FAO FHA GDP GIS GNA GoI HA HFA HI HQ HVCA IDEA IFRC IMF INGO IOM Complementarity, coordination and coherence ASEAN Agreement on Disaster Management and Emergency Response Action Contre la Faim ASEAN Coordinating Centre for Humanitarian Assistance on Disaster Management AADMER Partnership Group Arbeiter-Samariter-Bund Association of Southeast Asian Nations (Brunei, Cambodia, Indonesia, Laos, Malaysia, Myanmar, Philippines, Singapore, Thailand and Vietnam) Australian Government Overseas Aid Programme Badan Nasional Penanggulangan Bencana (National Board for Disaster Management) Badan Penanggulangan Bencana Provinsi (Provincial Disaster Management Agency) Community Based Disaster Risk Management Community Based Disaster Risk Reduction Civil Protection DG Development and Cooperation European Commission's Humanitarian Aid and Civil Protection Directorate General Disaster Preparedness ECHO Disaster Management Disaster Risk Management Disaster Risk Reduction European Commission European External Action Service Evaluation Question Early Rapid Assessment Team Emergency Response Center Emergency Response Team European Union Early Warning System Food and Agriculture Organisation of the United Nations Kyoto Framework for Action Gross Domestic Product Geographic Information System Global Needs Assessment Government of Indonesia Humanitarian Aid Hyogo Framework for Action Handicap International Headquarters Hazard-vulnerability capacity assessment International Institute for Democracy and Electoral Assistance? International Federation of Red Cross and Red Crescent Societies International Monetary Fund International non-governmental organisation International Organisation for Migration Final Report PARTICIP GmbH vi

7 ISDR See UNISDR JC Judgement Criteria JICA Japan International Cooperation Agency? KPDT Kementrian Daerah Tertinggal (Ministry of Neglected Areas) LRRD Linking Relief and Rehabilitation to Development MDG Millennium Development Goals MIC Monitoring Information Center MoU Memorandum of Understanding NAP National Action Plan (for Disaster Risk Reduction) NFI Non Food Items NGO Non-Government Organisation NTT Nusa Tenggara Timur Province NZ New Zealand NZAID New Zealand Agency for International Development ODA OECD Official Development Assistance OECD/DAC Organisation for Economic Co-operation and Development/Development Assistance Committee OFDA US Office of Foreign Disaster Assistance OGB OXFAM Great Britain PDNA Post-disaster needs assessment PMI (Indonesian Red Cross) PPP Purchasing Power Parity READI Regional EU-ASEAN Dialogue Instrument RSO Regional Support Office SMS Short Message Service SNAP Strategic National Action Plan ToR Terms of Reference (used as a synonym for specifications for this evaluation) UN United Nations UN OCHA (OCHA) United Nations Office for the Coordination of Humanitarian Affairs UNDAC UN Disaster Assessment Coordination Team UNDP United Nations Development Programme USAID United States Agency for International Development WFP World Food Programme Final Report PARTICIP GmbH vii

8 EXECUTIVE SUMMARY Context of the evaluation Indonesia with a population of million is highly vulnerable to earthquakes, volcanic eruptions, and floods, in both metropolitan areas and those less urban. The tsunami in 2004 and the earthquakes in 2009 are still very much present in the population s memories. Indonesia is also a nation with a steadily growing economy (6%/year) and a strong new disaster risk reduction law. The National Disaster Management Organization (BNPB) is increasingly competent and keen to affirm its leadership with international actors. Indonesia is also the HQ of ASEAN (Association of South East Asian Nations), the only regional organization with an Agreement on Disaster Management and Emergency Response (AADMER) which is legally binding for its Member States. Since 2007, ECHO presence has shifted from humanitarian assistance ( 10.1 Mio mostly in the first three years) to community based disaster management projects, initiatives for handicapped or technical support and liaison with AADMER (over 5.7 Mio, a modest amount compared to the Disaster Risk Reduction DRR - budget from other donors and Indonesia itself). ECHO s policy to increase its focus on aid effectiveness, results-orientation and impact, led to a progressive decrease in the number of new Indonesian projects, in preparation for a potential exit from a country that is considered to be increasingly resilient. Over the period covered by the evaluation, strategic priorities emphasized progressively the importance of a multi-hazard approach (2010) and later a coherence of partners activities with the national vulnerability index and development priorities, as well as making replication / dissemination a priority at sub-national level (2012). Purpose and methodology of the evaluation The purpose of the evaluation is to assess the performance and added value of ECHO-funded programmes from 2007 to Ten Key questions have been formulated by DG ECHO, some specifically forward looking on the contribution of EU Civil Protection, now part of DG ECHO, and a DG ECHO exit strategy from Indonesia. The methodology included an extensive review of documents and publications both internal and external to DG ECHO, briefing meetings with key DG ECHO staff, interviews with ECHO Regional Support Office in Bangkok, two extended visits to Indonesia where a broad range of stakeholders were interviewed individually and five projects areas were visited. A consultative workshop was held in Jakarta with all stakeholders at the end of the first visit to the country. To conclude the process, consultations with officials from ECHO, including Civil Protection, the DG for Development and Cooperation (DEVCO) and the European External Affairs Services (EEAS) took place in Brussels. Relevance and appropriateness All interlocutors and observations pointed to DG ECHO emergency humanitarian assistance (HA) being one of the most relevant among the international community. DG ECHO directed its assistance to the most vulnerable groups affected by disasters often largely ignored by other Evaluation report Particip GmbH 1

9 donors: a neglected food crisis in Eastern Indonesia and the West Java earthquake largely underfunded by external donors were some of the forgotten crises. ECHO went a step further by identifying those with special needs as some of the most affected by a disaster: for instance children with disabilities, especially mental disabilities, a group normally hidden from sight by the communities. Considerable information indicated that DIPECHO Disaster Risk Reduction (DRR) activities were also well targeted. Communities spread over the many vulnerable districts, were selected based on the national risk mapping complemented by an objective assessment. This evaluation endorses the conclusion of the 2007 evaluation of DIPECHO in South East Asia, All partners conducted area-specific risk assessments as a basis for designing their projects, some more comprehensively than others. ECHO s bottom-up approach was very appropriate. Relevance and appropriateness are great assets of DG ECHO projects in Indonesia. Effectiveness, efficiency and coverage The effectiveness of the emergency HA was noted both in terms of timeliness and results. Several examples were still praised by beneficiaries and interviewees years after completion. At the DIPECHO projects visited, a positive impact on the preparedness and resilience of local beneficiaries could be seen, occasionally evidenced by visible improvements of the local response in a subsequent disaster: floods or others. Effectiveness on a wider scale (beyond the targeted communities: from District to Province and National levels) has been more elusive. The projects that most clearly had benefits beyond the direct beneficiaries were those involving participation and buying in by local institutions which were motivated and supported by ECHO implementing partners such as line ministries (education, health etc.) or the national society of Red Crescent. Although most community based projects included capacity building of the District Government, few significantly and sustainably targeted the Provincial Disaster Management Institutions and none addressed the institutional needs of the national BNPB. This national objective is not mentioned in ECHO strategic documents, which only emphasized replication at the sub-national level. This can be seen as a shortcoming in a rather centralized disaster management system. The bottom up strategy of ECHO failed to meet and merge with the top down approach adopted by BNPB. At ASEAN level, ECHO promoted the establishment of a dialogue between AADMER and the few International NGOs members of the AADMER Partnership Group. This dialogue still needs to meaningfully involve the Indonesian civil society. In terms of coverage, expectations should remain modest given the comparatively low level of ECHO funding, the vulnerability to disasters of 40% of the population and the size of the country. Nevertheless, the fine-tuned selection of targets (in particular of specific vulnerable subgroups such as handicapped and women) and the partnership with development implementing partners with a large network of contacts has resulted in a very efficient coverage for ideas and models developed by ECHO. The production and dissemination of material and publications was a key factor. Evaluation report Particip GmbH 2

10 Impact and sustainability The findings point to some long term impact on awareness or preparedness of the direct beneficiaries in most cases. There are a few examples where this impact on actual resilience and response in a disaster, could be documented following the end of the project. This observation is important as the benefit of imparting new knowledge or promoting a change in attitudes remains somewhat intuitive or speculative until proven in a real crisis to lessen vulnerability or improve the local response. Evidence suggests that the community impact of ECHO funded projects will outlive the partners activities. The potential for sustainability of the capacity building carried out by Partners at district level is also positive in view of the political commitment backed by legislation, the existence of financial government resources for this purpose and the involvement of the Civil Society including local universities. Replication and scaling up are another matter. The perspectives appeared more limited. The projects most successful in building a DRR mechanism at district level have generated interest from neighbors. Some of the latter are progressively adopting selected aspects of the model. Similarly, thematic or specialized projects involving (or owned by) line ministries are being replicated at national level. General community-based DRR projects showed little prospects for wide scale replication. There is no national sponsoring or endorsement by BNPB which has limited direct knowledge of these community-based models and still less sense of ownership. BNPB is the only institution able to stimulate adoption and expansion of ECHO models. The issue is a deficient marketing by ECHO or EU not a lack of national resources as is generally the case in many countries where DIPECHO is operating. Triple C s The 3Cs include coordination, complementarity and coherence in the sense of the Maastricht Treaty. At national level, BNPB expressed dissatisfaction regarding the coordination and consultation by DG ECHO on DRR issues. Although DIPECHO activities fall under the officially stated national priorities, BNPB s immediate concern is building its own institution top-down from central to Province level and exercising their mandated overview or coordination over all foreign partners. The lack of ongoing dialogue between ECHO and the national disaster management agency was illustrated by the conspicuous absence of DG ECHO at a large consultation meeting between the Humanitarian Country Team (of which ECHO is a Member) and the Government of Indonesia. BNPB invited all donors except ECHO. Coordination with other donors is generally effective. The interlocutors see a complementarity between their projects and the very specialized and appreciated niche of ECHO. Participation of DG ECHO in donors meetings and its advocacy for adoption of its communities models are noted. This coordination does not lead to a planned complementarity as other donors channel their funds and projects through BNPB. Coherence within the EU is another issue. DG ECHO is looking at DEVCO, the Development and Cooperation General Directorate for continued support to its beneficiaries and partners. It is a key component of an exit strategy. The evaluators observed little substantive dialogue in Brussels and still less in Indonesia in the sense of mutually influencing the selection of projects and operational decision making. DEVCO just launched an initial project to strengthen the coordination mechanisms of AADMER. Civil Protection, part of DG ECHO, appeared to play a substantive role in the gene- Evaluation report Particip GmbH 3

11 sis and design of this project. The evaluators noted a lack of dialogue or influence of other units in DG ECHO (Jakarta or Regional Support office) on this project. ECHO has not exploited its potential for complementarity and coherence between DIPECHO (targeting communities and working through NGOs) and Civil Protection (building governmental institutions capacity). In brief, the issue is not the quality of or the need for DIPECHO product but its promotion at GoI national level. This may derive from a lack of recognition of the long term developmental nature of DRR and an unbending application of humanitarian principles by DG ECHO in a developmental (that is non humanitarian) context. Within the EU, there is also room for improvement in terms of coherence between DG ECHO and DEVCO and within DG ECHO itself where the two cultures (humanitarian and Civil Protection) have so much to benefit mutually from a collaboration in Indonesia to start with. Recommendations One recommendation stands out: (#1) DG ECHO should continue to closely monitor the situation in Indonesia, a country with a high potential for major disasters. This monitoring should benefit from the expertise of the Emergency Response Coordination Centre and from the onsite experience and observation of ECHO partners. Indonesia will remain a critical pivot for DRR and HA in the region, whatever the ECHO decision regarding an exit. Other main recommendations are divided into critical strategic recommendations (the prerequisites that should determine an exit strategy), other strategic recommendations and finally operational recommendations. Conclusions Recommendations Critical Strategic Recommendations DG ECHO s lack of programmatic dialogue and consultation with BNPB is turning out to be the main impediment for scaling up and sustaining ECHO models. Maintaining a direct dialogue does not mean establishing a formal inter-institutional relationship, something outside the remit of a DG mandate. ASEAN and AADMER are offering an opportunity to advocate DIPECHO expertise in community based DRR and civil society. DEVCO funded cooperation with AHA offers an entry point for a coherent and powerful message from both the humanitarian and Civil Protection arms of ECHO. Other Strategic Recommendations The dialogue between ECHO and DEVCO is not mutually influencing the decision making effectively. It should more substantive and sustained. #2 To build a constructive relationship and dialogue with BNPB. #3 To exercise a significant influence with ASEAN and Indonesia s fast development as DRR leader in South East Asia. # 4 To advocate complementarity between the long-term development initiatives of DG DEVCO and DIPECHO activities. #5 To present an internally coherent message advocating the top down management principles promoted by Civil Protection (A5) and the bottom up community based and civil society approach modeled by DIPECHO. Evaluation report Particip GmbH 4

12 The Indonesian trend is to decrease reliance on international partners and focus on national NGOs. DIPECHO and in particular the APG regional project do not strengthen sufficiently the capacity of local NGOs. Operational Recommendations The scaling up and even the sustainability of the impact on capacity building cannot be achieved without a minimal presence in Jakarta. Lack of BNPB knowledge and ownership of DIPECHO projects is a major obstacle to scaling up. This is the only institution with the power to scale up DIPECHO models. Other donors will follow BNPB s lead. The success of partners' collaboration with the Ministry of Education offers a good potential complementarity with DEVCO s strong support of this sector. Operational implication of recommendations #1 and 6. Local DRR budget exists but is difficult to access. Lack of Government funding is not the main issue in Indonesia. Requiring a budgetary commitment from Province authorities is possible and desirable. Dialogue with ASEAN mechanisms does not involve or benefit sufficiently the national Civil Society Operational implication of recommendation #6 Common vision and outcomes resulting from working together as a team rather than as individual agencies is a long term undertaking worth pursuing particularly in a country with a strong national disaster management organization. The projects specifically targeting persons with disabilities is a highlight of DIPECHO in Indonesia. They are worth replicating elsewhere. DG ECHO has an understanding with OCHA for such post disaster deployment of experts. It may be convenient to open an additional option for hosting locally those experts under the ASEAN regional coordinating mechanism. #6 To give more priority to strengthening national NGOs. #7 To maintain an effective DG ECHO presence in Indonesia. #8 To invite the National Disaster Management Organization to designate an expert to participate in the onsite visits of future external evaluations and, when appropriate, in the monitoring visits by the TA. #9 To engage DEVCO in Jakarta to incorporate learning from inclusive education programs in Indonesian based education initiatives. #10 To consider the inclusion of Indonesia in the next two DIPECHO funding decisions. Projects should meet three requirements: approval from BNPB, sponsoring of local NGO partners and identification of future matching funding from local authorities. #11 To request the AADMER Partnership Group (APG) to expand its membership to include local NGOs (or their networks) with regional, national or multi-provincial reach. #12 To continue promoting a consortium approach especially in countries with a strong leadership such as that of the BNPB. #13 To replicate the successful approach of projects dedicated to the Handicapped and priority groups in other countries. #14 To initiate negotiation with AADMER/AHA of a MoU for potential emergency deployment of EU Emergency Response Coordination Centre experts under the AHA umbrella. Evaluation report Particip GmbH 5

13 1 INTRODUCTION: INDONESIA IN THE ASEAN CONTEXT 1.1 The Country 1. Indonesia, with an estimated population of approximately million (2012), is the most populous country of the Association of Southeast Asian Nations (ASEAN), representing approximately 39.7%. Consisting of 17,508 islands, Indonesia is the largest archipelagic country in the world. 2. Indonesia is classified by the World Bank as a lower middle income country. Its economy expanded at a rate of 6% / year over the period of The International Monetary Fund (IMF) estimates the GDP PPP per Capita 2 at 4,977 USD in 2012, projected to reach 7,487 USD in 2018, making Indonesia the 5th nation out of the 10 in the ASEAN Region. Lower income countries are Philippines, Vietnam, Laos, Cambodia and Myanmar (by decreasing order of GDP) In terms of selected Millennium Development Goals (MDG), Indonesia s performance reflects its economic ranking within ASEAN countries. Table 1 documents the relative ranking of Indonesia compared to 4 less economically developed nations of ASEAN, and to Thailand, HQ of the ECHO Regional Support Office (RSO). Country Population in Million 4 (% of ASEAN total) Table 1: Profile of 6 selected ASEAN countries GDP (PPP) USD 5 Selected MDG 6 GNA Projected MDG MDG MDG Final 2012 increase Index 7 (ECHO (2018) (2010) (2010) (2011) 2013) Thailand 64.4 (10.5%) 10, % Indonesia (39.7%) 4, Philippines 95.8 (15.6%) 4, Vietnam 90.4 (14.7%) Cambodia 15.3 (2.5%) Myanmar 63.7 (10.3%) NA In brief, Indonesia is well positioned to become an emerging power in the region, provided political and economic stability is maintained. 1 World Bank ( accessed 15 Sept GDP (PPP) is gross domestic product converted to international dollars using purchasing power parity rates. An international dollar has the same purchasing power over GDP as the U.S. dollar has in the United States. 3 (source IMF 2012) accessed 15 Sept (source IMF 2012) accessed 15 Sept 2013 Rounded to one decimal 5 As above 6 Asian Development Bank (ADB 2013) accessed 15 Sept A3/EN D(2011) TECHNICAL NOTE Methodology for the Identification of Priority Countries for the European Commission Humanitarian Aid: GNA and FCA" 8 MDG 1.1 Proportion of population below $1.25 (PPP) per day 9 MDG 1.8 Prevalence of underweight children under-five years of age 10 MDG 4.1 Under-five mortality rate Evaluation report Particip GmbH 6

14 1.2 The Vulnerability to Hazards 5. According to a global risk analysis by the World Bank, Indonesia is among the top 35 countries that have high mortality risks from multiple hazards. Approximately 40 % of the population are at risk, i.e. more than 90 million lives. 11 Situated in the earthquake belt and Pacific ring-of fire, Indonesia is highly vulnerable to earthquakes and volcanic eruptions. Sumatra alone has suffered over 15 large earthquakes in the past 100 years. Indonesia also has 129 active volcanoes, 70 of which are classified as dangerous. Within the past century, floods have been the most frequent disaster for Indonesia. Floods often impact major population centres such as Jakarta (with a population of more than 13 million), Medan (more than 2 million), and Bandung (more than 4 million). The government estimated that the 2007 flood in Jakarta created total damage and losses of more than US$900 Mio The same study outlines the factors of vulnerability as follows: Currently, more than 110 million people live in or around 60 cities that are predominantly located in the coastal areas, exposing them to common hazards such as earthquakes, flooding and communicable diseases. The limited capacity of urban centres to provide adequate shelters and infrastructure, has led to the emergence of many unplanned settlements. Poor quality and enforcement of land use zoning in turn has led to many hazard prone locations being occupied by settlements The Ministry of Public Works estimated that a quarter of the urban population (or around 25 million people) lives in slums and informal settlements. The combination of the poor quality settlements and inadequate infrastructure has increased Indonesia s vulnerability, especially when larger scale disaster events occur. 7. Data from recent disasters in the selected ASEAN countries illustrate the comparative vulnerability of Indonesia. It should be noted that the period covered ( ) includes the Indian Ocean Tsunami; that smaller disasters are going unreported in less advanced or organized countries and finally that the larger the area, the higher the number of events. For the latter reason, losses are calculated as a proportion of population size or Gross Domestic Product (GDP). Country No. of events Table 2: Natural Disasters Total people killed (yearly average as a % of total population) 14 Total people affected in M. (yearly average as a % of total population) Economic losses in M. USD (yearly average as a % of GDP) 15 Indonesia ,693 (0.13%) 10.2 (0.4%) $15,041 (0.17%) Thailand ,283 (0.20%) 42.9 (6.4%) $42,547 (1.16%) Philippines ,589 (0.30%) 66.3 (6.5%) $4,858 (0.19%) Vietnam 68 44,093 (0.05%) 14.2 (1.6%) $5,500 (0.40%) Cambodia 14 41,903 (0.31%) 2.6 (1.9%) $592 (0.42%) Myanmar ,952 (0.39%) 6.7 (1.1%) $4,732 (0.89%) 8. Table 2 shows that although Indonesia has suffered high losses from disasters in the last 10 years, their magnitude is in part linked to its size. Finally, it is noteworthy that the DG 11 World Bank (2011): Advancing a National Disaster Risk Financing Strategy Options for Consideration 12 Laporan Perkiraan Kerusakan dan Kerugian Pasca Bencana Banjir Awal Februari 2007 di WIlayah Jabodetabek, National Development Planning Agency (BAPPENAS) Raw data obtained from EM-DAT: The OFDA/CRED International Disaster Database Université Catholique de Louvain, Brussels (Belgium) accessed 09/Oct/ Using total population estimates from: accessed 09/Oct/ Using Nominal GDP data from the International Monetary Fund 2012 Estimates. Data retrieved from: accessed 09/Oct/2013 Evaluation report Particip GmbH 7

15 ECHO Global Needs Assessment final index 2013 ranked Indonesia as the lowest (least vulnerable) of all six selected ASEAN countries (see Indonesia, with an estimated population of approximately million (2012), is the most populous country of the Association of Southeast Asian Nations (ASEAN), representing approximately 39.7%. Consisting of 17,508 islands, Indonesia is the largest archipelagic country in the world. 9. Indonesia is classified by the World Bank as a lower middle income country. Its economy expanded at a rate of 6% / year over the period of The International Monetary Fund (IMF) estimates the GDP PPP per Capita at 4,977 USD in 2012, projected to reach 7,487 USD in 2018, making Indonesia the 5th nation out of the 10 in the ASEAN Region. Lower income countries are Philippines, Vietnam, Laos, Cambodia and Myanmar (by decreasing order of GDP). 10. In terms of selected Millennium Development Goals (MDG), Indonesia s performance reflects its economic ranking within ASEAN countries. Table 1 documents the relative ranking of Indonesia compared to 4 less economically developed nations of ASEAN, and to Thailand, HQ of the ECHO Regional Support Office (RSO). 11. ). Out of the 10 top natural disasters in Indonesia (1900 to 2013), eight are geological and only two are meteorological (See table 3). Table 3: Top 10 Natural Disasters in Indonesia Top 10 Natural Disasters in Indonesia for the period 1900 to 2013 sorted by numbers of killed: Disaster Date No Killed Earthquake 26-Dec ,708 Earthquake 21-Jan ,000 Drought Jan ,000 Earthquake 27-May ,778 Volcano ,500 Volcano May ,000 Earthquake 12-Dec ,500 Storm Jun ,650 Volcano 3-Jan ,584 Volcano ,369 Source: EM-DAT CRED accessed 15 Sept Disaster Risk Reduction in Indonesia and the ASEAN 12. In 2007, a few years after the 2004 Indian Ocean Tsunami, the government of Indonesia passed a new Law on Disaster management. The law allowed for the establishment of the National Board for Disaster Management (Indonesian: Badan Nasional Penanggulangan Bencana or BNPB) in 2008 and provided the legal basis for the formation of provincial, district and city level Local Disaster Management Agencies (Indonesian: Badan Penanggulangan Bencana Provinsi or City/District). The law formed the legal basis for the Evaluation report Particip GmbH 8

16 National Disaster Management Plan. 16 The plan included some of the strategies most highly valued by the EU: 17 Community-based disaster management Establishment of the Indonesian National Rapid Response and Action Team (SRC- PB) Specific risk reduction programs for groups with special needs Enhancement of the role of NGOs and government partner organizations The plan further developed this into the following program 18 cycle and clarified the broad scope of the mandate of the BNPB: from preparedness, to response and reconstruction including DRR. 13. The BNPB was established in 2008 to replace the National Disaster Management Coordinating Board (Badan Koordinasi Nasional Penanggulangan Bencana or Bakornas PB) that was established in The BNPB is divided into four departments respectively for emergency response; prevention and preparedness; rehabilitation and reconstruction; and logistics and supplies. Headed by a General with the rank of Minister, the BNPB is often seen as mostly focused on response and preparedness. Initially staffed by transferring whoever was available from other ministries, the quality of the staff started to steadily increase once a budget was approved and allocated in The law projected the development of similar structures at province and district levels. The provincial agency (BPBD) requires the local adoption of laws and regulations. Although formally set up at province level, the BPBDs vary considerably in resources, expertise and scope of work. They remain, as at national level, strongly oriented towards response. 15. The GoI s commitment to disaster risk management is illustrated by its financial support: Although Indonesia and the Philippines were, respectively, the second and fourth largest recipients of international DRR funding over the two decades, the amounts received pale into insignificance compared with financing for such activities from domestic government sources. Averages of available data suggest that the Philippines government invests 20 times more than the international community in DRR and the government of Indonesia almost 10 times more 19.Indonesia requested technical assistance from the World Bank to improve its financial response capacity in the aftermath of natural disasters. Recommendations include: Increasing the annual budget allocation up to US$500 Mio for postdisaster rehabilitation and reconstruction; Securing a contingent credit line of US$500 Mio; and purchasing catastrophe risk coverage of US$800 Mio. 16. BNPB receives international assistance from several sources: Australia, New Zealand, Turkey, USA, Asian Development Bank, UNDP, WFP and the French Red Cross At ASEAN level, institutional capacity building has accelerated since the 2004 Tsunami. AADMER, a legally binding ASEAN Agreement on Disaster Management and Emergency Response was signed by the ASEAN Foreign Ministers in July 2005, after four months of negotiations, making it one of the fastest negotiated agreements in ASEAN's history. 16 The National disaster management plan (English translation) page 71, see 17 The National disaster management plan (English translation) page 81-83, see 18 The National disaster management plan page 87-91, see 19 GFDRR September 2013 Financing Disaster Risk Reduction - A 20 year story of international aid by Jan Kellett and Alice Caravani 20 Evaluation report Particip GmbH 9

17 Since the signing of AADMER in July 2005, ASEAN has put in place measures to implement many provisions under the Agreement. 21 In particular, it set up the ASEAN Coordinating Centre for Humanitarian Assistance (AHA Centre) and formally established coordination with the Civil Society. The AADMER Partnership Group (APG) constituted of all major relevant International NGOs. 18. In 2009, all 10 governments formally ratified, the ASEAN Agreement on Disaster Management and Emergency Response (AADMER) Work Programme The programme has a comprehensive risk reduction focus from Risk Assessment, Early Warning and Monitoring and prevention/mitigation to Preparedness/Response and Recovery. 19. Finally, the ASEAN Secretariat is widely credited for opening the door to international humanitarian assistance after the Nargis cyclone in Myanmar (2008). 1.4 DG ECHO s Programme in Indonesia Humanitarian Assistance 20. Since 1994 ECHO has allocated a total of Mio to provide humanitarian assistance to people affected by natural and man-made disasters in Indonesia. Of this total amount, 84.5 Mio (80%) was in response to natural disasters, and 22 Mio (20%) was to assist the victims of conflicts Since 2007, ECHO has invested 10.1 Mio in 30 projects from six complete Ad-Hoc decisions and the 2010 preparedness and response to epidemics decision 24. These projects targeted over 1 million people 25, although one coordination project claimed to reach 500,000 individuals as direct beneficiaries. 26 Annex F details statistical information about beneficiaries and assistance provided, since In 2007, ECHO contributed 1.6 Mio in response to floods in Jakarta and in North Sumatra. This mostly consisted of support to water and sanitation as well as NFI s. 23. In 2008, ECHO contributed 1 Mio, mainly food security initiatives, to vulnerable groups in a forgotten emergency in Eastern Indonesia. 24. In 2009, DG ECHO funded two humanitarian interventions in response to two major earthquakes, in West Java and West Sumatra, for a total amount of 4.5 Mio. This amount financed temporary shelters with a strong focus on earthquake resistant techniques, as well as non-food items, water and sanitation, logistics, transport, and coordination of relief. In 2010, ECHO supported the response to the twin disasters of 25 and 26 October (a 7.7 Richter Scale earthquake causing a tsunami which hit Mentawai islands, and the eruption of Mount Merapi, near Yogyakarta in Central Java, which directly affected some 100,000 people (See Photo 1). As ECHO/DRF/BUD/2010/ Using final beneficiary numbers from ficheops, see Annex F for more details 26 Photo 1: Mount Merapi eruption (from DG ECHO Annual Report 2010) Evaluation report Particip GmbH 10

18 early as 27 October, the Commission adopted a primary emergency decision of 1.5 Mio, which allowed five projects to be funded, providing assistance to some beneficiaries. These actions focused on support to shelter, water and sanitation as well as NFI s. Additionally, ECHO funded IoM and Mercy Corps in 2010 to undertake additional actions in Mentawai under the preparedness and response to epidemics 2010 decision Disaster Risk Reduction 25. Since 1994 ECHO has allocated 8.3 Mio for disaster preparedness; this represents about 7% of the total Humanitarian and disaster preparedness assistance in Indonesia during this period. 26. Between 2007 and 2013, four DIPECHO cycles have contributed, in Indonesia alone, over 5.7 Mio, reaching an estimated 621,282 people, the equivalent of a little over 9 per person. This involved fourteen contracts and 10 partners (see Annex) and represents15% of the total DIPECHO investments of 38 Mio in South East Asia. 27. Over 1.5 Mio was allocated in (8th DIPECHO Action Plan). In addition to Indonesia-specific programs, DIPECHO South East Asia contributed 1.7 Mio to two regional programs (OGB and IFRC) and one multi-country program. ECHO is involved with supporting the AHA centre 27 based in Jakarta, through a regional DIPECHO project led by OXFAM (ASEAN Partnership Group -APG) 28, and by providing input for assistance channelled via the EU Delegation / Regional EC-ASEAN Dialogue Instrument (READI facility), a mechanism for multi-sectoral cooperation. Given the high collective commitment of ASEAN (AADMER work plan is legally binding every country), the modest investment is relevant especially in the light of the ever-closer cooperation between the EU and ASEAN, two economic giants ECHO office in Jakarta 28. Since the very beginning in 1999, ECHO has maintained an office in Indonesia. With peak activities and staffing following the Tsunami and the 2006 and 2009 major earthquakes, for several years, there is no assigned annual budget (just operating funds covering for instance the rental of the premises from the EU Delegation). 2 PURPOSES AND SCOPE OF THE EVALUATION 29. As stated in the specifications for the Evaluation of the European Commission's Humanitarian Activities in Indonesia the purpose of this evaluation was to review and assess the impact of ECHO-funded assistance programmes in Indonesia. The evaluation includes assessment of the impact of humanitarian aid and DIPECHO projects. For the DIPECHO interventions the evaluation assesses the extent to which preparedness helped the population when they were afterwards affected by an adverse natural event. 27 AHA is the ASEAN equivalent of the Monitoring Information Center (MIC). Based at the European Commission in Brussels, the MIC is accessible 24/7 and can spring into action immediately when it receives a call for assistance. The MIC works in close cooperation with national crisis centres throughout the 32 countries participating in the Mechanism (EU 28, the former Yugoslav Republic of Macedonia, Iceland, Liechtenstein and Norway). Accessed 16 Sept APG was first funded in 2009 by Canada 29 Additionally, over 11 Mio were invested in 2006 but finished in early These projects were response to the 2004 Indian Ocean Tsunami, the 2006 Yogyakarta earthquake and a 2006 food security crisis in Eastern Indonesia. Evaluation report Particip GmbH 11

19 30. The previous South East Asia DIPECHO Humanitarian Implementation Plan expires at the end of The discussions on the next DIPECHO Humanitarian Implementation Plan have already started. The evaluation is expected to feed-in this process with a substantial contribution, by reporting on the evolution of existing disaster risk management strategies supported by DIPECHO in the region, as well as encouraging the development of coherent and pertinent programme strategies in the next action plan. 31. The evaluation covered DG ECHO funded projects between 2007 and The evaluation includes measuring the impact of humanitarian aid (emergency) and DIPECHO (Risk reduction). 3 METHODOLOGY AND LIMITATIONS 3.1 Evaluation Design 32. The evaluation ran between February and Sept 2013 and included the following activities: An inception phase with included initial briefings in Brussels, initial document review and the production of an Inception Note. A Desk phase, including a review of secondary data, an update of the intervention logic and an analysis on which judgement criteria (JC) and indicators were established for evaluation questions (EQ). Methodologies were developed further during this phase. A two-prong Field phase: a first visit which incorporated visits to Bangkok, Jakarta and Central Java, Yogyakarta, NTT 30 and West Sumatera and a second visit to Jakarta, Bangkok and Brussels. Extensive discussions were held with key informants. The first field phase ended with a workshop in Jakarta. Debriefings held in Brussels and a final reporting stage that involved iterative data collection and analysis. Table 4: Summary of evaluation phases Phases Activities Methodologies and tools Outputs Desk Initial team meeting Literature review. Inception note Inception meeting Summary of key documents Guide questions Drafting of inception and policies. field tools and field reports Completion of intervention logic. Desk Report Development of judgment criteria and indicators. Development of initial assumptions and preliminary analysis. Development of selection Field Initial briefing with ECHO Indonesia. Interviews and field visits. Additional secondary criteria for field visits. Semi-structured interviews with key stakeholders in Brussels, Bangkok and Indonesia Field visits including inter- Field report Case study report Draft report I 30 NTT or Nusa Tenggara Timur includes West Timor Evaluation report Particip GmbH 12

20 data collection. Final field workshop. Synthesis Debriefing meeting. Analysis and reporting. Presentation of findings views, focus group discussions and participation, observation in five locations. Final field workshop and debriefing. Triangulation and analysis of data. Formulation and testing of conclusions and recommendations. Integration of comments. Draft report 3.2 Conduct and Methodology 33. Methodological and evaluation tools are in Annex B. 34. Ten evaluation questions (EQs) were given in the Terms of Reference (ToR), and a set of Judgment Criteria and indicators was developed against these. The evaluation questions, judgement criteria and indicators were based on the desk review and a preliminary analysis of the situation. The evaluation matrix is set out in Annex B. 35. An extensive literature review was carried out to refine the evaluation questions and to understand the broad context of ECHO s commitment in Indonesia. In total, the team analysed over 100 documents including operational and project documents, ECHO reports, risk assessments, agency reports and others (see Annex G). 36. The evaluators visited available agencies/stakeholders involved in ECHO s Humanitarian Aid (HA) and DRR activities and met with ECHO, DEVCO and European External Affairs Services (EEAS) staff in Brussels, Bangkok and Jakarta. Key staffs were interviewed covering the main question areas of the evaluation. Annex B details methodologies used. Interviews with community members at the site visits were less formalised, but conformed to the EQs. 37. The team visited five project areas (Jakarta, Yogyakarta, West Java, NTT and West Sumatera). See Figure 1- Site visit locations: Padang, Jakarta, Garut, Ciamis, Yogyakarta and Kupang from East to West. Evaluation report Particip GmbH 13

21 Figure 1- Site visit locations: Padang, Jakarta, Garut, Ciamis, Yogyakarta and Kupang from East to West 38. These visits involved meetings with Government and non-government agencies as well as focus groups with beneficiaries and some key stakeholders. The whole team visited the flood-affected areas in Jakarta that allowed them to calibrate and refine their approach. Later, splitting the team enabled them to visit a maximum number of sites. Annex C includes a list of the organisations that the evaluation team met. 39. A workshop was held in Jakarta following the initial visit. It included donors and partners such as AusAID, NZAID, UNDP and WFP. The programme, presentation, and the minutes of this workshop are included in Annex B. 3.3 Limitations 40. Travel within Jakarta was particularly difficult leading to the division of interviews among the two international team members during the first visit. 4 KEY EVALUATION QUESTIONS: FINDINGS, SPECIFIC ANALYSIS AND CONCLUSIONS 41. The presentation of the findings is organized around the Evaluation Key Questions with some reordering. The main Judgment Criteria (JC) and indicators have been regrouped and condensed in a box at the beginning of every EQ. To facilitate reading, the findings are immediately followed by specific analysis and conclusions. Overlapping between the questions has been reduced to a minimum. Evaluation report Particip GmbH 14

22 4.1 To what extent have DG ECHO strategic orientation and intervention logic addressed the needs of the most vulnerable communities and categories of population, also in the areas most exposed to frequent natural hazards? (EQ1 - Relevance) Judgement criteria DG ECHO policies, programmes and plans are based upon and address needs and risk assessments. ECHO actions are congruent with national priorities based on risk assessment. Indicators Extent to which ECHO-funded projects are based on risk and needs assessments with special attention to vulnerable groups. Extent to which actions include an analysis of Government priorities and a dialogue with all actors Findings 42. In the period , the Indonesian context significantly changed, as have ECHO strategies and activities. DG ECHO priority both at global level as reflected in the Humanitarian Implementation Plans and at country level in the DIPECHO Action Plan has consistently targeted the most vulnerable and remote areas based on needs of the communities. In the first three years, ECHO DRR projects continued addressing community level resiliency, soon to be complemented by larger humanitarian assistance projects for the two earthquakes in In the last three years, community resiliency has been progressively overshadowed by regional support activities, one particularly directed to multi country support to ASEAN disaster management instruments. In 2010, strategic documents - DIPECHO action plan and the operational guidance stressed the importance of replication and dissemination at sub-national level. 43. At country level, the new law on disaster management, adopted in 2007, put into motion a paradigm shift from disaster response to risk reduction. This required building new institutions starting at national level and progressively at province and later district levels. It is classical top-down approach of institutional building contrasting with ECHO bottom-up strategy (from communities to district and then province). 44. In interviews GoI disaster management officials (particularly at local level) and other donors and partners stressed that one of ECHO s strengths is the systematic targeting of communities most vulnerable and in need. Community-based disaster management is one of the key strategies in the GoI National disaster management plan This statement was in reply to a specific question or, in several cases, spontaneous when asked what the strength of ECHO is. 45. ECHO s decisions and partners proposals (fichops) suggest that resources were allocated based on partner assessments and, when available, with reference to wider scale Government risk assessments. The approaches were consistently multi-hazard as stressed in ECHO policies and strategic documents. 46. Many stakeholders noted that ECHO encouraged its partners to focus on needs not covered by other actors: Most notably needs based and independent decisions included ECHO investments in the 2008 food crisis as a forgotten emergency in Eastern Indonesia, an ar- 31 The National disaster management plan (English translation) page 82, see Evaluation report Particip GmbH 15

23 ea and crisis which otherwise received little support. Similarly in 2009 ECHO invested in an emergency response decision which included not only an earthquake response in West Sumatra (which received significant international support), but also in West Java (which received very little international support) 32. The local Government officials in the West Java region recognised this independent and needs based decision. See Case Study 1. Case study 1: ASB temporary shelters in Ciamis, West Java; In 2009 earthquakes struck first West Sumatra, then a month later West Java. Government, media and non-government actors focused their response mostly to the West Sumatra earthquake. ECHO responded to the needs in West Java by partnering with ASB to provide cost efficient and fast earthquake resilient Bamboo houses, whilst people wait for longer term housing funds and construction. Families that received the bamboo shelter had multiple benefits. Firstly, they were able to have basic shelter, whilst awaiting permanent housing grants (up to three years). Secondly, after building their house, they used the Bamboo shelter as extra rooms, an extension, storage or premises for a small shop/business. Thirdly, house owners used the knowledge of resilient building techniques, taught by ASB, when building their own house. Key Lessons Addressing needs or emergencies overlooked by other donors is the strength of ECHO. Recovery actions which allow people to make their own longer term decisions allow them to better address their specific needs. Using or improving successful strategies or locally available construction techniques and material is cost-effective. 47. A review of the documents, interviews and observations concurred that among the populations in need, the most vulnerable were identified by surveys and assessment. For instance, partners specialising in handicapped persons, Arbeiter-Samariter-Bund (ASB) and Handicapped International, had in-depth analysis of handicapped persons vulnerabilities. Similarly WFP and FAO had in-depth food security knowledge. 48. ECHO clearly uses needs and vulnerability assessments for global and regional decisions. The 8 th DIPECHO action plan specifically refers to the GoI vulnerability index. Partner proposals reference national hazard and vulnerability assessments when available. All emergency decisions reference initial rapid assessments. ECHO emergency decisions were based on preliminary needs analysis from post disaster assessments. Partners and ECHO stated that dialogue and initial assessments often started within hours of a disaster occurring. 49. Field visits are also consistent with the above findings: ECHO has supported a number of initiatives that improve the identification and targeting of vulnerable communities and categories of people: food security 33 and other hazard risk mapping, targeted contingency planning, and programs to specifically address issues of the most vulnerable groups. 50. In West Java and Yogyakarta, (ASB) supported an inclusive education program. The evaluation team visited the program and met with school and community leaders as well as children with disabilities. Communities explained that children with disabilities, especially mental disabilities, are often referred to locally as a hidden population, as due to cultural 32 OCHA Financial Tracking System reports that out of emergency contributions to Indonesia totaling USD 79.7 in 2009, USD 77.7 M were earmarked for Sumatra earthquake. It suggests that ECHO (which contributed 1.5 Mio) was by the far the main donor if not the only one for Java. 33 FAO Food security mapping in NTT Evaluation report Particip GmbH 16

24 constraints they are not allowed to have a full place in society. Based on an in-depth analysis of this situation of these highly vulnerable people, the project worked with local leaders, communities and institutions to bring children back into communities, education and disaster preparedness actions. One core strategy of the GoI national disaster management plan, is specific risk reduction programs for groups with special needs Analysis and Specific Conclusions 51. The relevance of past ECHO activities under the existing context may not necessarily apply to the very fast changing environment of tomorrow. The continued relevance of DIPECHO activities will be discussed under section ECHO partner strategies are in line with key strategies of the GoI national action plan as required in the operational guidance for proposals under DIPECHO in South Asia. This is especially so for the promotion of community-based disaster management; specific risk reduction programs for groups with special needs; and the enhancement of the role of NGOs and government partner organizations. 53. The finding that ECHO projects refer to government assessment does not automatically mean that ECHO projects are based primarily on the Government s classification and priorities. First, Indonesia uses several methods to identify districts which are vulnerable or in need of special attention. While BNPB focuses on 336 districts/cities (out of 508) prone to disasters, the Ministry dedicated to the special regions includes factors such as economic status, infrastructure, accessibility and human resources to identify 183 special districts, of which 128 are prone to natural disasters whereas 143 are exposed to conflict 34. Districts and communities targeted by ECHO funded projects fall into one of those high risk categories. This is well documented in the proposals by partners. 54. The inclusion of the communities selected by partners in the list of most vulnerable in the BNPB Vulnerability index is a pre-requisite, which is easy to meet as it includes 66% of the districts. The selection by the partner is mostly, and often rightly so, based on operational or institutional considerations. All this being said, all the beneficiaries were vulnerable and in need of ECHO assistance. 55. This evaluation endorses the conclusion of the 2007 evaluation of DIPECHO in South East Asia, All partners conducted area-specific risk assessments as a basis for designing their projects, some more comprehensively than others. The humanitarian relevance of the targeting is one of ECHO s great assets. 34 There are 410 districts and 98 cities/municipalities. Out of the total of 508, one district and five municipalities do not have local autonomy or local Parliament. accessed 19 Sept 2013 Evaluation report Particip GmbH 17

25 4.2 To what extent have ECHO (humanitarian aid) and DIPECHO-funded activities contributed to reducing vulnerabilities, enhancing the capacity to cope, and boosting the resilience of targeted at risk populations, national institutions and the wider disaster risk reduction stakeholder community to recurrent natural hazards in Indonesia? (EQ2 Effectiveness). Judgement criteria Changes in resilience at local, provincial and national level desegregated for HA and DRR. Appreciation of causal relationship between ECHO activities and changes. ECHO DRR actions are replicated outside the action area. Indicators Extent to which stakeholders and beneficiaries report that ECHO funded projects have increased their respective resilience and / or coping capacity (Community, local and national level). Observations, if possible in recently affected areas Findings 56. The findings will be divided into those related to humanitarian assistance and those related to DIPECHO. Humanitarian assistance: 57. If there is a point of agreement in all interviews with stakeholders and partners it is the effectiveness both in speed and results of DG ECHO humanitarian assistance. This point was stressed by other donors who recognize the comparative advantage of DG ECHO. There were several examples observed by the team confirming this overall opinion. Several partners stated that constructive dialogue with ECHO started within days (some stated hours) of rapid onset disasters. 58. The International Organization for Migration (IOM) project in West Java (shelters and psychosocial assistance) was seen as effective in delivering 1000 temporary shelters and psychosocial support in spite of adverse weather conditions (monsoon). It filled gaps in the assistance provided by others (Government, Red Cross). Disaster preparedness was mainstreamed through various elements of the action, such as providing training on safe earthquake resistant construction techniques and by aiming at improving provisions for mental health. The model of prefabricated shelters was found appropriate for the widespread area. See Case Study The ASB project in West Java was seen by beneficiaries as well-timed and effective: ASB anticipated that the cash grant from the Government to rebuild destroyed homes would be delayed and provided assistance to build temporary bamboo houses 35. In various meetings, homeowners and their families stated that the temporary shelters provided them with small (over 30m 2 ) but good accommodation for up to three years. Most beneficiaries kept the temporary bamboo shelter after receiving a government grant for permanent housing. Some used the government-housing grant to pay off debts. Most used the grant for housing complemented with family remittances, and continue to use the temporary bamboo house for extra space, storage or as small local shops. 35 It was indeed delayed for several years. Evaluation report Particip GmbH 18

26 DIPECHO assistance: 60. Appraising the effectiveness of DRR is complex and somewhat subjective. To quote IFRC, Overall, what we know is more aspirational than evidencebased We know that much of what has been done in the past has not necessarily either empowered communities or enhanced long-term resilience 36. The most credible indicator is the improved response and resiliency of beneficiaries after another disaster. 61. In Patay village (NTT), FAO activities associated with the rehabilitation of the mangroves acted as an awareness raising process regarding natural hazards. The mangrove rehabilitation reduced the potential impact of tsunamis and storms on the coast (notably coastal erosion) and the mangroves provided an enhanced food source (fish and shrimp) for the villagers and an increased household income which in turn increased resilience. Photo 2: Evacuation routes and facilities 62. In Jakarta s flood prone areas, Action Contre la Faim (ACF) has been active in DRR (mostly preparedness, since flood prevention in a metropolitan area is beyond the reach of a NGO or DIPECHO) for a decade. The main author of the report first visited the ACF project and area in It was striking to note the change of awareness, ownership and resiliency resulting from this sustained effort funded by ECHO. The most often praised achievement is the Early warning system relying on direct (as well as through district) SMS information from communities upstream. ACF communication equipment is still in use and maintained by the community. Evacuation routes and facilities (see Photo 2 2) are considered by community leaders as less effective. The evacuation route identified in the ACF scenario was not adapted to the actual flood pattern in This project was effective in the short term, but the beneficiaries recognized that the long-term solution is a land use management plan to relocate households presently within 50 m of the riverbed and returning the river to its original width. 63. The effectiveness for special groups (women, disabled, etc.) will be discussed in section 4.7 (cross cutting issues). It is one of the high points of DIPECHO programs and strategies in Indonesia. 64. The interview with the Indonesian Red Crescent Society (PMI) offered a very positive perspective. The regional DIPECHO project implemented by the International Federation of Red Cross and Red Crescent Societies (IFRC) was seen as highly successful in introducing DRR in the PMI and strengthening all levels of this national institution. Success stories were identified and largely disseminated. Some were however indirectly linked with the PMI response (for instance, improvement in the management of blood banks). 65. ECHO s partners have developed a number of models, success stories, publications and leaflets that are well done, instructive and practical. Similarly, IoM supported the Universitas Gadjah Mada to develop Geographic Information System (GIS) modelling after the 2010 Merapi volcanic eruption. This contribution has been appreciated. The National 36 Quoted from IFRC tender for a two year research. Harvey, P. Harmer, A Background paper 1: Building Trust: Challenges for national authorities and international aid agencies in working together in times of disasters. International Dialogue on Strengthening Partnership in Disaster Response: Bridging national and international support. London: Humanitarian Outcomes. Evaluation report Particip GmbH 19

27 Government has requested this University to develop software for android systems for GIS modelling elsewhere. Those examples, among many, are tangible contributions to resiliency and an asset for replication by civil society and Government. 66. Some divergence of opinion between ECHO staff and partners affected the outcome. Communities in an OXFAM project developed a Village Action Plan meeting their needs and the objectives of DRR (See Photo 3: Padang - community evacuation road). Issued from the community itself, the plans also addressed what they perceived to be, and correctly so, the most important underlying factor of vulnerability: poverty. Modest income generating activities such as tree planting and fish processing capacity were not approved Photo 3: Padang - community evacuation road. by ECHO. They may not be traditional mitigation measures, although they are often funded by DIPECHO in other countries. 37 A similar observation was made by IFRC. 67. At regional (ASEAN) level, has ECHO contributed to greater resilience? The drive towards a response capacity and DRR strategy shared among 10 countries is a complex and long term initiative. The direct beneficiaries of DIPECHO s regional project are institutions and civil society. Our ASEAN interlocutors felt that this project has indeed placed the civil society into ASEAN DRR agenda and work plan and therefore promoted it to its member states that formally approve all documents. DIPECHO contributed to a formal dialogue between the AADMER Partnership Group of NGOs and ASEAN. This result still needs to be pursued and translated into a concrete outcome for vulnerable communities Analysis and specific conclusions 68. Strengthening a local government is a lengthy and at times frustrating undertaking. In East Lombok, OXFAM sustained effort to sensitize the Local Agency for Disaster Management (BPBD) to DRR. One of the expected results was the formation of the DRR forum, as mandated by the district regulation number 3-year 2012 on disaster management. 38 The process of establishing this forum began in There were many obstacles along the way, from the official rotation of staff to the lack of understanding in DRR. On October 17, 2012, the BPBD convened the founding conference on the DRR forum. Unfortunately, since there were few participants beyond related government officials, the forum could not be established. At last, the DRR forum in East Lombok was established on November Unlike the first conference, it was attended by a diverse range of stakeholders including related line agencies, universities, schools, the National Search and Rescue Team, the Indonesian Red Crescent (PMI), the military, police, civil society organizations, the media and the private sector In southern Africa, ECHO is funding crop tree planting, irrigation systems and vegetable growing as source of income to support the DRR local committee and community expenses. Experience has shown in this region that only those local DRR committees with a self-generating income were sustained over time (exit strategy). 38 This DRR forum is an advocacy structure, not an implementing structure. 39 Abstracted from TANGGHU Newsletter published by OXFAM (January-May Edition) Evaluation report Particip GmbH 20

28 69. OXFAM clearly summarized the importance of DIPECHO community work as follows: The district is the key to disaster resilience. In this regard both at community and district level, partners have been effective in increasing resiliency of the direct beneficiaries and the corresponding district agencies. There are however limits on what small short-term DRR projects can achieve. For instance, in the flood prone area in Jakarta, cleaning drainage, disposing of waste (activities not maintained by the population), do very little to minimize the floods (see Photo 44). In earthquake prone regions sound building principles can go a long way to reduce the loss of lives. 70. At province level and above all national level (PMI is an exception) the contribution to resiliency is perceived as limited. BNPB declared that ECHO projects, although nominally in Photo 4: limits on small DRR projects; disposing of waste. compliance with the national strategic plan, did not contribute to strengthen the capacity of the national or provincial official institutions. Our BNPB interlocutors stressed in successive visits that their immediate and most pressing concern is capacity building of their new institution at national and provincial level, and praised other donors for sharing BNPB priority and focussing their resources in this direction. 71. This does not affect the impact at district level (rather autonomous), but is a major obstacle for a broader national impact (see section 4.4 Replication). Bypassing the National disaster Management Agency may not bear consequences in many countries, where DIPECHO is most active, but is becoming a major impediment in a country with ambition to be a DRR leader in the ASEAN region. The top down approach of the national institution in charge (BNPB) and the bottom-up strategy of an international partner (ECHO) are not likely to meet without direct dialogue. 72. It is not that the GoI is not interested in local projects; the policy is towards decentralization and local ownership. The GoI is initiating with World Bank an Empowerment project including local budget for provincial and district level contingency planning and a flagship resilient village program. In May 2013 at the global platform, the Minister in charge of the BNPB restated the importance of community-based processes in Indonesia s disaster management strategy. 73. At regional (ASEAN) level the contribution is positive, but will trickle down only slowly to national and provincial levels. There is one important caveat, well expressed by a senior seasoned UN National staff member: the main beneficiaries of the OXFAM-led project with AADMER are the International NGOs, not the emerging local NGOs interested in DRR. However, the future in the views of the GoI (and several interlocutors) is a phasing out of the direct role of INGOs on behalf of local actors. Inclusion of Indonesian national partners in the APG is overdue. Evaluation report Particip GmbH 21

29 4.3 To what extent have DG ECHO-funded DRR activities contributed to reinforcing subnational response capacities, including preparation and awareness of local communities? (EQ3 - Effectiveness) Judgement criteria ECHO funded projects have increased the response capacity of stakeholders and partners at Community, local and provincial level. Demands are made for good practice documentation or support from other sub-national communities or institutions (preparedness and awareness campaigns). Indicators Extent to which partners and stakeholders report that ECHO-supported CBDRR/CBDRM or sub-national institution planning processes were updated or replicated. Observations, if possible in recently affected areas Findings 74. In Air Manis village the Early Warning System that was developed by Mercy Corps was supported by a local and continuing community disaster management group. This group continued to work with Mercy Corps and BPBD to monitor the EWS and evacuation system and to progressively develop the EWS into an automated 24 hour system that did not depend on a human presence in the mosques to receive and further transmit warning messages (see Photo 55). 75. In Jakarta, Air Manis and Patay, local awareness of hazards and appropriate preparedness measures (evacuation routes, team of volunteers, etc) were established and maintained. As indicated earlier, community leaders in Kampung Melayu spontaneously expressed that ACF projects did improve their response to the 2013 flood through networking (organized solidarity among neighbours), collaboration with BPBD and minimal equipment. This flood reached about 3 m engulfing the first story or most houses as shown in Photo 6. Photo 5: the Early Warning System developed by Mercy Corps 76. The benefits are not limited to household beneficiaries. Most of the projects include capacity building of local authorities (district level mostly). An example is the IOM DIPECHO project in West Java. Building up on its relief experience in the communities, IOM contracted Red Cross and a well-respected local NGO, IDEA. As a result, the district disaster management body had tested and updated its standard operating procedures and produced hazard maps. Six local communities developed and simulated local response and disaster risk reduction plans for volcanoes. In addition, local health centres also tested their disaster management plans through simulations. These plans were put into practice in 2011 during flash floods. During the floods, basic trauma units were formed and resources were mobilised as per the plan. Evaluation report Particip GmbH 22

30 77. In 2008, the Danish Red Cross project along with PMI, established community Emergency Response Teams (ERT) selected by the local leaders. Those genderbalanced ERTs are reportedly still operational. 78. Interviewees failed to see a significant impact on the quality of response at provincial level. A number of interviewees felt that on-going decentralisation and the lack of clarity of the role of the provincial government were limiting factors. However, it can be assumed that an improved local response does facilitate the response of higher levels. Photo 6: flood level 2013 Case study 2 : IoM in Garut district, West Java; In September 2009, ECHO funded IoM to provide temporary shelter and physiological support in the aftermath of the earthquake in Garut, Java. In line with LRRD approach, DG ECHO approved a follow up DIPECHO project focused on improving response and disaster risk reduction capacity of the newly formed district disaster management bodies. At the end of the action, the district disaster management body had updated standard operating procedures and produced hazard maps. Six local communities developed and simulated local response and disaster risk reduction plans including for local health centres. These plans were reality tested with success in 2011 during flash floods. Key lessons: Disasters offer a window of opportunity for DIPECHO initiatives. It is a good example of LRRD. Administrative change takes time, but attitude change or creating a culture of safety is essential, but also more time consuming. Manuals, guidance and processes developed with and supported by a government agency, such as the Ministry of Health, are better accepted and come with a system of support (retraining, continued learning, monitoring and guidance) Analysis and Specific Conclusions 79. DIPECHO resources are modest, compared to the scale of resources available to the Indonesian local and provincial governments. Its contribution to improving the response capacity must be seen in this context. IOM reflections in the Garut district, West Java are interesting and perhaps representative of the country situation: District disaster management agencies were starting to realise the scale of the challenges they face in improving disaster management. The action had effectively provided support for six village structures in risk awareness and developing standard operating procedures and action plans. However, there are 442 villages in the district. During the meeting it was estimated that it would take decades to replicate activities at the required scale, at current resource levels. Garut is but one of well over 500 districts in Indonesia. Staff capacity is an important limiting factor. In addition, the gains in staff capacity were likely to be eroded by a civil service policy of three yearly staff rotations. 80. IOM noted also that knowledge gained by the projects focusing on established health institutions was more likely to remain in the health sector. In the health sector, as with the Indonesian Red Cross, personnel, manuals and training protocols are better established and have a system of support. Similar observations were made in ASB supported activities in the education sector. Evaluation report Particip GmbH 23

31 4.4 To what extent do ECHO funded DRR programmes complement actions developed, endorsed and/or supported by national government, EC cooperation instruments or other external assistance, taking into account DG ECHO s mandate? (EQ4 Coordination- Complementarity and Coherence) Judgement criteria (JC) Priorities and action areas set out in ECHO decisions and partner programs are aligned with Hyogo Framework for Action (HFA), GoI priorities and other EU relevant instruments. DG ECHO shares information and consults with those institutions prior formulating ECHO Plan of Action and approving the projects. Priorities and opinions of those agencies have been taken into account in the selection of projects. Indicators Extent to which Government is informed and endorses ECHO funded activities. Extent of dialogue and consultation between ECHO and EU instruments involved in DRR related activities Findings The findings relevant to the 3Cs (coordination, complementarity and coherence) in the sense of the Maastricht Treaty, will be grouped according to the categories of main interlocutors in EQ The consortium approach will be discussed under section 5.3. National Government 81. New disaster management laws, strategies and plans established since 2007 have improved the disaster management and risk reduction enabling environment. These laws, strategies and action plans are based on international conventions (for example the Hyogo Framework for Action HFA ). The national disaster management plan is very closely aligned with the HFA 41. Documents show that ECHO, partners and donors have aligned their strategies with both international and national frameworks. This is also true for the ASEAN plan of action adopted by all countries as their own master plan (1 st JC). 82. As indicated earlier, BNPB expressed dissatisfaction regarding the coordination and consultation by DG ECHO. This BNPB concern is more related to the 2 nd JC: information and consultation by ECHO. It is noteworthy that the senior BNPB interlocutor while expressing concern on ECHO modus operandi praised IOM and OXFAM (both funded by DIPECHO) for their collaboration and on-going consultation. 83. The OXFAM project (Building Resilience in Eastern Indonesia), funded first by AusAid through the Australia-Indonesia DRR Facility, was formally approved by the BNPB in its capacity of chair of this facility. Apparently, BNPB was not informed of its extension and replication under ECHO/DIP/BUD/2012/93000 (8 th DIPECHO). 84. ECHO, according to all international interlocutors, participates actively in meetings and workshops, for example the monthly Disaster Management meetings organized by OCHA. The strong advocacy by ECHO for Community-based initiatives was praised. However, Evaluation report Particip GmbH 24

32 ASEAN: during the second visit, an important meeting between the GoI and the Humanitarian Country Team was held by OCHA/BNPB with the objective of jointly reviewing the Indonesian Contingency Plan. Although ECHO is a member of the Humanitarian Country Team and a major responder, it was not invited. UNOCHA indicated that the invitations were issued by BNPB. Other donors with a relationship with BNPB (USA, Australia and New Zealand), as well as main partners participated. 85. The focal point for contact on AADMER and READI is with DEVCO due to the crossthematic content. A positive note is ECHO RSO (Bangkok) informal consultation with the ASEAN focal point regarding the proposed regional DIPECHO projects. Other donors 86. There are only four major bilateral donors for DRR in Indonesia: Australia (AusAid), USA Agency for International Development (USAID), New Zealand, and Japan (JICA). They all channel their assistance through the Government or joint mechanisms. The most active one which also has the largest budget is AusAid 42. JICA is more focused on infrastructure than capacity building. The three others, providing DRR support, expressed strong appreciation of ECHO s focus on communities, praised its active participation in the donor forum, but noted its relative absence from the BNPB dialogue, instruments and structures (international convergence meetings and national platform). EU Instruments 87. At the 19 th ASEAN-EU Ministerial meeting held in Brunei on th April 2012, it was agreed that: ASEAN and the EU will forge closer cooperation and coordination on disaster risk reduction and management at the global and regional levels; enhance dialogue and cooperation in crisis management, notably, in the form of knowledge transfer and exchange of best practices; establish a regional network of information sharing and early warning systems for emergency situations In particular, ASEAN and the EU will work together in strengthening the ASEAN Coordinating Centre for humanitarian assistance and disaster management (AHA Centre) through close cooperation in implementing the AADMER 5-Year Work Programme ( ) in the following areas: (i) Risk Assessment, Early Warning and Monitoring; (ii) Preparedness and Response; (iii) Prevention and Mitigation; and (iv) Recovery. 89. Towards the end of 2012, in line with this political commitment, DEVCO launched a 27- month project to strengthen the ASEAN emergency response architecture and early warning systems as well as the AHA centre and a number of individual national crisis response centres in ASEAN. The project will transfer DG ECHO expertise in Monitoring and Information Centres (MIC) and Emergency Response Centres (ERC) to Indonesia. The team leader, a civil protection expert with considerable experience in Indonesia, indicated that close consultation has taken place directly with Civil Protection in Brussels. The DEVCO project team leader, however, had no knowledge of the regional APG/AADMER project funded by ECHO. There is a clear complementarity between these two projects but the lack of onsite communication between DIPECHO and this project may affect the future coherence of DG ECHO and EU message. 42 AusAid for instance is covering the salary of the staff of the AHA. As a recent development, AusAid is being integrated into the Ministry of Foreign Affairs. It is unclear what the change of Australian Government will have on AusAID support to Indonesia and in particular for DRR 43 accessed 28 Sept 2013 Evaluation report Particip GmbH 25

33 90. The staff dialogue between ECHO/Jakarta and DEVCO in the EU Delegation is seen by both parties as cordial at collegial level. Nevertheless, the DEVCO interlocutor in Jakarta indicated that he was not consulted regarding the selection of projects to be funded by DIPECHO among the 30 reportedly submitted. Conflicting information was received from RSO, as the OXFAM DRR project in Eastern Indonesia (Papua, a location which is not the top priority of ECHO), was approved primarily based on the stated interest of DEVCO in this region, making a scaling up of the project with DEVCO funding more likely This is in line with DG ECHO aspiration that EC Cooperation instruments...provide support to beneficiaries for a DIPECHO exit strategy (EQ 10) 46. It is too early to determine whether this flexibility from ECHO will effectively contribute to sustainability and coherence. An interview with DEVCO in HQ did not permit to a thematic area dedicated to DRR to be identified. The closest is climate change but oriented, at the request of the GoI, towards forestry, green energy and similar priorities not including community-based DRR Analysis and specific conclusions 91. Although the DIPECHO projects complement and in fact fill critical gaps in the programs and plans at national level content-wise, there is a major problem in the form that is counterproductive. The issue is neither the quality nor the need for the product, but its marketing at GoI national level. At sub-national level partners are doing a commendable job of sensitizing the authorities, stimulating their support and ownership and ensuring that they benefit. At ASEAN level ECHO s sharing of information is appreciated. The gap is at BNPB level, THE institution with mandate and authority to coordinate and monitor all partners. 92. BNPB is a young institution with a promising future, but still weak and therefore eager to affirm its authority. As in many emerging countries in Asia, there is an acute sense of pride, sovereignty and own potential (in the near future for Indonesia) of the country. Indonesia is no longer giving a blank check to humanitarian responders and development NGOs. The same national criticism was heard in the evaluation of DIPECHO in Ethiopia, a country however with conflicts, severe discrimination and a lack of resources. In Indonesia, there is openness to the foreign expertise that DG ECHO and its partners offer. 93. The evaluators do understand the importance for ECHO of preserving its decisions from undue political or other influence in order to save lives in case of disasters and especially conflicts. ECHO benefits from considerably easier regulations to speed humanitarian assistance for those most in need, rather those most in political favour. This being said, in a country like Indonesia, whose condition is closer to EU Member states than to that of many poor countries eager to receive ECHO assistance, some flexibility must be exercised in the management of DRR (not HA). Considerable opportunities exist to market community-based programs to the Indonesian government, a core value of DIPECHO programs; this requires interaction with the national institutions. 94. ECHO is not an independent institution but a DG of the European Commission and therefore cannot liaise formally with the GoI. A mechanism (OXFAM-APG) has been found to cooperate, share information and informally consult with ASEAN, a political institution. The same is possible for BNPB through a partner with an overall coordination mandate (UNOCHA or ISDR as in Ethiopia or through a wider involvement of the EU Civil Protection), see section Agreement number: ECHO/DIP/BUD/2012/ OXFAM is a major recipient of ECHO funding although some reservation on their performance as partner has been occasionally questioned. 46 Specifications for the Evaluation (3.3 paragraph 5) Evaluation report Particip GmbH 26

34 95. In Ethiopia, the Delegation played an active role in formal liaison with the centralized Government on humanitarian issues far more substantive (and sensitive) than those identified in Indonesia. Could the same happen here? If the DG ECHO office in Jakarta is not allowed, or not in a position to technically consult with BNPB counterparts, should a focal DRR point be designated in the Delegation (DEVCO) and speak on behalf of both DEVCO and DIPECHO/Civil Protection? This issue will be elaborated further on in the conclusions regarding the exit strategy and ECHO's office future (section 4 10). 96. Formal coordination and mutual consultation between the EU s two main instruments is lacking at the time of formulation of DRR projects, which presupposes a give and take in the process of deciding on projects. It may be more time consuming and require some compromise, but will ensure that the EU speaks in a coherent voice and that ECHO s valuable efforts are sustainable. This is the price of advocacy within the same institution. 97. A future opportunity may lie in the new DEVCO project to strengthen AHA and the ASEAN countries. There is some advantage in balancing the chain of command and situation room approach of the Civil Protection, with the local community (quite autonomous in Indonesia) and civil society perspective of DIPECHO and its regional APG project. This cannot be done in DG ECHO in Brussels, but at country level. 4.5 To what extent have DG ECHO-funded DRR activities had an impact at national level, through replication, scaling-up, advocacy, or complementarity with national initiatives (EQ5 Effectiveness - Sustainability) Judgement criteria ECHO actions are cited by the National Government and key stakeholders as examples or models for replication. Indicators Extent to which Government and key stakeholders report that they sought DIPECHO models or project for potential replication. Extent to which DIPECHO DRR message is reaching national emergency response institutions, BAKORNAS PB, DRR and scientific institutions and actors not directly funded by ECHO Findings 98. The complementarity with national initiatives has been addressed under EQ4 (above). DIPECHO initiatives are ahead of the national program that is not yet significantly reaching districts and communities. Success stories and limitations in scaling up or replication will be summarized below. 99. ECHO, through partners, has developed models for community-based disaster management; contingency planning; school based disaster management; including the handicapped in disaster management, and models for mainstreaming DRR at sub-national level. There are many examples of limited replication: PMI through the IFRC DRR project stressed that they trained their own volunteers as trainers while NGOs are recruiting them. That is a factor which facilitates sustainability and replication in other districts. ECHO partners' and sub-national Government officials indicated that the Australia- Indonesia Facility for Disaster Reduction replicated district level contingency planning Evaluation report Particip GmbH 27

35 processes piloted by IoM and ASB supported initiatives in seven additional districts in Java. Three of five district disaster management agencies visited, listed plans to replicate Community-Based DRR in additional communities. Although national plans mention that it is a requirement, districts claimed that they have limited capacity to replicate on a wider scale. School based disaster management initiatives, implemented by ASB in central Java were replicated at a local level with support from the Mayor. Similarly, ASB material from the project for persons with disabilities is widely used by the Ministry of Education. Representatives of the Yogyakarta DRR forum have been asked by other provincial and district forums to share lessons learned in workshops and publish them in other documents There is no indication of a massive scale up by the GoI. However, donors do refer to and include some lessons and initiatives in their cooperation with BNPB Analysis and specific conclusions 101. The initiatives of DIPECHO are replicated on a limited scale by other local authorities, donors and stakeholders. The lack of national endorsement and massive scaling up is due to several factors: The absence or weakness of the specialized institutions at province and furthermore at district level: when the structures are formally established, the staff is poorly trained and unfamiliar with the concept of DRR. This will require years of sustained investment from the GoI. A perceived focus of provincial (and some will say national) authorities on response at the expense of longer term prevention measures: this is a predisposition in all Civil Protection-like institutions in many countries. It is why DIPECHO contribution is particularly valuable. Last but not least, the inability of ECHO to advocate its approach and models directly with BNPB, the indispensable interlocutor for any decision at national level. The fact that all donors share a comprehensive DRR approach is encouraging. 4.6 To what extent LRRD aspects have been taken into account in the design and implementation of ECHO and DIPECHO strategies and activities? (EQ6 Effectiveness) Judgement criteria HA response projects are linked to long-term development. Indicators To what extend HA emergency response projects consider and demonstrate a longer-term impact Findings 102. LRRD is especially relevant in emergency humanitarian assistance. In the urgency to save lives and provide relief, humanitarian actors tend to overlook the more long term, possibly negative, impact of their activities The evaluation identified several instances where LRRD was built in ECHO HA projects. Evaluation report Particip GmbH 28

36 The IOM shelter project disaster preparedness was mainstreamed through various elements of the action, such as providing training on safe earthquake resistant construction techniques and by aiming at an improved preparedness for mental health provision. The fact that IOM identified the need for a follow-up DRR project illustrates this LRRD preoccupation. The ASB bamboo shelter (see Photo 77) project in West Java showed definite LRRD success: It was designed as an interim measure waiting for the materialization of the promised reconstruction by the Government. Photo 7: ASB bamboo shelter When questioned, house owners could still cite key earthquake resilient design features in the bamboo shelter three years later and reported using this knowledge when building their own house. The project in Yogyakarta (funded in 2006 and implemented in 2007) linked GIS mapping of emergency camps, lahar hazards and city planning By design, DRR projects (DIPECHO) should link past relief with future preparedness and development. A distinct aspect of LRRD is whether ECHO partners have developed the local capacity of local NGOs with a view not to perpetuating their own role. This was not observed with two interesting exceptions: IFRC does not implement projects directly, but monitors and strengthens its national society - PMI. This approach also contributes to sustainability (EQ 9). OXFAM does not sponsor a local counterpart, but plans ahead to transform its international operations in Indonesia (as well as in South Africa and Brazil) into a genuine local NGO Analysis and Specific Conclusions 105. LRRD is a trademark feature of ECHO's Humanitarian Aid Activities. It is particularly positively seen in Indonesia. One of the key factors is that ECHO implementing partners for Humanitarian Aid and DIPECHO are development organizations with a long presence in the country with their own long-term development vision The standing ineligibility of local NGOs to receive funding and implement ECHO projects on their own remains an obstacle to developing the long-term capacity of the national civil society. A country like Indonesia would gain more if ECHO partners would identify and include local counterparts in their priorities and by sponsoring them and strengthening their capacity help they become future implementers, rather being effective subcontractors. 47 Lahar is a mudflow or debris flow composed of slurry of pyroclastic (volcanic) material, rocky debris, and water. Evaluation report Particip GmbH 29

37 4.7 To what extent the following cross-cutting issues have been taken into account in the design and implementation of ECHO and DIPECHO strategies and activities? (EQ7 Cross cutting Issues) 107. The cross cutting issues include: Children, Effects on environment, Women, Handicapped people, Community networks in project implementation and Participation. Judgement criteria Cross-cutting issues were specifically identified and considered in the design and implementation of ECHO/ DIPECHO strategies and activities. All stakeholders including vulnerable groups, Government at all levels, private sector, civil society and community groups have been consulted. Indicators Extend to what ECHO funded projects include specific vulnerability criteria. Project planning and implementation are routinely a widely consultative and participatory process at local, sub-national and national levels. Number of actions which focus on specific cross-cutting issues. Extent to which ECHO and EC instruments maintain an ongoing dialogue on a common vision of long term outcome Findings 108. The analysis of the fichops, confirmed by the field visits, showed that the HA and DIPECHO projects adequately cover the most vulnerable groups: women and children: In particular OXFAM stressed the priority given to the promotion of the participation of women in the DRR forum. A similar observation was made in the visit to the ACF project in Jakarta. Although this project was completed, the local leaders expressed the importance of women as volunteers and explained their absence from the meeting by the timing coinciding with the end of the school day. PMI mentioned the gender balance in the community-based Emergency Response Teams Field visits and partners' documents show that organisations differed in their depth of analysis of the cross cutting issues. Organisations with long term investments and specific projects in cross cutting issues have a more in-depth analysis, such as for example ASB and Handicap International on handicapped persons in Indonesia Specifically targeted, rather than mainstreamed, cross-cutting issues also had a greater effectiveness and impact. The influence of ASB, Handicap International and Child Fund Indonesia projects has been felt beyond the direct beneficiaries as illustrated by ASEAN translating some of its most important guidance material into Braille, and by other advocacy initiatives. The projects collaborate with the Ministry of Social Affairs and the Ministry of Education at sub-national levels whilst also advocating for change at national level. The projects involved regulatory change at various levels, impacting long term projects. The GoI includes specific risk reduction programs for groups with special needs as a key strategy Partners have established and do contribute to regional networks and forums. ASB and HI launched the Disability Inclusive Disaster Risk Reduction network at the 5th Asian Minis- Evaluation report Particip GmbH 30

38 terial Conference on DRR in See case Study 3.This network is committed to ensuring that people with disabilities are better accounted for in DRR, and broader development, post Similarly partners contribute to regional and global initiatives such as the emergency capacity building and Partners for Resilience regional initiatives/projects. Case study 3: ASB focus on children with disabilities In Indonesia, children with disabilities are often excluded from society, schools and disaster preparedness activities. ASB implemented two DIPECHO actions in 2008 and 2010 to focus on children with disabilities in Yogyakarta. The initial actions sought to widen longer-term access to disaster risk reduction information and procedures for disabled children in inclusive schools. The second action focused on disabled children outside the school structure. ASB designed and implemented the actions in close collaboration with the district department of education and later the department of social affairs (for children outside the school structure). The actions resulted in mainstreaming tangible of aspects of disaster preparedness for children with disabilities in both ministries. Attitude changes within communities were noted towards children with disabilities. This approach is being replicated by the ministries in other communities and districts and promoted at national level. Key lessons Cross cutting projects specifically targeted to vulnerable sub groups were ground breaking. Collaboration with institutions outside the field of disaster management, such as line ministries and universities, increases effectiveness. Getting local regulatory support from line Ministries and synchronising actions with their budget cycles is important for sustainability Networking among partners leaves room for improvement. On the one hand, some NGOs felt that ECHO should organize more regular meetings with all its partners (the last one reportedly was for the launching of the 8 th DIPECHO plan in 2012, on the other hand, encouraging the networking of consortia for presenting proposals has not met a great welcome from partners. They felt that the competition for diminishing resources runs counter to collaborative action This does not mean that a consortium is not a useful structure for effective collaboration. Member agencies need to be encouraged to have a common vision, to contribute to common outcomes, to minimise competitive approaches and to work together as a team rather than as individual agencies. A positive example is the Partners for resilience and disability forum Participation, both in HA and DIPECHO projects, seems to be a strong built-in component. This is particularly true of community-based DRR projects where participation is a major result and outcome in itself Analysis and Specific Conclusions 115. Cross cutting issues are very well integrated. Having specific projects on persons with disabilities has been a positive initiative that would deserve a lessons learned publication outside the country (possibly with ASEAN?) 116. Partners have explored further regional crosscutting initiatives in ASEAN and through regional advocacy and learning bodies. Positive linkages exist to ECHO programs but outside existing programmes. Evaluation report Particip GmbH 31

39 4.8 Which elements in the institutional context and in the relations between the different actors have had a higher impact in the effectiveness and scaling up of the DRR interventions? (EQ8 Efficiency) Judgement criteria Factors identified by stakeholders or evaluators contributing to the success (lack of) of projects. Indicators Analysis of success and failures stories. Strengths and weaknesses of GoI partners. Level of commitment and available resources of stakeholders Findings 117. In the sections above, several factors for success were identified: they can be summarized below: DG ECHO s insistence on replication in contract negotiations Strong political commitment (Indonesia and ASEAN); Partners with long experience in development in Indonesia and their own distinct longterm vision; Good communication with (and endorsement by) local authorities: district level and when required province level leading to a sense of local ownership; In the case of the Red Cross, strong partnership with the local counterpart (PMI) guaranteeing sustainability; The most innovative projects (Disabled, Women Focus in DRR by OXFAM, Urban Poor by ACF) are usually those most easily replicated. Close quality control and monitoring by ECHO Factors in relations that impede scaling up are also noted: Competition for survival and growth among International NGOs at the cost of developing national counterparts that will one day replace them; Lack of ownership by BNPB resulting from ECHO failing to acknowledge and engage meaningfully with the coordinating and monitoring mandate of this new agency. The main issue is that ECHO has not been able to re-evaluate and adjust to Indonesian context the relationship it has with national governments in less developed countries. Lack of give and take in the early stage of project design and selection between ECHO and other donors (EC, bilateral or World Bank) with the potential to massively expand the pilot coverage of DIPECHO projects. Poor and rotating human resources at district level, a problem addressed locally and at high cost by the projects but impeding wider and faster spreading of the successes to other districts. Lack of financial resources locally is not an issue Analysis and Specific Conclusions 119. DG ECHO developed innovative community-based initiatives and is genuinely interested in and strongly advocates their take over and expansion at national level. Indonesia is a country with the commitment and financial resources to do this over time. However, the lack of ownership as well as direct familiarity with DIPECHO s local achievements as the only body with mandate and authority, and a feeling of being bypassed by a foreign organ- Evaluation report Particip GmbH 32

40 ization, albeit well intentioned and competent, is generating frictions and preventing the desired national impact The solution is not establishing a formal relationship between two institutions. This is done by the EU Delegation as DG ECHO is not an institution on its own. It is by encouraging technical frequent contacts with BNPB, i.e. meaningful consultations short of abdicating ECHO decision power on the use of its funding on DRR matters (a developmental issue) while carefully protecting the independence and autonomy of DG ECHO in Humanitarian Assistance matters. This is difficult but indispensable in the special situation of Indonesia The understandable preoccupation of international partners to maintain and expand their presence (staff, projects) is preventing them from preparing their own exit strategy by mentoring local NGOs and civil society. That is only delaying the time when most of the DRR and other civil society activities will be carried out by local partners with some specialized external technical assistance as they deem fit. 4.9 To what extent have the project activities and results had lasting effects after hand over? (EQ9 Sustainability) Judgement criteria ECHO-funded projects and programmes have caused anticipated (or unanticipated) long term effects.support has been provided by National authorities, EU Co-operation instruments and the other external development assistance to beneficiaries for a DIPECHO exit strategy and/or continuation of DIPECHO programmes and activities Indicators Extent to which action and decisions have an effective sustainability strategy. Extent to which stakeholders or beneficiaries identify a long term impact after termination of ECHO support and to which projects link with other donor initiatives In particular, whether national authorities, EU cooperation instruments and other external development assistance have provided support to beneficiaries for a DIPECHO exit This EQ has been addressed in the earlier sections. Interlocutors and evaluators noted that the impact on community awareness is outlasting the duration of the projects. Some achievements such as safer housing or mangrove rehabilitation are likely to remain sustainable. In brief, there are encouraging signs that in most projects, a sustainable impact will remain especially if the structures mandated by law at district level take hold. The establishment of DRR forum in some districts and their chairmanship by local universities is another potentially lasting impact. Government officials sensitized and engaged by the partners are likely to remain committed even if some are rotated elsewhere. Of course, the problem is the small relative scale of the impact in a large country. Expansion of ECHO efforts by other EU instruments or donors would have been an additional positive factor. This contribution was not evidenced in the evaluation. Evaluation report Particip GmbH 33

41 4.10 What were the added value and the relevance of DG ECHO presence in Indonesia for the last five years? (EQ 10 Impact) Judgement criteria ECHO strategy is based on the analysis of its added value in a changing environment. Indicators Benefits of ECHO presence in Indonesia as perceived by main stakeholders and partners. Plans of action reflected changes of GoI capacity and policies of other donors. Extent to which ECHO strategies and programs are based on a comparative (regional) analysis of its added value in Indonesia Findings 124. Stakeholders appreciated the remarkable diversity of hands-on activities in communitybased projects funded by ECHO. Activities have ranged from HA to long-term food security and community development; from early warning systems to evacuation, supporting gender and youth based programs and hazard mitigation work in local communities. This was demonstrated in Jakarta for example, where flood based activities covered support to schools in disaster management planning, support in training for the provisions of psychosocial support to affected people, and the establishment of community disaster planning groups, women s groups and youth groups. Case study 4: Mercy Corps in West Sumatra In response to the September 2009 Earthquake, ECHO, along with other donors, supported Mercy corps shelter and water and sanitation humanitarian response. Building on this and its other work, Mercy corps started its first DIPECHO contract in Indonesia. Shortly after the disaster, the provincial government established a West Sumatra provincial level disaster management agency, policy and procedures and s Padang city and Agam district established their own local disaster management agencies in compliance with the 2007 law. DIPECHO Project helped establish and train twenty disaster preparedness teams at district, community and local schools with awareness activities at provincial level. The provincial government supported the formation of village disaster preparedness teams in all 54 villages of Padang city, expanded the early warning system and later with funding from UNDP, PMI and others further improved and replicated models established by this DIPECHO Project. Key Lessons Indonesia political commitment as illustrated by the adoption of the 2007 Law was critical to the expansion throughout the province. Additional time and effort from EU and ECHO are needed to create a supportive environment for expansion of community, district and provincial initiatives at national level. Humanitarian response: 125. ECHO partners stated that ECHO makes fast and flexible emergency decisions with consistent and regular two-way communication within its network of partners, which is relevant to meeting critical post-disaster needs. This is reflected in the speed at which ECHO announces its decisions concerning partners assessments. In spite of their relative speed Evaluation report Particip GmbH 34

42 in the immediate aftermath of disasters, those decisions are based on well documented needs assessments by partners or the GoI ECHO emergency decisions and partner actions promoted key instruments, such as community-based disaster management, linking relief to rehabilitation and development initiatives and participation of stakeholders. Not all humanitarian donors shared the same concern for the midterm impact of their emergency assistance There is a general consensus that ECHO s humanitarian presence in the aftermath of a disaster was one of the most relevant and best-adapted to the changing needs of affected populations. DIPECHO Strategy and Presence 128. Between, ECHO issued 30 HA contracts (including 2 for preparedness and response to epidemics), and 14 DIPECHO contracts in Indonesia. In addition, DIPECHO funded regional initiatives from Oxfam and IFRC and a Handicap international action shared with the Philippines 48. The analysis of the objectives of the non-emergency projects shows a consistent emphasis on community benefits, development of models and a concern for replication and sustainability In interviews, beneficiaries of both humanitarian assistance and Disaster Risk Reduction (DRR) consistently stressed the change DIPECHO made in their lives Other donors (Australian Government-AusAid, US Assistance for International Development USAID, New Zealand and Japan) are providing a far more important financial commitment but directed to the Government of Indonesia (GoI) rather than the end of line beneficiaries or civil society. Those donors believe that this ECHO strategy has offered a unique and original contribution during this period As UNOCHA noted, the field of disaster management has evolved considerably and rapidly over this period of time. ECHO is credited for its sharp analysis of the changing context. This analysis was critical for ECHO in view of its interest to collaborate with ASEAN structures and to promote its standards and models developed over those years and tested on site All other donors and the GoI, noted that DIPECHO is the only major donor which consistently focuses on community-based disaster management, specific risk reduction programmes for groups with special needs, and enhancement of the role of NGOs and Government partner organisations. The 7 th and 8 th DIPECHO decisions state this added value and focus, calling for support to systems of replication based on this analysis. DIPECHO programs focused on these areas, which were identified by the 2007 external evaluation of DIPECHO s South East Asia programs 49. In OCHA s view, ECHO s added value and strengths include advocating in all meetings, the adoption of their community-based models and standards Many ECHO partners have decades of experience in Indonesia and are therefore very familiar with the social, economic and political environment. ECHO support has enabled those mostly developmental partners to update and improve their skills in DRR, a concept not always well understood and practiced ECHO supports community focused and community-based actions. People interviewed felt that this focus on community-level provided a balance to other donors, Government and agencies top-down approaches. 48 Seventeen projects were also started in 2006 but completed in early AguaConsult (2007) Evaluation of DIPECHO Action Plans, South East Asia Evaluation report Particip GmbH 35

43 135. There were some discordant notes: GoI documents emphasised that locally based capacity needs to be developed ( bottom-up approaches) as do national level capacity ( top-down ) approaches. However, interviews with BNPB during the second field visit suggested that strengthening the institution (national and provincial level structures) should be the main immediate priority for external assistance and that ECHO community level DRR action is a far more distant concern. This was often the case in the interviews with National Disaster Management Organizations in DIPECHO target countries Interviewees believed that DIPECHO s presence was necessary and relevant to Indonesia s disaster risk management policy and programmes 50. Most stakeholders stated that ECHO had a niche working at sub-national level, and that this was particularly useful not just in addressing community needs, but also in supporting the under-trained and underresourced staff of the provincial and district disaster management bodies (BPBD) Apart from the IFRC no partner has identified a local NGO or Civil Society Organization that would act not as subcontractor but as a mentored successor. It is understood that DG ECHO can only partner directly with European registered NGOs or UN agencies which have signed a Memorandum of Understanding 51. Building the capacity of national NGOs to carry on the activities would have been played a major role in ensuring a long lasting impact Analysis and Specific Conclusions 138. ECHO s engagement with International NGOs and its variety of interventions at community or local level was alone in aiming to strengthen civil society and in being directed immediately towards the end beneficiaries. Other actors channel resources to the institutional building of BNPB ECHO has met a need that no other agency was addressing directly and properly. Community based DRR is mentioned among the GoI stated priorities (as every other aspect of DRR is). In practice, BNPB challenge was and still is to build its own capacity and authority (internally and externally) with a top-down approach contrasting sharply with ECHO bottom up policy (from the people to sub national level). Both are justified and indispensable. In many non-asian countries where DIPECHO is implemented, the institutional top down capacity building remains far less concrete. In Indonesia, BNPB has limited impact at district level and ECHO is not effectively reaching the provincial level ECHO s absence from Indonesia in this critical period would have left a vacuum. The strong advocacy might not have the national success hoped for, but one can assume that, in ECHO s absence, all the key policy documents and work plans issued by the GoI or ASEAN would not have had the same comprehensive content and humanitarian (peopleoriented) note. 50 In Indonesia, the evaluators found a particularly highly motivated and competent cadre of nationals working in the GoI or international institutions. They were not afraid to speak out when some international activities or actions were not perceived in the long term interest of the country and its institutions. This was made more credible by their opinion and analysis of ECHO strengths and weaknesses. 51 In September 2013, there are 226 potential partners. Evaluation report Particip GmbH 36

44 5 OVERALL CONCLUSIONS 5.1 DRR and Development 141. The initial focus of DIPECHO, the disaster preparedness programme of ECHO, was to raise awareness and prepare the population for the impact of disasters. It was closely linked to improving the local response. Overtime, the international community has adopted the concept of risk management and risk reduction. DG ECHO within the EU, rapidly realized the importance of a wider and broader approach and became one of recognized advocates for risk reduction at community level DRR is more closely linked to long term development than awareness and preparedness. ECHO management tools and in particular the short duration of the projects have not always been the most suitable instrument. This evaluation, like many others, underlined the inherent difficulty of pursuing a long term goal (resilience) with short term instruments (18 month contracts). The constraints impeding further change are understood and acknowledged In the absence of major investment and projects from EU development instruments, DIPECHO has played an indispensable pioneering role. Now, DEVCO and other donors are becoming increasingly involved with resources and mechanisms which are more adapted to the size and developmental nature of the project. Once this involvement is fully established and DRR becomes a standard part of EC development programmes, the role of DIPECHO may need to be reassessed. 5.2 From Communities to National and Regional Levels 144. ECHO has gained an enviable reputation as the funding agency best able to reach the communities most in need in the most vulnerable areas. Evaluation after evaluation has confirmed that ECHO partners improved the awareness of their beneficiaries, increased their resilience to future disasters and sensitized local authorities to this issue In addition to its classical community-based approach, DIPECHO has launched regional projects of institutional capacity building in Indonesia. One through the IFRC has definitely improved the DRR capacity of a key Indonesian partner: PMI 52. With regard to the second one, the evaluators are still unsure to which extent the AADMER Partnership Group APG project implemented by the OXFAM consortium, benefitted national or regional civil society or the INGO s members of the APG 53. In Indonesia, local NGOs, albeit weak, are seen by national interlocutors (GoI or UN) as the key to long-lasting impact. Participation as members (not observers) of Indonesian NGOs is envisaged by APG but has not yet happened. There is conflicting information on the reasons for the delay to an expansion that would be in the best interest of DRR and in line with ECHO policy of strengthening civil society Between the community (and district) level and ASEAN projects, there is a gap at national and provincial levels. As necessary, implementing partners reached the provincial level to secure concurrence for their activities. For operational purposes, that may be sufficient given the broad autonomy of the provinces. However, the national level is essential for scaling up in a country with strong political commitment to DRR and great national pride. 52 According to ECHO RSO, success would be more mitigated in other ASEAN countries. 53 APG is led in Malaysia by Malaysia Mercy Corps, a locally established NGO. Evaluation report Particip GmbH 37

45 147. The relationship between ECHO and BNPB is poor. An explanation may lie in the fact that Humanitarian Principles (humanity, neutrality, impartiality and independence strongly endorsed by DG ECHO) are too broadly applied. DG ECHO succeeded in ensuring that its emergency response is based on needs only and, as far as possible, not influenced or delayed by political considerations. This is to its credit. Clearly, those principles essential in the context of a natural disaster, an armed conflict or a complex emergency 54 do not call for development programmes such as DRR to be entirely independent from the National Government supervision. On the contrary, Government ownership is essential for sustainability and as an exit strategy DG ECHO seeks a massive replication of its success stories in a limited number of communities, on a national scale. Only national authorities can achieve that. Even with their support, other obstacles will remain. The number of DIPECHO beneficiaries in any country and particularly in Indonesia, is very small compared to the masses of those with similar needs elsewhere. The task is immense. As in many pilot projects, ECHO quality control was remarkably demanding and human resources mobilized were among the best available in Indonesia. Large-scale expansion will not benefit from the same advantages. Funding per capita is not excessive for the outcome but is still difficult to sustain at national level. Indonesia may however be the exception as a lack of funding is not the main constraint In future evaluations, including in the tender specifications, the requirement to invite the National Disaster Management Organization to designate an expert to integrate the evaluation team at least for the field visits would go a long way to stimulating ownership and support. Not being funded by DG ECHO or benefitting directly, this national designated team member would be indeed external! This is not applicable in every country and in particular in complex emergencies The strategic changes noted in ECHO funding decisions from humanitarian assistance to community based DRR were well adapted to the changing context in Indonesia. The shift to replication and dissemination was appropriate but too limited by de facto excluding the national level where authority and resources for scaling up are located. There is still some work to do before the top down and bottom up approaches meet and merge. 5.3 Consortium Approach 151. The Consortium approach is both facilitating the grant awarding process for ECHO/HQ and rewarding partners who make the effort to coordinate and complement their activities into one single proposal. The second aspect seems to be lost by most partners in Indonesia 55. Reportedly, many DIPECHO partners entered into a consortium primarily to increase their chances of winning contracts, rather than to improve the impact of their work Some partners noted that competition, especially as external funds available are decreasing in Indonesia, is more the norm than collaboration and sharing. Consortia and member agencies need to be managed to have a common vision, to share common procedures and outcomes, to minimise competitive approaches and to work together as a team rather According to RSO/Bangkok, the consortium approach is more successful in Philippines and other countries not visited / covered by the team. Success stories are reported by The Asian Disaster Preparedness Center (ADPC) in Vietnam where consortium approach is reportedly also working well. Evaluation report Particip GmbH 38

46 than as individual agencies. It is a long term undertaking worth pursuing particularly in a country with a strong national disaster management organization Building Consortia is a sound approach. ECHO is planning to adjust its procedures by allocating funding to all members of the consortium rather than only to the lead agency. That will go a long way to assuage NGO criticisms that this additional burden is primarily to facilitate ECHO paper work. 5.4 DIPECHO Exit Strategy DIPECHO Projects 154. The team was requested to offer an exit strategy from Indonesia, keeping in mind the relative needs of other countries. However, the process of readjusting priorities in the Region is well advanced as the number of projects funded in Indonesia has declined sharply over the last few years, and exit is near completion in two other countries Furthermore, delays in launching and implementing the evaluation, meant that formulating next year s Humanitarian Strategy is near completion. The evaluators were informed that the forthcoming strategy would not be contemplating any additional funding for community-based initiatives in Indonesia. That is unfortunate as achievements in this country need to be consolidated Progress, some sustainable, has been made in a few districts. Scaling up requires some additional effort at national level (BNPB) and Civil society. Indonesian civil society organizations with a national or at least multi-provincial vision have not benefitted sufficiently from DIPECHO and are not ready to lead this process on their own. Recruiting them as cost-effective local subcontractors is not sufficient to strengthen their management capacity. They need to be mentored and built up to, in part, replace ECHO implementing partners once external funding dries up and national considerations favour local partners In many other countries, where DIPECHO programmes are implemented, this vision of scaling up with national resources is not realistic, as the only hope for replication and scaling up or merely local sustainability is finding another external donor. Not so in Indonesia, where districts and provinces have received budget allocations for DRR. Tapping this source is likely to be time consuming and complex. Could ECHO and its partners, once they have sponsored / mentored one or more local institutions, assist them in this arduous local fund-raising process?. Echo's ultimate contribution would be to link further DIPECHO funding with provincial authorities strongly committed to further support local civil society and NGO partners. Doing so, Indonesia would be used as a testing laboratory for new exit approaches that may be needed in other countries with fast growing DRR management capacity In brief, the evaluators believe that ECHO should include in its forthcoming strategic plan, room for one or two community-based projects aiming primarily at strengthening selected national NGOs. In a first cycle, the capacity of these NGOs should be built, and modalities of securing local funding for them should be explored. A prerequisite for a second and final cycle would be a firm local commitment of matching funding by districts or provinces. 56 The APG OXFAM lead consortium is seen by partners as a success but competition is minimized as each member plays the lead role in a country. Evaluation report Particip GmbH 39

47 5.4.2 ECHO Office in Jakarta 159. The need for an ECHO office is normally determined by the volume of projects and the workload for their monitoring. Clearly, this is not the case anymore in Jakarta where DIPECHO projects are phasing out. The incumbent is currently dedicating significant amount of time to helping RSO monitor projects in other countries (Philippines for instance). Keeping an office just in case a major disaster might occur in the near future (a possible occurrence in a country called a supermarket for disasters by a major partner of ECHO) is not in itself a cost effective approach. DG ECHO can quickly and easily reopen and expand its presence in Jakarta should emergency Humanitarian Assistance be required on a large scale The main issue is whether DG ECHO is committed: to influencing ASEAN instruments, to playing an influential role in advocating civil society participation within the new DEVCO project in support to the AHA centre, and to strengthening the genuine national Civil Society (rather than only ECHO international partners -see 5.4.1). The aspiration of Indonesia to play a leadership role in regional DRR is another aspect that may be leveraged to pursue DRR advocacy in all ASEAN countries. The evaluators are familiar with most regional DRR institutions in Africa, Latin America and Caribbean. None of them have an economic cloud and potential DRR influence, as ASEAN demonstrated. The AADMER plan is the only one binding its Member States. ECHO and its partners have played a role in shaping the priorities in this Plan Other bilateral DRR actors have faced the same issue regarding the level of their presence. Although the financial commitment from JICA and AusAid largely requires a significant presence, a smaller contributor, the US Office for Foreign Assistance that had initially downgraded its presence has reintroduced a senior US advisor for the above reasons (probably among others). WFP has sharply decreased its food assistance but shifted its priorities towards high level policy influence in food security and general disaster risk management on a cost-sharing basis with the GoI. This influence required a sustained presence in Jakarta All these heavy weight actors shared their concern that nobody but ECHO provides advocacy and expertise in community-based DRR, a stated priority in all national or regional plans but practically overshadowed by the urge to build the institutional and operational instruments (BNPB, AHA). The ISDR office in Bangkok arrived at the same conclusion: Although Indonesia is the only country where the national platform is led by universities and NGOs, it is too early to close down ECHO s modest staff presence in this country Some readjustment of the tasks of the ECHO staff in Jakarta will be required and are shown in a proposed post description in Annex H. The Programme Officer should contribute to ensure that DG ECHO programmes (humanitarian assistance and DIPECHO) are coordinated with, complementary of and coherent with those of other EC instruments (in particular DEVCO), the National Disaster Management Organization (BNPB), and the relevant Work Plan and guidelines as approved by ASEAN countries This prime function of this post would be: 57 ISDR is directly supporting national NGOs and universities through its small grants project (Up to 50,000 USD for civil society local actors across the world). There is an opportunity for complementarity with ECHO in this initiative. 58 The terms of Coordination, complementarity and coherence (3 Cs) as used in the Maastricht Treaty are defined in the publication: evaluating coordination, complementarity and coherence in EU development policy: a synthesis by EU Evaluation Services (2007). Evaluation report Particip GmbH 40

48 improving substantively the coordination with BNPB, offering technical support to ASEAN institutions in coordination with the AHA project funded by DEVCO in consultation with Civil Protection (A5) and ensuring that future DIPECHO projects with selected implementing partners do strengthen a national counterpart. 5.5 EU Civil Protection Role 165. The natural counterpart of European civil protection institutions (bilateral or regional) is the National Disaster Management Organization. EU Civil protection, now integrated into ECHO, has plenty to offer to these counterparts in countries where DIPECHO programmes are implemented. The experience of the Monitoring and Information Centre (MIC) and its successor, the Emergency Response Centre (ERC) would be most valuable for the larger developing countries with the commitment and human resources to sustain it. Indonesia and most ASEAN Countries are good examples. This assistance will fill existing gaps in the support provided by DIPECHO The EU Financing Decision funding the Civil Protection includes interventions in third countries in its Budget article This funding is presently limited to emergency response: Dispatching expertise and transport and related logistics for a total of 5 Mio for No funding is allocated for the transfer of expertise before an emergency occurs In the case of Indonesia, an excellent and productive dialogue was developed between ECHO-Civil Protection and DEVCO through the technical assistance project with AHA (and ASEAN Member States) 60. (See section 4.4.1). The scope of the agreement between ASEAN and EU Ministers of Foreign Affairs is much broader than Civil Protection as it covers risk management from preparedness to recovery This is a very positive and encouraging development towards complementarity between DIPECHO and Civil Protection but it also has the potential for a loss of coherence within DG ECHO if the community dimension is overlooked in this regional DEVCO funded project Ongoing field cooperation and complementarity should not be contingent upon the approval of a DEVCO project. A Civil Protection (CP) presence on the side of DIPECHO projects in Indonesia (and for cost-effectiveness in other ASEAN countries) would fill the critical gap identified in relations with BNPB. This implies a small CP budget for cooperation prior to emergencies and perhaps a post in the RSO. Management supervision will be required to ensure coherence as DIPECHO and CP cultures are distinct albeit complementary (bottom up community participation versus top down chain of command for response) Another important consideration is the unintended consequences that the present modalities of an MIC/ERCC deployment may have in Indonesia or other ASEAN country: Historically, regional disaster management organizations have faced a difficult challenge asserting their coordination role in large scale disasters attracting worldwide attention: Coordinating their members contribution cannot be done without taking into account the 59 EU C(2012) 9518 final accessed 1 Oct AHA - ASEAN-ERAT accessed 28 Sept 2013 Evaluation report Particip GmbH 41

49 global response and they are grossly under-resourced (size and expertise) compared to UN-OCHA and the UN Clusters system. EU MIC/ERCC support is usually provided through UN-OCHA. It is convenient but also indirectly contributes to the imbalance between the local or regional management authority and the international community s own coordination mechanisms. It may be the best alternative in the absence of other appropriate partners. This is not the case with ASEAN. Indirectly weakening the Coordinating Centre for Humanitarian Assistance (AHA Centre) would be particularly undesirable if the EU provides technical cooperation to strengthen this centre prior to the disaster. ECHO should be in a position to provide direct support from EU MIC to the national / regional Civil Protection, should it choose to do so. DG ECHO would place itself in a privileged position to bring local and international assessment teams closer (the UN Disaster Assessment and Coordination Team UNDAC- and the ASEAN Early Rapid Assessment team - ERAT). A MoU should be negotiated with ASEAN for such potential deployment, if required. 6 RECOMMENDATIONS The most valuable worldwide contribution of DG ECHO is its rapid emergency humanitarian response directed to those most in need. The humanitarian response from DG ECHO in Indonesia was found to be very effective and well-targeted. Only one recommendation is offered: Recommendation 1. DG ECHO should continue to closely monitor the situation in Indonesia, a country with a high potential for major disasters. This monitoring should benefit from the expertise of the Emergency Response Coordination Centre and from the onsite experience and observation of ECHO partners. 6.1 Critical DIPECHO Strategic Recommendations Almost all other recommendations address the DRR / DIPECHO programme in Indonesia which constitutes most of the recent ECHO activities, covering the evaluation period. DG ECHO should consider two critical decisions or pre-requisites regarding its DRR actions and exit strategy in a country such as Indonesia. Should DG ECHO accept these first two recommendations, other strategic and operational recommendations should be considered. If not, the exit as currently planned is the best alternative. Recommendation 2. DG ECHO should build a technical relationship and constructive dialogue with the National Disaster Management Organization, BNPB. DG ECHO should recognize the long-term developmental nature of DRR and critically the importance of collaborating and coordinating with a National Disaster Management Organization. This cooperation at programmatic level could be complemented by a more formal policy level relationship between EU Delegation and BNPB. The technical relationship is especially important in a country which has a strong political commitment to DRR, and has the resources to replicate or widely expand the successful experience, including that of ECHO s international partners. Accepting the national coordination role on DRR issues should not be perceived to conflict with ECHO mandate or the humanitarian principles that must prevail in emergency humanitarian assistance. In Indonesia, BNPB is the only mechanism for sustainability, replication and scaling up of ECHO s community-based risk reduction models. Evaluation report Particip GmbH 42

50 Recommendation 3. DG ECHO should adapt to the almost unique situation in Indonesia in order to exercise a significant influence with ASEAN and Indonesia s fast developing leadership in South East Asia. In assessing the importance of a continuing presence in Indonesia, ECHO should consider not only the diversity of hazards and risks but also the advocacy benefit of ECHO influence in the development of the future national and regional DRR plans. ASEAN mechanisms such as AHA may offer ECHO a powerful platform for advocacy CBDRM and humanitarian principles, in response not only in Indonesia but also in other ASEAN countries. It is also an opportunity to present a coherent DG ECHO message from both DIPECHO and Civil Protection (A-5) programmes. 6.2 Other Strategic Recommendations Recommendation 4. DG ECHO should continue advocating complementarity between the long-term development initiatives of DG DEVCO and DIPECHO activities. Advocating this complementarity requires a sustained and meaningful dialogue and ongoing mutual consultation from early stages of the planning and formulation of strategies to final selection and/or approval of specific projects in Indonesia. This essential advocacy and dialogue during critical moments of both instruments financing cycles should take place in the EU Delegation in Jakarta as well as at HQ. The ultimate objective is to include DRR, in particular at community level, in the long-term thematic development agenda. Recommendation 5. In Indonesia, ECHO should present an internally coherent message advocating the top down management principles promoted by Civil Protection (A5) and the bottom up community-based and civil society approach modelled by DIPECHO. Within DG ECHO, DIPECHO should join forces with Civil Protection Programme/unit (A-5) to provide technical advice to the ASEAN-AADMER plan and to its regional disaster management instruments such as AHA. Both mandates and financial mechanisms are different but the message should be identical. DIPECHO contribution should consist of technical expertise, may be indirect and does not presuppose larger financial commitment. Recommendation 6. DG ECHO should give more priority to strengthening national NGOs. In Indonesia, there is a strong drive towards self-reliance and a decreasing role for INGOs, especially those without a consistent presence in the country. ECHO has a stated objective to strengthen the resilience of national institutions. The best way to strengthening Indonesian civil society is through capacity building of genuinely local organizations with a national or sub national reach. This has been achieved by the IFRC through its National Society. Identifying and mentoring local NGOs should be a central focus of the last cycles of DIPECHO programme in Indonesia. 6.3 Operational Recommendations Recommendation 7. DG ECHO should maintain an effective presence in Indonesia if it seeks to genuinely exercise an effective advocacy role with ASEAN, Indonesia and the DRR community. The post description and duties of the programme officer should be adjusted with special emphasis on coordination with BNPB. Evaluation report Particip GmbH 43

51 This permanent presence should focus on substantially improving coordination with BNPB, and offer technical support to ASEAN institutions in coordination with the AHA project, funded by DEVCO in consultation with EU Civil Protection Programme (Unit A5). The presence should ensure that remaining DIPECHO projects and implementing partners strengthen national counterparts / civil society. A revised term of reference (tasks) is attached in Annex H. Recommendation 8. DG ECHO should consider inviting the National Disaster Management Organization to designate an expert to participate in the onsite visits of such future external evaluation or, when appropriate, in the monitoring visits by the TA. Part of the difficulty in scaling up DIPECHO models is the lack of BNPB s direct knowledge of the local success and achievements of DIPECHO funded partners. Inviting BNPB to join field visits (evaluation, routine monitoring or official tours) will be no substitute for a dialogue leading to shared ownership but is a powerful initial step of good will. Recommendation 9. DG ECHO should engage DEVCO in Jakarta to consider incorporating learning from inclusive education programs in Indonesian based education initiatives. Most of DEVCO support is oriented towards the education sector. It is a critical sector for sustained DRR and one where partners have developed considerable expertise. ECHO/RSO with the support of the delegation should concentrate on exploring possibilities for achievements, DRR advocacy agendas and lessons identified to be incorporated into DEVCO s education support agendas. Recommendation 10. DG ECHO should consider the inclusion of Indonesia in the next two DIPECHO funding decisions. Projects should meet stringent requirements. Eligible DIPECHO projects in Indonesia should: Primarily aim to mentor, sponsor and develop the DRR capacity of a local NGO partner to retain, evolve and sustain knowledge gained in community-based programming. Assist those partners in identifying and accessing local government resources for cost sharing of a second and final DIPECHO cycle project. Be endorsed by BNPB prior to their submission under DIPECHO forthcoming Action Plans. Recommendation 11. DG ECHO should request the AADMER Partnership Group (APG) to expand its membership to include local NGOs or their networks with regional, national or multi-provincial reach and ensure that the dialogue with ASEAN benefits Indonesian civil society more directly. The AADMER Partnership Group has produced excellent results by broadening the scope and vision of the ADMER plan. Although Government to Government relationship is the essence of ASEAN and AADMER, this project has contributed to broadening this approach by including community focused and civil society priorities. Now, it is time for the Indonesian Civil Society and local actors to take a greater and more visible role in this formal liaison with ASEAN mechanisms, contributing to facilitating an exit strategy for DG ECHO. Recommendation 12. DG ECHO should continue to promote a consortium approach especially in countries with a strong leadership such as that of the BNPB. The objective of the consortium approach is to present a common vision and share common procedures and outcomes. A common vision is particularly critical in Indonesia where there is a strong and increasingly competent government leadership in DRR. This effort should be pursued. Evaluation report Particip GmbH 44

52 Recommendation 13. DG ECHO should consider replicating the very successful approach of dedicated projects for Handicapped and priority groups in other countries. The projects implemented by ASB and Handicap have proven to be very effective, technically sound and well connected with the national line ministries. They have shown that direct collaboration with the Government at national level, albeit in this case not the disaster management organization, is possible and effective. Lessons from this experience should be disseminated across ASEAN and its replication should be encouraged by DIPECHO. Recommendation 14. DG ECHO should initiate negotiation with AADMER/AHA of an MoU for potential deployment of EU Emergency Response Coordination Centre experts under their umbrella. Currently, DG ECHO has an understanding with OCHA for such post disaster deployment of experts. It may be convenient to open an additional option for locally hosting those experts under ASEAN/AHA umbrella. The latter option would be most appropriate and coherent if the DEVCO-Civil Protection project nurtures a special relationship between the two regional organizations (EU and ASEAN). It is also more likely to facilitate the coordination with the affected Member State of ASEAN. The most opportune channel (OCHA or ASEAN) should be determined case by case. Evaluation report Particip GmbH 45

53 Annexes Annex A Specifications SPECIFICATIONS FOR THE EVALUATION OF THE OPEAN COMMISSIONS HUMANITARIAN ACTIVITIES IN INDONESIA TABLE OF CONTENTS 1) Background/ Introduction 2) Justification and timing of the evaluation 3) Purpose, objective and scope 4) Methodology, outputs and schedule 5) Management and supervision of the evaluation 6) Evaluation team 7) Amount of the contract 8) Timetable 9) Content of the Offer 10) Award 11) Complementary information 12) Guidelines for the Evaluation Team Evaluation report Particip GmbH 46

54 1. BACKGROUND/INTRODUCTION From 1995 to 2012 ECHO allocated a total of Mio to provide humanitarian assistance to people affected by natural and man-made disasters in Indonesia. Of this total amount, 84.5 Mio was in response to natural disasters, and 22 Mio was to assist the victims of conflicts. Since the South Asia Tsunami of 2004, which saw a massive influx of humanitarian aid, there has been no major crisis in the country. In 2009 DG ECHO funded two humanitarian interventions in response to two major earthquakes, in West Java and West Sumatra, for a total amount of 4.5 Mio. This amount financed temporary shelters with a strong focus on earthquake resistant techniques, as well as non-food items, water and sanitation, logistics, transport, and coordination of relief. In 2010, ECHO supported the response to the twin disasters of 25 and 26 October (an 7.7 RS earthquake causing a tsunami which hit Mentawai islands, and the eruption of Mount Merapi, near Yogyakarta in Central Java), which directly affected some 100,000 people. DG ECHO funded integrated relief activities in support of the displaced populations. Disaster Risk Reduction Over the past 12 years, DG ECHO has also been funding community-based disaster preparedness activities in Indonesia, through its Disaster Preparedness Programme (DIPECHO). From 1998 to 2009 the DIPECHO Programme has allocated 30.7 Mio to South East Asia, % of which (= 4.4 Mio) for Indonesia and 15.10% (= 4.6 Mio) for regional projects also including support to AADMER via civil society.(asean Agreement on Disaster Management and Emergency Response) via civil society. For the period DIPECHO allocated 1.47 Mio to Indonesia (including for earthquake and volcano preparedness activities in Yogyakarta area). Some 1.5 Mio are allocated to Indonesia for the period (8 th DIPECHO AP). Presently, the only ECHO activity in Indonesia is DIPECHO - community based activities covering the most vulnerable and poor regions (NTT, NTB, Papua). In addition, ECHO is involved with supporting the AHA centre (the equivalent of the MIC under the ASEAN/AADMER disaster management set-up) based in Jakarta, through a regional DIPECHO led by OXFAM (ASEAN Partnership Project), and by providing input for assistance channelled via the EUDEL READI facility. 2. JUSTIFICATION AND TIMING OF THE EVALUATION Given the recent history of ECHO funding in the country and in view of the next South East Asia DIPECHO Humanitarian Implementation Plan ECHO would like to evaluate the present added value of the ECHO funding instruments in Indonesia. To date, ECHO has had a role relegated to Disaster Risk reduction and emergency stand-by donor and ECHO would now like to re-evaluate its presence in the country. This evaluation is expected to contribute to defining ECHO's operational strategy in the next years in terms of both Humanitarian Aid and DIPECHO programming in Indonesia. A review of ECHO Indonesia s impact and strategy thus far is therefore opportune in preparation of the next South East Asia DIPECHO Humanitarian Implementation Plan ( ). This evaluation is expected to contribute substantially to informing the evolution of existing disaster risk management strategies supported by DIPECHO in the region, as well as encouraging the development of coherent and pertinent programme strategies in future action plans. The results of the evaluation should be available by the end of the first quarter of 2013 to ensure that preliminary recommendations can be expected from the evaluation consultants by April/May 2013/. This would allow sufficient time for the recommendations to be taken into account in the subsequent DIPECHO South East Asia strategy and operational conclusions to be drawn for ECHO's wider presence in Indonesia in the medium to long term. Evaluation report Particip GmbH 47

55 3. PURPOSES, OBJECTIVE AND SCOPE 3.1 Purpose and objectives The purpose of this exercise is to review and assess the impact of ECHO-funded assistance programs in Indonesia. The evaluation will include measuring the impact of humanitarian aid and DIPECHO projects. For the DIPECHO interventions possible angles include the extent to which preparedness helped the population when they were afterwards affected by an adverse natural event. 3.2 Specific objectives To have an independent, structured evaluation of past and current ECHO and DIPECHO actions in Indonesia since 2007 and in particular to evaluate: a. The relevance of the presence of ECHO in Indonesia. b. The relevance of DIPECHO strategies and projects for disaster risk reduction and risk management at all levels in the country. c. The effectiveness of individual programme strategies and the actions contained therein in addressing the needs, vulnerabilities and coping capacity constraints of target populations /institutions for both humanitarian aid and disaster preparedness programmes; d. The complementarities and synergies of DIPECHO programmes with other disaster risk reduction initiatives at local, national, regional and global levels. e. The role and effectiveness of DIPECHO/DRR actions as a catalyst for DRR actions by other stakeholders. f. The overall efficiency of DIPECHO projects, analysing in particular how it is affected by working in consortia. g. The evolution of the programme throughout its implementation. h. The impact of DIPECHO s actions since 2007 in enhancing the resilience of at-risk populations and relevant institutions and thereby reducing their vulnerability to natural hazards, including how well preparedness helped the population when they were afterwards affected by a disaster; ƒ. The sustainability and scaling up of actions, including LRRD aspects (Linking Relief, Rehabilitation and Development). The evaluation should focus primarily on the pertinence of ECHO's presence in the country as a whole since 2007, rather than an evaluation of individual projects. It will review the achievements of Humanitarian aid and DIPECHO actions in Indonesia since 2007, critically appraising lessons learned from past strategic approaches. The scope of the evaluation will cover the implementation of DG ECHO funded action between 2007 and The key users of the evaluation report include inter alia Commission staff at HQ, regional and field level, the implementing partners, other stakeholders with an interest in the evaluation findings and other humanitarian donors and agencies. Evaluation report Particip GmbH 48

56 The information requested in the evaluation questions listed in chapter 3.2 is the main subject of this evaluation. When addressing the evaluation questions, and whenever feasible/applicable, the evaluators will take due account of the OECD/DAC evaluation criteria: relevance/appropriateness, connectedness, coherence, coverage, efficiency, effectiveness and impact 62; the 3Cs 63 - complementarity, coordination and coherence; cross-cutting issues 64; the objective of LRRD (Linking Relief, Rehabilitation and Development) 65 ; and the 23 Principles and Good Practice of Good Humanitarian Donorship Evaluation questions The evaluation will be based on a set of evaluation questions. These questions reflect the Commission's needs in terms of information with a view to accountability and improved performance of humanitarian actions. The evaluation questions will be further discussed and validated at the Desk phase and other questions may be added at that stage. The evaluation will address the following questions: 1. Relevance: To what extent have DG ECHO strategic orientation and intervention logic addressed the needs of the most vulnerable communities and categories of population, also in the areas most exposed to frequent natural hazards? What were the added value and the relevance of DG ECHO presence in Indonesia for the last five years? Effectiveness: To what extent have ECHO (humanitarian aid) and DIPECHO-funded activities contributed to reducing vulnerabilities, enhancing the capacity to cope and boosting the resilience of targeted populations at risk, national institutions and the wider disaster risk reduction stakeholder community to recurrent natural hazards in Indonesia? To what extent have DG ECHO-funded DRR activities contributed to reinforcing sub-national response capacities, including preparation and awareness of local communities? 2. Complementarity: To what extent ECHO funded DRR programmes complement actions developed, endorsed and/or supported by national governments, EC cooperation instruments or other external assistance, taking into account DG ECHO s mandate? To what extent have DG ECHO-funded DRR activities had an impact at national level, through replication, scaling up, advocacy, or complementarity with national initiatives? To what extent have DG ECHO- funded DRR activities contributed to reinforcing sub-national response capacities, including preparation and awareness of local communities? 62 For further explanation of these evaluative criteria evaluators are advised to refer to the Active Learning Network for Accountability and Performance (ALNAP) guide "Evaluating humanitarian action using the OECD-DAC Criteria. An ALNAP guide for humanitarian agencies", ODI, Evaluators should also refer to the "Evaluation of humanitarian aid by and for NGOs. A guide with ideas to consider when designing your own evaluation activities", Prolog Consult, 2007 ( For example: gender, children HIV-AIDS, environment, protection, climate change, etc. 65 A communication from the European Commission to the European Council and European Parliament on LRRD policy can be found at: Evaluation report Particip GmbH 49

57 Regarding this question the evaluators will particularly consider: o Co-ordination between DG ECHO and other stakeholders shall be assessed at national and local levels. Consideration must be given to the relationship, if any, between the DIPECHO programme and the EC objective to link relief to rehabilitation and development (LRRD) 67. o The connectedness with the national emergency response institutions (e.g. Civil Defence or equivalent), the national disaster management coordinating board BAKORNAS PB) etc. as well as the scientific institutions assessing vulnerabilities and hazards (geophysics and oceanographic institutes etc.). o The different approaches and responses delivered by the NGOs, IFRC/European Red Cross Societies and International Organisations. 3. Coherence: To what extent LRRD aspects and the following cross-cutting issues have been taken into account in the design and implementation of ECHO and DIPECHO strategies and activities? Children; Effects on the environment; Gender, in particular women s involvement, consultation and participation in projects; Handicapped people; Community networks in the implementation of projects; Participation. 4. Efficiency: Which elements in the institutional context and in the relations between the different actors have had a higher impact in the effectiveness and scaling up of the DRR interventions? 5. Sustainability: To what extent have the project activities and results had lasting effects after hand over? In particular, the evaluator is expected to assess whether national authorities, EU cooperation instruments and the other external development assistance have provided support to beneficiaries for a DIPECHO exit strategy. The conclusions of the evaluation will be presented in the report in the form of evidencebased, reasoned answers to each of the evaluation questions. On the basis of the conclusions, as well as of any other relevant information collected during the evaluation, the evaluator will provide practical, operational recommendations for future adjustments and actions, particularly as regards the development of the next Action Plan. As a minimum the evaluator will provide recommendations on the following subjects: In view of the current and the foreseen response capacity of the Indonesian Government (physical and financial, potential gaps), the EC cooperation instruments and other external development assistance which might provide support to beneficiaries for a DIPECHO exit strategy in the country, recommendations on whether the Commission should continue with its DRR humanitarian interventions in Indonesia. In case of continuation, recommendations about field presence: should it be maintained? If so, should the ToRs of the ECHO operational staff covering Indonesia be amended? Might ECHO RSO BKK provide appropriate and sufficient monitoring of the humanitar- 67 When preparing the report, the evaluators are required to compare the Maastricht Treaty s definition of the 3Cs (coherence, complementarity and co-ordination) with the 1999 OECD-DAC / ALNAP definitions. Evaluation report Particip GmbH 50

58 ian situation in Indonesia? Would the speed of ECHO response capacity in the event of a major disaster be preserved, taking into account the constraints mentioned above? Recommendations on the means by which DIPECHO can most practically operate to improve the supported actions and strategies at the local and national level. Relevant, practical and feasible recommendations applicable to the following will be required: o the DIPECHO funding instrument and other DRR targeted operations; o the strategic formulation process for DIPECHO Indonesia; o the programme strategies and operational methodologies of DIPECHO partners, particularly at national, sub-national and local levels (including comment on the merits/disadvantages of the use of local implementing partners/sub-contractors). Specific and innovative recommendations to DG ECHO, DIPECHO partners, local, national and regional authorities for dissemination and replication of best practices. Recommendations for the strengthening of strategic and programmatic linkages with other relevant instruments of the European Commission, as well as other bi-/multi-lateral financing institutions, and national governments. Recommendations or analyse to which extent the contribution of the EC Civil Protection could have an added value in the ECHO DRR programming in the region. If the evaluation finds that ECHO/DIPECHO should be discontinued in Indonesia an exit strategy should be proposed. 3.4 Tasks to be accomplished The evaluators shall accomplish as a minimum the following tasks as a basis for their report: Analysis of existing DIPECHO documents, reports, monitoring documents, related literature and all relevant documents provided by the Commission concerning DG ECHO activities in Indonesia and South East Asia (where relevant) and the management and monitoring of existing projects. Analysis of existing evidence (databases, reports, statistics ) provided by other actors involved in Humanitarian aid, Disaster Risk Reduction and Disaster Preparedness fields in Indonesia and South East Asia (where relevant): national authorities, other donors, partners, international agencies Collecting the views and opinions of partners, officials, other donors, and beneficiaries. Interviews with Commission staff involved in activities in Indonesia and South Asia (where relevant). Collecting all necessary qualitative and quantitative data for providing answers to each of the evaluation questions above. Carrying out a number of case studies of DIPECHO projects on the field, with due regard to the diversity of disasters envisaged, target populations and regional contexts. The evaluators shall carry out visits to regions targeted by DRR operations since The duration of the field visit to Indonesia should not be shorter than two weeks, in order to ensure in-depth work on the field. At the same time it is imperative to have contacts with the ECHO field offices and the EC Regional Delegation, and regional organisations such as IFRC, UNDP, PAHO, OCHA, UNICEF. On-going as well as completed projects should be visited. Drafting the deliverables requested in these Terms of Reference. Evaluation report Particip GmbH 51

59 4. METHODOLOGY, OUTPUTS AND SCHEDULE The tender proposal should describe in detail the proposed methodological approach to tackle the evaluation questions listed above, as well as the tasks requested. This will include a description of one or more indicative judgment criteria 68 that they consider useful for addressing each evaluation question. The judgement criteria will be discussed and validated with the contractor during the briefing phase. This should also include a clear description of the proposed methodology for the field visits mentioned in point 4.2. To the extent possible, the methodology should promote the participation in the evaluation exercise of all actors concerned, including beneficiaries and local communities when relevant and feasible. The methodological approach will be refined with, and validated by, the Commission during the briefing phase. 4.1 Deliverables Within the framework of the present evaluation, the contractors will produce the following deliverables, in accordance with the schedule defined in this chapter: 1. Inception 2. Note Desk Report 3. Field Report 4. Final Report 4.2 Desk Phase The Desk Phase will have a minimum duration of 20 days and deal with The analysis of existing Commission documents as well as other donors' or partners information, The contact, and possibly, meetings with Commission staff, other donors, organisations and partners, the final definition of the methodology, the finalisation of the schedule for the field mission. The desk phase starts from the moment the contract is signed, and it will be based on both documents provided by the Commission and retrieved by the contractors from the start of their work. The documents may be produced by the Commission or any other relevant actor (other Commission services, international agencies, other donors, partners, communities of practice ). The contractor will ensure that an appropriate literature review is carried out throughout the contract. The Inception Meeting will take place in Brussels at DG ECHO headquarters with the relevant Commission staff. During this meeting, additional documents and necessary clarifications will be provided by the Commission. The consultants will present their understanding of the Terms of Reference, they will propose JC for each evaluation question as well as a detailed presentation of the methodology proposed (using a Power Point Presentation). Following the 68 A judgment criterion specifies an aspect of the evaluated intervention that will allow its merits or success to be assessed. E.g., if the question is "To what extent has DG ECHO assistance, both overall and by sector been appropriate and impacted positively the targeted population?", a general judgment criterion might be "Assistance goes to the people most in need of assistance". In developing judgment criteria, the tenderers may make use of existing methodological, technical or political guidance provided by actors in the field of Humanitarian Assistance such as HAP, the Sphere Project, GHD, etc. Evaluation report Particip GmbH 52

60 meeting the contractors will produce a short Inception Note. The Inception Note will summarise the contents of the Inception Meeting, refine the evaluation questions and the judgement criteria on the basis of the information received during the meeting and in the course of the interviews held around it, refine any draft methodological tools required in the Specifications or proposed by the contractors in their offer, identify limitations and challenges provide an outline of the next steps in the evaluation work, identify potential additional issues for consideration, suggest a final timing for the evaluation, including the schedule for the field work and case studies. The Inception Note will be commented on, corrected and, if satisfactory, approved by the Commission. At least ten days before the start of the Field Phase, the contractors will produce a Desk Report that will include: A short description of the data collection work implemented, including the meetings, reviews and interviews conducted. A refined version of the overall methodology for the evaluation on the basis of the work carried out so far. Evaluation questions should be completed with judgment criteria, and the information sources to be used for addressing them should be clearly identified at this stage. The initial assumptions concerning the evaluation questions, to be tested during the Field Phase, on the basis of the preliminary analysis carried out during the desk phase, A refined version of the methodology (tools, criteria) for the Field Phase, as well as a preliminary schedule and list of partners, stakeholders and projects to be visited explaining criteria used for choosing this sample of projects, A discussion of possible issues identified during the Desk Phase that had not been previously discussed with the Commission. The Commission will consider these issues and decide on whether they merit further consideration in the light of the evaluation.. The Commission will comment on the Desk Report. The contractors will take into account all comments from the Commission, as a condition for approval. In case of disagreement consultant will provide an augmented reply to comment explaining why the comment cannot be taken into consideration. In case of substantial disagreements, the contractors may be invited to come to Brussels for discussing the report. The expenses for such a meeting will be covered by the price of the contract. 4.3 Field Phase Following the formal approval of the Desk Report, the evaluation team shall undertake the field visits. The Field Phase should have a minimum duration of 20 days. The details of the field missions will be discussed and agreed with the Commission during the inception meeting, and will be refined in the inception and desk reports. The travel and accommodation arrangements, the organisation of meetings and the securing of visas will remain the sole responsibility of the contractor. The evaluation team will Evaluation report Particip GmbH 53

61 receive a security briefing regarding the specific evaluation assignment during the desk phase. If, during the Field Phase, any significant change from the agreed methodology or scheduled work plan is considered necessary, this will be explained to and agreed with DG ECHO Evaluation Sector, in consultation with the steering group. A final workshop in the field, with the participation of the EU Delegation, DG ECHO representatives and partners, shall be organised before leaving the country. The modalities and content of the workshop will be discussed and agreed with DG ECHO staff in the field and the Evaluation Sector will be informed about them beforehand. At the end of the field trip the team leader should ensure that a Field Report is drawn up and transmitted to DG ECHO Evaluation Sector. The Field Report will describe briefly the data collection activities implemented, with a special mention of those of a participatory nature (including in annex the list of sites and persons visited, minutes from the focus groups if organized, the minutes of the workshop and any other relevant technical documents); a brief description of the situation found; as well as any relevant items identified during the field visit, which could have an influence in the methodology or the conclusions of the evaluation. N.B.: the Field Report is not an evaluation of the intervention, and should not include overall conclusions and recommendations, nor it is a collection of project evaluations. It is an operational document to allow the evaluation managers an appropriate follow up of the work done, as well as identify any possible items to be tackled during the synthesis phase (changes in the methodology, aspects not raised during the desk phase, changes in the judgement criteria, suggested changes in the evaluation questions ). As a reminder, even if the evaluation will assess individual projects, conclusions and recommendations must be drafted with a view to the overall evaluation of the Commission's intervention in the area concerned, and will be based on the overall information collected during the evaluation process. 4.4 Synthesis phase A first Draft Report (maximum 50 pages excluding the annexes) in accordance with the format given in point 5 of the annex of the Terms of Reference shall be submitted by electronic transmission to DG ECHO Evaluation Sector. It should be submitted after the evaluators return from the field the basis of the timetable indicated in chapter 8, as well as the timetable included in the contractor's offer and agreed during the Desk Phase. A debriefing meeting will be organized in Brussels after the submission of the first draft report. The evaluators will make a PowerPoint presentation to DG ECHO and key staff of the main findings, conclusions and recommendations of the evaluation. The date for this meeting will be decided by DG ECHO Evaluation Sector in agreement with the contractors, the relevant desk(s) and the steering group. Prior to the meeting, DG ECHO may provide written comments on the first draft report to the evaluation team. On the basis of the results of the meeting on the Draft Report, and taking into due account the comments received before and during the meeting, a Draft Final Report (maximum 50 pages excluding the annexes) will be submitted to DG ECHO Evaluation Sector not later than 10 calendar days after the meeting on the draft final report. DG ECHO Evaluation Sector should mark its agreement, make comments or request further amendments within 10 calendar days. Evaluation report Particip GmbH 54

62 4.5 Final report On the basis of the comments made by DG ECHO, the evaluation team shall make appropriate amendments and submit the Final Report (maximum 50 pages excluding the annexes) within 10 calendar days. If the evaluators reject any of the comments they shall explain and substantiate the reasons why they do so in writing. The evaluation will result in the drawing up of a single report with annexes. The report shall follow the structure outlined in section To facilitate dissemination, together with the final report, the evaluators will provide a Power Point presentation in electronic form, with the main conclusions and recommendations of the evaluation. Before the expiration of the contract, the contractors may be required to present briefly DG ECHO's staff or stakeholders with the results of the evaluation. 4.6 Dissemination and follow-up The evaluation report is an extremely important working tool for DG ECHO. The evaluation report is the primary output of the work of the evaluators, and once finalised the executive summary and/or the entire final report will be placed in the public domain on the Internet. The report is meant to promote accountability and learning. Its use is intended for DG ECHO's operational and policy staff, other EU services, humanitarian beneficiaries, EU Member States and citizens, other donors and humanitarian actors. Whenever applicable, the executive summary and/or the final report shall be translated into relevant languages for dissemination purposes. Following the approval of the final report, DG ECHO Evaluation Sector will proceed to the dissemination of the results (conclusions and recommendations) of the evaluation. Therefore, whenever applicable the contractors shall provide a dissemination plan. 5. MANAGEMENT AND SUPERVISION OF THE EVALUATION The Evaluation Sector in DG ECHO is responsible for the management and the monitoring of the evaluation, in consultation with the responsible desk. DG ECHO Evaluation Sector, and in particular the internal manager assigned to the evaluation, should therefore always be kept informed and consulted by the contractors and copied on all correspondence with other DG ECHO staff. The DG ECHO Evaluation manager is the contact person for the contractors and shall assist the team during their mission in tasks such as providing documents and facilitating contacts. A steering group, made up of Commission staff involved in the activity evaluated, will provide general assistance to and feedback on the evaluation exercise, and discuss the conclusions and recommendations of the evaluation. 6. EVALUATION TEAM This evaluation will be carried out by a team of a minimum of 3 experts (this is an indicative number) with experience both in the humanitarian field and in the evaluation of humanitarian aid. If necessary, these experts must agree to work in high-risk areas. It is therefore recommended that the team should include national experts whenever possible. The inclusion of a native expert in the team will be considered positively in the evaluation of of- Evaluation report Particip GmbH 55

63 fers. Proficiency in English is compulsory. Knowledge of local language(s) would be an advantage. The experts profiles should include solid knowledge and experience in: (1) Evaluation of Humanitarian Aid activities (2) Disaster Risk Reduction and Disaster Preparedness (3) Experience in Indonesia or South East Asia would be an advantage Ideally, the team should be gender-balanced. Guidelines for the evaluation team are provided in chapter 12 of the Specifications. 7. AMOUNT OF THE CONTRACT The maximum budget allocated to this evaluation is TIMETABLE The work must be completed within 5 months from the date of the Inception Meeting. The contractor is expected to start the work immediately after the contract has been signed. At the latest, the final report will be delivered by the end of the 4 th month after the date of the Inception Meeting. Unless explicitly authorised by the Commission in written form, this deadline has to be strictly respected. The evaluation starts at the actual signature of the contract and by no means any contact and/or expense may occur before it. The largest part of relevant documents will be provided after the signature of the contract and during the desk phase. In their offer, the bidders shall provide an indicative schedule based on the table below: Dates Evaluation Phases and Stages Meetings Deliverables Desk Phase Inception Inception note Desk report Field Phase Field work Workshop Field report Draft final report Synthesis phase 9. CONTENT OF THE OFFER Meeting on the draft final report Final Report The technical part of the bidder's offer must include: 1. A description of the understanding of the Terms of Reference, their scope and the tasks covered by the contract. This will include a graphic reconstruction of the intervention logic of the Commission's humanitarian activities concerned. It will also explain the bidder's understanding of the evaluation questions, including Evaluation report Particip GmbH 56

64 a first proposal of judgment criteria to be used for answering the evaluation questions and the information sources to be used for answering the questions. The final definition of judgment criteria and information sources will be validated by the Commission during the desk phase; 2. The methodology the bidder intends to apply for this evaluation for each of the phases involved, including a draft proposal for the number of case studies to be carried out during the field visit, the regions to be visited, and the reasons for such a choice. The methodology will be refined and validated by the Commission during the desk phase; 3. A description of the distribution of tasks in the team, including an indicative quantification of the work for each expert in terms of person/days; 4. A detailed proposed timetable for its implementation with the total number of days needed for each of the phases (Desk, Field and Synthesis). 5. The CVs of each of the experts proposed. The financial part of the offer must include the proposed total budget in Euros, taking due account of the maximum amount for this evaluation as defined in chapter 7 of this Terms of Reference. The price must be expressed as a lump sum for the whole of the services provided. 10. AWARD The contract will be awarded to the tender offering the best value for money on the basis of the following criteria: 1. Quality: quality criteria will be assessed on the basis of the a) Quality criterion 1 (max. 50 points): The appropriateness of the proposed team. b) Quality criterion 2 (max. 50 points): The tenderer's understanding of the tasks and the quality of the methodology proposed. 2. Price Technical evaluation Quality criteria a) and b) will be evaluated on the basis of the information provided in the technical part of the offer (see chapter 9). Only those tenders with a mark higher than 25 points for each quality criteria a) and b), and higher than 60 points for the overall rating, will be considered for the award of the contract Financial evaluation For the purpose of evaluation and comparison of the financial offers, the Commission will use the price as submitted in the financial offer of the bidder (see chapter 9) Award of the contract The contract will be awarded to the tender achieving the highest score obtained by applying the following formula: Score for tender X = Cheapest price Price of tender X * total quality score (out of 100) for all criteria of tender X 11. COMPLEMENTARY INFORMATION Evaluation report Particip GmbH 57

65 GUIDELINES FOR THE EVALUATION TEAM Regulatory basis The Regulatory basis for the evaluation of the aid provided by DG ECHO is established in Article 18 of Regulation (EC) 1257/96 concerning humanitarian aid, which states "the Commission shall regularly assess humanitarian aid operations financed by the Community in order to establish whether they have achieved their objectives and to produce guidelines for improving the effectiveness of subsequent operations". Article 27 of the Council Regulation (EC, Euratom) 1605/2002 laying down the rules for the establishment and implementation of the general budget of the European Union states that : "In order to improve decision-making, institutions shall undertake both ex ante and ex post evaluations in line with guidance provided by the Commission. Such evaluations shall be applied to all programmes and activities which entail significant spending and evaluation results disseminated to spending, legislative and budgetary authorities" Specifications The Specifications set out the scope of the evaluator's mission, the issues to be considered and the evaluation timetable. They allow those commissioning the evaluation and/or the review to express their needs (guidance function) while providing the evaluators with a clear idea of what is expected from them (control function) Scope of the evaluation and topics of study In addition to the initial information contained in the Specifications, the Commission will provide throughout the evaluation process, and notably during the Desk Phase, all kinds of explanations and clarifications needed for the full understanding of the evaluation work. The Inception Meeting in Brussels provides everyone involved in the evaluation (DG ECHO requesting service and particularly the responsible desk, DG ECHO Evaluation Sector, the steering group, the evaluators and other Commission services) with the opportunity to discuss the contents of the Specifications and to establish the final modalities for the evaluation. This meeting should also allow the evaluators to clarify any doubts they might have about the scope of their mission. Any important complementary information provided by the Commission on the content of the Specifications at this stage will be considered an integral part of these, and will be set out by the team leader in the Desk Report. The evaluators are encouraged to raise any issue they may consider important or unclear for a better understanding of the evaluation scope or content. During the process of the evaluation the evaluators must try to address all the items listed in the Specifications. Any decision not to cover one or more of the main assignments described in the Specifications will be discussed with the Commission before drafting the Final Report. If the evaluators persist in their intention not to tackle a point, and the Commission considers such a decision as inappropriately justified, DG ECHO may choose not to accept the Evaluation report Particip GmbH 58

66 final report The evaluation team Each team member is jointly responsible for the final accomplishment of the tasks; however, the separate elements of work necessary for the accomplishment of the tasks should be allocated between the experts and clearly specified in the offers for the tender. The members of the team must work in close co-ordination. The team leader appointed in the contractor's offer shall have the added responsibility of the overall co-ordination of the tasks to be completed, of the elaboration of Executive Summary and of the final coherence of the report and related work, both in terms of content and presentation Methodological and professional guidelines The evaluators are required to carry out their work in accordance with international standards of good practice in approach and method. All conclusions must be substantiated with adequate data. All recommendations must be adequately based on evidence-based conclusions. In the conduct of their work the evaluators should use a multi-method approach and triangulate between different sources of information. These information sources should include, primary stakeholders (specifically humanitarian beneficiaries, members of the host communities), local government (or equivalent such as group/tribal leaders), national government, international agency staff, partners (both expatriate and local employees of partners), DG ECHO experts, EU Delegation, other donors and humanitarian agencies, non-beneficiaries etc. They should consistently use primary data (i.e. information collected directly by the evaluators during their work), and secondary information (programming and policy documents, monitoring reports and data, statistical data, relevant studies and evaluations, etc.), In order to substantiate evaluation findings the numbers, sex, ethnicity etc. of primary stakeholders should be noted, as well as ways in which confidentiality and dignity have been assured in the interview process. In this consultation, the evaluation team is encouraged to use participatory techniques. In carrying out their work, the evaluators should be vigilant as to any non-respect of international humanitarian law and principles, standards and conventions, UN protocols, Red Cross codes, and declarations. The evaluators should report any non-respect of such matters by DG ECHO-financed entities to DG ECHO in a duly substantiated form. During the contract, evaluators shall refrain from any conduct that would adversely reflect on the European Commission or DG ECHO and shall not engage in any activity that is incompatible with the discharge of their duties. Evaluators are required to exercise the utmost discretion in all matters during their mission The report By commissioning an independent evaluation and/or review DG ECHO expects to obtain an objective, critical, easy to read and transparent analysis of its interventions. This analysis should contain the information needed by the Commission for management, policy-making and accountability. It should also include operational, realistic recommendations at operation- Evaluation report Particip GmbH 59

67 al and/or strategic level. Above all, the report should be a document that can function as a learning tool. Therefore, while writing it, the evaluators should always bear in mind why the report is done, for whom, and how the results will be used. For each evaluation question quoted in the report the consultant will provide an evidence-base, reasoned answer. Conclusions 8 will be provided pointing out strengths and weaknesses of the evaluated intervention, with special attention paid to the intended and unintended results. Furthermore, the report is a working tool of value to DG ECHO only as long as it is feasible and pragmatic, keeping in mind DG ECHO's mandate constraints and it clearly reflects the evaluator's independent view. DG ECHO's concern is to respect this independence. The evaluation methods should be clearly outlined in the report and their appropriateness, focus and users should be explained pointing out strengths and weaknesses of the methods. The report should briefly outline the nature (e.g. external or mixed) and make-up of the team (e.g. sectoral expertise, local knowledge, gender balance) and its appropriateness for the evaluation. It should also briefly outline the evaluators biases and/or constraints that might have affected the evaluation and how these have been counteracted (past experiences, background, etc.). The report shall be written in a straightforward manner in English with an Executive Summary at the beginning of the document. Final editing shall be provided by the contractor. The report should be in the font Time Roman 12, have single line spacing and be fully justified. The final report should contain: An Executive Summary of maximum 5 pages. A list of Abbreviations and Acronyms The main report. Technical annexes. Other annexes as necessary. This format should be strictly adhered to: Cover page (a template is provided at the end of this annex) title of the evaluation, date of the evaluation report, name of the company, contract number, disclaimer in the sense that "The opinions expressed in this document represent the views of the authors, which are not necessarily shared by the European Commission." Table of contents Executive Summary A clearly drafted, to-the-point and free-standing Executive Summary is an essential element. It should be short, no more than 5 pages. It should focus on the key purpose or issues of the evaluation, outline the main points of the analysis, and contain a matrix made of two columns clearly indicating the main conclusions and specific recommendations. Cross-references should be made to the corresponding page or paragraph numbers in the main text. EU Member States receive each Executive Summary, which is also published on DG ECHO website. The evaluation team should take this into account when drafting this part of the report. Main body of the report Evaluation report Particip GmbH 60

68 The report should include at least a description of the purpose of the evaluation, the scope of the evaluation, the design and conduct of the evaluation, including a description of the methodology used, the evidence found, the analysis carried out, the conclusions drawn in the form of reasoned answers to each of the evaluation questions in the Specifications(the questions must be quoted fully in the report when providing the conclusions and conclusions should be evidence-based and fully substantiated, and derive in a logical manner from the data collection and analysis carried out), recommendations for the future. Recommendations should be clearly linked to the conclusions and as realistic, operational and pragmatic as possible; they should take careful account of the circumstances currently prevailing in the context of the implementation of the humanitarian activities, DG ECHO's mandate and of the resources available to implement it both locally and at the Commission level. Recommendations should be prioritised, directed at specific users and where appropriate include an indicative timeframe. The report should have separate sections for the evaluation work in each of the regions visited. Annexes of the report (as a minimum): Annex A: Specifications; Annex B: A detailed description of the methodology implemented and the tasks carried out by each expert; Annex C: List of persons interviewed and sites visited; Annex D: Summary presentation of individual case study visits; Annex E: Map of the areas covered by the operations financed under the action and the sites visited by the contractors; Annex F: Statistical information about beneficiaries and assistance provided; All possible confidential information shall be presented in a separate annex. While finalising the report and its annexes, the evaluators will always highlight changes (using track changes) and modifications introduced as resulting from the meeting and the comments received from DG ECHO Evaluation Sector. Each report and all its annexes shall be transmitted in electronic form to DG ECHO Evaluation Sector (ECHO-EVAL@ec.europa.eu). The final report should be sent by to DG ECHO Evaluation Sector (ECHO- EVAL@ec.europa.eu) in three separate documents in PDF format each containing: the executive summary, the report without its annexes (also removed from the table of contents) and the report with its annexes. Evaluation report Particip GmbH 61

69 Annex B Detailed Description of the methodology implemented and the Tasks carried out by each Expert Evaluation Matrix Primary EQs Judgement criteria and indicators Tools EQ1: To what extent have DG Judgement criteria Desk review ECHO strategic orientation DG ECHO policies, programmes and plans are Semistructured and intervention logic addressed the needs of the most based upon and address needs and risk assessments. interviews vulnerable communities and Site visits categories of population, also ECHO actions are congruent with national priorities based on risk assessment. Case studies in the areas most exposed to frequent natural hazards? EQ2: What were the added value and the relevance of DG ECHO presence in Indonesia for the last five years? EQ3: To what extent have ECHO (humanitarian aid) and DIPECHO-funded activities contributed to reducing vulnerabilities, enhancing the capacity to cope and boosting the resilience of targeted populations at risk, national institutions and the wider disaster risk reduction stakeholder community to recurrent natural hazards in Indonesia? EQ4: To what extent have DG ECHO-funded DRR activities contributed to reinforcing sub-national response capacities, including preparation and Indicators Extent to which ECHO/ DIPECHO projects and programs are based on risk and needs assessments with special attention to vulnerable groups. Extent to which actions include an analysis of Government priorities and a dialogue with all actors. Judgment criteria ECHO strategy is based on the analysis of its added value in a changing environment. Indicators Benefits of ECHO presence in Indonesia as perceived by main stakeholders and partners. Plans of action reflected changes of GoI capacity and policies of other donors. Extent to which ECHO strategies and programs are based on a comparative (regional) analysis of its added value in Indonesia. Judgment criteria Changes in resilience at local, provincial and national level desegregated for HA and DRR Appreciation of causal relationship between ECHO activities and changes ECHO DRR actions are replicated outside the action area. Indicators Extent to which stakeholders and beneficiaries report that ECHO funded projects have increased their respective resilience and / or coping capacity (Community, local and national level). Observations, if possible in recently affected areas. Judgement criteria Stakeholders and partners report that ECHO funded projects have increased their respective response capacity (Community, local and provincial Desk review Semistructured interviews Site visits Desk review Semistructured interviews Site visits Case studies Desk review Semistructured interviews Site visits Evaluation report Particip GmbH 62

70 Primary EQs Judgement criteria and indicators Tools awareness of local communities? level). Indicators Case studies Demands are made for good practice documentation or support from other sub-national communities or institutions (preparedness and awareness campaigns). Extent to which partners and stakeholders report that ECHO-supported CBDRR/CBDRM or subnational institution planning processes were updated or replicated. Observations, if possible in recently affected areas. EQ 5: To what extent ECHO Judgement criteria Desk review funded DRR programmes Priorities and action areas set out in ECHO decisions Semi- and partner programs are aligned with Kyoto structured complement actions developed, endorsed and/or supported by national govern- Site visits Framework for Action (FHA), GoI priorities and interviews other EU relevant instruments. ments, EC cooperation instruments or other external Consultation with those institutions is made prior Case studies assistance, taking into account formulation of ECHO Plan of Action or approval DG ECHO s mandate? of projects. EQ6: To what extent have DG ECHO-funded DRR activities had an impact at national level, through replication, scaling-up, advocacy, or complementarity with national initiatives? Indicators Extent to which Government is informed and endorse ECHO funded activities. Extent of dialogue and consultation between ECHO and EU instruments involved in DRR related activities. Extent of collaboration between Partners under the consortium approach. Judgement criteria ECHO actions are cited by National Government and key stakeholders as examples or models for replication. Indicators Extent to which Government and key stakeholders report that they sought DIPECHO models or project for potential replication. Desk review Semistructured interviews Site visits Case studies Extent to which DIPECHO DRR message is reaching national emergency response institutions, BAKORNAS PB, DRR and scientific institutions and actors not directly funded by ECHO. EQ7: To what extent LRRD aspects have been taken into account in the design and implementation of ECHO and DIPECHO strategies and activities? Judgement criteria HA response projects are linked to long-term development. Short-term DIPECHO projects and EU long-term programs are complementary and consistent. Desk review Semistructured interviews Site visits Case studies Evaluation report Particip GmbH 63

71 Primary EQs Judgement criteria and indicators Tools Indicators Extend to which HA emergency response projects are considering the medium-term impact. To what extend is DRR Project responsibility handed over to national partners or embedded within the institution? Extent to which ECHO and EU instruments maintain an on-going dialogue on a common vision of long term outcome. EQ 8: To what extent the following Judgement criteria cross-cutting issues Cross-cutting issues were specifically identified have been taken into account and considered in the design and implementation in the design and implementation of ECHO and DIPECHO of ECHO/ DIPECHO strategies and activities. strategies and activities? Children Effects on environment Women Handicapped people Community networks in project implementation Participation EQ9: Which elements in the institutional context and in the relations between the different actors have had a higher impact in the effectiveness and scaling up of the DRR interventions? EQ10: To what extent have the project activities and results had lasting effects after hand over? (in particular assess whether national authorities, EU cooperation instruments and other external development assistance have provided support to beneficiaries for a DIPECHO exit) All stakeholders, including vulnerable groups, Government at all levels, private sector, civil society and community groups have been consulted. Indicators To what extend are specific vulnerability criteria taken into account in ECHO funded projects? To what extend are project planning and implementation widely consulted and became a participatory process at local, sub-national and national levels? Number of actions which focus on specific crosscutting issues. Judgement criteria Factors identified by stakeholders or evaluators contributing to the success (lack of) of projects. Indicators Analysis and presentation of success and failures stories. Strengths and weaknesses of GoI partners. Level of commitment and available resources of stakeholders. Judgement criteria Level of sustainability of ECHO-funded projects. Programmes and actions have caused anticipated or unanticipated long term effects. Support provided by other actors to facilitate ECHO exit strategy Indicators Extent to which action and decisions have an effective sustainability strategy. Extent to which stakeholders or beneficiaries identify a long term impact after termination of ECHO support and projects link with other Desk review Semistructured interviews Site visits Case studies Desk review Semistructured interviews Site visits Case studies Desk review Semistructured interviews Site visits Case studies Evaluation report Particip GmbH 64

72 Primary EQs Judgement criteria and indicators Tools donor initiatives. To what extend national authorities, EU Cooperation instruments and the other external development assistance have provided support to beneficiaries for a DIPECHO exit strategy and/or continuation of DIPECHO programmes and activities. Evaluation Team roles Particip contracted a team of consultants. Initially, two senior international evaluators, Mr Philip Buckle (team leader) and Mr John Ievers (team member), were assisted by two Indonesian consultants, Ms Hening Parlan and Mr Joseph Viandrito. Dr. De Ville De Goyet (final team leader) led the second mission to Jakarta, with Hening Parlan as well as interviews in Bangkok and Brussels (with John Ievers). Mr. De Ville De Goyet led in writing the final report in October. Mr. John Ievers led in desk review, preliminary analysis and desk report, field methodology, and presentations for final field feedback workshop. He produced the field methodology guidelines, guide questions and desk report. Mr. Ievers led field trips to West and Central Java as well as Yogyakarta, with Joseph Viandrito. Philip Buckle (initial team Leader) led in project administration and scheduling. He designed the online survey and delivered the inception and field reports. The desk report was drafted by Mr. Ievers, reviewed by Mr. Buckle. Mr. Buckle led interviews for team 1 with GoI, donors, INGOs, NGOs, local Government and community members. He undertook interviews in Bangkok and with EU in Brussels. Led visits to Jakarta, NTT and West Sumatra with Hening Parlan. Led in drafting inception and field reports. Hening Parlan (national consultant) logistic support, local meeting arrangements, translation, document acquisition, interviews with GoI, donors, INGOs, NGOs, local Government and community members. Joseph Viandrito (national consultant) logistic support, local meeting arrangements, translation, document acquisition, interviews with GoI, donors, INGOs, NGOs, local Government and community members. Limitations Given the size of the country, travel time was considerable and the field visits required flights and then road transport, or, in Central Java, long periods of land transport. A significant portion of time was set aside for travel. This was compensated for in part by the team splitting into sub teams, this enabled the coverage of a larger area and a greater number of site visits. Travel time within Jakarta was both long (often very long) and unpredictable due to very heavy traffic flows. The team thus split into two and planned for 2-3 meetings per day. On a number of occasions there were no staff remaining in agencies that had first-hand knowledge of their own agencies project activities. Therefore, the staff met had to refer to their own files. Local corporate memory was sometimes weak. Evaluation report Particip GmbH 65

73 Semi-Structured Field Questionnaire Semi-structured interviews Interviews were based on a check list of guideline questions Although all guideline questions have been asked during most interviews albeit with the possibility of varying order the focus occasionally centred on specific issues requiring triangulation or of special relevance to the unique role or expertise of each interlocutor. The interview process remained flexible. This kind of flexibility allowed the interviewee to describe events, observations and issues in personal terms and offer the evaluators new avenues for reflexion. Key GUIDE questions for semi-structured interviews 1) What do you think are ECHO/DIPECHO main contribution to humanitarian aid and Disaster risk reduction in Indonesia? a) The aim of this question is to explore relevance and complementary. b) Prompts include LRRD, PDNA, CBDRR, Scaling up and replicating DRR initiatives, Humanitarian reform/cluster, Setting DRR agenda etc. c) Explore how ECHO/DIPECHO differs from other Donors (and other EU instruments) to establish ECHO footprint d) Explore how ECHO/DIPECHO programs complement and work with other donor programs or EU instruments e) How have key cross cutting issues being incorporated into the design of programs priorities - How has this being influenced by DIPECHO? Prompts Children; Effects on environment; Women; Handicapped people; Community networks in project implementation; Participation. 2) How do you think ECHO/DIPECHO s strategy and priorities contributed to government / national / organizational priority setting and visa-versa? a) How have ECHO funded activities contributed to government priority settings? Prompt at (sub) national level / (line) ministry or mainstreaming in planning b) This should explore links over the timeline of how ECHO has evolved with government (complementarity and impact at a strategic level) question varies according to interviewee c) How did this sharing take place (the methods and depth) d) How or what mechanisms did ECHO / DIPECHO, Partners and government use to share or align priorities? Prompt - Participation support to development of NAP, SNAP; Workshops e) What tools/information was used for national/echo priority settings - prompt (Geographic) vulnerability index or risk assessment; Social vulnerability assessment (slum, gender ); Government priorities / NAP / SNAP; National forum workshops; PDNA or other post disaster (sector) assessment f) Do you think there are vulnerable areas or categories of people who needed assistance but did not get it? Prompt Should ECHO have a role here, who meet or did not meet these needs); If unmet is it a systemic or political issue g) What were the main contributions of ECHO in developing policies for humanitarian AID and Resilience (DRR)? h) Prompt if needed LRRD; PDNA; what opportunities could ECHO have taken? Evaluation report Particip GmbH 66

74 3) How do you think ECHO funded activities contributed to reinforcing sub-national response capacities? a) Prompt activities campaigns, IEC materials, CBDRR processes, and contingency planning processes. b) Have ECHO funded preparedness (DIPECHO or LRRD) activities being replicated elsewhere c) Are plans updated locally d) Do you have a strategy to link relief, recovery, rehabilitation and development? How influenced is this by ECHO? e) Prompt links to institutions and hand-over; links to development policies and plans; link to DIPECHO; use of HVCA or similar? f) Is there a demand for good practice in re-enforcing sub national response capacities? g) How has this demand being met / good practice shared h) Are ECHO products used? How? 4) What ECHO/DIPECHO funded activities are or should be replicated or scaled-up? a) What ECHO supported actions were replicated (or you think should be replicated)? b) What worked and what did not? c) What are/were constraints or opportunities? Prompt if needed CBDRR; Mainstreaming DRR; LRRD / PDNA d) What requests were made for replication / scale-up by who 5) What do you think are the main opportunities and challenges to scaling up or replication of DRR interventions? How have the changes in Institutional context helped or hindered? What other changes need to happen? a) Prompt National law / NAP (both) / SNAP; ADDMER / regional changes; BNBP and sub-national; Finance/policy/institutional/political b) What level should be targeted for replication / scale-up c) Was Advocacy successful 6) Do you think interventions will continue after financing stops? What aspects are most likely to continue and why? a) Prompt - policy framework; institutional framework; financial framework; political framework support b) How have other EU instruments or ODA assisted in sustainability/continuation 7) Should the commission continue with humanitarian and DRR interventions in Indonesia? a) What form (or opportunities) should that take? b) How should the existing DIPECHO initiatives be supported / sustained or continued? If not what is a good exit strategy c) Do you think ECHO (emergency) should remain presence in Indonesia or better to have a focal point(s) in the region? What is the added value of ECHO having a presence in Indonesia? d) How should good practice be best disseminated and replicated? (Based on what has and has not worked) Evaluation report Particip GmbH 67

75 e) How should ECHO and DIPECHO better work with other instruments (commission and bi/multi lateral) f) How could EU civil protection be of added value to ECHO in the region Programme and outcomes of the final field workshop Date: 8 May 2013 Time: Venue: Purpose: Puri Asri 2 Le Meridian Hotel, Jalan Jenderal Sudirman, Kav 18 20, Jakarta To engage with key stakeholders by: 1. Setting out the work programme of the evaluation team in Indonesia 2. Describing the provisional findings regarding ECHO s activities Discussing the findings and other relevant issues 4. To provide advice to the evaluation team on further work and issues Programme: Registration Participants: Welcome and Introduction Philip Buckle, Team Leader the evaluation purpose and methodology Philip Buckle the Evaluation teams work programme in Indonesia (and the rationale for this) John Ievers (Senior Evaluator) Provisional Findings Team Tea Break Plenary discussion lead by John Ievers and Philip Buckle Summation, Thanks and Close Philip Buckle Philip Buckle (Team Leader) John Ievers (Consultant) Hening Parlan (Consultant) Joseph Viandrito (Consultant) Bharat Pathak, Program Director for DRR and Climate Change, Mercy Corps Farah Rachmad National Representatives Coordinator Médecins du Monde (MdM) Linda Rupidara Programme Officer ECHO Alex Robinson Country Director Indonesia Arbeiter-Samariter-Bund (ASB) Santi Dharmawan Development Programme Coordinator NZAID Peter Guest Deputy Country Director UN World Food Programme Jeong Park Disaster Management Adviser AUSAID Angela Staiger Senior Project Development Officer International Organization for Migration Evaluation report Particip GmbH 68

76 Nasrullah DM officer German Red Cross Fabrice Carbonne Country Director Action Contre la Faim Ade Reno Sudiarno Emergency Officer Oxfam Indonesia Kartika Juwita Program Manager The Netherland Red Cross Lina Sofiani Emergency Officer UNICEF The field workshop brought together key stakeholders from NGOs, UN and donors. The (dis) agreement with, understanding of and prioritisation of initial findings and conclusions were tested. The participants brought clarity on areas of particular importance or complexity. It highlighted that in the context of Indonesia a significant amount of money is available from national and international sources. The availability of money, from all sources, was regarded as less important than the capacity to spend that money in an efficient and effective manner. The group expressed that ECHO had a relatively unique role as a consistent advocate for needs and community based approaches. The group compared ECHO, with New Zealand Aid, both relatively small members of the top five DRR donors in Indonesia. New Zealand s Aid has a defined niche working at national and strategic levels. The participants also clarified that since 2007 Indonesia has made significant advances in the disaster management and reduction but significant weakness remain. The group highlighted as important weakness in capacity especially in disaster risk reduction at sub-national level. Similarly the group focused on the large scope of work needed to increase disaster management capacity at sub-national a level in a country as large and diverse as Indonesia. It highlighted the critical importance of working with government to achieve impact at scale from communitybased models. The group also highlighted that Indonesia faces significant challenges in DRR regarding the ongoing process decentralisation. Concerns were expressed regarding a changing risk profile noting slow-on set disasters, climate change and that frequent small disasters are under-recorded. Evaluation report Particip GmbH 69

77 Annex C List of Agencies and Sites Visited, incl. Persons Interviewed Mission in April/May Day/Date Venue Meeting with Title Organization 16 April ECHO Office Bangkok David Verboom ECHO 20 April Joelle Goire ECHO Office AmelieYan- Bangkok Gouiffes ECHO Country Director Paul Jeffery Director of Monday 22 Program Mercy Corps Erin Carter, MPH April Program Director Bharat Pathak for DRR and Climate Tuesday, 23 April c/o The Johanniter International, 7th floor, Menara Kartika F 701 Compound of Kartika Chandra Hotel, Jalan Jend. Gatot Subroto Kav Jakarta Selatan UN FAO office MenaraThamrin Building 7th floor, Jalan M.H. Thamrin, Kav. 3Jakarta Telephone: EU/ECHO Office Intiland Tower, 16th floor Jl. Jend. Sudirman 32 Jakarta Indonesia Tel: Isabel Simpson Farah Rachmad Dr. Mustafa Imir Dr.Ageng S. Herianto Linda Rupidara Change Coordinator Health Program National Representatives Coordinator Assistant FAO Representation (Programme) Programme Officer Médecins du Monde (MdM) the Netherlands Médecins du Monde (MdM) FAO Com- European mission Country Representative Coffee Bean, Pacific Place, Dr. Alex Robinson Country Director Indonesia Samariter-Bund) ASB (Arbeiter- SCBD, Jakarta New Zealand Kirk Yates Development NZ Embassy & Evaluation report Particip GmbH 70

78 Day/Date Venue Meeting with Title Organization Wednesday, Embassy Counselor NZAID 24 April Sentral Senayan Firliana Purwanti 2, Floor 10 Senior Development Jl. Asia Afrika Pro- No. 8, Gelora gram Coordinator, Bung Karno, NZAID Jakarta Pusat WFP Office - Wisma Keiai, 9th Deputy Country Director Programme UN World Food floor, JalanJend. Peter Guest Sudirman Kav. 3, Jakarta Starbucks Coffee Australian Government Shop at Setiabudi Over- Thursday, One, close to Jeong Park DM Adviser seas Aid Programme 25 April Australian Embassy AU- SAID First Secretary, EU/ECHO Office Head of Regional & Eco- Intiland Tower, Delegation of the 16th floor, Jl. Andreas Rottger nomic Cooperation EU to Indonesia, Jend. Sudirman Brunei and 32, Jakarta Muamar Vebry Project Officer ASEAN Indonesia. Tel: + Post Disaster and Reconstruction Friday, 26 April ACF Indonesia office. Kompleks D'Best, Blok J No. 36 (Jl. Fatmawati Raya No. 15), Jakarta Selatan. IFRC Office Jalan Wijaya I No. 63, Kebayoran Baru, Jakarta Selatan, Indonesia IOM office, Sampoerna Strategic Square North Tower Floor 12A, Jl. Jend. Sudirman Kav.45-46, Jakarta 12930, Indone- Rayendra Thayeb Martius Marzuki Dr. Enhard Bauer Nasrullah Ingeborg Ponne Denis Nihill Angela Staiger Assistant Country Representative Ex Program Manager Country Director DM officer Country Director Chief of Mission Senior Project Development Officer Action Contre la Faim German Cross German Cross Red Red The Netherland Red Cross IOM Evaluation report Particip GmbH 71

79 Day/Date Venue Meeting with Title Organization sia Tel: USAID office Meeting room 1741, Gedung Sarana Jaya, Jl. Budi Kemuliaan 1, Jakarta The American Club, Jl.Brawijaya IV #20 KB, Jakarta Harlan Hale Helen Vanwel Wouter Fabrice Carbonne Rayendra Thayeb II. Senior Regional Adviser Country Director Country Director Country Director Assistant Country Representative Ex Program Manager USAID CARE International Indonesia CARE Netherland Action Contre la Faim Saturday, 27 April FIELD VISIT to Project ACF at Cipinang Besar Utara, East Jakarta Community at Cipinang Besar Utara (CBU) village, Jakarta: Rahmat Junaidi Sakyo Idris Darusman Untung Etc. (Total 11 people) Subardi Maria SriHastuti SriSuhartini Etc. (Total 15 people) SitiAminah Budi NurulIman Head of DRR Task Force DRR Task Force Team DRR Task Force Team DRR Task Force Team DRR Task Force Team DRR Task Force Team DRR Task Force Team Head of School Teacher Teacher Teacher Teacher CipinangBesar Utara (CBU) DRR Task Force. SD (Elementary School ) No. 11 & 12, Cipinang- Besar Utara village Youth Group at RW 12, CipinangBesar Utara village Women Group at Evaluation report Particip GmbH 72

80 Day/Date Venue Meeting with Title Organization Husni RW 12, Cipinang Alwi Youth leader Besar Utara village Youth leader Youth leader Youth leader Youth leader Monday, 29 April Tuesday, 30 April Kementrian Daerah Tertinggal (Ministry of Neglected Areas) Office BNPB Office, Jln. Tanah Abang Jakarta Pusat PMI Office, Jln. GatotSubroto, Jakarta Selatan OCHA office, MenaraThamrin, 10th floor, Jl. Thamrin 3, Jakarta Oxfam Country Office Jl. Taman Margasatwa No 26,Ragunan, Jakarta World Bank Office, Tower 2 Pacific Place Maimunah Wati Jumiati Karsi Etc. (total 13 people) Dr. Suprayogahadi Ir. Sugeng Tri Utomo DESS Bevita Rajan Gengaje Titi Moektijasih Richard Mawer S. Srikaran Nanang Subana Dirja Ade Reno Sudiarno Iwan PhD. Gunawan Women leader Women leader Women leader Women leader Women leader Vice Minister Deputy 1 Program Manager Indonesian Red Cross (PMI) Head of OCHA Humanitarian Affairs Analyst Country Director Programme Development and Funding Coordinator DRR Manager Emergency Officer Senior Disaster Management Adviser - Ministry of Neglected Areas (KPDT) National Agency for Disaster Management (BNPB) PMI (Indonesian Red Cross) UN OCHA Oxfam Indonesia Senior Disaster Management Adviser - World Evaluation report Particip GmbH 73

81 Day/Date Venue Meeting with Title Organization RubbyMangunsong World Bank Bank Program ficer UNDP Office, Thamrin Building, Jhn. MH. Thamrin Jln. Teratai, No. 11 Kupang Telp WFP Sub Office Kupang, Jln. S.K. Lerik No. 15 A, Kelurahan Kelapa Lima, Walikota, Kupang Indonesia Telp Telephone : UN FAO Kupang Office Jln. Sam Ratulangi No. 3, Kupang, NTT Indonesia Telp/Fax : KristantoSinandang Drs.Tinitadeus Magdalena Lany Dandy Rahman Nunuk Supraptinah Blasius Agus Rekso Of- Head of Crisis Prevention and Recovery Unit Head BPPD, Province of NTT Mitigation and Preparedness staff at BPPD Programme assistance for Food For Work Di- Program rector Senior Programme Assistance Program Manager FAO 2011 Graphic, Information and Media assistance Program Officer Head of Crisis Recovery and Prevention Unit (CRPU) UNDP BPPD Nusa Timur Province Tenggara WFP Office Staff Kupang, NTT FAO Staff BPBD Garut Office, West Java Health Centre (Puskesmas) Cipanas, Garut, West Java Ruslan Anas Aolia Malik Atik Rahmat Agus Dinar Secretary of BPBD Garut Head of Prevention and Preparedness, BPBD Garut Head of Health Centre (Puskesmas) Cipanas, Garut BPBD West Java BPBD West Java Garut, Garut, Health Centre (Puskesmas) Cipanas, Garut District Health Department, Garut Evaluation report Particip GmbH 74

82 Day/Date Venue Meeting with Title Organization Officer at District Health Department, Garut Vice Bupati (Head of District) Wednesday, 1 May Sindang Sari Restaurant, Ciamis, West Java Parsley Restaurant, Yogyakarta Iing S. Arifin Dicky Erwin Erlan Suherlan Arie Ahmad S. Nyoto Khadijah Rani Rachmat Sutrisna Muntari Pujaningsih MPd Dr. Haryanto Head of BPBD Ciamis Head of Prevention and Preparedness. BPBD staff, Ciamis BPBD staff, Ciamis Badan Keluarga Berencana Dan Pemberdayaan Masyarakat (BKBPM) staff, Ciamis Bappeda staff, Ciamis Bappeda staff, Ciamis Beneficiary Beneficiary Beneficiary Lecturer Lecturer Ciamis District Office, West Java BPBD Ciamis, West Java BKBPM Ciamis Bappeda Ciamis Yogyakarta State University Sugeng local government officer Dept. of Education, Yogyakarta Thursday, 2 May SD Pelem Dlingo, Gunung Kidul, Yogyakarta. Joglo Restaurant, Wonosari, Gunung Kidul, Yogyakarta Surahmi Harto Drajat Martiono Kardi Budhi Hardjo Nugraha Irfan Ratnadi Purwanto Head School Teacher Teacher Teacher Teacher of Head of BPBD Sub-unit Head of BPBD SD Pelem Dlingo, Gunung Kidul, Yogyakarta (elementary school) BPBD Gunung Kidul, Yogyakarta Bupati office Evaluation report Particip GmbH 75

83 Praditya Alex Robinson Day/Date Venue Meeting with Title Organization District Head Social Department Sutrisno office Social Department Sunarti, Muryani, of- Nglipar village, Harsi, Sumiati, ficer Nglipar Village Gunung Kidul, Ngatini, etc. (totally Yogyakarta. 24 people) Cadre/beneficiaries Head of Village at Nglipar village Cadre/beneficiari es ASB ASB staff ASB Country Director Community Leader in Patay Friday, 3 May 2013 Field Visit to WFP Community in Patay village, Kupang, NTT Welhamus Mena Nomleni Lexi Tamonop Romana Wife of Community Leader Member Team Community of of WFP Community Partner at Patay Village (4 peoples). Community in Sulamu, Kupang, NTT Dinner Meeting with NGO s at NTT Mama Fons Papa Fons Muhammad Mela Bano Raymond us Yus Nakmofa Dr. Jonathan Lassa Irawan Community active at WFP project Community champion on Livelihood Leader of Community Farmer Farmer Expert of DRR in Kupang Director of Local NGO in Kupang FAO Community Partner at Patay Village (4 peoples). Local NGO s in Kupang, NTT Evaluation report Particip GmbH 76

84 Day/Date Venue Meeting with Title Organization Head of Study and Research Policy Community Organizer Handicap International office, Catherine Gillet gramme Direc- Regional Pro- Handicap International Yogyakarta tor BPBD Sleman office Karina/Cordaid office, Yogyakarta Diana Setiawati Taufiq Wahyudi Asih Kushartati Dr. Trias Aditya Aris Sudiyono National Project Manager Head of BPBD Head of Sub- Unit BPBD Head of Participatory mapping and Planning Team Core staff IOM Yogyakarta BPBD Sleman BPBD Sleman University of Gajah Mada DRR Forum, Yogyakarta Anat Prag Representative CORDAID Coord Logistic Sunday 5 May 2013 Filed Visit to West Sumatra Nurhayati Edrizaldi Vita Karmila Preparedness Team Communication Team Community at Air Manis, Padang, West Sumatera Rudi Dalius Dr. Euyet Nazmar Preparedness Team Leaders of Mosque Economic Expert (6 persons) Government Padang City of Monday 6 May 2013 Local Government at Padang City,Jln. Prof. Yamin No. 70 Padang Telp (0751) Drs. Dedi Henidal MM Yunisman Fuji Asrtomi Supriyanto Igusnofaldi Law and Politic Expert Secretary of BPPD Padang City Mitigation and Preparedness BPPD Padang City - MERCY Padang Sumatera Corp, West Evaluation report Particip GmbH 77

85 Day/Date Venue Meeting with Title Organization READY Program Manager 8 May Jakarta MERCY Corp Staff Kompleks Puri Asri A 6 RT 01 RW 07, Kelurahan Ampang, Kuranji Padang Sumatera Barat Supriyanto Igusnofaldi ACF ASB AUSAID CARE International German Red Cross IOM Medecine du Monde Mercy Corps NLRC NZAid Oxfam Unicef WFP ECHO - Observer READY Program Officer READY Program Manager READY Program Officer MERCY Corp Evaluation report Particip GmbH 78

86 Mission in September Day/Date Meeting Venue Person, Title Organization Contact Jakarta Dr. ACHMAD HARIJADI Expert of DKI Jakarta Hotel Morissey Former Head of Bappeda (Planning and Budgeting Board) DKI Jakarta Sunday, 15 September 2013 Monday, 16 September 2013 Tuesday, 17 September 2013 Wednesday, 18 September 2013 Jln. Wachid Hasyim Jakarta Pusat Hotel Morissey In the part of AHA Centre Meeting EU/ECHO Office Intiland Tower, 16th floor, Jl. Jend. Sudirman 32, Jakarta Indonesia. Graha 55 Building Jln. Tanah Abang II No. 57 jakarta INDONESIA Hotel Le - Meredian NZ Embassy & NZAID New Zealand Embassy SentralSenayan 2, Floor 10 Jl Asia Afrika No 8 Gelora Bung Karno Jakarta Pusat Setia Budi Building, Kuningan, Jakarta ERNIE WIDIANTY R. MD. M.Kes, MBA. - (Nessy) Expert of Province DKI Jakarta ADELINA KAMAL Head, Disaster Management and Humanitarian Assistance, Division Socio Asean Cultural Community (ASCC) Asean Secretariat LINDA RUPEDARA Programme Officer Ir. SUGENG TRI UTOMO, DESS Former Deputy Mitigation and Preparedness BNPB WAYNE ULRICH Community Safety and Resilience Coordinator IFRC Indonesia Country Office FILLIANA PURWANTI Senior Development Program Coordinator, NZAID PHILIP HEWIT Development Counselor New Zealand Aid Programme JEONG PARK DM Adviser Australian Government achmad.harjadi@gmail.com ernie.ness@gmail.com, ernieness@cbn.net.id lina@asean.org Linda- Ivonne.Rupidara@echofield.eu Telp , , Fax ; Mobile: sriutomo@hotmail.com Mobile : wayne.ulrich@ifrc.org Skype wayne.ulrich T : Firliana.purwanti@mf at.govt.nz T : M : Philip.hewit@mfat.govt.nz Jeong.park@ausaid.gov.au Evaluation report Particip GmbH 79

87 Day/Date Meeting Venue Person, Title Organization Contact Pusat. Overseas Aid Programme AUSAID Wisma Harapan Kita Jakarta Pusat BEVITA Program Manager PMI (Indonesian Red Cross) bevitad@yahoo.com Thursday, 19 september 2013 Friday, 20 September 2013 Ministry of Neglected Areas (KPDT) Kementrian Daerah Tertinggal (Ministry of Neglected Areas) Office Jl. Abdul Muis No. 7 Jakarta Cemara 2 Hotel, Jln. Wachid Hasyim Jakarta Pusat Cemara 2 Hotel, Jln. Wachid Hasyim Jakarta Pusat BNPB Graha 55 Building Jln. Tanah Abang II No. 57 jakarta INDONESIA EU/ECHO Office Intiland Tower, 16th floor, Jl. Jend. Sudirman 32, Jakarta Indonesia. Tel: OXFAM Office Jln. Margasatwa 26 Jakarta Selatan Dr. Ir. SUPRAYOGA HADI, MSP Deputy Minister for the Development of Special Regions BRUNO MAESTRACCI Team leader ASEAN Crisis Centres DEVCO project ALEX KESPET Knowledge Management expert TITI MOEKTIJASIH Humanitarian Affairs Analyst United Nations Office for the Resident Coordinator/Humanitarian Coordinator Menara Thamrin, 10th floor Jl. M.H. Thamrin no. 3 Jakarta 10250, Indonesia Ir. DODI RUSWANDI, MSC Deputy Head for Prevention and Preparedness MUAMAR VEBRY Project Officer Post Disaster and Reconstruction RICHARD MAWER Country Director Oxfam in Indonesia Jakarta T : M : E : yogahadi@gmail.com, supryayoga@kemenegpdt.go.id bruno.maestracci@gmail.com Alex.kespet@gmail.com Telp.: (62-21) ext. 107 Mobile: (62-811) Fax: (62-21) moektijasih@un.org Phone : Facsimile : dodyruswandi@gmail.com Mobile : muamar.vebry@eeas.europa.eu Telp ext 201; Mobile skype: r.mawer Evaluation report Particip GmbH 80

88 Day/Date Meeting Venue Person, Title Organization Contact nesia Saturday, 21 September 2013 Sunday, 22 September 2013 Monday, 23 September 2013 Wednesday 25 September 2013 Thursday 26 Kampung Melayu RT 14/RW 02 Jakarta Timur DKI Jakarta Breakfast Meeting Plasa Indonesia EU/ECHO Office Intiland Tower, 16th floor, Jl. Jend. Sudirman 32, Jakarta Indonesia. AHA Centre BPPT 1 st Building 17 th Floor Jln. MH. Thamrin No. 8 Jakarta Hotel Ibis Riverside WFP IFRC Regional Office ADPC (Skype conference) ADE RENO SUDIARNO DRR Coordinator Oxfam in Indonesia Jakarta LINDA RUPEDARA Programme Officer JANGGAM ADHIT- YAWARMA Senior Disaster Monitoring and Analysis Officer Bangkok ORLA MURPHY Regional Humanitarian Manager OXFAM Regional Office PETER GUEST Programme Advisor Regional Bureau for Asia INDIRA KULENOVIC Community Safety & Resilience Unit NGUYEN HUNG HA Programme Manager DRR ASLAM PERWAIS Head, Disaster Risk Management Systems Telp ARSudiarno@oxfam.org.uk Contact person : Jordan : achmad.harjadi@gmail.com SEVEN PEOPLES (All of man) COMMUNITIES KAM- PUNG MELAYU Beneficiaries ACF Project Dr. ACHMAD HARIJADI Expert of DKI Jakarta Former Head of Bappeda (Planning and Budgeting Board) DKI Jakarta ERNIE WIDIANTY R. MD. M.Kes, MBA. - (Nessy) Expert of Province DKI Jakarta ernie.ness@gmail.com, ernieness@cbn.net.id Linda- Ivonne.Rupidara@echofield.eu Telp , , Fax ; Mobile: M : jannggam.adityawarma@a hacentre.org omurphy@oxfam.org.uk +66 (0) peter.guest@wfp.org Indira.kulenovic@ifrc.org Hunga.nguyen@ifrc.org +66 (0) aslam@adpc.net Evaluation report Particip GmbH 81

89 Day/Date Meeting Venue Person, Title Organization Contact HANG THI THANH PHAM ISDR Programme Officer Friday 27 September In addition: ECHO / RSO ROJO GARCIA Pedro Luis Head or RSO GOIRE Joelle TA for Philippines and Indonesia YAN-GOUIFFES Amelie DRR Coordinator TOUCH Thearat DRR Assistant Pedro.luis.rojo@echofield. eu Joelle.goire@echofield.eu Amelie.yangouiffes@echofield.eu Thearat.touch@echofield.eu 6 Health Officials from Jakarta and provinces were interviewed at the HELP course in Japan Focus group discussion with 7 community leaders and volunteers in Jakarta (ACF program) Brussels: Day/Date Meeting Venue Person, Title Organization Contact Anne-Marie Renner Desk Officer for the Philippines and Indonesi, DG Humanitarian ECHO Office Aid and Civil Protection (ECHO), B.5 - Asia, Latin America, Caribbean, Pacific October 1 October 2 European External Action Service (EEAS) Europe aid office, Rue Joseph II ECHO Office Morgan Mc Swiney European External Action Service (EEAS), Asia and Pacific, South East Asia Division Giuliano Porcelli, Crisis Response and Operational Coordination, EEAS Giulio Gentile, Desk officer, Unit H1 - Geographical Coordination for Asia and Pacific, Directorate General for Development and Cooperation - EuropeAid Yves Dussart Prevention, Preparedness and Disaster Risk Reduction. DG Humanitarian aid and Civil Protection ECHO- A Civil Protection Policy, annemarie.renner@ec.europa.eu EEAS/ CORT, 03/87, Rue de la Loi, 242, 1046 Brussels Belgium Tel giuliano.porcelli@eeas.europa.e u European Commission Rue Joseph II - 54, 7th floor, Office No. 51 Tel Tel.: yves.dussart@ec.europa.eu Evaluation report Particip GmbH 82

90 Annex D Summary presentation of individual case study visits IoM in Garut district, West Java Building sub-national capacities from a low starting point In 2007, the Government of Indonesia wrote a new law on disaster management. This law marked a paradigm shift of policy focus from emergency response to disaster risk reduction. The law recognises that to enact the new policy Indonesia needs new institutions, attitudes and knowledge. These institutions were needed at national and sub-national level. Indonesian has over 500 districts, which require district disaster management bodies (BPBD). One such district is Garut in West Java. The national disaster management body recognises Garut district in West Java as one of the most hazard prone districts in Indonesia. Hazards include earthquakes, tsunamis, two active volcanoes, floods, dry periods, landslides and wind storms. Landslides are the most frequent disaster in Garut. Historically, district government focused only on building capacity to response to these hazards. In September 2009, an earthquake struck Java Island causing widespread damage in Garut and other districts. This came at a time when, Garut district was struggling to establish its new district disaster management agency. ECHO funded IoM to provide temporary shelter and physiological support to those most in need. IoM also recognised the need to invest in local government capacity. Thus, IoM proposed and was awarded its first DIPECHO action in Indonesia. The action focused on improving response and disaster risk reduction capacity of the newly formed district disaster management bodies (BPBD). This involved risk mapping, training and simulation at district level as well as in six villages. In addition, collaborating with the Indonesian Red Cross (PMI), the action provided emergency response, disaster management and Basic Trauma Life Support training to the local health sector. The action was challenged by the low knowledge and limited experience of staff within the newly formed BPBD. The new body was formed in the offices and with staff from a recently closed narcotics agency. Few staff had experience of disaster management, even fewer were aware of basic disaster management or risk reduction concepts. Few institutions such as local health clinics had disaster management plans or training. However, the 2007 law and the 2009 earthquake provided an opportunity to start the process of changing attitudes and institutions to better manage and reduce risk from hazards. The relatively new legal and institutional environment provided some basic guidance to start this process of change. International and regional manuals and processes of planning and simulations existed within the Red Cross structure and other local NGOs. IoM contracted Red Cross and a well respected local NGO, IDEA within this action. The action used existing curriculum for training of trainers in disaster management of local health institutions. At the end of the action, the district disaster management body had simulated and updated standard operating procedures for specific hazard. The action with local government also produced hazard maps to better understand and prioritise disaster risk reduction actions. Six local communities developed and simulated local response and disaster risk reduction plans for some hazards (volcano s). In addition, local health centres also simulated disaster management plans. Evaluation report Particip GmbH 83

91 These plans were tested in 2011 during flash floods. During the floods, Basic trauma units were formed and resources were mobilised as per the plan. The district disaster management body recognises that improvement has being made. However, it also recognises that low staff capacity continues to limit advancement. In addition, district disaster management agencies were starting to realise the scale of the challenges they face in improving disaster management. The action had effectively provided support for six village structures in risk awareness and developing standard operating procedures and action plans. However, there are 442 villages in the district. During the meeting it was estimated that it would take decades to replicate activities at the required scale, at current resource levels. Garat is but one of well over 500 districts in Indonesia. Staff capacity is an important limiting factor. In addition, the gains in staff capacity were likely to be eroded by a civil service policy of three yearly staff rotations. It is noted that, knowledge gained by the action is more likely to remain in the health sector. In the health sector, as with the Indonesian Red Cross, personal, manuals and training protocols are better established and have a system of support. The actions and government staff also identified key lessons for learning Relatively new sub-national disaster management bodies are limited by low staff capacity, especially in disaster risk reduction. Time and a system of support is needed to build this capacity. Institutional change takes time, but attitude change or creating a culture of safety is more critical and time consuming. Manuals, guidance and processes are available and do not need to be rewritten. Using existing materials and resources (such as the Indonesian red cross, Local NGO and government) improves effectiveness and efficiency. Manuals, guidance and processes already supported by a government agency such as the Basic trauma life support and (health) disaster management are better accepted and come with a system of support (retraining, continued learning, monitoring and guidance) Basic information such as hazard mapping is important for planning but this needs to be linked with budget cycles, resource and support institutions to be more effective. Finance was available for some initiatives but accessing national level resources was difficult due to budget and planning cycles, unclear processes and procedures. Over two thirds of local/district budgets are used for salaries. Ciamis, West Java Linking shelter relief to rehabilitation and development In 2009, earthquakes struck first West Sumatra then a month later West Java. Government, media and non-government actors committed significant resources to the West Sumatra earthquake. After the West Java earthquake, fewer resources were available from national institutions, which were already committed to West Sumatra. Government figures indicate that in West Java upwards of 65,000 houses were severely damaged and over 185,000 people displaced. Government and other humanitarian actors prioritise house rehabilitation. Learning lessons identified in previous disasters, the Government promoted and supported a self-build model for house rehabilitation. National government committed to paying for reconstruction of completely destroyed houses and provincial government paying for the rehabilitation of significantly dam- Evaluation report Particip GmbH 84

92 aged houses. District government committed to pay for a larger number of slightly damaged houses. ECHO responded to the needs in West Java, adding to financing decision for West Sumatra. ASB recognised that housing was a priority. It also recognised that reconstruction of completely damaged houses using the self-build model, especially with resources focused on West Sumatra would take 2-3 years. ASB working closely with the local government and learning from good practice during the 2006 Central Java earthquake proposed a temporary housing solution. The solution involved providing cost efficient and fast to build earthquake resilient Bamboo houses whilst people wait for longer term housing funds and construction. Bamboo houses have a lower life expectancy than the preferred concrete block houses which were largely destroyed. In addition, ASB focused on raising awareness on resilient building techniques within the majority selfbuild population as well as local government. Three years later, national government had dispersed IDR 15 Mio (about 1000) to households to support reconstruction. While waiting for this grant and reconstruction, people lived in the Bamboo houses. The bamboo houses cost the project a little over 300 using mostly locally sourced and renewable materials, or about 100 per year of occupation. Families that received the bamboo shelter received multiple benefits. Firstly, they were able to have basic shelter whilst awaiting permanent housing grants. Most families reported that it took between 2 and 3 years to receive the grant and build their house. Secondly, after building their house they used the Bamboo shelter as extra rooms, an extension, storage or premises for a small shop/business. Thirdly, when questioned, house owners could after three years cite safe construction techniques for earthquake resilience using in building the bamboo shelter. People also reported using this knowledge when building their own house. However, a few of these families elected to remain in the bamboo shelter, and use the government grant to pay off their debts or invest in their children s education or businesses rather than build a concrete house. Families who built permanent homes did so at their own pace with the Government grants as well as money sent back to them from family members who worked abroad or in Indonesian cities. People reported that the use of Bamboo shelters provided them with the time to recover, and make key decisions. Additionally, when questioned people and government knew and had applied some key principles of earthquake resilience promoted by the project. This action highlights a number of good lessons The humanitarian and recovery environment in parts of Indonesia is becoming more predictable from one similar disaster to another. This increases predictability. Partners can thus collaborate and have better linkages of relief to reconstruction and development providing gains in effectiveness and efficiency. Actions which allow people to make their own decisions, such as providing basic shelter while waiting for self-build housing reconstruction grants, allow people to better adjust to their specific needs. Actions, which build on previous learning s, can increase effectiveness and efficiency. This is done by using or improving successful strategies, awareness raising materials already existing and locally sources materials and resilient construction techniques. The use of Bamboo shelter over for example tents provided addition benefits to people. After, building their permanent house they were able to use the bamboo shelter as extra rooms, storage or business premises. Evaluation report Particip GmbH 85

93 Cross-cutting issue Mainstreaming disaster preparedness focusing on children with disabilities In Indonesia, children with disabilities are often excluded from society, schools and disaster preparedness activities. Barriers include social stigma, especially for intellectually disabled, as well as legal, regulatory, physical and communication barriers. Deaf, blind and wheelchair bound children may not be able to hear, read or access schools and disaster preparedness messages. Children with intellectual disabilities are often hidden from society, receiving less attention during disasters due to stigma felt by family. The Government of Indonesia is making efforts to address the needs of this highly vulnerable group. One such effort is in the principal of education for all including persons with disability. This principal is articulated in a 2003 education law and reinforced by a 2009 ministerial decree on inclusive education services. Additionally, in 2011, Indonesia ratified the convention of the rights of persons with disabilities. These significant changes at the nation level are challenged to evoke changes at local level. ASB received two DIPECHO actions in 2008 and 2010 to focus on children with disabilities in Yogyakarta. The initial actions sought to widen longer-term access to disaster risk reduction information and procedures for disabled children in inclusive schools. The second action focused on disabled children outside the school structure. ASB designed and implemented the actions in close collaboration with the district department of education and later the department of social affairs (for children outside the school structure). The actions were closely intertwined within and designed to adapt existing structures and procedures. The actions included DRR messages including for children with disability in the teacher training curriculum of Yogyakarta State University, for the training existing and future teachers and school principals. Similarly, The action adapted existing manuals and guidance on school infrastructure for school principals and inspectors. In addition, monitoring and supervision structures in both education and social affairs were enhanced with key disaster and inclusive education messages. This provided a regular forum for the discussion of DRR and children with disabilities. Outreach workers (cadres) which work with the department of social affairs (and health) and community leaders were utilised to widen longer-term access to DRR information and procedures for children with disabilities outside of formal education The actions resulted in mainstreaming tangible of aspects of disaster preparedness for children with disabilities in both ministries. Attitude changes within communities were noted towards children with disabilities. Most children with disabilities were able to go to school. Communities and teachers reported local campaigns including with local politicians for children with disabilities to access high schools as well as primary schools. The projects provided working models for district and provincial government to inform local regulations, provincial and district education strategies. This model provides a structure and system, which could be, and is being replicated by the ministries in other communities and districts. In addition, ASB worked with the parliament and planning and development authority on the 2011 to 2015 education strategic plan and budget allocations. Thanks in part to this co-funded project, other districts in Yogyakarta are have already or are working on new regulations, plans and resource allocations adapted from this project. Evaluation report Particip GmbH 86

94 Lessons identified for learning Focusing actions within existing institutions, such as universities and the departments of education and social affairs, increases effectiveness, is more efficient and more likely to continue after the action stops. Synchronising actions within existing institutional and regulatory structures can provide models that are more easily adapted elsewhere. Getting local regulatory support and synchronising actions with budget cycles is important for success and continuation. West Sumatra Enhancing Community Based Disaster Management and Capacity Building In 2007 Indonesian passed a new law on disaster management. This law recognised the need for a paradigm shift from disaster response to reducing risk from disasters and preparing for disasters. The law highlighted the important role of village, district and provincial disaster management bodies. These disaster management bodies needed to be formed to make the intension of the law a reality. In September 2009, an earthquake hit West Sumatra, causing widespread damage affecting over one million people in urban and rural areas. The 7.9 (richer scale) magnitude earthquakes damaged or destroyed about 200,000 houses as well as a number of hotels and government building. The damage highlighted the need and value for better disaster risk reduction measures as well as disaster preparedness, echoing the 2007 disaster management law. Mercy corps is a leading and innovative International NGO in Indonesia, since It is part of the emergency capacity building project and lead in establishing rapid assessment and response protocols. In 2009, Mercy corps had 15 projects valued at about 10 Mio across Indonesia. Mercy corps established a presence in West Sumatra in 2008, with a two-year disaster risk reduction projects funded by OFDA and Boeing Corporation. Thus when the earthquake struck, Mercy corps was well positioned with relationships with local government, communities and NGOS. In response to the September 2009 Earthquake, ECHO, along with other donors, supported Mercy corps in a primary emergency decision to provide shelter and water and sanitation support. Building on this and its other work, Mercy corps started its first DIPECHO contract in Indonesia. The action built on Mercy Corps experience and relationships in Agam district and Padang city. The action focused on training staff and developing standard operating procedures and contingency planning. In November 2009, the provincial government established a west Sumatra provincial level disaster management agency, policy and procedures. Following this Padang city and Agam district established their disaster management agencies guided by the 2007 law. The action helped establish and train twenty disaster preparedness teams. Eight teams focused on district and community groups, including women and youth groups. Twelve were Schools Disaster Preparedness Teams (SDPTs). In addition, Mercy corps helped establish 2 taskforce teams partnering community leaders and district authorities. The action focused on 189 training staffs in two districts, in DRR and DRM, along with awareness training in four other districts and the province. Communities and districts established standard operating procedures, (tsunami) evacuation routes and learnt from desk top simulations. Community developed plans and actions for disaster risk reduction. Communities sought finance from local government to make these plans a reality. The government acknowledged these plans but did not support them finan- Evaluation report Particip GmbH 87

95 cially. However, the government supported the formation of village disaster preparedness teams in all 54 villages of Padang city. The action itself, encompassed a number of good practices including the use of local resources in addition to local government structures, for example the use of mosques for early warning systems. Government supported early warning systems in the communities as they had an ongoing early warning system project. After the action, Mercy corps maintained a presence in the area. Additionally, the Government with support from UNDP, PMI and others further enhance the knowledge and models established by the action and established disaster preparedness teams in a reported 115 villages. The establishment of these disaster preparedness teams was based on knowledge gained from this action and the experience and learning of other non-government actors. Learning Coherence: Local government is open to partnering with external actors to establish disaster management bodies at various levels. Coherence: Disaster management actions from this ad similar ECHO funded actions, encompass learning, are based upon and continued by other actors. Local government and organisations connect these often-fragmented initiatives. Sustainability: The community, district and provincial teams need to be sustained within a system of support, which is emerging. Sustainability: The project was greatly aided by the emerging legal and institutional environment but further time is needed. Annex E Map of Areas covered by the Operations Financed under the Action and the Sites visited by the Contractors Map of Areas covered by the Operations Financed under the Action: Site visit locations: Padang, Jakarta, Garut, Ciamis, Yogyakarta and Kupang from East to West Evaluation report Particip GmbH 88