Evaluation Report. Catholic Relief Services Indonesia West Sumatra Transitional Shelter Program

Transcription

1 Evaluation Report Catholic Relief Services Indonesia West Sumatra Transitional Shelter Program 31 May, 2010

2 Table of Contents Executive Summary... 3 Introduction... 5 Evaluation Overview... 6 Overview of Methodology and Limitations... 7 Program Overview... 8 I. Relevance II. Accountability and Appropriateness III. Coordination IV. Effectiveness V. Efficiency VI. Impact Cross-cutting themes Gender Vulnerable Households Environment Conclusions and Recommendations Annex 1: Performance Indicators Tracking Table (PITT) Annex 2: Terms of Reference of the Evaluation Annex 3: Methodology Annex 4: Limitations Annex 5: List of team members Annex 6: Sites visited Annex 7: People and organizations consulted... 40

3 Executive Summary In response to the September 30, 2009 earthquake in West Sumatra, launched an emergency response in partnership with local NGO WALHI (WAhana Lingkungan Hidup Indonesia), to distribute emergency shelter kits to 8,700 affected families. This response was followed in November 2009 with a large-scale transitional shelter (t-shelter) program targeted at 10,000 households with cash grants and technical assistance for building Safe, Adequate / Comfortable, and Durable (S.A.D) transitional shelters. The t-shelter program continued through May 2010, reaching more than 11,000 households in 135 communities/villages in three districts of West Sumatra. The CRS t-shelter program provided a comprehensive package of assistance which began with program introduction meetings in targeted villages, establishment of committees comprised of community volunteers, and a house-to-house damage survey. Based on these activities, a beneficiary list was compiled and posted in a central location in the community, offering households the opportunity to review and challenge beneficiary selection by contacting a CRS feedback hotline or directly addressing their concerns with CRS field staff or community shelter committee members. Approximately one week after beneficiary lists were posted and one or two days before cash distribution, CRS and WALHI engineers gave technical instructional presentations to all beneficiaries and other interested individuals in a given village and provided them with safe construction posters. Cash grants were distributed directly to beneficiaries in the village through an innovative partnership with the Indonesian Post Office (PT Pos). The cash grant value was IDR 2,500,000 (USD $267), disbursed in two tranches. The first tranche provided 80% of the cash grant. Following distribution of the first tranche, CRS 1 field teams monitored progress and provided on-site technical assistance to households, who were expected to complement the cash grant by salvaging materials from their original homes or using other resources. Special attention was paid to vulnerable households in each community. Once the vulnerable beneficiaries had completed their shelters and the whole community had made significant construction progress, all beneficiary households whose shelters were complete or near-complete received the remaining 20% of the cash grant. This report provides an overview of the CRS t-shelter program and presents the findings from a comprehensive program evaluation process which took place during January-April Two separate program evaluations assessed progress toward the program's goals and indicators, and a cash grant learning assessment studied in-depth the efficiency and impact of the cash grant aspects of the program. Based on the findings, the program's achievements are discussed in terms of relevance, accountability and appropriateness, coordination, effectiveness, efficiency and impact. Cross-cutting themes discussed are gender, progress of vulnerable households, and the environment. Relevance: The t-shelter program was in line with local needs and priorities: shelter, and in particular earthquake-resistant shelter that would keep families safe in the event of another earthquake, was considered the most urgent need in post-earthquake West Sumatra. Still, beneficiaries were somewhat reluctant to move out of their unsafe houses, often because of limited understanding of structural safety or else out of concern for securing their assets. The program was well-received by beneficiaries and stimulated active participation in the target communities, and the committees of community volunteers were an appropriate mechanism for implementation of t- shelter activities. The program design promoted ingenuity in t-shelter construction and ensured beneficiary dignity, especially for vulnerable households. Accountability and Appropriateness: The t-shelter program encouraged ownership and accountability from the targeted communities. By working through local partners, especially village shelter committees, CRS's visibility and relationship with the community was ensured in a culturallyappropriate way. Community members were not often aware of CRS' identity, nor of the program's targeting criteria, but they trusted committees and stakeholders to serve as a bridge to the t-shelter 1 The t-shelter program was implemented jointly by CRS and Walhi, a local NGO with a strong presence in West Sumatra that implements primarily environmental programming. All program and field teams referred to as CRS in this report were in fact comprised of both CRS and Walhi staff. 3

4 program. In this way, accountability was ensured through the partners involvement, regular CRS monitoring visits, and informal conversations in the field. This meant that beneficiaries who built their own t-shelter were more likely to be interested in interacting with CRS than those who hired labours. Site visits and technical presentations were highly appreciated. Coordination: CRS worked systematically and in coordination with various stakeholders in West Sumatra to deliver humanitarian assistance in a cohesive and effective manner. Strong lines of collaboration were developed and maintained between CRS, local communities, government stakeholders and the private sector. The partnership with PT Pos was a highlight of coordination, and set a new precedent for partnership between NGO s and the private sector in Indonesia. The shelter cluster coordination group was also particularly valuable, enhancing CRS' influence, credibility and reputation among key stakeholders in West Sumatra and beyond. CRS was active and intentional in coordinating with local authorities, who nonetheless sometimes hoped for more consultation and information-sharing from CRS. Overall, though, the local authorities expressed strong appreciation for the t-shelter program, and for CRS' effectiveness in managing well any issues as they emerged. Effectiveness: The t-shelter program achieved its purpose according to its intended outputs, reaching more than 11,000 households, more than the original target of 10,000, in the months following the earthquake. The program design was successful. The cash grant amount was appropriate for fulfilling the program goals; the use of salvaged materials helped people build their shelters; and the approach of providing the cash grant in two tranches, the second of which was conditional upon community completion and individual progress, was ideal. The full-completion standard, in which individual progress was assessed as well as community progress, kept beneficiary accountability high and led to high rates of success. Effectiveness improved over time, as Spherecompliance of shelters built, completion rates, and speed of construction all improved between the mid-term evaluation and the final evaluation. This is partially due to the program design which served as a jump-start for beneficiaries to rebuild their lives and livelihoods, as well as to the flexibility of the program team to adapt the implementation guidelines according to feedback from the field and monitoring data. Efficiency: The t-shelter program achieved a high level of success with a comparatively low number of inputs. It was cost-effective, timely, and easily understood and supported by beneficiaries. One of the keys to the program's success was the simplicity of its goals and methodology, making expectations clear, both for CRS to implement and community members to support the program. The participation of PT Pos also contributed significantly to program efficiency by managing the cash and freeing up the field teams to focus on community engagement. Delays, when they happened, were generally caused by issues that emerged within the targeted communities such as shortages of available labour and some materials. The CRS teams were seen as being highly productive and prompt. Communication between CRS and traders was limited but beneficiaries themselves were active in negotiation and planning in order to achieve optimal completion of their own shelters. Impact: The CRS t-shelter program stimulated and empowered community members, including vulnerable households and women, to take charge of their own recovery and helped speed their return to normal life. It provided an increased feeling of safety and security to beneficiaries, and accelerated the process of economic recovery in the region. The t-shelter program operated effectively and appropriately within the existing economic structure of the region to facilitate recovery at the community level. A very small degree of community conflict surrounding receipt of assistance or cash and related issues of social jealousy occurred but was quickly resolved. The cash grant programming provided an economic stimulus to households, skilled labourers and traders of construction materials. Cross-cutting: Women were engaged in all phases of the program and were decision-makers in the design and construction of their shelters. Vulnerable households had better completion rates than average. They usually hired skilled labour or received assistance from family, as opposed to benefiting from community support. They built their shelters depending on culturally-appropriate forms of assistance, their own initiative, and the support of the CRS field teams. There was a high level of awareness about environmental issues and beneficiaries made decisions with environmental concerns in mind. 4

5 Introduction: Background and Context On September 30, 2009, an earthquake with magnitude of 7.6 on the Richter scale struck West Sumatra. Five districts of West Sumatra province were affected, and the Indonesian National Disaster Management Agency (BNPB) reported a death toll of 1117 people, with over three thousand injured. A total of 249,833 homes were damaged by the earthquake, including 114,797 heavily damaged and 67,198 moderately damaged homes.2 In response, the CRS Emergency Response team deployed staff to the region to work with local environmental NGO WALHI in conducting rapid assessment in Agam and Pasaman Barat districts and surrounding areas, and subsequently distributed emergency shelter kits to 8700 households. CRS quickly identified transitional shelter (t-shelter) as an urgent need and began to design its response strategy during the early days of the emergency. Since the completion of emergency distributions, the CRS West Sumatra program has focused on providing t-shelter assistance to families who continued to live in emergency shelter or who put up sub-standard transitional shelters that were not safe, adequate or durable enough to last until they would be able to rebuild permanent homes, and cash grant methodology was developed for implementing this t-shelter program. The total budget for the CRS West Sumatra earthquake response program was USD4.6 million (including USD29,687 worth in kind contribution from the Action Contra le Faime or ACF) and approximately USD3.4 million of which came from CRS private funds. Other donors include USAID s Office of Foreign Disaster Assistance (OFDA), UN-OCHA's Emergency Response Fund, Caritas Australia, Caritas New Zealand, Trócaire/Irish Aid, PLAN International and private foundations through CRS. By the time CRS completes its program in May 2010, CRS have provided transitional shelter assistance to more than 11,000 households in 135 sub-villages across Agam, Pasaman Barat and Padang Pariaman districts. The Kabupaten (Regency, or District) of Agam was selected at the beginning of CRS's intervention in West Sumatra to be its base of operations. CRS observed significant unmet needs in the district while many other agencies were focusing their efforts closer to the city of Padang. The district contains, and is in close proximity to, many remote communities to which access is difficult and which could therefore be easily neglected. Agam has a population of 430,000, which represents 9% of the total population in the province of West Sumatra. In Agam, according to local government data, there were as many as 86 fatalities (not all bodies were found), and 137 people injured as a result of the earthquakes. A total of 22,845 affected houses were identified, 17,059 of which were classified as severely or moderately damaged. Pasaman Barat, immediately north of Agam was also immediately selected for intervention, for the same reasons. In Pasaman Barat alone, a total of 9,148 affected houses were identified, 6,286 of which were classified as severely or moderately damaged. Padang Pariaman district suffered the greatest damage from the earthquake. In Padang Pariaman district, the government of Indonesia estimates that 675 people were killed in the earthquake and another 1055 seriously or moderately injured. More than half of the most heavily damaged houses were in Padang Pariaman, a total of 57,931 houses having been designated as sustaining heavy damage. A further 16,291 were classified as having medium damage, and 12,945 light damage. Because most agencies were focusing their efforts in and around Padang Pariaman, CRS originally worked further north. However, as the emergency phase ended and transitional shelter became the main priority for many agencies, significant gaps were revealed in coverage for Padang Pariaman, and CRS began implementing the t-shelter program there as well. 2 All contextual statistics presented in this section are based on: West Sumatra and Jambi Natural Disasters: Damage, Loss and Preliminary Needs Assessment. BNPB, Bappenas, and the Provincial and District/City Governments of West Sumatra and Jambi and international partners, October

6 Evaluation Overview This evaluation was designed to measure and document the success of the CRS West Sumatra Earthquake Response Transitional Shelter Program against the indicators set at the start of the program. Due to the innovative nature of the program design, the programming and M&E teams agreed on the importance of documenting the process through a comprehensive evaluation process. 3 The objectives for the program evaluation were as follows: 1. Assess t-shelter project progress focusing on the quality of technical assistance, cash grant, feedback mechanism, timeliness, and shelter completion. 2. Assess the quality of pondoks 4 in compliance with the standards that were established for the project: measuring Safety, Adequacy, and Durability (SAD). 3. Identify good practices and critical gaps in the project implementation in order to provide recommendations for program quality improvement in the later stages of this programme and in future responses, as well as for general organizational learning. 4. Provide CRS program accountability to community, partners, stakeholders and donors. During February-April 2010, CRS participated in an inter-agency assessment of cash grant programming led by Oxfam. Other participating agencies were Mercy Corps and Save the Children. As a part of this process, CRS conducted a separate in-depth assessment of the cash grant aspects of its t-shelter program. The data collection tools for the cash grant assessment were entirely separate from the program evaluation, although findings from the evaluation were incorporated into the cash grant lessons learned, and the findings from the cash grant assessment informed the final evaluation findings. The objectives for the cash grant assessment were: 1. Document the initial process in which cash grants were determined as the optimal methodology for the t-shelter project design; 2. Assess the impact of cash grants in communities, especially in terms of relationships between the various project stakeholders; 3. Measure the efficiency of the chosen disbursement mechanism; and 4. Provide a clear record of lessons learned for future programming. This report addresses key findings from the evaluations and feedback from key staff in response to initial discussions of findings, according to selected standard evaluation criteria in emergency contexts 5 : Relevance: the degree to which the program was in line with local needs and priorities Accountability and Appropriateness: the tailoring of activities to local priorities, focusing on ownership and accountability (discussed separately from relevance in order to address findings regarding community ownership and accountability with CRS more in-depth) Coordination: systematic use of policy instruments to deliver humanitarian assistance in a cohesive and effective manner, managing information and ensuring accountability to local stakeholders Effectiveness: the extent to which the program achieved its purpose, and to which its objectives were achieved Efficiency: the outputs, both qualitative and quantitative, achieved as a result of inputs Impact: the wider effects of the project on individuals and communities in the region Cross-cutting themes investigated were gender, vulnerable beneficiaries, and environmental concerns See Annex 3: Methodology, and Annex 4: Limitations, for details about the program evaluation process Pondok is a locally accepted term in Bahasa Indonesia for the transitional shelters being built Based on the criteria definitions in Real-time evaluations of humanitarian action, An ALNAP Guide, Pilot Version by John Cosgrave, Ben Ramalingam and Tony Beck, Overseas Development Institute,

7 Overview of Methodology and Limitations A mid-term evaluation was conducted as a final evaluation for the communities served through one funding stream, during January-February, 2010, assessing the project's impact on beneficiaries and communities and progress toward the program indicators. In addition to providing findings for early reporting, the findings from this mid-term evaluation were shared with the program team, who incorporated the information into decisions for improving program quality. During March, 2010, a final evaluation was conducted in those communities where CRS's intervention had already been completed, using the same methodology as applied in the mid-term evaluation. The dates of this evaluation were determined according to donor reporting requirements, but are considered by the CRS West Sumatra team to provide information on the entire t-shelter program, since the program methodology was mostly unchanged after this date. Both evaluations used the same tools, which included a household survey administered to 216 beneficiaries in the first evaluation and 260 beneficiaries in the final evaluation, and a shelter completion checklist filled out by the survey team for all beneficiaries included in the survey sample. Qualitative data was collected through focus groups and individual interviews with beneficiaries, including vulnerable beneficiaries, non-beneficiaries, shelter committee members, and stakeholders. Furthermore, a number of case studies and human interest stories were collected and the monitoring and evaluation team recorded observations made during visits to communities. Findings were discussed with program team members at the end of each evaluation process. The findings from the final evaluation were both combined with, and compared to, the findings from the mid-term evaluation for measuring project success. Throughout this report, findings from both evaluations are used as appropriate and relevant, and each reference will be identified by evaluation or as consolidated findings from the two evaluations. This evaluation report also incorporates findings from the CRS Cash Grant Learning Assessment conducted 11 February 5 March, as a part of an inter-agency learning process. This also used a combination of quantitative and qualitative data, culminating in an internal lessons-learned workshop to discuss all cash-related aspects of the program. Overall, the evaluation process was thorough and provided a large volume of valuable information. However, there were some notable limitations mainly due to the use of different types of sampling and the timing of the two evaluations that was dictated by donor reporting requirements. For more details on the methodology, sampling and limitations, please refer to Annexes 3 and 4. 7

8 Program Overview The distribution of cash grants was selected as an effective way to ensure T-shelters would be built quickly while also providing support to the local economy. Targeted beneficiaries were defined as all households in the earthquake-affected areas of Agam, Padang Pariaman and Pasaman Barat Districts whose homes were determined by CRS-WALHI qualified engineers to be unsafe for living. Many of these households continued to live in emergency shelter or had put up sub-standard transitional shelters that were not safe, comfortable or durable enough to last until they would be able to rebuild permanent homes. Poor and vulnerable households received priority assistance. The objectives of the program were as follows: 6 Goal: People affected by the 2009 West Sumatra Earthquake rebuild their lives and their livelihoods. Strategic Objective: Earthquake-affected households in target villages of West Sumatra province have a safe, adequate and durable place to live in the interim period before their permanent houses have been repaired or (re)- constructed. Intermediate Result: Target families make use of shelter support in targeted time frame. Output 1.1: Target households have received adequate Technical Assistance to (re)-build Sphere compliant transitional shelter. Output 1.2: Target households have received adequate cash support to (re)- build Sphere compliant transitional shelter. Initial reports from the CRS and WALHI staff working in the field in West Sumatra indicated that cash grants were the most effective means of assisting families to provide for their own transitional shelter. As opposed to procuring and directly distributing materials, the cash grants could be quickly disbursed to large numbers of households, providing them with the flexibility to purchase materials and hire labour according to their specific needs, while simultaneously supporting the local economy by encouraging the purchase of materials and supplies from local vendors and the hiring of local labourers. CRS began its transitional shelter program by designing and piloting a methodology for providing large-scale t-shelter assistance in affected communities in West Sumatra. Key technical staff developed a set of standards for building SPHERE-compliant Safe, Adequate and Durable (SAD) pondoks that would be easily understood by beneficiaries who would lead the process of building their own t-shelter. These SAD standards are as follows: Safe (resistant to earthquakes and other hazards) Adequate/Comfortable (with good ventilation and drainage, of a good size, gendersensitive with appropriate privacy) Durable (materials are strong and durable enough to last months) On the basis of these guidelines, teams composed of staff from both CRS and local partner WALHI piloted the methodology in four sub-villages, testing the effectiveness of different tranche sizes, community groupings, and conditions for disbursement of the second tranche of the cash grant. A primary goal was to determine the best way to ensure access of vulnerable households to the assistance they needed in order to complete their t-shelters. The project design was finalized as follows 7 : Pondok committees were established to engage the community, ensuring fairness and comprehensive coverage of a full sub-village. Committees were formed of approximately 10 community members, in equal numbers of men and women, who volunteered their time to accompany the entire process. The cash grant size was IDR 2,500,000. This was less than the total amount that team members calculated would be required for building a SAD pondok, but would be enough once 6 7 See Annex 2 for more details on the objectives, indicators, and reported progress achieved, according to evaluation findings See program document, CRS West Sumatra Transitional Shelter Implementation Guidelines, developed in late December

9 complemented with salvaged materials from their destroyed houses and/or other available resources. This amount was also in accordance with the general standards for cash grant amount that were adopted by the West Sumatra Shelter Cluster Coordination Group following consultations among relevant agencies. The cash grants were disbursed in two tranches: the first provided 2,000,000 IDR (80% of the total grant) and the second provided the remaining 500,000 IDR (20% of the total). The pilot exercise indicated that this distribution would provide a good balance between significant assistance to get started and motivation to work for the second tranche. For the sake of community cohesion and project efficiency, the first tranche of the cash grant was distributed together to 100% of beneficiary households in a sub-village. After satisfactory progress at the community level was achieved (see conditions below) the second and final cash grant tranche was also distributed together at one time to all eligible households. Technical assistance was provided in three forms: a technical presentation by CRS and WALHI engineers and social mobilizers was given to all beneficiaries shortly before receiving the first tranche; at the technical presentation, beneficiaries received a poster summarizing the SAD standards; and regular ongoing technical support was provided by the CRS field team members during the weeks following disbursement. The progress of the most vulnerable households was the primary condition for receipt of the second tranche in the whole community. This was intended to provide increased incentive for community solidarity with vulnerable families and this condition was accepted in the pilot communities. It was measured in terms of all vulnerable households' pondoks being Safe, Adequate/Comfortable, and Durable (SAD). Meanwhile, the overall progress of a community was also monitored and once it reached approximately 80%, the second tranche disbursement was arranged. Those few houses whose progress lagged considerably were excluded from the second tranche disbursement, but were informed that they would receive the final 500,000 once they demonstrated significant progress (to the 80% level) on their pondoks. This program design was developed so as to place special emphasis on ensuring that the most vulnerable households were able to complete their pondok along with the other members of their community. The list of vulnerable houses was developed by CRS field teams in close communication with the pondok committee members, and was mainly based on identification by members of the target community. These households were generally characterized as being physically and/or socially isolated from the rest of their communities. Most vulnerable households were headed by elderly residents who were frail and lacked the family support necessary for building their pondoks; femaleheaded households, usually widows with several children; families with many small children and possibly a pregnant mother; and others whose need for a pondok was urgent but their ability to provide limited. CRS developed an agreement with the Indonesian Post Office (PT Pos), Bukittingi branch, to disburse cash grants in the target communities. The Indonesian Post Office had significant experience setting up mobile field offices to redeem cash vouchers. This innovative partnership greatly improved staff security during distributions in the field, and it also enhanced the speed and effectiveness of the overall intervention. Cash grants after January 11, 2010, when the agreement was first implemented, were disbursed by PT Pos. Cash grants before this date were disbursed directly by CRS teams so as to expedite project activities during the interim period until the details of the arrangement between CRS and PT Pos were finalized. In addition, to ensure the sustainability of the intervention in these highly earthquake-prone locations, CRS met regularly with relevant government entities and other specialized actors, and collaborated with them whenever possible and appropriate to implement all project activities and maintain an open discussion about the importance of building earthquake-resistant shelters. 9

10 I. Relevance This section will describe how the t-shelter program was in line with local needs and priorities. Shelter was the most urgent need in post-earthquake West Sumatra and beneficiaries were eager to participate in the reconstruction of their homes. The program was well-received by beneficiaries and stimulated active participation in the target communities. Key points to be discussed: The identification of t-shelter as the most urgent and fundamental need in West Sumatra was confirmed in the evaluation. Specifically, a priority for beneficiaries was having a safe place to live. Communities were interested, engaged, enthusiastic and active in their engagement with the program. Beneficiaries were reluctant to move out of their unsafe houses. Many beneficiaries did not know the government-assessed damage level to their house. Pondok committees were appropriate for ensuring quality program implementation and appreciated both by beneficiaries and by the CRS field teams. The cash grant was enough for providing basic shelter, but more often provided families with a stimulus to build quality shelter for their families. The program design created a space for beneficiaries, including vulnerable households, to demonstrate ingenuity and feel dignity in their ability to contribute to ensuring shelter for themselves and their families. T-shelter was an urgent need in the target communities. It was described by stakeholders and beneficiaries alike as the most basic need. A representative from Agam District said, Transitional shelter was relevant to the context and needs of our communities after the disaster. Many communities did not have a safe place to live after the earthquake. This project gave them hope and encouraged people to build their lives. While shelter was confirmed as the most immediate need, community members had other urgent needs including support for rebuilding their livelihoods, and for repair of water and sanitation infrastructure. Government stakeholders were committed to meeting these needs, but they reported feeling strained in their limited capacity for implementing a more long-term rehabilitation project and were feeling overwhelmed in all sectors. They were committed to providing for their constituents in areas such as livelihoods, and water and sanitation, as well as permanent housing reconstruction, meaning CRS' assistance was well-received. The pondok was seen by many beneficiaries as an earthquake shelter, that is, many saw it as a long-term solution to their need for an earthquake-resistant shelter because they expected that the government money for permanent house reconstruction would not be enough to ensure this. Regardless of what they do with their original house, many said they planned on keeping the pondok as a safe place to live in the event of another major earthquake. When the evaluation team visited beneficiaries during a week in which there were a large number of tremors, they observed that more people were living in their pondoks than at an earlier time, and that in interviews they emphasized the importance of a safe place to live more. The flexibility of the program design was good in that it enabled people to buy a variety of materials according to their needs. Materials and labour were readily available in most communities. The one thing that was not always readily available was access to land on which to build a shelter safely removed from potential beneficiaries' damaged houses. The CRS field teams strove to work with these individuals to situate their pondoks appropriately. The program was economically beneficial, and socially fulfilling, for skilled labour. For example, one non-beneficiary man explained how he benefited from the program: The project helped us economically as I also have skills as a construction labourer. My economic situation improved. Before, I was a food seller in the market but I am now a paid skilled labourer. We have finished building 5 pondoks so far. And now the community has pondoks, a safe place to stay. 10

11 In the final evaluation, 64.6% of beneficiaries surveyed were living in their pondok, which represents an increase from the mid-term evaluation in which only 53.4% were living in their pondok. Most but not all of the occupied pondoks were surveyed as complete: 73.3% of those with completed pondoks were living in their pondoks, so some respondents were likely waiting until their pondoks were complete before moving in. However, many others had complete pondoks but were still not living in them. Both completion rates and pondok occupation increased notably between the two evaluations (discussed below under Effectiveness). There were some instances of people using their pondoks for other purposes, such as animal housing or storage, but these were isolated and also apparently decreasing over time. These findings indicate that many people were moving into their pondoks once they applied the final touches, others preferred to use their pondoks as a safe shelter only in the event of another earthquake, and still others had other plans for how to use their shelters complete living elsewhere complete but living in unsafe home complete and lived-in incomplete 50 0 Mid-term Evaluation Final Evaluation Figure 1: Complete pondoks, by where their owners are currently living Many respondents were still living in their unsafe houses: 27.2% in the final evaluation, representing only a small decrease from the mid-term evaluation (32.7%, which included mostly incomplete pondoks). The field teams were aware of this and invested heavily in transmitting a message to communities about the dangers of living in unsafe houses. There were only very occasional instances of beneficiaries who did not want to build a pondok nor understood why they should do so. Most community members valued having an earthquake-resistant home, but they did not always seem aware of the urgency of having a safe shelter. A number of beneficiaries used poles to prop up the walls of their houses and concluded that this makes it safe, even though structural engineers concluded otherwise, and communicated that to them. Other beneficiaries may have seen the pondok as a place to stay in the event of another earthquake but preferred to stay in their current house as long as they could, out of a concern for keeping their assets secure since many still kept many items in their damaged houses. Survey team members asked the respondent what level of damage their house had been designated by the Indonesian government's survey team, which was represented by a color-coded poster stuck to each damaged house. In the two evaluations, 45.7% of respondents' houses were identified by the government's survey team, which conducted a house-to-house survey immediately after the earthquake, as heavily damaged. Another 12.7% were identified as medium or lightly damaged. There was no notable relationship between damage level and completion rates. However, another 41.4% of respondents' houses' status was not identified by the survey team, for a variety of reasons. Often, the respondents did not know and the survey team did not see a government poster identifying the status. Other times, the respondents were unhappy with the performance of the government's survey team, or were not assessed. However, since the CRS engineers determined these houses were unsafe, it seems likely that many of these houses were also heavily damaged. The high number of those houses whose status could be identified as heavily damaged supports the CRS targeting criteria. 11

12 heavy damage (212) 46% unidentified (192) 41% medium damage (37) 8% light damage (22) 5% Figure 2: Government identified damage to original homes of all sampled households Pondok committees played an active role in coordinating community involvement, the extent of which varied from community to community. Among other things they were described as helpful, useful for speeding up the process, engaged, and knowledgeable. They energetically helped CRS with beneficiary selection and helped beneficiaries with coordination and communication. In the cash grant surveys, 63.4% of respondents learned about the distribution process directly from the committees, and feedback from the beneficiaries confirmed that they saw committees as their main point of contact regarding all program details. There was a high level of community involvement and interest in the program. One female committee member said about the t-shelter program: It involved the community. Almost all the community attended the socialization and they were excited for this pondok program. They are building their pondoks now. They are willing to participate in any meeting. Overall, communities were very enthusiastic and actively participated in all activities. In most communities, there was almost universal and active support for the program. However, beneficiaries looked out for their own needs, did not often help others but instead focused on building for themselves. Focus Group Discussion (FGD) participants, who were non-vulnerable beneficiaries, were adamant in pointing out that their community as a whole did not provide assistance to vulnerable members. They said it is not a part of the region's tradition to do so. Instead, vulnerable families built their pondoks with the assistance of hired labour or family members. However, vulnerable beneficiaries fared quite well, and in fact better than average on a number of program indicators, as will be discussed further below. Vulnerable beneficiaries often took charge of designing and constructing their own pondoks, and were in fact empowered by the cash grant program. Beneficiary testimony: Mrs. Gustimar, 39 years old, from Kapa Timur sub-village in Pasaman Barat district, used the cash grant to build a shelter for her family with a little food stall attached. Her shelter not only provides her children with a safe place to sleep, but the food stall helps her to provide for her family: The only thing that matters is that I now have a place to stay and I have my food stall. I feel very fortunate to have help from CRS and WALHI because if you had not come to help, then I'm sure I could not have built a pondok or have a shop like this now. 12

13 II. Accountability and Appropriateness This section will explain how the targeted communities were active in the t-shelter program, and how CRS ensured it was accountability to targeted communities. By working through local partners, especially village shelter committees, CRS's visibility and relationship with the community was ensured in a culturally-appropriate way. The program's targeting criteria was known to community stakeholders and shelter committees but not always to community members; nonetheless, all stakeholders agreed that CRS was fair. Key points to be discussed: Local partners, especially pondok committees, and informal conversations in the field were effective mechanisms providing program accountability. This was appropriate to the community context and key to project success since not all beneficiaries could identify CRS. Beneficiaries who built their own pondok, as opposed to hiring labour, were most likely to be interested in interacting with CRS. The presentation was considered the most useful form of technical assistance by the majority of survey respondents. Community members sometimes expected the government to provide t-shelter assistance, which led to some blurring of roles. Community leaders and stakeholders were familiar with CRS and targeting criteria, while community members often were not. Even those who did not know the targeting criteria, however, agreed that CRS was fair in its selection of beneficiaries. When surveyed, beneficiaries consistently reported that they were very happy with the program, the process of implementation, and with the services provided: 88.4% responded the program was quick when asked about timeliness 99.6% responded that the program met their needs 98.2% responded that they were satisfied or neutral regarding the overall program % of cash grant survey respondents reported that they were overall satisfied with the cash grant process 100% of cash grant survey respondents were satisfied with PT Pos's service (48.6% reported great service) Overwhelmingly, throughout the qualitative data, the CRS field teams were described as friendly and actively involving the community. In the cash grant assessment FGDs, CRS and WALHI staff felt that their role in the community was enhanced through the partnership with PT Pos. It freed them up to focus more on community relationships and program quality, and it also helped ensure their own safety and that of beneficiaries. Though beneficiaries were not particularly concerned about their own safety when receiving a sum of cash, the field teams were concerned on their behalf. One committee member said: The field staff's service was good: they discussed things with the community and involved the committee. During the socialization and surveying and monitoring visits, they also involved the community. Stakeholders and beneficiaries appreciated the teams for their thorough approach to ensure coverage of entire communities. They surveyed every house instead of relying on government data, and provided assistance to all households whose houses were unsafe. Both community members and stakeholders, including beneficiaries and non-beneficiaries, felt that CRS was fair, and said that the beneficiaries of the project were in fact people who needed assistance. However, most interviewees also mentioned that there were some people in the community who needed t-shelter assistance but were not included, such as people who lived in very isolated locations. The field teams went back and re-surveyed when these claims were determined as valid. There were also some instances in which non-beneficiaries did not know why they did not 8 These figures are consolidated from the two evaluations 13

14 receive assistance, as they did not understand the CRS designation of safe or unsafe. Community members generally were not aware of the targeting criteria defined by CRS, namely provision of t-shelter assistance for those with unsafe houses. Many beneficiaries, especially vulnerable, did not know why they received the assistance, even though they appreciated it and used the cash grant to build a pondok. However, stakeholders and community leaders were aware of the criteria. This is another area in which the role of the pondok committees was important for the project's success. The committee members helped to finalize the beneficiary lists and so were usually aware of the criteria. They took pride in playing this role and were able to explain to their fellow community members the goal of the program and how the cash grant should be spent. Not all community members attended the socialization meetings held to introduce the project in their communities. This was the first and most direct point of contact with CRS, during which the criteria was explained and CRS introduced. Sometimes, people who attended the socialization did not understand or did not pay attention. As a result, most beneficiaries, when asked, could not identify CRS as the source of their cash grant. They often thought the PT Pos or a government body provided the funds. Some recognised CRS but were not sure who or what CRS was. Again, though, pondok committee members and community leaders were well-acquainted with CRS and CRS' work. The CRS field teams were concerned about this lack of visibility, particularly as it could impact program accountability to the beneficiaries. Extensive efforts were made to brand CRS in the communities, including stickers, posters, t-shirts, and the CRS and WALHI logos on all program materials, including the vouchers submitted to PT Pos in exchange for the cash grant. A 24-hour hotline was established with a phone number, posted in a central location in each community by the beneficiary list, which community members could call to provide feedback or submit complaints. This phone line received some feedback, but most community members did not know about this option. They told the interviewers that if they had any concerns, they would talk to their pondok committee. A vulnerable woman explained, I don't know about the 24-hour helpline service. But if I have a problem I can talk to the committee and CRS staff during monitoring visits. But I have no problems with the service. Furthermore, the field teams made frequent visits to all beneficiaries' homes, both during the initial survey and as progress was being monitored. These visits received high acclaim, both for their technical value and for the personal relationships built as a result of the direct interaction. However, this relationship was only as strong as the beneficiary was directly involved in building his/her pondok. One vulnerable woman said, I trusted my labourers to build the pondok. What is considered good for them is good for me. She did not feel any need to directly address CRS. In the final evaluation, a full 73.4% of respondents did not know of a way to provide feedback to CRS. However, when asked if they felt any concerns they had were taken into consideration, 85.5% said they had no concerns to be considered. Most of those who did have concerns said that those concerns were in fact taken into consideration. Of the 14.5% of respondents who did have feedback or concerns, 92% knew of a way to contact CRS. 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Was there a way to provide feedback? Were concerns taken into consideration? Yes No Had no concerns Figure 3: Reporting any concerns to CRS/WALHI, according to the final evaluation 14

15 Again, most people were happy to interact with their pondok committees about the program, and when they felt the need to speak to CRS, they found a way to make that contact. Beneficiaries and government stakeholders alike indicated that the pondok committee represented the community to CRS well. They were active and proud of the opportunity to be involved, and maintained open lines of communication with CRS. Though the site visits were highly appreciated, for technical information, 50.4% of respondents in the final evaluation thought the presentation (given immediately before cash grants were disbursed) was the most useful form of technical assistance. The presentation was appreciated for giving beneficiaries a clear idea of what was expected and giving skilled labourers valuable tips on building transitional shelter. Posters were only considered most useful by 5.1% of respondents, and only found their way into 65.2% of homes, according to the final evaluation. However, some interviewees expressed a strong level of appreciation for the poster, which provided them with something akin to a blueprint for their shelters. Of those who received the poster, completion rates were somewhat better, indicating that it was still a valuable tool. The amount and type of technical assistance needed by a household varied significantly depending on the degree of their dependence on skilled labour to construct their pondok. Those who hired all skilled labour were generally disinterested in technical assistance, and in fact were unconcerned about having any opportunity for direct interaction with CRS. Most vulnerable beneficiaries hired builders and knew little about CRS. However, on the other hand, there was the example of a vulnerable man who explained that he had no prior knowledge of construction but was inspired by the project to learn. He built his pondok based on the information he received from CRS. 9 Pondok committee testimony: Mrs. Desmiati, 44 years old, Pauh Kambar Hilir village, Padang Pariaman District: I and people in my community here are aware of the 24-hour helpline service, the number is clear and it is visible on the road to our village. If they did not contact CRS it means they did not have any problems. Many people in the community come to talk to me or others on the committee whenever they have problems or issues or questions about the project. Most of their questions were about the cash grant distribution date. Because CRS staff often visited us, we could also contact them. I was involved in beneficiary selection and also in the technical monitoring 9 This man's story was published on the CRS blog, April 12, See rvicesblog+(catholic+relief+services+blog) 15

16 III. Coordination In this section, the relationship between CRS, community leaders and other stakeholders will be discussed. The t-shelter program systematically maintained open lines of communication with members of local government at all levels, and strove to use locally appropriate mechanisms to deliver humanitarian assistance in a cohesive and effective manner. Strong lines of collaboration were developed and maintained between CRS, local communities, government stakeholders and the private sector. The partnership with the PT Pos was a highlight of coordination. Key points to be discussed: PT Pos partnership was a highlight of coordination. It set a new precedent for partnership between the non-profit and private sectors in Indonesia. Pondok committees played a key role in coordinating between CRS and communities. Shelter cluster coordination was good for enhancing CRS influence and credibility. CRS was active and intentional in coordination with local authorities, and local authorities appreciated the t-shelter program. Some local authorities hoped for more consultation and information-sharing from CRS. CRS managed well any issues that emerged. The innovative collaborative partnership between CRS and PT Pos was productive, positive and a key to program effectiveness. Partnership with an NGO, particularly an INGO, is new to PT Pos 10. Their past experience of similar programming has been in partnership with the government. PT Pos representatives have expressed that they appreciate this new partnership because it improves their image and reputation in community and government relations, and they are glad to play a role in the rehabilitation of the region following the earthquake. Procedurally, working with an NGO has been simpler and more efficient for PT Pos than working with the government, since CRS takes such an active role in each target community. Feedback from PT Pos is that they are hoping for continued opportunities to work together, possibly in other parts of Indonesia. They are committed to continue, and maximize on, this type of involvement with NGOs and to share their positive experiences with other branches of the PT Pos. Other INGOs have already followed CRS' example in partnering with PT Pos for cash disbursement in t-shelter programming for West Sumatra. CRS' partnership with pondok committees was also seen by the various stakeholders as productive and effective, an accepted way to interact with the community. In the cash grant survey, 94.5% of respondents reported a great relationship with their pondok committees. One committee member said, I, as a representative of my community felt informed and involved in the process of project design and implementation. The CRS team leader often asked me about the progress and my ideas about the best way to continue the project. I felt honored. People in the community have often asked me or the staff questions during monitoring so we don't have any issue or concerns about coordination in our community. Committees, as well as local government stakeholders, have been a consistent bridge between CRS and the residents of the target communities. From the early days following the earthquake, CRS was an active participant in the Shelter Cluster established by the UN coordinating body. CRS assisted with assessments and strategy development for meeting shelter needs in West Sumatra. Through close coordination with the Shelter Cluster, CRS kept abreast of policy developments regarding reconstruction plans for West Sumatra, and in fact often had access to information that had not yet filtered down to local or district governments through official mechanisms. CRS was asked by the Shelter Cluster to serve as liaison between the district governments of Agam and Pasaman Barat, however information sharing was not always easy in a context where communication between the Indonesian and local governments was complicated. 10 The Indonesian Post Office (PT Pos) is a State Shareholder Company - an independent for-profit company with government affiliation and oversight. 16

17 CRS was one of the first agencies in the West Sumatra response to design a cash transfer program for shelter and some agencies were finding that donors initially expressed some reluctance to fund cash transfer programming, a problem CRS was not facing due to a strong base of private funding. In response, CRS worked closely with the Shelter Cluster and with other agencies to design a justification for the use of cash grants in shelter programming in the West Sumatra response. This document was shared widely, and following its release, a number of other agencies began designing their cash grant programming. CRS was in a position of leading the way in many of these decisions, and other agencies regularly approached CRS for advice in designing their programs. CRS was very intentional in coordinating and meeting regularly with the authorities at the district, sub-district, village and sub-village levels. One sub-district official said, CRS often conducted coordination in this sub-district office. They showed very good initiative, especially when they started the project. We also appreciate CRS' proactive response to counter some issues that came up about being Catholic, that is, the 'C' in CRS. Some issues and misunderstandings were clarified. During the coordination period, CRS asked for our consent and shared their plan, which is a good thing. Most government stakeholders similarly reported being kept informed about the project. The C issues were of significant concern to some people and a few full communities, but they were explained and openly clarified. One vulnerable woman said, Our whole community participated in this project. Everyone was consulted... and issues that were raised, like the 'C' in CRS, were also clarified. In many communities this issue was discussed before the decision was made to welcome CRS and there were many isolated incidents of potential beneficiaries refusing, or initially refusing then later changing their minds, assistance for fear of religious pressures. In a handful of communities, controversy surrounding the issue did significantly slow progress. While this was mentioned at all levels, from beneficiaries to district-level stakeholders, all interviewees confirmed that it became clear to everyone that CRS provides aid to those who need it, without conditionality regarding beliefs or other prejudice. Stakeholders had suggestions for how this could have been mitigated by CRS, including engaging local religious leaders in early coordination and avoiding listing beneficiary names and identity numbers. However, all agreed that CRS explained itself well and proved by example that religious fears with regard to CRS' assistance were unfounded. While feedback from government stakeholders regarding CRS' coordination efforts was positive, there was some frustration regarding the way in which coordination was conducted. One districtlevel official and a representative from one of the smaller sub-districts in Agam confirmed the relevance and appropriateness of the program design, but also recognised that relations between CRS and their government offices were somewhat strained. 11 They suggested that if CRS had consulted with them before developing the program design instead of assuming the needs of the community based on their own data, relations may have been better, even if the actual intervention was appropriate regardless. They also mentioned that they would like more regular progress reports from CRS, more information about beneficiary lists for cross-checking, and other more frequent responses to their queries. One challenge for communication was that district and sub-district authorities seemed busy and under a high degree of stress, often sorely feeling the lack of sufficient capacity to coordinate a reconstruction effort after a major earthquake. Some of them have approached CRS for help, in search of assistance or advice for dealing with the challenges they face in rebuilding their communities. But others, notably the district-level authorities, said they have not had time to work with CRS as they have too many time constraints. They acknowledge that CRS has been active in engaging them and regret that they could not participate as actively in coordination efforts. 11 As an outcome of the interview with this district-level stakeholder, the district government sent a delegation to observe the evaluation process and visit beneficiaries over the course of two days. This was a positive relationship-building exercise for both CRS and the district government. 17

18 Stakeholder testimony: Mr. Arman Datuk Marajo, Malalak sub-district secretary, Agam district: CRS often met with us in this (sub-district) office to discuss their program. They consulted with us, especially when they started the project. It is a very good initiative. We appreciate how CRS was proactive to counter some of the questions people in the communities had about them being Catholic. Issues and misunderstandings were clarified effectively. During the coordination period, CRS asked for our consent and shared their plan. It is a good thing. The government here supports CRS. 18

19 IV. Effectiveness This section will discuss evaluation findings which describe progress toward the indicators of success defined at the beginning of the t-shelter program, drawing the conclusion that the t-shelter program achieved its purpose according to its intended outputs. The program design was successful, and effectiveness improved over time. The program team was responsive to issues as they emerged and made adjustments to the implementation strategy accordingly. Key points to be discussed: Quality of construction increased over time. This was partially due to improved project implementation by the CRS field teams, but also confirms that families used the cash grants as a stimulus to take charge of their own recovery. Use of salvaged materials helped people build their pondoks. The cash grant amount was good. It provided sufficient assistance to those who could not provide for themselves and a useful stimulus for most beneficiaries to begin rebuilding their homes. The approach used, providing a second tranche of 80%, conditional upon community completion and individual progress, was ideal. The full-completion standard, in which individual progress was assessed as well as community progress, kept beneficiary accountability high and led to high rates of success. Progress toward key indicators was high (see annex 2). One of the main measures of program success was the percentage of target beneficiaries who built shelters that adhered to Sphere guidelines, defined for this t-shelter program as SAD-compliant pondoks. In the mid-term evaluation, 81.3% of households surveyed had SAD-compliant pondoks, while in the final evaluation, that figure had increased to 89.1%. SAD-compliance and completion were evaluated separately, and similarly, the percentage increased of households with pondoks considered complete by the standards set by the CRS field teams, that is frame, walls, flooring and roof installed. 12 Since most respondents continue to improve on their shelters, the program team expects these indicators for completion to continue to improve over time, even well beyond program end. SAD compliant Complete Mid-term evaluation Final evaluation 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure 4: Improvement in SAD-compliance and pondok completion over time Furthermore, the amount of time required to build a pondok decreased from the mid-term evaluation to the final evaluation, from 17.2 days to 15.4 days. SAD-compliant pondoks took slightly longer to complete than those that were complete but not SAD-compliant. 12 A complete but not SAD-compliant shelter might be fully constructed but too small, or lacking a partition for women's privacy, for example. Such a pondok will never meet the SAD requirements although it will serve the purpose for the family's shelter. A SAD-compliant pondok could meet the Sphere-based standards established by the field teams but not yet have a floor installed or all walls finished, for example. 19

20 Average SAD Vulnerable Mid-term Final Figure 5: Days required for completing pondoks, change between the two evaluations These improved completion rates are likely due to a number of factors, including beneficiaries' felt urgency for safe shelter and the continually eroding state of emergency shelters which was affecting people's health. However, the field teams also made two changes in its implementation strategy that can help explain the improved completion. The first change was in the initial surveying. The teams interacted more with potential beneficiaries before listing them as eligible for assistance, explaining to them that the money was to be used for pondok construction only and asking them if they were willing and wanting to build a pondok. Since this program focused on transitional shelter, and there was a clear message from the government that they intend to provide for permanent shelter reconstruction, only individuals willing to build a pondok were eligible to receive the assistance. However, in communities of later arrival, some people had already begun renovation of their house and did not want to undertake a pondok project in addition to that. One woman explained her situation: My house was heavily damaged after the earthquake. We soon renovated our house as there was a baby and elderly people here. Then we did not receive a cash grant, because we did not want to build a pondok. However, she understood why she did not receive and agreed that the program's targeting was fair. The second change was the establishment of a program quality team, which double-checked the monitoring reports of each field team. Second tranches were not disbursed until a program quality team member approved the progress of the entire community, as well as of each individual who would receive the second tranche. The conditionality of the second tranche was an accepted factor for ensuring compliance with the program expectations. High standards and clear communication of those standards from the beginning increased pressure on both communities and field teams to commit to completing good pondoks quickly. Most respondents (89.1% in the final evaluation) continued to improve on their pondoks, and the program design was built around the assumption that beneficiaries would to contribute with their own resources and continue to do so over time. In keeping with this expectation, the average amount spent on building a pondok increased significantly between the two evaluations. In the midterm evaluation, average expenditure was IDR 4,017,157, while in the final evaluation, average expenditure was IDR 5,066,016, and the average expenditure for completed pondoks was slightly lower than the average. Some beneficiaries undertook large and ambitious projects which would require a long time to finish, and by the time of the evaluation those pondoks were not yet complete. The high level expenditure undertaken by these beneficiaries may have skewed the data somewhat, however, their example highlights the program design putting ownership of construction into the hands of beneficiaries, and providing them with a stimulus to rebuild their lives and livelihood. For these beneficiaries, not only was a safe, adequate and durable pondok important, but furthermore, a construction project served to provide them with new meaning in life after a devastating experience. 20

21 Between 1.25 and 2.5 million (42) 9% less than half of cash grant (16) 3% 2.5 mil exactly (61) 13% more than 2.5 mil (341) 74% Figure 6: Expenditure for pondoks, combined from both evaluations Families willingly and eagerly contributed with their own resources and the average expenditure on pondok construction was well above the cash grant amount; in addition, 87.1% of respondents in the final evaluation spent the full cash grant or more on construction. Most of the remaining 12.9% spent more than half of the cash grant to build their pondoks. Even taking into consideration the isolated instances of beneficiaries who did not use the cash grant for shelter, this finding confirms that not only did most of the project money go directly to its intended use, but many families complemented project money, further injecting cash straight into the economy by hiring builders and purchasing materials. The following chart illustrates what cash grant money was spent on: 13 Walling Frame Roofing Flooring Labor Tools Transport 13% 30% 78% 76% 71% 69% 89% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure 7: Use of cash grant money for construction, according to percentage of responses regarding purchases in both evaluations In the final evaluation, a full 75.0% of respondents used salvaged materials from their damaged homes to build their pondoks, increased from 62.5% in the mid-term evaluation. In both evaluations, people with heavily damaged houses by the government's criteria were more likely to use salvaged materials. The average amount spent on a pondok that used salvaged materials was not significantly different from the amount spent on a pondok that did not use salvaged materials, nor was there a relationship between SAD-compliance and use of salvaged materials. Some beneficiaries used salvaged materials to offset the cost of pondok construction, while others used their access to improved resources to improve the quality and uniqueness of their new homes. Again, decision-making was left in the hands of each individual beneficiary to use their resources as they saw fit, as long as they built a transitional shelter, which almost all beneficiaries did. While such instances were rare, it is important to note that there were households who built a complete SAD-compliant pondok with less than the full cash grant amount and with little or no 13 See cash grant assessment report ( Lessons Learned: Cash Grant Programming, March 22, 2010, CRS Indonesia West Sumatra Program) for a discussion of non-construction expenditures using cash grant money, which were widespread yet small in value 21

22 access to salvaged materials. Others completed small pondoks which mostly adhered to Sphere standards, except for size, or which were intended for only one person to sleep. Some vulnerable beneficiaries had little to begin with, so could salvage little from their original houses, and they had limited resources to tap into in order to complement the cash grant. However, these households were still able to provide shelter for themselves, and in some instances for their families, through the t-shelter program. The majority of beneficiaries appreciated the stimulus provided by the cash grant, energetically pursued resources to complement the amount they received, and built transitional shelters for themselves which can serve as a home for as long as they need it. Beneficiary testimony: Mrs. Asnidar, 41 years old, Hulu Banda Jorong, Malalak Barat, Agam District, mother of four, whose husband is a migrant worker in another province. She paid attention to the technical information she received and led the process of building a safe shelter for her family: Since my husband isn't always at home, I decided to hire labor. But I was the one who decided on the design and how the pondok would be built. I sometimes helped the builder, and I checked up on his work to ensure the quality of the pondok. We spent 4 million Rupiah to build this big pondok for my whole family. It was not very expensive since we used materials salvaged from our old house. The cash grant was enough to start our pondok, and if we had not received it, I don't know whether we would have been able to build this pondok or not. 22

23 V. Efficiency This section will discuss the program's timeliness and success in relation to the number of inputs invested in implementation. The t-shelter program achieved a high level of success, reaching a large number of households with significant shelter assistance in the months following the earthquake, with a low level of input: it was cost-effective and timely. The objectives and the implementation process were easily communicated to beneficiaries, which helped them to take an active role in building their shelters according to the standards set by CRS. Key points to be discussed: PT Pos's role contributed significantly to program efficiency, increasingly so as time went by. The CRS teams were freed up to focus on working directly with beneficiaries, and the process of cash disbursement was streamlined. Hindrances to productivity and slowing factors were relatively insignificant. When such issues emerged, they were within the communities themselves. Meanwhile, the CRS teams were seen as being highly productive and prompt. Communication with traders was limited and may have slowed progress. In many communities, beneficiaries coordinated with traders on behalf of a group of people. The simplicity of the goals and methodology was a key to success: beneficiaries understood clearly why they were receiving assistance and what their role was to be in building their shelters. This program reached more than 11,000 households with transitional shelter support in less than 6 months. As a district representative said, From the information I have heard that CRS has already distributed more than 10,000 first cash grants to the community, the project is quick. Of the 11,000 beneficiaries, evaluation findings indicated that 89.1% had shelters that adhered to Sphere standards, while most of the remaining beneficiaries continued to improve on their shelters. Operating costs were low: as of the time of writing this report, 73% of the total project budget had been injected directly into the target communities through cash grants, and once implementation is complete, it is anticipated that this figure will reach 80%. One of the key factors contributing to this high level of program efficiency was the simplicity of the program, with clear goals and methodology. In the cash grant surveys, 95.9% of respondents reported that the procedure for collecting their cash grants was clear to them, and feedback from interviews supports the conclusion that the program goals and methodology were widely understood. Beneficiaries were aware that the cash grant was given to build a shelter, and the SAD guidelines were not only clear to them, but also aligned well with their own priorities for their shelter. For CRS and WALHI, focusing the entire program on transitional shelter, once emergency distributions were complete, provided the field teams with time and energy to invest fully in quality t-shelter programming. The partnership with PT Pos also contributed to efficiency. PT Pos has an established set of Standard Operating Procedures (SOP) pertaining to community cash distribution which are clearly understood among their staff, and became increasingly clear to CRS staff and better communicated to beneficiaries, over time. They were developed for PT Pos'. partnership with the government assistance program, and have been applied to the partnership with CRS. In the cash grant assessment focus groups, CRS and WALHI staff stated that one of the main reasons for, and benefits of, working with PT Pos was the existence of these clear procedures. PT Pos has been consistent in its role and practices, which helped facilitate efficiency and clear roles for all involved in the distribution process. It also freed up CRS and WALHI staff to focus on implementing other aspects of the program, namely social and technical assistance for community members. Disbursements were implemented punctually and efficiently, and on the day they were scheduled to happen, although due to factors such as bad roads and maps, there were some instances of same-day delays that did not have an impact on project activities. 23

24 CRS'. service was fast once community socialization began. Beneficiaries and stakeholders commented on CRS' late arrival in some communities, in some cases 5 months or more after the earthquake happened, but in the larger context of humanitarian assistance in West Sumatra, this was still relatively punctual. In some communities, project start-up was delayed due to discussions ensuring potential beneficiaries that CRS'. mandate is humanitarian and not proselytizing in nature. Nonetheless, once community activities began, progress was quick. The efforts of the field teams contributed to efficiency, as they pressed community members to complete their pondoks quickly. Frequent site visits, and reminders of the second tranche that was contingent on successful completion rates, were tools used by the CRS field teams to help beneficiaries complete their shelters quickly. Interviewees did often mention the fact that many pondoks in their communities were still not complete, and that progress was not as fast as they would like. One commonly cited reason was that the cash grant amount was not enough to cover the full construction cost, and that the time needed for saving more money or pooling family resources slowed progress. The program team was aware of these comments, but other findings in this report confirm that the resulting community ownership made such delays acceptable. Most families were well on their way to completing their shelters and took pride in taking charge of building their own shelters. In addition, the same interviewees also acknowledged that the main problems for completion were external to the program activities, or else due to beneficiaries' own lack of initiative. For example, skilled labourers were in scarce supply throughout West Sumatra in the months following the earthquake, so beneficiaries often had to wait their turn for someone to help them build their pondok. Similarly, difficulty accessing materials or equipment sometimes slowed progress. Generally, when asked, stakeholders and community members attributed these minor slowing factors to community conditions. They appreciated CRS'. assistance and took responsibility for instances when progress was less than ideal. When interviewed, traders reported that CRS did not negotiate directly with them, and while they usually had sufficient stock available to meet demand, CRS may have been able to minimize price inflation for certain construction materials by discussing the program and schedule for cash grant disbursements with them in advance. Consolidated transportation expenses, for example, could have been discussed so as to keep prices stable. However, even without CRS playing a role in negotiation, traders were kept informed by beneficiaries themselves and were generally able to plan. Often community members went to traders in groups to order their materials, so in this particular context there was not sufficient evidence to conclude that CRS'. support was needed for engaging suppliers to play an active role implementing t-shelter programming. Beneficiary testimony: Mr. Cangkuak, 83 years old, Salimpaung, Malalak Utara, Agam district, identified as vulnerable by the field teams in coordination with the pondok committee. He hired skilled labour to do most of the construction on his shelter: "I live alone here. My house was flattened by the earthquake. I heard about the CRS pondok program, and they came and surveyed my house. Then suddenly a committee member came to my house, asking me to accept the cash grant. I was surprised! It was such a quick process. My pondok was completed in 8 days, which is good since I really needed it." 24

25 VI. Impact This section will describe how the t-shelter program empowered community members, including vulnerable households and women, most notably by providing them with the tools they needed to take charge of building their own homes and providing for their families. The program also provided an increased feeling of safety and security to beneficiaries, and sped the process toward economic recovery in the region. Key points to be discussed: Beneficiaries and communities had an increased feeling of safety and security. T-shelter provided a stable platform for individuals and households to focus on economic recovery once they had homes to live in. Cash grants provided a stimulus for households to take charge of their own recovery. Minor community conflict, including social jealousy and religious suspicion of CRS as a Catholic organization, was quickly resolved by the field teams and pondok committees. The t-shelter program operated effectively within the existing economic structure of the region, and had little impact on the regional economy. The cash grant programming provided an economic stimulus to households, skilled laborers and traders of construction materials. Pondoks alleviate fear. Beneficiaries spoke of how they no longer run hysterically from their houses whenever they feel a tremor because they feel safe in their pondoks. A village level stakeholder explained, I noticed my neighbors when a tremor happened last weekend. People felt safer and did not run out for evacuation. The pondok can be used to anticipate the next earthquake and aftershocks. The most strongly-felt impact of the program is a sense of safety, security for sleeping and everyday living. This sense of security has freed people up to start rebuilding other aspects of their lives. Economic recovery was another impact of the program. Once beneficiaries had a safe place to stay, they could turn their attention to their livelihoods, as this vulnerable woman described: I stayed in a tent before and now I have a bigger and safer place to stay and live. It enables me to think about my economy and other things in my life. I can start a new enterprise now by opening a small food stall. This pondok is really meaningful for my family. In addition, individuals other than beneficiaries have benefited economically from the program. Local skilled labourers and traders, for example, profited. These benefits were usually felt in the immediate vicinity of target communities, and usually only in the immediate aftermath of disbursements. In one rural area, where there is minimal competition for sale of construction materials, a local trader reported that his sales doubled during the days immediately following CRS cash grant disbursement, but it quickly returned to normal after 3-4 days. In an area closer to the city, only the trader in the very immediate vicinity of the disbursement site experienced a notable boom in sales, while a trader just a few kilometers away reported no notable change. 14 When asked, community members reported that they still have significant livelihood needs which CRS has not directly met, but has often helped by providing a stable platform for beneficiaries to invest in their own livelihoods, and by providing shelter which freed up the local government to support livelihoods. Especially in Malalak sub-district, one of the CRS t-shelter target areas that suffered an enormous landslide in the aftermath of the earthquake, many people's livelihoods were not yet recovered since the earthquake: agriculture infrastructure was damaged and shops collapsed, and the resulting economic impact filtered down to food sellers and traders throughout the region. 14 This report will not discuss the economic impact of the earthquake itself. For more information, please see refer to the cash grant assessment report ( Lessons Learned: Cash Grant Programming, March 22, 2010, CRS Indonesia West Sumatra Program) 25

26 The economy of West Sumatra is large enough and stable enough that the CRS t-shelter program, even though it was extensive and covered entire sub-districts across three districts in the province, injecting more than USD $3,000,000 into rural and semi-rural communities, did not have any notable economic impact on, nor cause any disturbance to, the regional economy. Rather, it provided an economic stimulus to households, skilled labourers and traders. Beneficiaries spent their cash grants in the local economy, often directly paying individuals for labour or timber without other supply costs, and those who they paid in turn spent the money earned in the local economy. One estimate is that this filter-down effect resulted in a five-times multiplication of the cash grants' impact on local economies in post-earthquake West Sumatra 15, usually at the community level in areas suffering economically after the earthquake. Community conflict did occur, but only occasionally, surrounding receipt of assistance or cash and related issues of social jealousy. There were occasional accusations of unfairness. Such instances were rare, and as they were explored by the field teams, often revealed to be rooted in other preexisting family feuds or interpersonal differences. In any large-scale distribution, especially one involving the transfer of cash, some accusations of unfair treatment are expected, but in this program they were notably rare. One likely reason for this is that the assistance was provided to all households whose homes had been rendered unsafe by the earthquake and who were willing to build pondoks, leaving little room for community members to question CRS's fairness. The overall impact on the communities was very positive. The t-shelter program provided earthquake-affected households with a stimulus to begin rebuilding their lives and livelihood. Many beneficiaries told the evaluation team that the program gave them hope. By building a pondok using their creativity and resourcefulness, they can take pride in a tangible asset which they made. In the aftermath of a traumatic event, this program was important for helping people return to normal life. Vulnerable households, especially, often were eager to undertake the project and begin construction of their homes, and invested time and energy in building something they could enjoy. Beneficiary testimony: Mr. Khairul, 42 years old, Jalan Bantiang, Malalak Barat, Agam District, who built a spacious pondok for his family of seven, using many salvaged materials from his damaged house to complement the CRS cash grant: I am so grateful that we received a cash grant from CRS and WALHI to build transitional shelter. As a result, I could make a better and more spacious pondok for all my family members. I made this pondok so large because I want everyone in my family to stay here and feel secure. I don t consider this pondok to be transitional shelter, instead, it is our home. Now I feel safe and don t worry about finding a safe place for my family to stay and sleep. After all, we could not stay in our damaged house anymore as it was unsafe. Thank you CRS. 15 Martin Aspin, at April 6, 2010 workshop: Assessment of Cash Transfer Programmes (CTP) in West Sumatra 26

27 Cross-cutting themes Gender Women were engaged in all phases of the program. Women were most likely to interact with program staff, as they handled money and often attended the meetings. Women were involved throughout the t-shelter process and they had a clear voice. Women actively participated on pondok committees, and other women were regularly consulted. The data indicates that the only families in which women were not actively involved were families with only male adults, usually elderly widowers. Women's involvement was in both design and in implementation, and in fact women were often even involved in the physical labor of building their shelters. One woman, who also is a member of a pondok committee, said, I myself am working together with my husband to complete our pondok. We do the same things, I am also involved in the physical work. I prepare the foundation, cut and fill, paint, nail, erect the frame, etc. We respect each other. We discussed all the ideas for design and its implementation. We have equal positions. Because West Sumatra's culture is matrilineal, team leaders estimated that a full 60% of beneficiary households were female-headed. Most of these women are married but the home ownership is in their names. Most household representatives collecting cash at the disbursements were women (69.5% of the cash grant exit survey sample), indicating that women manage the money for their homes. While this would suggest reason to include increased precautions in the disbursement procedures, when asked, the women themselves did not express any concern regarding their own security. During technical assistance, the CRS teams emphasized the importance of putting partitions in pondoks so as to ensure women's privacy, and women played an active role in the design of their pondoks. Even so, not all pondoks have partitions. All families that were interviewed were aware of this issue but when asked, both women and men said that their priority was saving money and ensuring the family's safety. However, one interviewee went on to explain that his family planned on using second tranche money or other resources later on to add rooms for the girls, so he did continue to take this concern into consideration. The field teams concluded that women were aware of privacy issues and made their own decisions regarding partitions, so did not push the issue since women were in a position to ensure their own needs were met. Vulnerable Households Vulnerable beneficiaries had better completion rates than average. They and their families took charge of guaranteeing their own success in building their shelters, while the program design also helped ensure they had the support they needed. Vulnerable beneficiaries were most likely to hire skilled labour or receive assistance from family, as opposed to community support. The vulnerable households that were identified in the target communities, by the pondok committee members along with CRS teams, included primarily elderly, families with many children, and widows. Most were also characterized by isolation, sometimes due to living in a remote location, other times because they had children to care for, and others due to age or disability. The vulnerable interviewees had a minimal level of understanding of the project. Isolation meant that they had very little information or opportunity to interact with CRS or the pondok committee about the project, except for when they received monitoring visits. They expressed appreciation for being selected to receive the cash grant and committed to using it to build a t-shelter. Overall, vulnerable households fared very well. Some of the highlights of vulnerable progress, according to the final evaluation, include: Vulnerable households were slightly more likely than average to have SAD-compliant pondoks 27

28 (91.7% of vulnerable in the two evaluations, compared to 89.1% overall) Vulnerable households took an average of 13.9 days to complete their pondoks, compared to the average of Vulnerable survey respondents were less likely than average to report availability of labor as an obstacle, even though 86.1% of the vulnerable beneficiaries surveyed spent cash grant funds on labour, compared to the average of 71.5% of respondents. Vulnerable households spent less money on their pondoks, but the average was still above the cash grant amount: The average amount spent by vulnerable households was IDR 4,466,667 (IDR 599,349 less than the overall average). They did not cite community help as an important factor for completing their pondoks. It was not a part of their community culture to work together to provide assistance to their vulnerable neighbors. Nonetheless, vulnerable households could count on the strength of family ties, especially family obligation to help their needy relatives. In addition, vulnerable households felt the need for housing acutely, and were likely to have limited resources but high motivation to use what little they had wisely. Many vulnerable beneficiaries felt empowered by the opportunity to use a cash grant to build their own shelter. One female committee member said, Here, the vulnerable did not receive help from others in the community. They hired labour and some of their families supported them. Many of the vulnerable, even the elderly, wanted to do the physical tasks themselves. Furthermore, there was a degree of peer pressure, mostly positive, from community members, following the CRS socialization meetings in which communities were informed that success was defined by the specific success of vulnerable families for building SAD-compliant pondoks. It seems likely that, even though labour was not offered for free, skilled labourers made themselves available to vulnerable beneficiaries before accepting assignments from other households in the community, so as to expedite the vulnerable households' completion. Environment There was a high level of awareness about environmental issues. Beneficiaries made decisions, mainly about use of trees and timber, with environmental concerns in mind. Beneficiaries, skilled labour and stakeholders alike, were aware of environmental concerns. Since WALHI is an environmentally-focused NGO, environmental issues were openly discussed during program socialization and site visits. However, community members were already careful about what trees they cut down and where, and made decisions regarding pondok construction with the goal of minimizing the environmental impact of the program. Vulnerable respondents were less likely to understand, but it seems likely that the labour who built their pondoks did know. There was no evidence of negative impact on the environment. Almost all reports of timber use was from trees cut down within the village boundaries. In many cases, people used trees from their own land or else paid for timber from a neighbor's land. They were selective in choosing which trees to cut, taking into consideration sustainable growth for their own livelihoods. Environment emerged as a non-issue, largely because people used their own timber, mostly coconut trees, on their own land, and expressed an understanding of environmental concerns regarding cutting down their trees. 28

29 Conclusions and Recommendations Evaluation findings regarding the t-shelter program were overwhelmingly positive. Targeted progress toward the key objectives of the program was achieved and surpassed. A large number of beneficiaries received significant assistance in a short period of time, and used the cash grant money for its intended purpose. The t-shelter program provided individual beneficiaries with a jumpstart to economic and social recovery after the earthquake, and provided communities and the local government with a stable platform for investing in long-term recovery. The technical expertise offered by the CRS teams was well-received and put into practice. CRS played a key role in the West Sumatra earthquake response, especially with regards to shelter programming. This strengthened relationships and set a precedent for new innovative partnerships in Indonesia in the future. Very few issues emerged, and these did not generally deter from successful completion of the program's goals and activities. However, based on the evaluation findings, the CRS West Sumatra program team recommends the following for future similar programming: 1. Develop a strategy for raising awareness about the risks of living in unsafe houses. This should take into consideration many people's valid concern for the security of the assets stored inside their houses. 2. Continue to be conscientious in considering the isolation of beneficiary households and their special needs, especially vulnerable households. In particular, consider local culture in determining community support for the vulnerable members of the community. Support mechanisms for vulnerable households exist, but they may not be the ones immediately obvious to programming staff arriving in a new setting. 3. Take advantage of any and all opportunities to explain CRS and beneficiary criteria to the community, remembering that community members are often not as interested in the identity of the institution providing the assistance as in the assistance itself. However, CRS staff should invest heavily in increasing its visibility so as to maximize opportunities for program accountability. 4. Increase the visibility of feedback mechanisms and provide beneficiaries with multiple opportunities to meet directly with CRS staff 5. Define carefully roles with local authorities as early in the process as possible, with an individualized approach according to local context, especially in cases where the project might be construed by some as overlapping with the government's role 6. Explore ways to engage religious and traditional stakeholders along with government stakeholders 7. Consider culturally appropriate forms of compensation for community volunteers Consider that there may be differences in the needs of semi-urban vs. rural communities, and investigate these further in future evaluations 16 For more details on the findings which led to this conclusion, see Lessons Learned: Cash Grant Programming, March 22, 2010, CRS Indonesia West Sumatra Program. 29

30 Annex 1: Performance Indicators Tracking Table (PITT) Transitional Shelter Programming Overall Goal: People affected by the 2009 West Sumatra Earthquake rebuild their lives and their livelihoods PROFRAME ELEMENT: INDICATORS BASELINE TARGET FINAL RESULT Strategic Objective: Earthquake-affected households in target villages of West Sumatra province have a safe, adequate and durable place to live in the interim period before their permanent houses have been repaired or (re)-constructed. Intermediate Result: Target families make use of shelter support in targeted time frame. Output 1.1: Target households have received adequate Technical Assistance to (re)-build Sphere compliant transitional shelter. Output 1.2: Target households have received adequate cash support to (re)- build Sphere compliant transitional shelter. 1. Percent of total affected population receiving Transitional shelter assistance 2. By end of project, 75% of households in target villages live in shelters that meet Sphere standards 3. 80% of targeted households report the support they received was useful, timely and of good quality ,000households have received transitional shelter, pursuant to Sphere standards 2. 80% of vulnerable households have improved a majority of checklist recommendations 3. % of targeted households have built shelters that meet most Sphere standards for risk reduction, comfort and durability at completion 4. 80% of target households use majority of cash grant for construction cost 1. Within 6 weeks of start of project, 80% of target HH have attended a TA demo or presentation 2. All target HH have received poster / flyer before receiving cash grants for T-shelter construction 1. Within 6 weeks of start of project, 5,000 HH have received cash grants 2. (specific for OFDA funded activities): $236,000 (59%) of approved project budget for Transitional shelter spent in the affected local economy % of total heavily or moderately damaged houses in West Sumatra 6.22% N/A 75% 89.1% 17 N/A 80% 96.0% 0 10,000 households 11,324 households N/A 80% 91.3% N/A 75% 85.6% N/A 80% 96.6% (87.5% spent the full cash grant or more) 0 80% 87.9% 0 100% in targeted communities 34.9% (a total of 70.0% received posters at some point during project implementation) 0 5,000 4,657 HH of 93.14% 0 70% To be calculated at program completion (72.3% at time report was written) 17 This figure is an estimate based on the CRS's field team's assessment of safe/unsafe houses at the time of socialization, compared against the SAD-compliance achieved according to the final evaluation. 30

31 Annex 2: Terms of Reference (ToR) of the Evaluation Term of Reference (ToR) Final Evaluation CRS West Sumatra Earthquake Recovery Transitional Shelter Project Purpose: The purpose of this Term of Reference is to describe tasks and activities of Final Evaluation. This final evaluation will be conducted internally during the period of 25 January 10 February and Mid March April 2010 (TBD) and organized by CRS West Sumatra M&E team. Background: On 30 September, 2009, a 7.6 Richter Scale magnitude earthquake struck off the coast of West Sumatra at a depth of 71 km, followed by a second one (6.2 RS) 20 minutes later. A third earthquake of 7.0 Richter Scale hit Jambi and Bengkulu provinces (south of West Sumatra province) in the morning of October 1st. The earthquakes affected five districts of West Sumatra province. The National Disaster Management Agency (BNPB) reported a death toll of Over three thousand people were injured. A total of 249,833 homes were damaged by the earthquake, including 114,797 heavily damaged and 67,198 medium damaged homes. 18 In response, CRS-Indonesia, in partnership with local NGO Walhi, provided emergency shelter toolkits to 8,700 households during the period of October-November After completion of emergency distribution, CRS/Walhi has implemented a Transitional Shelter (t-shelter) project to help earthquake victims speed their return to normal life. The main objective of the project is to assist earthquake-affected households to have a safe, adequate and durable place to live in the interim period before their permanent houses have been repaired or (re)-constructed. T-Shelter is an urgent need, and cash grants were identified as an effective way to ensure t-shelters are built quickly and with minimal disruption to the local economy. Targeted beneficiaries are all households who were affected by earthquake in identified communities, whose house is determined as unsafe for living, and who have limited ability to provide their own transitional shelter, for lack of materials, lack of finance or lack of labour skill. Objectives: Assess t-shelter project progress focusing on the quality of technical assistance, cash grant, feedback mechanism, timeliness, and shelter completion. Assess the quality of pondoks (the locally used term for t-shelters) in compliance with the standards that were established for the project: measuring Safety, Adequacy, and Durability (SAD). Identify good practices and critical gaps in the project implementation in order to provide recommendations for program quality improvement in the later stages of this programme and in future responses, as well as for general organizational learning. Provide CRS program accountability to community, partners, stakeholders and donors. Deliverables: Final evaluation report that describes the project progress and achievements against the stated and approved overall strategic objectives, intermediate results and outputs to share with all relevant stakeholders and donors. Internal final evaluation report focusing on lessons learned. Methodology: Two separate evaluations of the project will be conducted, the first in January-February, 2010, assessing the project's impact on the beneficiaries served under one specific funding mechanism, and the second in March, 2010, which will provide a comprehensive overall evaluation of the full project. The evaluation will be used for reporting on project 18 West Sumatra and Jambi Natural Disasters: Damage, Loss and Preliminary Needs Assessment. BNPB, Bappenas, and the Provincial and District/City Governments of West Sumatra and Jambi and international partners, October