Evaluation. USIP Grant-making and Grants in Afghanistan. November 27, 2013

Transcription

1 Evaluation of USIP Grant-making and Grants in Afghanistan November 27, 2013

2 Table of Contents EXECUTIVE SUMMARY... 4 BACKGROUND... 7 METHODOLOGY... 8 KEY FINDINGS AND RECOMMENDATIONS: USIP Grant-making USIP s Grant-making Process Capacity Building KEY FINDINGS AND RECOMMENDATIONS: The Seven Grants Afghanistan Watch (AW), Kabul, Afghanistan BBC Media Action, Kabul, Afghanistan Bond Street Theatre Coalition, Ltd., New York, NY The Killid Group (DHSA/TKG), Kabul, Afghanistan Future Generations Afghanistan (FGA), Kabul, Afghanistan Help the Afghan Children (HTAC), Fairfax, VA Sanayee Development Organization (SDO), Kabul, Afghanistan APPENDIX List of In-Depth Interviews Conducted Evaluation of USIP Grant-making and Grants in Afghanistan 2

3 Acknowledgments This study was conducted by InterMedia, based in Washington, D.C., for the United States Institute of Peace (USIP). All materials resulting from this study belong to USIP. The study was carried out in conformity with the standards of the ICC/ESOMAR International Code on Market and Social Research ( All aspects of the project were managed by InterMedia Project Director Najim Dost, and Project Assistant, Ahmad Hemmat, with oversight from other InterMedia staff. About InterMedia InterMedia is an independent, global consultancy specializing in strategic research and evaluation. Since 1995, our quantitative and qualitative research alone has engaged 1.4 million respondents in more than 100 countries, including many of the world s most challenging places. Our clients include international development agencies and philanthropic foundations, media, government agencies engaged in strategic communications and conflict resolution, and commercial firms. We help our clients understand and enhance the impact of policy, programs and investment in the 21 st century s rapidly evolving economic, social and technological environments. InterMedia s offices are based in Washington, D.C., London and Nairobi. InterMedia 2013 Evaluation of USIP Grant-making and Grants in Afghanistan 3

4 EXECUTIVE SUMMARY For more than a decade, the United States Institute of Peace (USIP) has been supporting programs designed to promote public understanding of peaceful alternatives to the violent resolution of conflict, the rule of law, and transitional justice in Afghanistan. In addition, USIP has been providing technical and capacity building support to their implementing partners throughout the grant-making process, particularly in designing projects and developing grant proposals. To date, USIP has awarded 24 grants in Afghanistan as part of its Priority Grant Competition (PGC). USIP contracted InterMedia to conduct an evaluation of USIP grant-making and a set of seven grants implemented by USIP s implementing partners in Afghanistan. InterMedia developed a multi-faceted evaluation methodology involving three components: a thorough desk review of important documents about the USIP grant-making and the set of seven grants was conducted; indepth interviews and consultations were held with a total of 28 USIP staff, grantee representatives and other key stakeholders; and 15 project participants and beneficiaries, per grant, were surveyed to elicit their feedback on the impact the projects may have had on their views, attitudes and behaviors. After mapping the USIP grant-making process, 1 a number of strengths and weaknesses were identified. The strengths include a meticulous review and selection process of grant applications by USIP s qualified technical team. However, the rather lengthy application process, which can sometimes take up to a year, impedes perceived efficiency, especially from the perspective of those implementing partners who can attract much larger grants from other donors that have less strict review processes. The review process also leads to long gaps between one project and another, leading some grantees to view USIP as a one-time donor and not as a long-term partner. One of the main factors contributing to the lengthy process is the application development phase, during which the USIP technical team and the applicants engage in several rounds of exchanges to ensure that the project is well designed and the proposal is well-developed. It is also during this critical phase that most of the capacity building interactions with the applicants take place. This process may be especially challenging and time-consuming for those grantees that lack permanent staff, and may not have one or more of the following advantages: familiarity and prior experience with USIP s requirements and application format, the right mix of expertise in both project design and subject matter, and English-language proficiency, especially writing skills. USIP established an office in Kabul in 2008 to increase its contact with and understanding of events, programs, and attitudes in Afghanistan, and to strengthen its capacity to execute innovative and effective programs in the country. However, significant confusion surrounds the role of the USIP Kabul Office in the grant-making process. Quarterly coordination meetings held in Kabul, in which only those partners that are actively implementing USIP grants are invited, provide the main forum for interaction between the grantees and the USIP Kabul Office. 1 Please refer to Figure 1 (page 10) for an illustration of the USIP grant-making process. Evaluation of USIP Grant-making and Grants in Afghanistan 4

5 USIP s grant-making process could benefit from clarifying the role of its Kabul Office. The quarterly coordination meetings would be even more helpful in cementing the relationships between the implementing partners and USIP if all partners, including those that are currently not actively implementing USIP grants, were invited to participate. USIP s Kabul Office should take a more active role in offering targeted capacity building programs to the implementing partners. In particular, USIP should consider focusing its capacity building programs on key priority areas including project design, monitoring and evaluation, and English-language training for the specific purpose of developing proposals and writing reports. Such targeted capacity building programs will also help streamline the USIP grant-making process by reducing the time it takes USIP s technical team to help grantees in developing proposals. In addition, the USIP Kabul Office could organize a special seminar for the key staff of all implementing partners to orient them to the USIP application format and expectations. Implementing these recommendations will require earmarking additional resources for training and capacity building. However, it can yield significant returns by making developing a tighter network between USIP and its implementing partners. InterMedia also assessed the performance of a set of seven grants against three evaluation criteria: the grants relevance to USIP priorities, the extent to which the grants achieved their stated objectives, and their sustainability. The majority of project participants and beneficiaries interviewed believe these projects make a direct contribution to promoting peace in Afghanistan. For example, in one of the grants, BBC Media Action produced educational radio programs to raise awareness and create a better understanding among rural audiences of the peaceful alternatives to the violent resolution of conflict, which is directly in line with one of the USIP s key priorities in Afghanistan. In fact, each of the seven grants evaluated address at least one of USIP s four priorities, in particular improving public understanding of the costs of conflict and peaceful alternatives to the violent resolution of conflict, and promoting education and discussion about conflict resolution, rule of law and transitional justice. In particular, one grant implemented by the Killid Group (TKG) focused exclusively on fostering transitional justice in Afghanistan. Many of the grants were successful in achieving their specific objectives, albeit to varying degrees. For example, in one of the most successful grants, Help the Afghan Children (HTAC) developed and piloted a grade-specific peace education curriculum, which is well on its way to becoming a part of the national school curriculum. The Sanayee Development Organization (SDO) was similarly successful in promoting community based peacebuilding and conflict resolution in Badghis Province through the establishment of peace shuras (councils). Future Generations Afghanistan (FGA) also succeeded in identifying Positive Deviant (PD) communities, those that are successful in achieving peace, security and development through defying local traditions and applying innovative strategies. They also succeeded in identifying factors that led to these communities successes, but because the Evaluation of USIP Grant-making and Grants in Afghanistan 5

6 grant activities were significantly delayed and were operating beyond budget, they were unable to apply the lessons learned from the PD communities to non-pd communities, as was planned. Afghanistan Watch (AW) also achieved its stated outputs (i.e., producing newsletters called the Truth, and organizing conferences on issues related to transitional justice, reconciliation and corruption). However, they were less successful in contributing to an improved understanding of these issues among national and international policymakers, practitioners and analysts. They were less successful particularly because the newsletters did not always reach their target audiences, and when they did, they were not always read. Among those that did read them, not all the readers were satisfied with the newsletter s quality and depth of analysis, thus falling short of achieving its objectives through the production and distribution of the newsletters. Dependency on donors continued financial support casts doubt on the sustainability of these grants. For example, although past publications of AW s newsletter will continue to benefit researchers in the future, no additional publications may be published nor will any conferences be held after the grant period is over, unless they are funded by another donor agency. Similarly, although the Bond Street Theater (BST) provided the four Afghan art partners (i.e., the theater companies) with business management training, in addition to training on interactive mobile theater, the art partners are unable to generate revenues. They cannot continue their performances without additional financial support beyond the grant period. The abundance of donor funds in the last decade has reduced the ability and incentive of Afghan organizations to think creatively about generating revenue through project activities. Therefore, future capacity building programs should also include sessions on revenue diversification and financial self-sufficiency. This evaluation revealed that USIP is already making significant strides in building the capacity of Afghan civil society organizations to implement effective and innovative projects in the area of peacebuilding and conflict resolution. According to many grantee representatives, what distinguishes USIP from other donors is the USIP s ability to carefully review grant applications, select projects with significant potential, experiment with new ideas, build the capacity of its implementing partners through the grant-making process, and exhibit commitment to peacebuilding in its interactions with the implementing partners. Implementing the recommendations in this report will further enhance USIP s ability to remain a key player in Afghanistan s peacebuilding efforts, especially in this critical time of uncertainty associated with the 2014 transition period. Evaluation of USIP Grant-making and Grants in Afghanistan 6

7 BACKGROUND For more than a decade, the United States Institute of Peace (USIP) has been supporting programs designed to promote public understanding of peaceful alternatives to the violent resolution of conflict, the rule of law, and transitional justice in Afghanistan. The overall theory of change in USIP s grant-making in Afghanistan, although not explicitly stated, is that if USIP can help build the capacity of the Afghan civil society to implement projects on peacebuilding and conflict resolution, we will be one step closer to attaining sustainable peace in the country. Consequently, USIP also has been actively involved in local capacity building efforts that directly engage Afghan citizens and institutions in the understanding of these issues. USIP grants are awarded mainly through two channels, the Annual Grant Competition (AGC) and the Priority Grant Competition (PGC). Afghanistan, along with Pakistan, Iraq and Sudan, fall under the USIP s PGC, which focuses on regions of particular importance to the institute. Grants under the PGC usually range from $45,000 to $140,000. In 2008, USIP established an office in Kabul to increase its contact with and understanding of events and programs in Afghanistan. To date, USIP has awarded 24 grants in Afghanistan, focusing on four priorities: 1. Improving public understanding of the costs of conflict and peaceful alternatives to the violent resolution of conflict. 2. Reducing strife and insecurity through religious and inter-communal dialogue by supporting traditional approaches to non-violent conflict management. 3. Promoting education and discussion about conflict resolution, rule of law and transitional justice. 4. Improving the capacity of the Afghan civil society for dialogue and conflict resolution. In July 2013, USIP commissioned InterMedia to conduct an evaluation of USIP s grant-making and a set of seven USIP-funded grants in Afghanistan. This report covers the period from the launch of the evaluation on Sept. 19, 2013, through Nov. 27, Evaluation of USIP Grant-making and Grants in Afghanistan 7

8 METHODOLOGY USIP s objectives for this evaluation included the following four key questions: 1. To what extent did the grantees in Afghanistan achieve their proposed objectives? 2. To what extent did the Afghanistan grant-making team achieve the objectives proposed in its grant-making strategy? 3. To what extent did USIP s grant-making improve the capacity of the grantees to conduct peacebuilding activities? What, if any training or assistance, did USIP provide in order to help grantees better meet their objectives? Are there additional technical resources or support that USIP could have provided? 4. What lessons should USIP draw about effective ways to conduct grant-making to civil society organizations in Afghanistan? Based on these evaluation questions and initial consultations with key USIP staff, the following criteria were used to measure the performance of USIP s grant-making and the set of seven grants: USIP grant-making evaluation criteria: Process Performance: to map the grant-making process and assess its performance. Capacity Building: to assess the extent to which the implementing partners needs have been addressed and the extent to which gaps are still remaining. Grant evaluation criteria: Relevance: the extent to which each grant addresses one or more of USIP s priorities. Impact: the extent to which the projects achieved their stated objectives. Sustainability: the extent to which the projects impact could last beyond the grant period. In order to assess USIP s grants and grant-making against these criteria, a multi-faceted approach was developed, including the following three components: first, desk research to review important documents; second, in-depth interviews and consultations with USIP staff, grantee representatives and other key stakeholders; and third, a survey questionnaire with project participants and beneficiaries to elicit their feedback on the impact the project had on their views, attitudes and behaviors. In particular, the desk research included a thorough review of the following documents for each of the seven grants: grant proposals, reviewers feedback and grantee response, amendments, progress reports, existing monitoring and evaluation reports, and any other relevant data and documents. In addition, we reviewed USIP s website and key documents, including the Grant Management Guide Memo, Grant Management Guide, Guidelines for Subcontracts and Grants for Spending outside the U.S., Concept Note Template, Application Form, Application Instructions, Financial Management Assessment Form, External Reviewer Sample Letter, and report templates. Evaluation of USIP Grant-making and Grants in Afghanistan 8

9 After a careful review of the above documents, and initial consultations with key USIP staff, interview guides and survey questionnaires, based on logic models created for each of the seven grants, were developed. After the interview guides and questionnaires were finalized in consultation with USIP staff, fieldwork was launched in Kabul, Jalalabad and Sheberghan provinces. In some cases, where program participants were located outside of these three regions, such as those participating in the SDO s grant, Community Based Peace Building and Dispute Resolution in Badghis Province, interviews were conducted over the phone. Overall, 28 in-depth interviews were conducted with key stakeholders including grantees, key staff from the USIP, both in Washington, D.C., and Kabul, and external experts on peacebuilding and conflict resolution. 2 In addition, 15 interviews, per grant, also were conducted with participants or beneficiaries, with the exception of the FGA s grant, Engaging Community Resilience for Security, Development and Peacebuilding in Afghanistan, which did not have typical participants or beneficiaries because it was a research project. For this grant, in-depth interviews were held with key project staff and a key member of the Steering Committee. After the completion of fieldwork, interview transcription and data entry, all of the data from the evaluation components (desk review, in-depth interviews, and survey questionnaires) were analyzed for emerging common themes, trends and insights. Further analysis focused on those particular themes that responded to the key questions the USIP wanted explored. From the analysis, key findings and recommendations were elicited. For the set of seven grants, these include recommendations on how to more effectively achieve desired objectives. In addition, key lessons learned emerged from the analysis that can be applied to future strategies for conducting effective grant-making to civil society organizations in Afghanistan. 2 For a complete list of all interviewees, please refer to Appendix 1. Evaluation of USIP Grant-making and Grants in Afghanistan 9

10 KEY FINDINGS AND RECOMMENDATIONS: USIP Grant-making USIP wanted its grant-making evaluated, with a particular focus on the extent to which its objectives have been achieved as stated in the grant-making strategy, the extent to which it has contributed to the capacity building of its implementing partners, and lessons learned for effective grant-making to Afghan civil society organizations in the future. Given that USIP s grant-making strategy is not currently documented, we sought to first map the USIP grant-making process before assessing its performance, especially relating to building the capacity of its implementing partners. USIP s Grant making Process The following diagram is a simplified illustration of USIP s grant-making process, broken down into four distinct phases: pre-application, application development, review and approval, and project implementation. Figure 1. USIP Grant-making Process Pre-application Phase Applicants hear about USIP grant funding and submit concept note using the USIP template. If it passes the internal review process, applicants receive the USIP Application Package. Application Development Phase USIP technical team works closely with the applicant to design the project and develop the proposal. This phase involves crucial and fequent capacity building interactions with the applicant. Review & Approval Phase Upon receipt of final proposal, the formal review process begins by Andrew Wilder & the Front Office. If the BoD approves, applicants receive Offer Letter & Award Package. Project Implementation Phase Grantees submit the following reports: one-page monthly summary, interm reports and final report, due three months after project completion. Mapping USIP s grant-making process revealed a number of unique features. First, it combines elements of both flexibility and structure. Interim reporting requirements, for example, are flexible; the frequency of reports varies depending on the nature of the grant. USIP is also flexible in terms of accepting applications at any time. The proposal review process, which includes reviews by two external experts, brings structure to the process. Many of our interviewees considered the review process one of the key strengths of the USIP grant-making process. In fact, for some grantees, what Evaluation of USIP Grant-making and Grants in Afghanistan 10

11 differentiates USIP from other donors is the USIP s due diligence, as exemplified by a careful review of applications by USIP s technical team during the pre-application and application development phases. Grantees know the reviewers will read all submitted materials carefully and raise important questions about the project s significance, design, likelihood of completion and impact, and qualifications of the project personnel. Actually we were pleasantly surprised to see how professional this whole process has been. At first, there was an independent group of consultants who checked our proposal and had very interesting questions. We worked hard not only to answer their questions but also to make our own program much more efficient than what we had originally imagined. [In-depth interviewee, implementing partner] USIP is one of the more comfortable donor organizations to work with because they manage to get close to you. All this is mainly because of Mr. Barmak Pazhwak; he makes you feel like he really cares. [In-depth interviewee, implementing partner] I think the strength of this process is that the grant officers usually work very hard with the applicants to develop a very strong proposal. [In-depth interviewee, USIP] For a few other interviewees, however, the process seemed bureaucratic and lengthy. The final review of the proposal by the USIP Board of Directors, many of whom are not sufficiently familiar with the projects and only review a summary of the complete proposal, is viewed as an additional layer to an already lengthy review process. Some grantees reported that it took them nearly a year to get their proposals finalized. For some, such a lengthy review process portrays the USIP grant-making process as inefficient and unreasonably timeconsuming, especially from the perspective of those implementing partners that can attract much larger grants from other donors that have less strict review processes. The abundance of funds in Afghanistan in the last decade, coupled with the multiplicity of donor agencies, with more lenient funding requirements, further exacerbates this perception. The review process also leads to long gaps between one project and another, leading some grantees to view USIP as a one-time donor and not as a long-term partner. During review process, we were bothered a lot! The process was very repetitive, boring and time consuming. We were about to give up on this project even then. The rest of the process went well. [In-depth interviewee, implementing partner] Evaluation of USIP Grant-making and Grants in Afghanistan 11

12 We have been working in Afghanistan for a while. I think we should be funding big and stay longer, and evaluate these activities frequently to make sure that it has the intended impact. If you really want a project to have an impact, I believe it should be longer than one year. [In-depth interviewee, USIP] Another key strength of the USIP grant-making process is the clearly laid out pre-application and application packages of forms and instructions. One section of the application form requires the grantees to carefully think through the project s theory of change and specify outputs, outcomes and methods of verification. A careful reading of the seven grant proposals, however, revealed that not all grantees were able to apply these concepts correctly. A review of the sample letter that is sent to external reviewers of grant applications also revealed important insights about the review process. For example, it clearly specifies four criteria against which to assess each proposal. These include significance, design, likelihood of completion & impact, and qualifications of the project personnel. However, these reviews do not generate quantitative measures of the performance of each grant according to these criteria, which would have made it easier to compare the merits of one grant against another. One of the key challenges facing the USIP grant-making process is the confusion about the role of its Kabul Office in the grant-making process. Almost all grantees, including USIP staff in Kabul, expect the Kabul Office to play a more direct role in all stages of the USIP s grant-making process than it currently does. Quarterly coordination meetings of USIP implementing partners, which started in 2012, provide the only forum for interaction between the grantees and the USIP Kabul Office. It provides the grantees with a unique opportunity to not only build new connections but also to learn from one another s experiences and develop synergies. As Ricardo Grassi of TKG pointed out, learning about other implementing partners activities was a welcome surprise for him during one of these meetings. Similarly, Ahmad Farid Sultani of BBC Media Action was happy to get other partners to deliver their magazines to remote regions across the country. Ajmal Shirzai of FGA wished he knew about the other USIP implementing partners earlier so that he could have selected members of the Steering Committee for his grant project from among them. On the other hand, the fact that only active implementing partners (those that are in the process of implementing a project) are being invited to these meetings creates a temporary disconnect with the inactive partners. Recommendations: Document USIP s grant-making strategy in Afghanistan and lay out its purpose and key objectives. Currently USIP does not have a specific strategy document for its operations in Afghanistan. Sharing this document with the present and future implementing partners would give them an opportunity to better understand USIP objectives and priorities and propose projects that contribute to the achievement of those objectives. Evaluation of USIP Grant-making and Grants in Afghanistan 12

13 Consider incorporating quantitative ratings of grant applications based on the existing evaluation criteria (significance, design, likelihood of completion and impact, and qualifications of the project personnel) to allow for easier comparisons across grant applications. Currently, the reviewers only produce a qualitative assessment of grant applications based on these criteria. Clarify the role of the USIP Kabul Office in the grant-making process and clearly communicate it to both USIP staff in Kabul and in D.C., and especially to the implementing partners. This will help achieve the office s objectives to increase USIP s contact with and understanding of events, programs, and attitudes in Afghanistan, and strengthen its capacity to execute innovative and effective programs in the country. Consider inviting all implementing partners to the quarterly coordination meetings so that all partners, including those that are not actively implementing a USIP project, are able to sustain their relationships with each other and with the USIP. This will also help in cementing the relationships between USIP and its partners and develop greater levels of trust, coordination, and collaboration. Promote USIP projects and grant opportunities more actively among Afghan NGOs and other relevant agencies interested in working in Afghanistan. One of the external experts interviewed believes USIP would benefit from increased efforts to raise public awareness about its grant-making among Afghan and international agencies working in Afghanistan. Diversify USIP s own funding sources in order to sustain its ability to take risks and to experiment with novel ideas and innovative projects. Dependence on a small set of donors can inhibit USIP s ability to remain innovative and flexible in its grant-making activities. Capacity Building Although capacity building is an important component of USIP grant-making, USIP does not have a department or position within its structure dedicated to this purpose. Instead, capacity building is implicitly embedded in the grant-making process. In particular, the USIP staff works closely with the applicants during the application development phase and provides support in designing the project and developing the proposal. Both the USIP staff and the majority of the grantees see this as one of the key strengths of USIP s grant-making. Several interviewees mentioned key USIP staff by name, whom they consider a critical factor in giving USIP a comparative advantage over other donors due to their commitment and hard work during the application development phase. Our capacity building component is one of our areas of strength. We see improvement over time as we are working along with our partners over the longer term. [In-depth interviewee, USIP] Evaluation of USIP Grant-making and Grants in Afghanistan 13

14 I see one main difference [between USIP and other donors], which is teaching through project implementation Creating an environment of discussion [through the quarterly coordination meetings] made an ideal opportunity for most individuals to learn and express their views. [In-depth interviewee, implementing partner] On the other hand, one of the main factors contributing to the lengthy application process lies also in the application development phase. It is at this stage that the USIP technical team and the applicants engage in several rounds of exchanges until the project is well designed and the proposal is well-developed. It is also during this critical phase that most of the capacity building interactions with the applicants take place. While this is a critical phase from the perspective of the USIP, not all grantees appreciate its value, partly because they perceive it as bureaucratic delay or because they don t feel the need for capacity building. In fact, many grantees don t distinguish between the three phases preceding project implementation. From their perspective, the application review process begins when they submit the first draft of the proposal and lasts until they sign the contract. The application development phase may be especially challenging and time-consuming for those grantees that do not have prior experience and familiarity with the USIP application format, requirements and expectations. An efficient process of developing proposals also requires staff members with not only good command of the English language, but also experience with project design and subject matter expertise. Some of these requirements may be lacking more in some grantees than in others. In fact, some of USIP s larger and well-established implementing partners do not feel the need for capacity building programs. A few others, however, are keenly interested in tapping into USIP s expertise through special training workshops, seminars and conferences. For example, SDO staff requested USIP s assistance in setting up a Randomized Control Trial (RCT) to evaluate one of their projects. They were happy to report that USIP provided them with extremely useful guidance on RCT design and implementation. Recommendations: Consider explicitly communicating the capacity building aspects embedded in the USIP s application development phase with the grantees. This will not only enhance their appreciation of the process but also help them develop more accurate expectations about the length of time it might take for their applications to be finalized. Conduct a thorough organizational assessment for all implementing partners to identify their capacity building needs beyond those identified through this evaluation. At this stage, USIP could focus its training and capacity building efforts on the following priority areas: project design, monitoring and evaluation, and English-language training for the specific purpose of developing proposals and writing reports. A well-designed training seminar on project design, monitoring and evaluation will also help USIP s implementing partners improve future grant performance. Evaluation of USIP Grant-making and Grants in Afghanistan 14

15 To continue providing capacity building support to its implementing partners, USIP should consider special resource allocation to either establish an in-house capacity building department or outsource capacity building to external training providers. These capacity building investments have the potential to yield significant returns by making the grantmaking process more efficient and developing a tighter network between USIP and its implementing partners. If capacity building is an integral part of USIP grant-making, USIP should select start-up Afghan organizations as its implementing partners and work with them over the long run to build their capacities both through the grant-making process as well as through more targeted capacity building programs. USIP Kabul Office should organize a special seminar for key staff of all implementing partners to orient them to the USIP application format and expectations. All these capacity building programs and training workshops will also help streamline USIP s grant-making process by reducing the time it takes the USIP technical team to help grantees develop project proposals. Evaluation of USIP Grant-making and Grants in Afghanistan 15

16 KEY FINDINGS AND RECOMMENDATIONS: The Seven Grants The grants were evaluated against three criteria: the grants relevance to USIP priorities, the extent to which the grants achieved their stated objectives, and their sustainability. Table 2. List of the Seven Grants Selected for Evaluation No. Grant No. Grantee Grant Title 1 SG Afghanistan Watch (AW) Media Monitoring, Analysis and Dialogue Group on Transitional Justice, Reconciliation and Corruption 2 SG BBC Media Action Encouraging Conflict Resolution 3 SG Bond Street Theatre (BST) Theatre for Social Development 4 SG The Killid Group (TKG) Fostering Transitional Justice in Afghanistan 5 SG SG SG Future Generations Afghanistan (FGA) Help the Afghan Children (HTAC) Sanayee Development Organization (SDO) Engaging Community Resilience for Security, Development and Peacebuilding in Afghanistan National School-Based Peace Education Program for Afghanistan Community Based Peace Building and Dispute Resolution in Badghis Province Overall Performance of Grants: Each of the selected grants was directly related to at least one of USIP s four priorities, particularly in improving public understanding of the costs of conflict and peaceful alternatives to the violent resolution of conflict, and promoting education and discussion about conflict resolution, rule of law and transitional justice. Noticeable examples include BBC s project to encourage peaceful conflict resolution through educational radio and print programs, TKG s project to foster transitional justice through trainings, conferences, and broadcasting life stories of victims, and SDO s project to promote community-based peacebuilding and conflict resolution by establishing peace shuras and preparing them to engage in local level conflict resolution. The project selection seems quite good. The grant proposals were at the heart of the process and the core of USIP s mission, so the types of issues that were addressed by these grants were very much consistent with the USIP s mission. They talked about peacebuilding, they talked about community awareness, and they were also to some extent experimental, and that they were trying different approaches using the non-traditional approaches. I believe these grants were very much consistent with the USIP mission. [In-depth interviewee, External Expert] Evaluation of USIP Grant-making and Grants in Afghanistan 16

17 Many of the grants were successful in achieving their specific objectives, albeit to varying degrees. For example, in one of the most successful grants, HTAC developed and piloted a grade-specific peace education curriculum and are well on their way to making peace education a part of the national school curriculum managed by the Ministry of Education. FGA also succeeded in identifying Positive Deviant (PD) communities, but they were unable to apply the lessons learned from the PD communities to non-pd communities because the project activities went beyond planned costs and duration. The sustainability of all of these grants depends heavily on the continued financial support of the donor agencies. For example, although past publications of AW s newsletter will continue to benefit researchers in the future, no additional publications may be published nor will any conferences be held after the grant period is over, unless they are funded by another donor agency. Similarly, although BST provided the four Afghan art partners with business management training in addition to training on interactive mobile theater for social development, the art partners cannot generate the revenue to continue their performances beyond the grant period without the additional financial support from donor grants. The abundance of donor funds in the last decade has reduced the ability and incentive of Afghan organizations to think creatively about generating revenue through project activities. Therefore, future capacity building programs should also include sessions on revenue diversification and financial self-sufficiency. 1. Afghanistan Watch (AW), Kabul, Afghanistan Project Title: Media Monitoring, Analysis and Dialogue Group on Transitional Justice, Reconciliation and Corruption AW monitored and analyzed print media outlets, and produced 12 publications of The Truth newsletter in Dari and English, with a particular focus on issues related to transitional justice, reconciliation and corruption. AW also organized public discussions on the above themes, moderated by a panel of experts. Together with the final report, AW also plans to submit an analytical paper on these themes to facilitate further debate and affect policy change. To evaluate this grant, in-depth interviews were conducted with Mr. Jalil Benish, Executive Director of AW, and Ahmad Zia Mohammadi, AW Project Manager. We also interviewed 15 individuals who had either read the newsletter or participated in a discussion session or both. Given the fact that copies of the newsletter were sent to organizations and not a specific individual, it was often hard to recruit interviewees from the recipient organizations because many people didn t know about or had not read the newsletter. Therefore, 13 of the 15 interviewees were recruited among those who participated in AW s discussion sessions in Kabul and two were those who had read the newsletter but not participated in a discussion session. Even among the 13 people who had participated in the discussions, where they should have received a copy of the newsletter, two of them had not heard about it. Evaluation of USIP Grant-making and Grants in Afghanistan 17

18 Although all interviewees said they learned something from this project, few could mention specifics, mainly for three reasons. First, both the newsletter and the conference were mainly in Dari language, rendering them less appropriate for some Pashto speakers. Second, the information presented in the newsletter simply summarized issues broadcast through the print media. Third, the information presented lacked in-depth analysis. Ten of the 15 interviewees said they applied what they learned in this project, mainly in their everyday discussions of these topics. Interviewees pointed to several key strengths of this project. For example, the newsletter makes an excellent archive for future researchers. The discussion groups provide a great opportunity to foster networking and learning opportunities among people from different backgrounds. In fact, one interviewee considered it a great example of democratic practices. Some interviewees found some of the panel speakers, like Ms. Kofi and Mr. Tawhidi, quite knowledgeable on the topics they covered. One interviewee applauded the efforts to redirect the focus of the media from the more trivial issues, such as entertainment, to more critical and serious issues like transitional justice, reconciliation and corruption. On the other hand, they also pointed to areas of improvement. For example, the newsletter could benefit from more in-depth analysis and more professionally written articles. Both the newsletter as well as the discussion groups could benefit from wider media coverage, especially since they currently focus on Kabul only. The selection of both keynote speakers as well as the participants could be more strategic. Some interviewees considered the issues of transitional justice, reconciliation and corruption so critical that each would deserve their undivided attention with follow up activities. Like all other conferences people will forget everything that was discussed in this discussion group if there is no follow up. [Male interviewee, Kabul] There has been extensive discussion on all of the three topics, but nothing has been put in action. From the assassination of the head of peace council to other criminal activities, no one has been tried. [Male interviewee, Kabul] Recommendations: Ensure that future newsletters and discussion groups are presented in both Dari and Pashto languages. Conducting them exclusively in one language may be viewed as marginalizing participants who speak the other language. Consider adding more value to the newsletter by incorporating more in-depth analysis of key themes and issues. Currently, the newsletter simply summarizes articles published in various print media, without taking into account professional editing practices such as distinguishing between paraphrases and quotes from the original sources. Evaluation of USIP Grant-making and Grants in Afghanistan 18

19 Expand the coverage of the project by inviting more media to the discussion groups and inviting guests from provinces other than Kabul. Currently, all the participants interviewed came from Kabul only and they said there was limited media coverage of the discussion groups, especially by popular television channels like Tolo TV. Ensure that future panel speakers are more informed about the topic they present, and represent opposing views in order to generate more discussion and debate. Some of the interviewees who participated in more than one discussion session asserted that not all panel speakers were experts on the topics they covered. Consider funding more follow-up activities, especially practical steps to address gaps on the themes of transitional justice, reconciliation and corruption. Without follow-up activities, the impact of the program will be short-lived and USIP projects will risk being viewed as one-off and disconnected. 2. BBC Media Action, Kabul, Afghanistan Project Title: Encouraging Conflict Resolution From 2002 to 2012, the Afghanistan Education Production Organization (AEPO) had been operating under BBC Media Action and producing educational radio and print materials including the popular New Home, New Life radio drama series. Although AEPO continues to provide the same services to BBC, they became an independent Afghan organization as of April Under this grant, AEPO produced 90 scenes of the New Home, New Life radio drama series, 20 scenes of Learning for Life drama series, 10,000 copies of the When there is peace booklet, and 16,000 copies of New Home, New Life magazines, all of which were in both Dari and Pashto languages. The print materials were distributed to local communities across the country through a network of AEPO staff and partner organizations. Since the specific objective of this project was to raise awareness and create a better understanding among rural audiences of the peaceful alternatives to the violent resolution of conflict, 15 regular listeners of these radio programs were interviewed. In addition, an in-depth interview was conducted with Ahmad Farid Sultani, AEPO project manager. Few people knew about BBC s print materials and magazines because they are not widely distributed to the public. On the other hand, all 15 people interviewed knew about BBC s New Home, New Life drama and listened to it on the radio at least once a week. Among the key strengths of the program are the locally relevant stories, familiar names and characters, and important lessons for life in Afghanistan s two main languages, Dari and Pashto. In fact, many of the interviewees could relate to the stories and remember key characters by name. They have also applied what they learned through these programs in their daily lives. Evaluation of USIP Grant-making and Grants in Afghanistan 19

20 The interviewees, however, also pointed to areas of improvement. For example, some listeners wanted to see some changes and innovations in the radio program, including broadcasting it in Afghanistan s other local languages, broadcasting parts of it through television, and making the stories more relevant to urban dwellers. A few others suggested that preaching messages of peace to ordinary citizens will have no impact when the ordinary citizens do not lie at the root of the conflict and those who are responsible for much of the national level conflict do not listen to these programs. AEPO Project Manager Ahmad Farid Sultani emphasized that three decade of war have negatively affected people s views. Changing people s attitudes and behavior about these issues requires a much more long-term approach. He proposed that future projects could focus on training provincial staff on peacebuilding and using theater as a means of societal change. Recommendations: Consider innovative modifications to the New Home, New Life drama series, including broadcasting it in Afghanistan s other local languages, broadcasting parts of it through television, and making the stories more relevant to urban dwellers. Much of the storylines and characters date back to the early 1990s when the program was first launched in Peshawar, Pakistan. While the continuity of stories and characters provides comfort and familiarity to some audiences, others view it as out-dated and repetitive. The recent rising trend in urbanization calls into question the relevance of programs overly focused on rural settings. Re-examine the root causes of conflict in Afghanistan and adjust the programs accordingly. Currently the majority of ordinary Afghan citizens believe preaching messages of peace to them is barking up the wrong tree. Future programs could, more specifically, emphasize the role of ordinary citizens in affecting national level change, including in bringing peace to the country (e.g., by making their voices heard through voting). 3. Bond Street Theatre Coalition, Ltd., New York, NY Project Title: Theatre for Social Development Under this project, BST organized a series of workshops on theater for social development for four art partners across Afghanistan, including Simorgh Theatre in Herat, White Star Theatre in Kabul, Nangarhar Theatre in Jalalabad, and Maiwand Theatre in Kandahar. The project aimed to build the capacity of these theater companies. The workshops were specifically designed to improve learning abilities, communication skills, self-confidence, teamwork, leadership, conflict resolution and prevention. The training also included a component on proposal writing, program evaluation and budgeting. Eight performances were conducted in four provinces, affecting nearly 8,000 male and female viewers. BST also produced a training manual to disseminate best theater practices among future artists and NGOs. In assessing the performance of this grant, interviews were held with 13 artists who participated in these workshops from Nangarhar, Herat and Kandahar. However, all the staffs of the White Star Theater, which was also supposed to be the resident theater at Kabul Evaluation of USIP Grant-making and Grants in Afghanistan 20

21 University, were unavailable during the time of fieldwork. 3 In addition, we conducted in-depth interviews with key project staff, including Anna Zastrow, Communications Director at BST, Abdul Hakim Hasehmi, Manager of Simorgh Theater in Herat; Ahmad Jawed Watanyar, Director of Kandahar Theater; and Said Karim Zhwandoon, Director of Nangarhar Theater. All the arts partners interviewed spoke highly of their relationship with BST and the knowledge and skills they received from BST trainers, particularly relating to interactive mobile theater. We have worked with several other organizations, including Norway, BBC, etc., but the workshops offered by BST were especially useful because they were teaching us new techniques. [In-depth interviewee, art partner] I think this particular project was extremely successful. We established a great cultural relationship with this theater group and also in the community, because we lived in the community and interacted with the community. We gave them the tools to continue the work by their own. [In-depth interviewee, implementing partner] The workshop participants were equally satisfied with the training they received. Key strengths of the workshops included the focus on new techniques, the importance of warm-up before any exercises, physical theater practices, and participatory theater practices in which the audiences are actively engaged. Many of the artists interviewed reported they applied what they learned through these workshops in performances and in training others. This training was unique. They showed us so many activities that I had neither heard nor seen in Kandahar. [Male interviewee, Kandahar] The training of female participants was seen as a welcome sign by several interviewees. Given the cultural context, however, they face particular socio-cultural barriers in participating in theater workshops and continuing to work in performances. some of the girls that were involved in the shows are no longer allowed by the families anymore to be involved in performing. I think it s typical and very controversial for girls to perform. [In-depth interviewee, implementing partner] 3 Two key staff at the management level are currently pursuing academic degrees outside the country and the third key staff was out of the province for Eid holidays and could not be reached by phone despite repeated attempts. Evaluation of USIP Grant-making and Grants in Afghanistan 21