Strategic Evaluation of the Japan Social Development Fund (JSDF)

Transcription

1 World Bank Japan Social Development Fund Strategic Evaluation of the Japan Social Development Fund (JSDF) FINAL REPORT 1 st June 2007 Prepared by

2

3 ITAD LTD 12 English Business Park English Close HOVE BN3 7ET United Kingdom Telephone:

4 Acknowledgements The study team would like to acknowledge and thank those who gave their time to the evaluation and contributed their ideas, knowledge and experience of the JSDF Programme. It is of course impossible to list the many contributors by name, but they include the World Bank Staff who participated in interviews and presentations in Washington and Bank staff in country offices who facilitated the project visits. We also wish to thank the representatives of participating governments, civil society members, project staff and the communities and individuals involved in the ten projects visited, who generously gave their time to describe their experience of JSDF. We would like to thank the Government of Japan personnel whose contribution was essential to the evaluation, in particular in the Ministry of Finance; Messrs Shigeki Kimura, Director Development Institutions, International Bureau, Kazuteru Sato and Osamu Yoshida; in the World Bank, Masanori Matsuo, Advisor to the Executive Director for Japan; and the representatives of the Ministry of Foreign Affairs, JICA, JBIC and staff of the World Bank office in Tokyo. We are grateful to David Potten, Head of Trust Fund Program Administration for his guidance and his detailed comments on the reports and Magdalena Manzo, Senior Operations Officer Trust Fund Operations at the Bank for managing and coordinating the evaluation exercise and for the support she provided to the team throughout the course of the study. The study was carried out by a core team of consultants from ITAD Ltd (UK), comprising Mike Felton (Team Leader), Ann Condy (Social Development Expert), Corrie Bell (Evaluator & Web survey Design), Charles Chandler (Programme Performance Evaluation) and Chris Barnett (Evaluator & Evaluation Methodology). Seven of the field Missions were led by a member of the Core Team and the remaining three were led by Johanna Pennarz (Senior ITAD Consultant), Muriel Visser, Tevan Poghosyan and Armen Galstyan.

5 Abbreviations ADB AF AFR CAS CBD CBO CDD CSO DAC DMC EAP ECA ED GDO GRM IBRD ICM IDA IEC IEG ISPRP JBIC JFPR Asian Development Bank Application Form Africa Region Country Assistance Strategy Community Based Development Community Based Organisation Community Driven Development Civil Society Organisation Development Assistance Committee Developing Member Country East Asia and Pacific Europe and Central Asia Executive Director Grant Development Objectives Grant Reporting and Monitoring International Bank for Reconstruction & Development Implementation Completion Memorandum International Development Association Information, Education and Communication Independent Evaluation Group Indonesia Strategic Poverty Reduction Partnership Japan Bank for International Cooperation Japan Fund for Poverty Reduction II

6 JICA JSDF LAC MDG MENA NGO NMPRP OECD OED ODA PCF PEP PRSP PRSTF SAR SC SD SDV TF TFO TTL Japan International Cooperation Agency Japan Social Development Fund Latin America and Caribbean Millennium Development Goals Middle East and North Africa Non Government Organisation Northern Mountain Poverty Reduction Project Organisation for Economic Cooperation and Development Operations Evaluation Department Official Development Assistance Post Conflict Fund Project Evaluation Pro forma Poverty Reduction Strategy Paper Poverty Reduction Strategy Fund South Asia Region Steering Committee Social Development Social Development Group (in the World Bank) Trust Fund Trust Fund Operations Task Team Leader III

7 Table of Contents Acknowledgements...I PURPOSE AND OBJECTIVES OF JSDF...i Executive Summary...ii Recommendations...xi 1. The Context - Strategic Evaluation of JSDF Scope and Context of the Evaluation Purpose of the Final Report A Note on the Presentation of Results... 1 ACHIEVEMENT OF THE CORE PURPOSE OF JSDF Assessment of JSDF s Strategic Purpose Aim of this chapter Reaching the Poorest and most vulnerable the JSDF Community Development Approach Participation and Empowerment and the Contribution of Civil Society Capacity Building How Innovative is JSDF? Sustainability Rapid and demonstrable results Re-focusing JSDF Objectives...19 PROJECT RESULTS AND PERFORMANCE Results of JSDF A study of Project Effectiveness JSDF Project Objectives Assessing Results the context Aim of this section The meaning of result terms a clarification An Assessment of Results Outputs...28 IV

8 Assessment of Outputs - Findings from the Desk Review (43 projects) Assessment of output performance for the ten site-visit projects Assessment of results by component type or theme Conclusions on output performance Outcomes achievement of Grant Development Objective Achievement of higher level results Assessment of impact by Project Project Level Impact - Conclusions The Value Added of JSDF Operations The potential for lesson learning Field Visit Evidence of Value Added Informing wider development practice in the Bank...44 IDENTIFICATION, IMPLEMENTATION & PROJECT MANAGEMENT Project Identification & Relevance Project Identification Relevance to Bank and Country Strategies Rationale for Grant vs. Bank Lending Links to IDA projects, strategy alignment and sustainability Complementarity with World Bank Projects Achievement of JSDF outcomes vs. other sources of funding Project Design Clarity of intervention logic & defining the objectives at design stage Stakeholder involvement at design stage The Seed Fund Partnerships, Collaboration & Visibility of Funding Project Implementing Partners Identification & selection of implementing partners Performance of Implementing partners (NGO / Local Government) Visibility of Source of JSDF Funding...61 JSDF PROGRAMME MANAGEMENT...62 V

9 7. The World Bank as Manager of JSDF Grants Role of the Bank in Managing JSDF Grants JSDF Management Performance Programme Level Perspective Recent changes and the simplification process Conclusions on Grant Application Process Management Roles during Grant Implementation Project Level Management by the Bank Recommendation for New Approaches to JSDF management Key Features Description of Quality Standards for new and existing grants Monitoring, Evaluation and Links to Lesson Learning Introduction JSDF monitoring and reporting systems Project-based M&E design Objectives and linkage to well-defined indicators Use of indicators Performance of project-based M&E Findings from the country visits Lesson Learning and Information Dissemination Concluding remarks (Monitoring and Evaluation) The JSDF Portfolio: A Quantitative Assessment Value of Fund Regional Distribution Country Distribution Sector Classification Project Duration Grant Values...87 VI

10 Appendices (Under separate cover) A. Terms of Reference B. List of Grants in JSDF Portfolio C. Methodology C.1. Evaluation Framework C.2. Document Review C.3. Interview Topic Guide C.4. Field Visit C.5. Web Based Survey D. Results D.1. D.2. Document Review Field Visit D.2.1. PEP Scores D.2.2. Assessment Frameworks (for each site-visit project) D.3. Web Based Survey VII

11 PURPOSE AND OBJECTIVES OF JSDF The main purposes of the JSDF are to 1 : Support innovative programs which directly respond to the needs of the poorest and most vulnerable groups in society; Provide rapid and demonstrable benefits, which can be sustained, to the poorest and most vulnerable groups; Build capacity, participation and empowerment of civil society, local communities and NGOs The objective of the Japan Social Development Fund (JSDF) 2 : To provide grants in support of innovative social programs to help alleviate poverty in eligible client countries of the World Bank Group. The focus: JSDF Grants complement Bank-financed operations and programs compatible with the development objectives of the CAS, PRSP or poverty reduction elements of Sector Strategies. The Grants are intended to focus on activities which: (i) respond directly to the needs of the poorest and most vulnerable groups; (ii) encourage the testing of innovative methods; (iii) support initiatives that lead to rapid, demonstrable benefits with positive prospects of developing into sustainable activities; or (iv) build ownership, capacity, empowerment and participation of local communities, non-governmental organizations (NGOs) and other civil society groups to facilitate their involvement in operations financed by the World Bank. Approximately 50% of total JSDF funds should go to eligible countries in East, South and Central Asia. 1 EXTMAINPRO/EXTJSDF/ 2 FY07 Policy Guidelines and Program Allocation (JSFD) i

12 Executive Summary Introduction 1) The Final Report of the Strategic Evaluation of the Japan Social Development Fund (JSDF) presents the results and synthesis of the information obtained from the desk review and stakeholder interviews carried out as part of Task I activities and the findings from country visits to 10 projects and the web based survey undertaken as Task II. It also draws on the three meetings / presentations which took place with the Evaluation Steering Group in May, July and November 2006 and on the written comments received from the World Bank and the Ministry of Finance, Government of Japan on the Draft Final Report. This June 2007 version of the Final Report responds to a request from the Advisor to the Executive Director for Japan, the World Bank for clarification on the specific points raised by the Japanese Ministry of Finance in February. 2) The objectives of the evaluation are to: Review the progress and development impact to date of JSDF Assess the achievements of JSDF as a whole and of selected grants, in the light of the Fund s objectives Make recommendations on ways in which the Fund might better achieve its objectives in the future. 3) The purpose of the Report is to present the results and conclusions of the study and a set of recommendations. Note on ratings / scores: The Evaluation Team ratings: 1-high; 2-substantial; 3-modest; 4-low 3 World Bank ratings: Highly satisfactory; Moderately satisfactory; satisfactory; not satisfactory RESULTS & CONCLUSIONS Achievement of the Core Purpose of JSDF 4) JSDF has achieved considerable success based on the assessment of the extent to which the core purpose and project results have been achieved. There is no case for complacency, but JSDF, funded as it is by Japan, occupies a unique place among the portfolio of Trust Funds administered by the World Bank. 3 A summary table of scores / ratings on key questions is given in Appendix D ii

13 Reaching the Poorest and Most Vulnerable 5) Key Findings: Based on desk study results JSDF projects were rated as substantial overall in terms of reaching the poorest and most vulnerable, although this is reflected to a greater extent in particular types of projects: Projects with a safety net theme (e.g. childcare system reform) were generally successful in reaching the most vulnerable while projects with productivity objectives (e.g. through micro-credit funds) tended to work with more active poor households. Task Team Leaders (TTLs) believed strongly that JSDF projects focus more on the poorest and most vulnerable than other Bank managed interventions. 6) The field visit findings show the inclusion of beneficiaries in project design is usually limited to community group or Civil Society Organisation (CSO) involvement, but at implementation stage, attempts to avoid elite capture received one of the lowest ratings overall, with most projects judged as modest in this respect. 7) Conclusion: It is extremely challenging for any development project to target the poorest and most vulnerable effectively and all too often these initiatives are thwarted by the active efforts of more powerful groups to take control of the benefits. JSDF s modest to substantial success in achieving this objective, and the fact that TTLs consider JSDF takes on this challenge more than other Bank instruments, suggests that JSDF has developed a special niche in this area. Participation and Empowerment and the Contribution of Civil Society 8) Key Findings: The desk study rated participation and empowerment as substantial. The field visit findings suggested a more modest assessment, and frequently participation rather than empowerment was in greater evidence. In the Web survey, TTLs only seldom mentioned participation as being a most important feature of the JSDF programme and there was no mention of empowerment. 9) Conclusion: Empowerment can be regarded as the most successful form of (or result of) participation which tends to take a long time to achieve, requiring attitude and behaviour changes. While substantial participation took place during implementation in particular, there was virtually no evidence of empowermen t occurring, which should be viewed as an impact level goal extending beyond the time frame of JSDF grants. Capacity Building 10) Key Findings: The Bank s own ratings are satisfactory or highly satisfactory, with respect to capacity building, although not much evidence is provided. Anecdotal comments from TTLs indicated a relatively positive assessment. The field visits findings confirmed an overall substantial rating. Despite the common lack of indicators, the Evaluation Team found capacity was raised in iii

14 both technical areas and on participatory/facilitation processes and skills. TTLs believed JSDF enhanced the Bank s best practice in capacity building, but the lack of formal channels for lesson learning make it unclear how this is achieved. 11) Conclusion: Capacity building is a key feature of JSDF projects and is one where there is substantial achievement overall. It could be greater if there was a clearer link between capacity building and the project development objectives and if projects were implemented over a longer timeframe. There is a trade-off between hiring experts to ensure good quality capacity building (e.g. using consultants) and institutionalising the capacity which is built among government agencies and national NGOs. However, a true measure of achievements in capacity building is an organisation s institutional effectiveness, which would be a more appropriate objective. How Innovative is JSDF? 12) Key Findings: Projects were ranked as having a high achievement with respect to innovation. Specifically, 37% of the project portfolio reviewed at desk study phase and five out of ten of the projects visited rated as high (33% and four projects respectively rated as substantial ). The web survey found that 37% of respondents strongly agreed and 54% agreed with the statement that JSDF projects were innovative. Innovation was also identified as the single most important feature of the JSDF by TTLs. 13) The most common form of innovation was in community development components or in ways of working with institutions and in partnership e.g. working with national NGOs as implementing agents (even when the overall project was not innovative in itself). Some projects were innovative in the national context; others were said to be innovative in terms of reaching particular marginalised or remote groups (including mountain communities, squatters and trafficked children). In contrast, some Bank stakeholders expressed concern that innovation was either not a transparent concept or not relevant. 14) Conclusion: The majority of JSDF projects studied demonstrated a range of innovative characteristics. However, the Evaluation Team s view is that innovation is not an outcome oriented objective and tends to relate more to strategy or approach, and is therefore difficult to measure. Given that it is held to constitute the most significant core feature of JSDF, it should remain as one of JSDF s strategic features, but requires a more precise definition. Sustainability 15) Key Findings: The desk review found the Bank s own assessment of sustainability varied between satisfactory and highly satisfactory. The Evaluation Team produced a rating of modest for five of the projects visited, one low, three substantial and one high. The findings from the field visits iv

15 suggested therefore, that slightly less than half the projects had substantial sustainability prospects. 16) In all but one of the cases where sustainability was ranked as substantial the grants were (or had been) scaled-up into components of an IDA project, in some cases with other donor support. There were also examples of sustainability of certain project activities where local community associations have maintained former project activities. However, this does not equate with sustainability of the project approach and development objective. 17) Without the prospect of IDA support at project closure, the time available for project implementation is considered too short to develop and test sustainable models. There is a trade-off between focusing on ongoing learning and improvements of the innovative approach, and the need to rush to produce replicable models for a linked IDA project. 18) Conclusion: Because the majority of projects are of a pilot nature, include innovative features and, a substantial capacity building element all within a relatively short implementation period; they are unlikely to provide clear evidence of sustainability at project completion. However, including sustainability as a key feature does provide a check on what type of projects are approved. The broad term sustainability should be clarified in terms of the specific use of up-take / scaling up/ or replication of the project which should lead to a sustainable programme over the longer term. Project Results and Performance Project Component and Output Performance 19) Key Findings: The sheer variety of JSDF projects types and themes presents a challenge to synthesising project results. Undertaking a meaningful assessment of project performance using current JSDF monitoring forms is difficult as the standard documentation is too generalised and does not follow through on measuring appropriately selected key indicators to learn lessons from the reporting process. 20) There is confusion in the use of terms between outputs, outcomes and impact in describing JSDF projects both at the design stage and during reporting of implementation progress and performance. 21) The desk review, based on the Bank s own ratings showed a quarter of the portfolio had all project components implemented to a satisfactory level, with a further quarter rated at satisfactory to highly satisfactory. Only some 10% of projects had serious problems of non-achievement. Of the 10 projects visited in the field as a group seven were assessed as having output levels of good quality with some weaknesses, with three having moderate quality of outputs with several weaknesses. Four of the ten had discontinued one or more (mostly peripheral) components. v

16 22) An analysis of performance for the most common types of component found that projects with a policy reform agenda occurred in three instances. In two cases the projects had some success in reforming policy, although the generally short timeframe of JSDF projects mitigates this aim. On the other hand, the Evaluation Team consider that the Bank could exert more influence on policy reform through JSDF grants, and there are some examples of missed opportunities in this respect. 23) Where micro-finance is a central feature of a project (as in the case of the Lao Village Investment for the Poor) this approach can be successful (and can serve as a model for replication nationally). JSDF has successfully pioneered approaches to Community infrastructure development with beneficiaries taking full charge in terms of selection of type of infrastructure, project management and payment for the work. 24) On the other hand, where infrastructure involves larger projects (as in the Armenia day-care centres and in the case of community centres in Benin) structures may be left unfinished at project completion and / or not fully used. 25) Conclusion: Projects which have a relatively uncomplicated mix of components or sub-activities complete the planned components. The relatively short timescale of JSDF projects is thought to be a factor in under-achievement of project outputs, with examples where projects have been extended in order to complete the work envisaged. For other components, the complexity of the finance and procurement side, coupled with the recipients own internal management constraints, consistently led to significant delays in implementation. Outcomes - Achievement of Grant Development Objective 26) Key Findings: Bank monitoring reports gave 58% of projects as satisfactory and 21% highly satisfactory in terms of meeting the Grant Development Objective. However, the Team noted that comments provided in the GRMs and ICMs were quite weak and often none were provided. Of the 10 projects visited, only six had listed suitable performance indicators. 27) The evaluation did encounter some examples of impact in the projects visited, including child trafficking in Benin; improvements for children and their families in two towns in Armenia (albeit in the latter case at a relatively small scale); household income increases as in Lao (VIP), Indonesia (Marginal fishing communities) and in Uganda (Youth vocational training). However, there are difficulties of measuring attributable income changes, so these results should be treated with caution. Given the scale of the micro-loans provided, the evaluators formed the view that incom e benefits were supplementary rather than sufficient to make a fundamental change in people s lives. vi

17 28) Conclusion: The site visits confirmed that JSDF project have made some tangible difference to people s lives, albeit with the qualification that many of the projects studied are pilots and the scope is, in some cases, quite limited. Value Added of JSDF Operations 29) Key Findings: With the field experience gained from the range of JSDF grants completed, the potential for learning lessons and adding value to Bank and recipient country operations is immense. The desk study indicated that JSDF is most likely to add value by the introduction of new or enhanced community development approaches or in operationalising principles of working with NGOs. 30) Based on field visit evidence, while there are examples of the JSDF grants adding value to their, or recipient country operations, the results are mixed. Where grants were linked to an IDA project, there were clear examples of new practices being adopted. Ecuador provides an interesting example of law and justice being given greater visibility as an important factor in social development and poverty reduction, with the potential (if the mechanisms existed) for influencing both the country s and the Bank s policy on a wider scale. On the other hand, on other projects in India, Benin and Indonesia there was little evidence of JSDF adding value to Bank operations. 31) Conclusion: Where experience gained on JSDF grants is potentially relevant more generally, it is not evident how JSDF influences Bank practice as channels are not monitored. This raises a fundamental issue as to whether JSDF is a testing activity for the Bank s core business, and if so, what mechanisms should be in place for ensuring lessons are learnt and communicated. It is concluded that there is a significant gap in the lesson learning process which JSDF needs to address. Identification, Relevance and Design Relevance to National Strategies and JSDF Objectives: 32) Key Findings: The majority of projects scored highly in terms of relevance to national strategies (seven out of ten), and to JSDF core purposes (eight out of ten). There is a correlation between links with Bank IDA projects, alignment to National Strategies and sustainability. Some projects without a close link to an IDA project also had some degree of sustainability, but the challenges to ensuring sustainability tend to be greater. 33) Although six projects visited had a close link to an IDA project, implementation delays in the JSDF grant weakened the complementarity of this relationship in three projects; in two cases, the JSDF grant emerged following problems with an IDA loan project. vii

18 34) Conclusion: JSDF has been successful in identifying and funding projects that are relevant to National strategies, country contexts and JSDF core principles. The Evaluation Team does not recommend any specific changes in the way in which projects are currently identified. Project Design 35) Key Findings: In relation to the level of stakeholder participation identified earlier, the web survey results suggest (with caveats due to limited data) that Seed Funds have increased the active participation of stakeholders in the project design stage particularly for national government, provincial/local government and NGOs. Partnerships Collaboration & Visibility of Funding 36) Key Findings: JSDF is successful in identifying and selecting its partners. The performance of implementing partners is generally good and the relationship that projects forge with NGOs is fruitful to both the JSDF project and the NGOs themselves. Japanese NGOs have played an important role in a number of projects and have expressed an interest in increasing their participation. In spite of efforts by the Bank to explain how NGOs can become engaged as implementing partners, there appears to be a lack of understanding of the process. 37) Visibility of source of funding: The Ministry of Finance and Ministry of Foreign Affairs in Tokyo have indicated that the funding source should be acknowledged, with the level of visibility being proportionate to the scale of the grants. However, there are no specific guidelines in this regard, and as a result, visibility of source of funding (and knowledge of source of funding by stakeholders) was expressed in a variety of ways in the projects visited. 38) Conclusion: The positive working relationship between JSDF and NGO partners has been good for both the NGOs and successful achievement of project outcomes. Communication between the Bank and NGOs (in particular those in Japan) on the prospects and opportunities for NGO participation in JSDF projects needs to improve. 39) Visibility on source of funding expected by the Government of Japan: Simple, straightforward guidelines on the level and type of visibility on source of funding should be provided to TTLs and grant implementing agencies. JSDF Programme Management The World Bank as Manager of JSDF Grants 40) Key Findings: JSDF grants are managed by the Trust Fund Operations (TFO). The role of TFO is primarily exercised during review of proposals and approval of grants (in cooperation with the Government of Japan). The site visits indicated viii

19 that Bank performance was generally of a good standard with a score of high or substantial for six of the projects, with one rated as modest and one low. Grant Application and Implementation Issues 41) The mostly commonly voiced concern is the time it takes to approve a grant (commonly about a year). Opinions of any TTLs indicate that the application process was cumbersome and bureaucratic. The time taken may also result in TTLs who start the process being re-assigned before the project is complete. Similarly Japanese staff in Embassies who are involved at the initial stages might also be re-assigned by the time the project is implemented. A further issue relates to the lack of incentives for TTLs to dedicate time to working on relatively small projects rather than large loans. Thus, it tends to be the really committed TTLs who will apply for grants. 42) Once grants are approved, operational responsibility is handed over to a TTL, with the TFO exercising very little oversight during implementation. One specific management problem identified during the site visits was the issue of disbursement. Implementing partners often did not have the capacity initially to deal with the Bank procurement procedures and where there are a number of implementing partners, channels for accountability and payment are complex. This has been a factor leading to quite serious delays in implementation in a number of cases. 43) Conclusion: While there may appear to be no specific incentives for TTLs as such, there is a sufficient number of high quality applications from TTLs who have the interest and commitment to apply and follow through with the implementation of appropriate projects. The Evaluation Team believe that the simplification process underway during the period of the study will have addressed many of the issues identified in particular in terms of speeding up the application process and in improving the level of transparency in terms of giving clear reasons for grant approval or rejection and that overall the process does not limit the level of quality applications or the efficient utilisation of the grant. 44) The Evaluation Team identified opportunities for improving quality assurance at key stages of JSDF management. This includes a Quality at Entry Process; Quality of Supervision and Closure / handover, and; Management Oversight Process. Monitoring, Evaluation and Lesson Learning 45) Key Findings: Monitoring and evaluation has an important role to play in the context of JSDF grants as part of sound management practice and as a central element to learning lessons from pilot projects and innovative approaches. Achievements in this area are however disappointing, especially at JSDF programme level and work is underway to address many of these failings. ix

20 46) Issues Identified: The field visit assessment indicated that many interventions have a relatively strong design logic and rationale, although this is not explicitly demonstrated in grant proposal documents. Indicators (linked to objectives) are often poorly articulated. Performance of M&E could be improved through greater attention to design issues (especially the development and articulation of outcome-level indicators) and improved consistency and adherence to quality standards for M&E. Urgent attention needs to be paid to lesson learning and communication mechanisms, which is integral to achieving JSDF s strategic objectives. 47) Conclusion: M&E is clearly an area that needs strengthening with an emphasis particularly on capturing lessons and communicating performance to a wider audience, as well as improving governance and accountability. Currently monitoring and reporting is characterised by a weak focus on outcomes/results and a lack of consistency between project-level M&E systems. Approaches need to be developed to improve lesson learning and communication, from the early design stage (e.g. support to develop communication strategies) to follow-up support (e.g. additional financial resources to enable scaling-up). The JSDF Portfolio A Quantitative Assessment 48) Key Findings: Since grants valued at over US$135 million have been implemented in almost 50 different countries worldwide, with just over US$33 million worth of grants approved in Overall approximately 39% of JSDF grants are in East Asia and the Pacific region (EAP), although the proportion is falling annually with only some 30% of projects in this region in If EAP and SAR are taken together this accounts overall for 52% of the portfolio (compared to the Policy Guidelines target of 50%). The most significant increase in the number of grants by region has occurred in Africa, with this region accounting for some 30% of the portfolio grants in Of the 116 eligible countries only 42% (49) received one or more grants. 49) The Fund is not sector specific and 37 sector/themes are covered according to the Bank s classification. In an attempt to rationalise the classification, the Team condensed these to 12 categories. Using these, the most common sector is multisector with 36%, followed by Health 12%, Rural Development 11% and Education 10%. 50) Conclusion: The Fund has shown a steady growth, with changes in regional distribution of Funds with the EAP region receiving less than at the start. Although this trend, and in particular the increase in grants to Africa may not have been planned as such, it does reflect the current international focus on that region. In spite of the increase in grants to the Africa region, some 27 of the 43 eligible countries had not received any grant. Countries in the North Africa and Mediterranean region are poorly represented. x

21 Recommendations The purpose of this section is to make recommendations on the way in which the Fund might better achieve its objectives in the future. A tentative road map (Table R.1) has been drawn up to illustrate the changes that might be expected to result from implementing these recommendations: In summary, the recommendations are framed around the following key areas: A. Re-defining the purpose statement as a result-oriented statement with measurable outcomes at the Grant Development Objective Stage B. A re-formulation of JSDF s Strategic Aim to include the majority of the current elements of the purposes as key features which define and guide JSDF, but are not objectives in themselves C. Developing a strategy for improving the sustainability of results after the completion of the JSDF grant into the uptake, scaling-up or replication stage D. Enhancement of lesson learning and wider communication of results, built into the programme level and individual project design and implementation process E. Improvements to the programme cycle management process and the adoption appropriate quality standards for each stage F. Incorporation of the lessons and knowledge from JSDF into the Government of Japan ODA programme and other enhancements to visibility G. Making adjustments to the Policy Guidelines on regional distribution A. A re-defined, result-oriented Purpose Statement 1) A re-formulation of JSDF purpose statement is recommended, which makes a clear statement that JSDF is a result-oriented programme, with measurable outcomes at the Grant Development Objective stage. However, JSDF is a unique programme, and the purpose statement needs to retain those characteristics of JSDF, which define its strategic nature (see recommendation B below). A new purpose statement would need to be developed and agreed in a workshop environment. For the purpose of this report and to initiate the discussion, the Evaluation Team propose, inter alia: 2) A purpose statement for the programme which states that each proposal for funding would need to show that its Grant Development Objective satisfies one or more of the following objective categories: xi

22 The productivity of the targeted poor is enhanced. The use of social and community services by the poorest and/or most vulnerable is increased. The protection and well-being of the most vulnerable through social safety nets or institutional support is enhanced. The use of comm unity level infrastructure by the poorest and/or most vulnerable is increased. 3) In order to enhance the assessment of overall performance of the portfolio, it is recommended that generic outcome-level indicators are identified for each of the objective categories (in the redefined Purpose Statement ) Outcome-level indicators can be tracked and then aggregated from individual project data. Indicators could include, for example: use of, and satisfaction with productive enterprises or services; increased level of access and use of social services; access to, take up of and, satisfaction with, for example, social safety nets; availability, use and satisfaction with community infrastructure. In each case, an attempt would be made to estimate the number of beneficiaries (by gender / poverty / vulnerability status). B. A re-formulation of JSDF s Strategic Aim to define and guide JSDF 4) The characteristics (currently in the programme s purpose statement) which define and guide JSDF include its focus on the poorest and most vulnerable, the use of participatory approaches, capacity building initiatives and innovative practices. These, together with the involvement of CBOs and civil society in the design and implementation process are fundamental pillars of JSDF and should be retained as part of the programme s strategic aim. But they are not objectives in themselves. Thus, concerning the current objectives in the existing purpose statement it is recommended that the following should be listed as JSDF core strategic features, rather than as part of its core objectives: Participation - Should be listed as one of JSDF s core strategic features. Empowerment should be dropped from the list of key features since it is a measure of impact, and cannot be adequately assessed during the course of JSDF projects, although emphasis can be given to including processes that are empowering Capacity building - Individual project documents should establish indicators for capacity building outputs and outcomes linked to development objectives Innovation - Should be retained as an essential and key defining feature of the programme. The term itself should be more precisely defined to include, for example, providing an alternative approach to CDD; introducing new partners (or partnerships) or including new target groups previously outside the mainstream development process. In other words, the project would include xii

23 ideas, themes and methodologies which had not been previously tested at country or sector level and did not replicate or supplement an existing project or development activity already taking place in that area. 5) Sustainability of grant achievements should continue to be a key feature of JSDF projects and a part of its core strategic aim. However, it is recommended that the broad term sustainability should be used (in the context of JSDF) in terms of the measures for up-take / scaling up/ or replication of the project (after grant closure) which in turn should lead to a sustainable programme over the longer term (see specific recommendation C below). 6) Conversely, it is recommended that the term rapid benefits should be dropped, as this grant criteria was introduced at the start of the programme to address a set of problems related to the then post-asia economic crisis. The term is thus no longer relevant. 7) In addition to the above, the revised statement on JSDF s core strategic features should state the requirement for good governance analysis, mechanisms and structures (especially transparency and accountability) in the design and formulation of grants. C. A Strategy for Improving Sustainability of Results 8) It is recommended that mechanisms to further enhance prospects for sustainability (as redefined above) are put in place so that results achieved can be both sustained and extended to a wider group. In the same way that the project design process has been improved by introducing a Seed Fund to encourage greater participation of stakeholders, so to the programme should introduce specific measures to improve up-take / scaling up/ or replication of the project. Specifically it is recommended that: Grant proposals should contain more detail in the application forms (in the relevant section) on the exit strategy and mechanism for long term sustainability, with very specific and costs proposals (with additional resources made available) for achieving this aim. These proposals for enhancing the prospects for sustainability would be managed and monitored by implementation teams (and supervised by the Bank) against targets and performance indicators. The grant implementation period should remain at the present average of 3 4 years, on the assumption that the exit strategy recommendations are taken up by the Bank. Joint Donor-Recipient Review: At a period approximately six months before project closure, a Joint Donor-Recipient Review of updated exit and xiii

24 sustainability options should be carried out. Funds for both the review and for bridging the period between the completion of the JSDF grant and the take-over of responsibilities by the identified organisation/s should be provided (see also recommendations under Section E below). D. A Strategy for learning lessons and wider communication of results 9) A strategy for sharing lessons learnt and communicating results should be developed both at JSDF programme level and for each project. Specifically, it is recommended that aims would include: Programme level: The strategy would focus on identifying and/or creating opportunities for the lessons from JSDF grants to be communicated to interested parties within the Bank. It would also seek to share learning across projects within the JSDF portfolio of projects. There are also opportunities for the lessons learnt to be taken up by the Japan ODA programme (see F: below) thereby adding value to their community development and poverty reduction initiatives. Project level: Each project would draw up and agree a communication strategy and plan from the outset. The principal aim would be to engage relevant organisations / institutions in the work the project is carrying out, creating a demand for the lessons learnt / results of the project and thereby forging channels for uptake either for necessary changes at policy level or for adoption at ground level. E. Enhancements to JSDF Programme Cycle Management 10) As part of the on-going process of change management within the programme, specific, additional recommendations aimed at improving performance throughout all phases of JSDF implementation, it is recommended that JSDF adopts a number of appropriate quality standards and other improvements at each of the key stages of the programme cycle : Quality-at-Entry Process Implement new Quality-at-Entry standard for approval of JSDF proposals within the context of an improved JSDF Purposes Statement. Revise the JSDF proposal form to improve the focus on outcomes. Ensure more consistent focus on outcomes (including Grant Development Objectives and Key Performance Indicators). Improve training and support for teams submitting a JSDF proposal. xiv

25 Quality of Supervision & Closure / Handover Processes Implement a new supervision standard for JSDF projects (which would include a joint donor-recipient review, as described above and a mid-term review). Revise GRM and ICM format to reflect the layout and content of revised proposal forms, with increased focus on monitoring outcomes. To improve the application of the above measures, it is recommended that guidelines are updated, training provided, web support is available to enhance the effectiveness of the GRM / ICM process. Management Oversight Process Implement a portfolio management standard for JSDF; Explore with the Government of Japan the benefits of using part of the existing fund or establishing a new companion fund to provide supplementary resources for the necessary core functions of JSDF, including: (1) evaluation of project outcomes in relation to objectives and adherence to JSDF key programme principles (e.g. innovation) (2) lesson learning, reporting and information dissemination; (3) strategic communication and planning, and (3) managing hand-over functions for up-take / scaling up or replication where agreed during the end of project review exercise. Specific recommendations in the context of the above would include: 11) Training workshops at Bank level: Under a separate, non-project budget line, the programme would plan, design and run a series of training workshops on programme management, in particular on project cycle management, monitoring, evaluation and on the communication of results (aimed at b oth TTLs and recipient stakeholders). 12) Training workshops at regional / country level: To improve the quality of project monitoring at project level and supervision at Bank level, draw up guidelines for an appropriate participatory M&E system for all JSDF projects. This would include the provision of a budget line in projects for training staff and stakeholders in project management, planning and monitoring. Emphasis would be on a training of trainers approach, so that the initial training initiative could be replicated in-country and in the region. 13) Bank Supervision: It is recommended that the facilities and resources available for Bank staff to supervise projects are reviewed, with the aim of increasing level of Bank supervision, especially given the high expectations placed on grant projects to produce innovative and highly relevant practices. In particular the facilities and resources required for a joint donor-recipient mid-term and end of project review would need to be made available. 14) Finance and Procurement: To speed up finance and procurement, build in training plans for implementing partner staff and stakeholders, involving improved guidance documents supported by training in-country. xv

26 F. Incorporation of the lessons and knowledge from JSDF into the Government of Japan ODA programme and other enhancements to visibility 15) The Government of Japan as the Fund s donor has a clear and well-defined role in the JSDF programme cycle, which includes agreeing the strategic aims and giving final approval of grants. The relevant government agencies in Tokyo (the Ministry of Finance, the Ministry of Foreign Affairs, JICA and JBIC) have limited numbers of people and resources to engage in the JSDF process over and above what they contribute at present. However, JSDF is a knowledge generating programme and there are significant opportunities for Japan s Official Development Assistance (ODA) programme to be enhanced, by benefiting from the ripple effect of knowledge, lessons and best practice information emerging from JSDF. In addition it is recommended that the guidelines on visibility are drawn up. It is specifically recommended that: The Bank and Government of Japan draw up a strategy for a formal and functioning link so that JSDF lessons can be communicated to Japanese ODA agencies for incorporation where appropriate. The World Bank and Government of Japan to draw up guidelines for implementing agencies on visibility requirements, vis-à-vis identifying Japan Government as funding agency. It is recommended that for example, official signs and documentation should state: JSDF funded by Japanese Government. In addition, stakeholders would be made aware of the role and involvement of the Government of Japan at key meetings during the process of project start up and during implementation. Other measures: While it is not the role of the Fund to enhance opportunities for Japanese NGOs, the Government of Japan itself may wish to put in place measures to improve the dissemination of information on JSDF programmes to prospective Japanese NGOs. For example, Japanese Embassies could be requested to contact Japanese NGOs in the respective country so that they are kept informed of possible opportunities for providing their services on JSDF grants. The World Bank Office Tokyo should continue to give out information on opportunities in the pipe line for civil society involvement grants to NGOs (including Japanese NGOs) and other relevant parties. G. Making adjustments to the Policy Guidelines on regional distribution 16) Given the changing nature of the portfolio, it is recommended that the policy guidelines on the proportion of projects going to East, South and Central Asia are reviewed, and that the guideline percentage for EAP and Central Asia is reduced in line with current trends in the portfolio. 17) It is recommended that JSDF investigates why some countries have not benefited from JSDF grants. Depending on the outcome, more may need to be done to xvi

27 encourage TTLs to take on JSDF funding in these countries (particularly in Africa). xvii

28 Table R.1: A Road Map: the present and future direction of JSDF Content Present Future Recommendation reference What is JSDF? JSDF is a funding mechanism to undertake a series of innovative, targeted interventions that benefit the poor in participating countries. No change from present basic structure, size and scope of programme, but reformulated purpose statement and objective categories (to include more results / outcome oriented goals) give proposals a clearer set of measurable objectives. Generic outcome level indicators enable performance of JSDF programme to be monitored and better managed. A re-defined, result oriented Purpose Statement What is JSDF s Strategic Aim JSDF s Strategic Aim is currently in its purpose statement (as objectives). Strategic Aim re -formulated to clearly identify the core features of JSDF to be participation; capacity building; A re-formulation of JSDF s Strategic Aim to define and innovation (itself redefined); sustainability (term clarified in context of JSDF), with good governance structures added. Rapid benefits feature is dropped. Proposals then designed with clearer set of objectives and less process and activity oriented. guide JSDF. Staffing of JSDF Very few staff administer the TF in Bank HQ, but Bank TTLs and Japanese resident mission No change from present Not applicable staff assist implementation in the field. JSDF project proposals Bank TTLs. No major change envisaged, except that some Redefining innovation as part are submitted by proposals developed in partnership with national civil society groups on a pilot basis. of a re -formulation of JSDF s Strategic Aim to define and guide JSDF. Presence of a Bank project in the same country / sector required? Yes No change at present. Not applicable xviii