Third Quality Assessment of DGF-Funded Global Programs and Partnerships (GPP III)

Third Quality Assessment of DGF-Funded Global Programs and Partnerships (GPP III) APPROACH PAPER Background 1. Global Programs and Partnerships (GPPs) are a prominent part of the Bank s work as global and regional issues increasingly influence the development agenda. The Development Grant Facility (DGF), the Bank s primary source of financial support to GPPs, is a key instrument for the Bank to engage with partners on high priority initiatives that complement the Bank s country programs. The GPP portfolio in FY09 has 219 programs and partnerships (of which 56 are DGF funded), with a DGF budget of $168 million. 2. The Board s expectations with respect to GPPs have been refined to respond to their growing importance. A review 1 in 2004 by IEG of the Bank s approach to global programs identified several main challenges, including the need for better management of the portfolio to set quality standards, add value, and enhance returns to Bank country operations and clients. In 2008, IEG submitted to CODE a progress report on global programs. 2 IEG found the objectives of most GPPs as highly relevant, but that the programs design and M&E framework were generally weak. IEG also stressed the importance of distinguishing between the overarching objectives to which a program is contributing from the specific objectives to which the program is accountable. 3. The newly-reorganized Global Partnership and Trust Fund Operations (CFPTO) has been focusing on improving the clarity of strategic priorities and selectivity in the Bank s GPPs, strengthening linkages with country priorities, proactively managing the tendency to proliferate new theme-driven facilities and institutions, strengthening the quality assurance and risk management processes, and increasing results-orientation. One of the instruments to help improve the effectiveness of GPPs is the QAG quality assessment of GPPs, including a synthesis report that identifies systemic issues to be addressed by management. 4. QAG has done two quality assessments of DGF-funded GPPs. GPP I covered six programs in FY06, while GPP II assessed eight programs approved in FY06 and FY07. Both GPP I and GPP II found several strong aspects in the programs reviewed. The programs objectives were viewed to be highly relevant and worthy of Bank support, with clear regional or global endorsement and strong alignment with the Bank s core institutional objectives. In addition, Panels highlighted the following strong aspects: (a) sound governance arrangements; (b) appropriate selection of partners or implementing agencies; and (c) dedicated TTLs and task teams who showed strong commitment to the program objectives and activities. 1 2 Addressing the Challenges of Globalization: An Independent Evaluation of World Bank Involvement in Global Programs, Phase 2 Report, IEG, 2004. Progress Relating to Global and Regional Partnership Programs, IEG 2008. - 1 -

5. Nevertheless, the following weaknesses were identified in GPP II: (a) lack of realism in setting objectives; (b) inadequate M&E arrangements; (c) poor assessment of sustainability risks; and (d) uncertainty about how to handle fiduciary aspects, particularly in the procurement area. The GPP II synthesis report recommended improvements in the following areas: (a) results framework; (b) risk assessment; and (c) the DGF disengagement strategy. In addition, the report noted scope for improving the quality of documentation and clarifying fiduciary processes. 6. CFPTO has requested QAG to undertake a third assessment of DGF-funded programs approved by the GPP Council in FY08 and FY09. CFPTO management has informed the Board that QAG would continue to assess the quality of GPPs. As in GPP II, the QAG assessment will focus on quality-at-entry, but will also review oversight aspects for the GPPs approved in FY08. A synthesis report will be prepared after the completion of the evaluations of the GPP III sample. Objectives 7. The objectives of the third quality assessment of DGF supported GPPs are to: Methodology Provide management and staff with real-time feedback on various aspects of the quality of specific programs assessed; Learn from experience of specific programs and disseminate lessons in key areas of quality and processes for improving the quality-at-entry and oversight of the Bank s participation in GPPs; 3 and Promote accountability through providing Senior Management with robust indicators of quality-at-entry and oversight. 8. GPPs are different from traditional Bank products. For example, the nature of the partnership arrangements is as important from a development perspective as the quality of the products themselves which are generated by joint efforts. The Bank s approach and processes for participation in GPPs have evolved considerably over the years. The criteria for priorities and parameters for depth of engagement of Bank s involvement in GPPs are summarized in Attachment 1; the Eligibility Criteria for Support from DGF grants are summarized in Attachment 2. 3 The synthesis report will include systemic recommendations to improve quality-at-entry and implementation; it will also include a status of recommendations from previous GPP synthesis reports. The synthesis report will also take into account the results of evaluations of closed GPPs and work done by some networks. - 2 -

9. In assessing the quality of GPPs, Panels will use a guidance questionnaire with some basic elements similar to those used for the assessments of Bank lending operations. The dimensions for the assessment of GPPs, which follow closely IEG s criteria for evaluating global programs, 4 will be as follows: Ratings (a) (b) (c) Relevance of Objectives of the Program: The Panel will assess the relevance of the objectives based on criteria established by the Bank. Shortcomings in relevance occur when the supply or the demand for the program is not well founded, or when the program s activities are competing with or substituting for activities which individual donors, beneficiary countries, or other GPPs could do more efficiently. Program Design and Monitoring and Evaluation Arrangements: The focus of this dimension is on assessing the quality of the program s design and the institutional arrangements for monitoring the progress of its activities, outputs and outcomes. The Panel will pay particular attention to the program s linkages with country-level activities and institutions. Governance, Management and Financing: The Panel will assess the effectiveness of both the formal partnership (among those involved in the governance of the program) and the participation and degree of ownership of other stakeholders. It will also assess the appropriateness and adequacy of financing and the effective use of resources, including adequacy and quality of procurement and financial management arrangements. (d) Focus on Development Effectiveness during Implementation: 5 Questions in this dimension refer to aspects/issues arising during implementation. In this regard, drawing from the progress reports and as needed from key reports on the various aspects of the program, including partnership arrangements, governance, management and technical aspects, the Panel will assess: (i) whether the problems were identified and actions were taken in a timely manner; (ii) whether there was adequate attention to the linkages to country-level activities and institutions; (iii) whether there was appropriate attention to the management and timeliness of the monitoring framework and to the use of performance indicators; and (iv) whether there was sufficient focus on sustainability issues. This section assesses the quality of Bank oversight of the program. (e) Bank s Performance as a Partner: The Panel will assess the Bank s performance as a partner both at the initiation as well as implementation of the program, and how its participation will contribute to the various aspects of the program. The Panel will assess the Bank s performance in playing up to its comparative advantages at both the global/regional level and the country level. 10. Panels will rate the quality of GPPs on a six-point scale, as follows: 1 = Highly Satisfactory: Excellent practices in most aspects and with the program being responsive to an issue of high priority to the Bank s client countries and/or global or regional 4 5 Sourcebook for Evaluating Global and Regional Partnership Programs Indicative Principles and Standards, IEG 2007. This section will be applicable to programs that are at least one year old from the date of GPP Council approval. - 3 -

community, and having a high probability of meeting the stated objectives. In addition, the program s risks are commensurate with rewards. 2 = Satisfactory: Satisfactory or better on all key areas--an effective response to an issue of high priority to the Bank s client countries and/or global or regional community, and likely to meet the stated objectives of the program. In addition, the program s risks are commensurate with rewards. 3 = Moderately Satisfactory: Satisfactory on most aspects but with some weaknesses and significant missed opportunities to respond effectively to an issue of high priority to the Bank s client countries and/or global or regional community. 4 = Moderately Unsatisfactory: Significant deficiencies in one or two key areas, which affect responsiveness to an issue of high priority to the Bank s client countries and/or global or regional community, and/or may affect prospects of reaching stated objectives of the program--the risks are not commensurate with rewards. 5 = Unsatisfactory: Significant deficiencies in several key areas needing correction. 6 = Highly Unsatisfactory: A broad pattern of significant deficiencies calling for major redesign or a substantially different approach. NE = Not Evaluable: (for the assessment questions that can not be evaluated on the basis of the documents and interviews). NA = Not Applicable. 11. In reaching an aggregate quality rating, the Panel will first rate each dimension, and then reach a judgment on overall quality. The Panel will be guided by the documentation provided by the Task Team and generated from the Operations Portal, as well as interviews with selected staff, managers and program partners. Ratings will reflect the Panel's best judgment considering the relative importance of various quality dimensions and are not meant to be arithmetic averages of subratings. Scope 12. The assessment will cover 15 Window 2 programs approved by the DGF Council for FY08 and FY09 with DGF funding of at least $300,000 (see Attachment 3). This sample covers about 70 percent of the total number of programs approved in FY08 and FY09. To minimize the evaluation overload on the Team Leaders and duplication of efforts, DGF grants that have been evaluated by IEG 6 in the last 12 months or are planned to be evaluated over the next 12 months will be excluded from the sample. Assessment Process 13. Preparatory Phase. In addition to the documents QAG will extract from the Operations Portal, the Team Leader will provide electronically other relevant, key documents that are not in the Bank s systems, lists of key persons in partner organizations and other stakeholders that the Panel may contact, if deemed necessary, and names of key Bank staff/managers who are involved with the 6 IEG reviews a DGF grant after completion of an external evaluation of the program, typically commissioned by the governing body of the program every 3 years. - 4 -

program. The Team Leader may also provide to QAG a short note on the context and salient features which s/he believes were not covered adequately in the documents made available (Attachment 4). 14. Assessment Phase. The QAG Panel will review documents provided by the Task Team. The Panel will also carry out phone interviews or other interactions with partners/clients and with other Bank staff and managers connected with the DGF grant, as the Panel judges necessary. No travel is planned in this context. 15. In the next step, the Panel will meet with the Task Team. To facilitate the meeting, the Panel will prepare questions for the Task Team on the main issues to be covered that emerged from its prior work. QAG will send the questions to the Team Leader at least two days before the meeting. The Panel may also choose to meet with the concerned Sector Board (SB) and other managers as appropriate. 16. Report Phase. Following the discussions, the Panel will prepare its informed judgment on the quality of the DGF grant and ratings. QAG will transmit the draft report to the Team Leader for comments within a week, with copies to the Sector Managers, Sector Board Head, VPU GPP Coordinators, the CFTPO managers, and the Regional Quality Director concerned with the program. Following receipt of comments from the Team Leader, the Panel will consider and where appropriate reflect them in the final report. 17. If the overall rating is less than Moderately Satisfactory and the Team Leader/sponsoring unit requests for a review, QAG would request an independent, senior-level validation of the Panel s assessment. The findings of the review will be sent to the Team Leader/sponsoring unit. Panel and its Composition 18. The Panel and its members are the heart of the QAG assessment approach. QAG will seek advice from the Sector Boards and Line Managers on possible panelists, especially outside technical experts, and on suitable outside partners that should be included in the Panel review. The assessment of quality of DGF grants puts more than the usual burden on QAG to ensure that the Panel has the needed capacity to view the DGF grant within its global context and assess the governance and management aspects with balanced professional objectivity. 19. The proposed Panel of two to three persons will consist of a senior (grade H+) staff or manager with extensive and diverse experience in Bank operations who will serve as Panel leader, and one to two senior/high-potential sector specialists; to the extent feasible, QAG will include one panelist from outside the Bank. The sector professionals will be individuals recognized for their technical skills, and breadth and practical experience in the area covered by the DGF grant; they should complement each other in technical terms to cover as large an area relevant to the program as possible. To avoid any potential conflict of interest, anyone who has recently been involved in the sponsoring SB/Region, who had had a role in the DGF grant, or who is currently in any way related to the activities of the program being assessed, will not be eligible to serve as a panelist. QAG Support to Panel Assessment 20. QAG will appoint a moderator to facilitate the assessment process, including the Panel s meetings. The moderator will have a critical role in ensuring consistency across panels and their review of different DGF grants. The moderator is independent of the Panel, has no vote on ratings, and thus is able to serve as a neutral channel of communication between the Panel and the Task - 5 -

Team and the sponsoring Unit. The skill-set for a moderator will be similar to that of a core panelist, combined with extensive familiarity with the QAG methodology. Outputs and Dissemination 21. There will be two specific outputs from this assessment: (i) the assessment of each DGF grant in the sample, which will be provided to the respective Team Leader, concerned managers and the GPP Group, and (ii) a synthesis note addressing the objectives in paragraph 7. The synthesis note will present analysis and highlight strengths and weaknesses in the quality of DGF grants and draw lessons in key areas of quality and processes for improving the quality of the Bank s participation in DGF funded GPPs. QAG will help disseminate findings and carry out further analysis as needed in collaboration with CFPTO. The synthesis report will include recommendations on how to improve learning and dissemination of the findings. Costs 22. The cost of the Assessment of Quality of DGF-funded programs, including moderators, synthesis note, and QAG overhead in management and logistics, is estimated at about $500,000. The 15 programs subject to full assessment have requested DGF funding amounting to about $18 million over the life of the programs; total program costs funded from all sources are about $260 million. QAG will cover direct costs of the assessments, and will reimburse time of staff participating as panelists. These costs do not include the time spent by teams in preparing for and interacting with assessment panels. Schedule 23. Assessments will commence with the notification to Team Leaders and line managers in the first week of April 2009; the synthesis note is expected to be ready for review by OVPs in mid-july 2009. Steps in the assessment process will be as follows: Issuance of Approach Paper April 1, 2009 Notify Team Leaders April 1-3, 2009 Collect documents and mobilize panelists April 1 15, 2009 Interviews of Task Teams April 16 May 29, 2009 Finalize assessment reports June 15, 2009 Working level review of draft synthesis report June 30, 2009 Review by OVPs of draft synthesis report September 2009 Issuance of final synthesis report September 2009-6 -

Attachment 1 The Bank s Work in Global Public Goods 1. To guide the Bank s work in global public goods, it is essential to establish criteria for setting priorities and parameters for the depth of engagement. The criteria set out in the 2000 Development Committee Communiqué on this subject are still valid but with adjustments based on the Bank s experience over the last several years: There should be an emerging consensus in the international community that global action is required; There should be an institutional gap that the Bank could help fill to encourage global action; The Bank should have the requisite capabilities and resources to be effective; Bank engagement should be consistent with its development mandate and relative strengths; and Global action by the Bank should support or catalyze other resources. 2. Consistent with its overall mandate to assist countries in achieving sustainable development and poverty reduction, these criteria allow the Bank to bring to bear its established tools: research, global advocacy, country level analytical work, technical assistance, lending operations, global partnerships, trust funds, and innovative financial mechanisms. Their specific use singly or in combination will be determined by the appropriate positioning and the depth of engagement of the Bank within given country, regional or global circumstances, and as part of overall World Bank Group resource allocation decisions. 3. Leadership in the provision of global public goods usually lies with entities that have primary responsibility for producing these goods. The United Nations and specialized international agencies are charged, for example, with monitoring the spread of communicable diseases, setting global standards on greenhouse gas emissions, and determining the rules of the international trading system. The Bank s contribution to the global public goods agenda is to complement their work by focusing on four major areas: working with countries to relate global concerns to national policies and programs; employing substantial experience in partnerships and financial mechanisms; providing constructive advocacy through objective analysis of global issues; and offering knowledge across economic and social sectors. 4. The focus of the Bank s work in Global Public Goods has been on five areas endorsed by the Development Committee in September 2000: preserving the environment; controlling communicable diseases; strengthening the international financial architecture; enhancing developing countries participation in the global trading system; and - 7 -

creating and sharing knowledge relevant for development. 5. The table below gives World Bank Priorities for support of Global Public Goods. World Bank Priorities for Support of Global Public Goods Source: World Bank: Global Public Goods: A Framework for the Role of the World Bank, (DC2007-0020), September 28, 2007. - 8 -

Attachment 2 Eligibility Criteria for Grant Support from the Development Grant Facility 1. Subsidiarity The program contributes to furthering the Bank s development and resource mobilization objectives in fields basic to its operations, but it does not compete with or substitute for regular Bank instruments. Grants should address new or critical development problems, and should be clearly distinguishable from the Bank s regular programs. 2. Comparative advantage 3. Multi-country benefits The Bank has a distinct comparative advantage in being associated with the program; it does not replicate the role of other donors. The relevant operational strengths of the Bank are in economic, policy, sector and project analysis, and management of development activities. In administering grants, the Bank has expertise in donor coordination, fund raising, and fund management. The program encompasses multi-country benefits or activities which it would not be efficient, practical or appropriate to undertake at the country level. For example, informational economies of scale are important for research and technology work, and operations to control diseases or address environmental concerns (such as protect fragile ecosystems) might require a regional or global scope to be effective. In the case of grants directed to a single country, the program will encompass capacity-building activities where this is a significant part of the Country Assistance Strategy and cannot be supported by other Bank instruments or by other donors. This will include, in particular, programs funded under the Institutional Development Fund, and programs related to initial post-conflict reconstruction efforts (e.g., in countries or territories emerging from internal strife or instability). 4. Leverage The Bank s presence provides significant leverage for generating financial support from other donors. Bank involvement should provide assurance to other donors of program effectiveness, as well as sound financial management and administration. Grants should generally not exceed 15 percent of expected funding over the life of Bank funding to a given program, or over the rolling 3-year plan period, whichever is shorter. Where grant programs belong to new areas of activities (involving, e.g., innovations, pilot projects, or seed-capital) some flexibility is allowed for the Bank s financial leverage to build over time, and the target for the Bank grant not to exceed 15 percent of total expected funding will be pursued after allowing for an initial start-up phase (maximum 3 years). 5. Managerial competence 6. Arm s length relationship 7. Disengagement strategy 8. Promoting partnerships The grant is normally given to an institution with a record of achievement in the program area and financial probity. A new institution may have to be created where no suitable institution exists. The quality of the activities implemented by the recipient institution (existing or new) and the competence of its management are important considerations. The management of the recipient institution is independent of the Bank Group. While an arm s length relationship with the Bank s regular programs is essential, the Bank may have a role in the governance of the institution through membership in its governing board or oversight committee. In cases of highly innovative or experimental programs, Bank involvement in supporting the recipient to execute the program will be allowed. This will provide the Bank with an opportunity to benefit from the learning experience, and to build operational links to increase its capacity to deliver more efficient services to client countries. Programs are expected to have an explicit disengagement strategy. In the proposal, monitorable action steps should be outlined indicating milestones and targets for disengagement. The Bank s withdrawal should cause minimal disruption to an ongoing program or activity. Programs and activities should promote and reinforce partnerships with key players in the development arena, e.g., multilateral development banks, UN agencies, foundations, bilateral donors, professional associations, research institutions, private sector corporations, NGOs, and civil society organizations. Source: World Bank, The Development Grant Facility: FY98 Annual Review and Proposed FY99 DGF Budgets, Oct. 28, 1998. - 9 -

Attachment 3 Assessment of Quality of DGF-Funded Global Programs and Partnerships DGF Grant Year of Approval Amount from DGF (in US$ Millions) ID GPP Title 1 P111021 Strengthening Health Stat. thru the Health Metrics Ntwk FY08 1 9.3 HDN Bos, Eduard R. 2 P105605 Global Initiative for Quality Assurance Capacity (GIQAC) FY07 1.28 9.1 HDN Hopper Richard R. 3 P110758 Promoting Youth Employment & Employability FY08 1.75 10 HDN Newhouse, David 4 P104884 Trade Data for Low Income Countries FY07 0.6 4.15 PREM Hoekman, Bernard M. Total Budget (in US$ Millions) Responsible Network TTL 5 P104701 Public Expenditure Management Peer Assisted Learning (PEM-PAL) F Y 07 0.3 6.14 PREM Taliercio, Robert R. 6 P110604 LAC MIC Gov. and Public Mgmt Partner. Facility FY08 0.3 3.95 PREM Bellver Vasquez-Dodero, Ana 7 Arab Water Management Institute 0.37 3.57 SDN 8 P110523 Alliance for Responsible Fisheries FY08 0.4 7 SDN Kelleher, Kieran 9 P105001 Affiliated Network for Social Accountability (ANSA)- East Asia/Pacific FY07 0.5 22.75 SDN McNeil, Mary L. 10 P099903 Global Partnership for Environmental Governance/Principle 10 FY07 0.7 14.51 SDN Ahmed, Kulsum 11 P105455 City Indicators Facility FY07 1.2 13.63 SDN Hoornweg, Daniel A. 12 P110161 Affiliation Network on Soc. Acct. & Gov in S.A. and the G.P.F FY09 1.5 28 SDN Sirker, Karen 13 P100336 Sub-National Development Program FY07 2 7.5 SDN Leigland, James 14 P105248 Global PPP & Multi-Donor TF for Youth Invest. FY08 2 109.4 SDN Patil, Pawan G. 15 P110242 Growing Forest Partnerships FY08 3.8 11.95 SDN Sander, Klas - 10 -

Attachment 4 Documentation List A. Documents to be collected by QAG: 1. From the Operational Portal: (a) (b) GPP Group Comment Sheet for the PRN; and Final Partnership Review Package to the VP for clearance (Program Data Sheet, DGF Module, PRN, Minutes of the Review Meeting) and the VP's comments. 2. From the GPP Group: (a) (b) (c) TORs for and report of the External Peer Reviewers; Memorandum/Emails from the Network VP/Sector Board Head to the GPP Group on the DGF grant application for the program; Minutes of the DGF Council s deliberation on the package of DGF applications; and (d) Legal documents, including DGF Grant Agreements, Trust Fund Administration Agreements, etc. B. Documentation to be provided by the Task Team: (Note: The Panel is expected to have about one day for reviewing documents. Accordingly, the Team Leader is requested to be selective in providing documents. To the extent possible, only documents (final or final drafts) that address aspects of the program in the Guidance Questionnaire should be provided. Documents to the extent possible should be sent electronically, and if they are from IRIS, Team Leader is requested to either attach the documents or identify them specifically in their note to QAG, rather than providing all of the program files in the IRIS.) 1. Key documents of relevance to the DGF grant (relating to the program's linkage with country operations, arrangements for governance, management and implementation of the program, terms of reference of Bank s representative on the governance body of the program, due diligence reports on DGF grant recipient institutions, in particular to assess the record of achievement in the program area and financial probity, arrangements for the Bank s oversight of procurement and financial management arrangements during implementation of the program, etc.); 2. Sector strategy report/paper relevant to the DGF grant that would assist the Panel to understand the program's context; 3. Back-to-office reports since approval of the DGF grant; - 11 -

4. Progress reports submitted to the GPP Group; 5. Optional: A short note on the context and salient features which the Task Team believes were not covered adequately in the documents made available; 6. Proposed Annual Financing for the next three years ($ million): DGF Grants Trust Funds External Financing BB Budget for Program Administration 7. Actual Bank budget allocated for program oversight ($000). 8. Contacts for the Survey of Partners/Stakeholders, and of Bank Staff/Managers. (a) (b) A list of key persons (full names and contact email addresses/ telephone numbers) that the Panel may contact, if deemed necessary, in partner organizations (e.g., on the governing body, on the secretariat, and others), and in countries, academia, research institutions, and NGOs. Please also indicate briefly their involvement/collaboration in the program; and A list of key Bank staff and managers in Regions and relevant Sector Boards who are involved in the DGF grant [e.g., for the program s links with country operations or as members of relevant thematic group(s)]. - 12 -

- 13 -