Evaluation of the Policy and Human Resources Development Trust Fund

Transcription

1 Evaluation of the Policy and Human Resources Development Trust Fund Volume II Armenia Country Case Study By: Universalia 5252 De Maisonneuve Blvd. West Suite 310 Montreal, Quebec H4A 3S5 Canada

2

3 Acknowledgements The World Bank resident mission in Yerevan provided outstanding support before and during the field visit to Yerevan. Special thanks go to Gayane Davtyan and Naira Melkumyan for their support with all logistical aspects of the mission. Thanks also to Lilit Simonyan for her excellent interpretation services during meetings, and to Karen Martirosyan for his help with completing interviews in Yerevan. i

4 Acronyms ASIF CAS DFID ECA EU FFPMC GEF GoA GoJ IBRD IDA IFAD KfW MCA MoFE PHRD PRSC PRSP RESCAD SAC SIDA TTL UNDP USAID WB Armenia Social Investment Fund Country Assistance Strategy (World Bank) Department for International Development Europe and Central Asia European Union Foreign Financing Projects Management Center Global Environment Facility Government of Armenia Government of Japan International Bank for Reconstruction and Development International Development Association International Fund for Agricultural Development Kreditanstalt fuer Wiederaufbau Millennium Challenge Account Ministry of Finance and Economy (Armenia) Policy and Human Resource Development Fund Poverty Reduction Support Credit Poverty Reduction Strategy Paper Rural Enterprises and Small Commercial Agricultural Development Project Structural Adjustment Credit Swedish International Development Cooperation Agency Task Team Leader United Nations Development Programme United States Agency for International Development The World Bank ii

5 Contents 1. Introduction Objectives of the Evaluation Methodology 1 2. Regional and Country Contexts The Regional Context for Case Study Country Allocation of PHRD among Countries in the ECA Region Profile of PHRD Grants for Armenia 3 3. Relevance of the PHRD Grants Relevance in Terms of National Development Priorities Relevance in Terms of the WB s Operations Grant Congruence with the CAS Relation of Grants to WB Lending Relevance to the Japanese Government Alternatives to PHRD for Project Preparation in Country Need for PHRD 7 4. Effectiveness of the PHRD Grants Preparation of Follow on Activities (QAE) Country Ownership Capacity Building Policy Development/Change Visibility of the Japanese Government Management of the PHRD Grants The PHRD Grant Application and Approval Processes at Country Level Implementation of Grant Activities Reporting, Monitoring and Evaluation Flexibility of the PHRD Grant The Simplification Process Cost-Effectiveness of PHRD Other Issues Implementation Grants Climate Change Grants Donor Coordination 21 iii

6 7. Conclusions Priorities/ Adjustments if PHRD Programme is Reduced Suggested Improvements to the Current Programme Other 23 Exhibits Exhibit 1.1 PHRD TA Grants to Armenia Reviewed for Case Study 1 Exhibit 2.1 Total of PHRD TA grants in Armenia since Fiscal Year Exhibit 3.1 Degree to which PHRD TA grants led to an increase of WB Lending to Armenia 5 Exhibit 4.1 Degree to which PHRD grants have improved the quality at entry of subsequent projects 9 Exhibit 4.2 Sub-Results of PHRD preparation grants 13 Exhibit 5.1 Flexibility of PHRD Grants 19 Appendices Appendix I List of Consulted Stakeholders 25 Appendix II Reviewed Documents 27 Appendix III PHRD Links to Armenia s PRSP, and WB CAS 29 Appendix IV Ratio of Amounts invested in PHRD and related WB Credits 30 Appendix V Selected Survey Results 31 Appendix VI List of Findings 33 iv

7 1. Introduction 1.1 Objectives of the Evaluation The case study assesses the effectiveness, relevance, and management of the Policy and Human Resource Development Fund (PHRD) Technical Assistance (TA) program in Armenia, and provides a summary of key issues, and potential areas for improvement of the PHRD grant operation in Armenia. In doing so, the study supports the objectives of the overall PHRD evaluation, which are: Review the progress and assess the development results of the PHRD TA Program since 1999, with special attention to its effectiveness, relevance and management; Assess the achievement of the objectives of the Program; and Make recommendations for improved operation of the PHRD Fund. 1.2 Methodology The study utilized a variety of approaches to data collection: document study, consultation, phone interviews, face-to-face interviews, and a focus group meeting. Most of the face-to-face interviews were carried out during a field visit to Yerevan, Armenia that took place from January 21-27, The field visit was led by Ms. Anette Wenderoth, (Universalia) with support from Mr. Karen Martirosyan (Armeria, Yerevan). Ms. Marie Hélène Adrien and Ms. Katrina Rojas from Universalia conducted additional interviews. A list of the consulted stakeholders is included in Appendix I. Since fiscal year 2000, Armenia has received and implemented a total of 11 PHRD TA grants in support of 10 WB operations. The table below provides an overview of these grants. While documents were reviewed for all ten projects, interviews were conducted only with stakeholders who had been involved in the eight grants that according to the data made available at study onset - had reached over 50% disbursement at the time of the evaluation. Exhibit 1.1 PHRD TA Grants to Armenia Reviewed for Case Study FISCAL YEAR PROJECT ID GRANT PURPOSE GRANT PURPOSE 2001 P Business Linkages 2 Preparation $192, P AMOUNT LENDER DISB% EXECUTED PROJECT STATUS 1 $ 199,540 IDA 65% 38% Recipient Executed Portion Bank Executed Portion Dropped Poverty Reduction Support Credit (PRSC) Preparation $620,000 IDA 70% 3 Recipient Closed 1 Source: SAP as of June 30 th The data available at the onset of the study indicated that this grant had 0% disbursement and the purpose was unknown. 3 Data initially received from the WB PHRD database indicated that this project had had 0% disbursement. 1

8 FISCAL YEAR PROJECT ID GRANT PURPOSE GRANT PURPOSE AMOUNT LENDER DISB% EXECUTED PROJECT STATUS P Utility Restructuring Preparation $500,000 IDA 86% Recipient Dropped 2000 P P P P P P P Fourth Structural Adjustment Credit Preparation $263,900 IDA 100% Bank Closed Health System Optimization Project Preparation $492,000 IDA 95% Recipient Active Education Quality and Relevance Preparation $370,250 IDA 90% Recipient Active Fifth Structural Adjustment Credit (SAC5) Preparation $400,000 IDA 98% Recipient Closed Rural Economy Modernization Project Preparation $925,000 IDA 78% Recipient Active Third Poverty Reduction Support Credit Preparation $960,500 IDA 100% 4 Recipient Pipeline Armenia Social Investment Fund III Preparation $316,000 IDA 100% Recipient Active Source: SAP and e-tf databases (as of April 2007) and SAP as of June 30 th Limitations For two of the reviewed grants we were not able to obtain confirmed up to date data on their actual disbursement rates at the time of writing, as the excerpts from the WB database that formed the basis for our analysis had a cut off date of June 30 th, Where we were able to obtain additional information on disbursement rates, this is indicated in the text. As we only realized after the field visit to Yerevan that the Poverty Reduction Support Credit (P078673) had actually reached over 50% disbursement, we were unable to interview any stakeholders in Yerevan in relation to this project. To the extent possible the team followed up by to obtain additional information. Another limitation was that the available PHRD documents for all projects provided only very limited detail on specific aspects of grant application processes, issues and challenges that were encountered during implementation and any analysis of the quality and relevance of planned and achieved results. The assessment therefore had to rely strongly on data collected through interviews, and focus group meetings. However, several projects had been completed several years in the past, and there were some challenges in identifying interview partners in Washington and Armenia, as individuals had left the positions they had held at the time of the project. Also, due to the time gone by, numerous interviewees could not recall many details on procedures, deliverables, or activities of the respective project. 4 The initial data we received from the WB PHRD database (SAP, as at June 30 th 2006) indicated a disbursement rate of 51%. 2

9 2. Regional and Country Contexts 2.1 The Regional Context for Case Study Country Armenia was selected for the PHRD evaluation as the country representing the Europe and Central Asia (ECA) region. Armenia is a lower-middle income country with a gross national income per capita of $1, A land-locked country, Armenia has been severely challenged by Azerbaijan and Turkey closing their borders with Armenia for political reasons soon after the country s independence. Armenia joined the World Bank (WB) in September One year later it became a member of the International Development Association (IDA). Since the inception of the World Bank program in Armenia, 44 WB projects have been approved 6 for a total amount of $935 million (including two operations of the Global Environmental Facility (GEF)). Among World Bank staff the country is widely regarded as a top performer in terms of credit disbursement, and with regards to the quality of grant and credit implementation. As an IDA country, Armenia currently has very limited access to other lending mechanisms. However, the country is bordering on eligibility for IBRD lending, and is expected to graduate in the near future. 2.2 Allocation of PHRD among Countries in the ECA Region Since 2000, a total of 190 PHRD grants have been developed in the ECA region. Of these, 11 grants (5%) were developed in Armenia. The total grant amount of the 190 PHRD projects in the ECA region since FY2000 was $ 99,931,514. Of these, $ 5,239,850 (5%) were allocated to Armenia. 2.3 Profile of PHRD Grants for Armenia All of the eleven PHRD TA grants Armenia has received since 2000 were preparation grants. In addition, Armenia received two PHRD Co-financing grants, which were not included in this evaluation. Exhibit 2.1 illustrates the allocation of the eleven grants over the time period FY Exhibit 2.1 Total of PHRD TA grants in Armenia since Fiscal Year 2000 FY00 FY01 FY02 FY03 FY04 FY05 FY06 TOTAL Preparation Climate Change/ Implementation Total Source: SAP and e-tf databases (as of April 6 th 2007) Through these eleven grants a total of $ 5,239,850 were allocated to Armenia. This represents 1.3% of the overall World Bank lending to Armenia during the same period ($ 392 million). Out of the 11 PHRD grants, 8 (80%) have since been followed on by a WB operation (7 approved projects, and 1 currently in the pipeline). 5 World Bank Website Armenia. 6 World Bank Website Armenia. 3

10 3. Relevance of the PHRD Grants 3.1 Relevance in Terms of National Development Priorities Finding 1: All consulted stakeholders found the PHRD mechanism highly relevant in the current Armenian development context. PHRD Preparation grants are aligned with Armenia s PRSP objectives All interviewed GoA stakeholders confirmed the relevance of PHRD grants objectives and outputs with regards to country priorities 7, and based their judgment on a number of observations: Alignment of PHRD objectives with PRSP goals and thus national priorities. (See also Appendix III) Approval cycles for WB credits tend to take considerable time, and during this time project concepts, if not developed thoroughly, can often change due to additional insights or unexpected contextual changes. Solid project preparation, as is made available through PHRD, allows for better-defined project proposals that are less vulnerable to changes. PHRD grants provide the GoA with access to first class international and national expertise, which in turn leads to improved quality of WB projects at entry. This is particularly important for sensitive areas and complex projects. PHRD grants help to develop feasible strategies for project implementation and financing. As approval times for PHRD preparation grants are seen as being relatively short, the grants serve as bridging projects, that allow the government to move ahead with activities fostering sector improvements, while waiting for approval of the main credit. Project preparation projects usually result in a number of valuable deliverables that serve as foundations for all further reform progress, and are thus crucial for moving ahead in the respective sector. Finding 2: Consulted Stakeholders expressed satisfaction with the types of sectors and projects supported by PHRD grants to date. Consulted stakeholders from Armenian government agencies, as well as WB staff, expressed content with the breadth and variety of sectors supported by PHRD TA grants in the past. All PHRD grants received to date were seen as having been relevant and useful in light of the country s development priorities and needs. There were no recommendations for changing the PHRD s strategic focus, e.g. to limit it to specific sectors or types of projects. Until now interviewees had not experienced any competition between PHRD applications from different sectors that would have required discussions about strategic choices between sectors. 7 25% (3 interviewees out of 12) rated PHRD outputs as very useful to Armenia, 66.7% (8) as useful. 4

11 3.2 Relevance in Terms of the WB s Operations Grant Congruence with the CAS Finding 3: All reviewed PHRD Preparation grants are aligned with the WB CAS. In 2001, the World Bank developed a CAS for FY , which was aiming to support the objectives outlined in the Government of Armenia s Interim Poverty Reduction Strategy Paper (March 2001). The current CAS FY was developed in response to the first full PRSP in Armenia that was approved in All reviewed PHRD preparation grants are in alignment with the respective WB CAS in place at the time of proposal writing, and made explicit reference to the CAS. (See Appendix III) 100% (10 out of 10) of the interviewed WB staff indicated that in their view the PHRD grants were very supportive of WB CAS priorities, and either very relevant (90%), or relevant (10%) to WB operations in Armenia. They also pointed out that the CAS and Armenia s PRSP were intricately linked: The CAS main purpose is to assist with the implementation of the PRSP, and CAS documents are developed in close cooperation of the Armenian government and the World Bank Relation of Grants to WB Lending Without the PHRD grants there is no way the project could have been done with that level of quality, speed and with a seamless transition between preparation and implementation. (WB Staff Member) Finding 4: Data suggest that PHRD grants have led to some increase of World Bank lending to Armenia. PHRD grants are also seen as having contributed to Armenia s high performance as an IDA borrower. While consulted stakeholders were unable to make statements on the PHRD s overall impact on Bank lending in Armenia, there were some example of individual project preparations having contributed to increasing WB lending. For example, the detailed studies and assessments developed during preparation of the Education Quality and Relevance project led the Ministry of Education and the WB to renegotiate the initial outline of the envisaged WB loan to ensure that it would sufficiently address identified needs. The analysis, combined with proven management capacity of the Education PIU resulted in an increase of the initial credit amount from $10 million to a total project cost of nearly $ 18 million, and to changing the credit modus to a more flexible Adaptable Program Loan. Exhibit 3.1 Degree to which PHRD TA grants led to an increase of WB Lending to Armenia SUBSTANTIALLY MODERATELY LITTLE NONE DON T KNOW FREQ % FREQ. % FREQ. % FREQ. % FREQ. % Armenian Government Representatives (n=12) WB staff (n=10)

12 Several stakeholders also stated that they felt that the quality preparation of projects had contributed to Armenia s successful track record with the World Bank i.e. its reputation as a top performer with regards to credit implementation and disbursement and indirectly to increased WB lending to the country. (See sidebar) The total estimated credit amount for the eight reviewed projects that were approved or are currently in the pipeline is approximately $ 221 million. The amount for the related PHRD preparation grants totals $ 4,347,650. Thus, for each dollar spent on PHRD, there is approximately $ 51 invested in loan Relevance to the Japanese Government Finding 5: There appears to be a need for clarification as to whether and how PHRD TA grants are supposed to be relevant to specific Japanese development priorities at country level. Our data suggest that PHRD grants contribute successfully to at least two of the main objectives of the overall PHRD grant mechanism: i) Strengthening human resources in developing member countries and ii) Assisting these countries in formulating and implementing development policy 9. To what extent the PHRD in Armenia is contributing to the third objective iii) Strengthening the partnership between Japan and the World Bank is not evident from our data. However, during the case study, the representative of the Japanese embassy in Moscow expressed concerns about PHRD grant applications not visibly and explicitly addressing Japanese development priorities in Armenia. The GoJ does not currently have a specific country strategy for Armenia, but has defined four programming priority areas for all countries in the Central Asia/Caucasus region (see sidebar). If we hadn t had PHRD the government could not have implemented the follow on projects so well and on time. And then we would not have qualified for new high case credits. (WB staff member) JICA priority Areas in the Central Asia/Caucasus region i) Support for transition to a market economy; ii) Socioeconomic infrastructure development; iii) Cooperation with social sectors such as health and education; iv) Conservation of the environment. While an explicit link of PHRD grant applications with JICA priorities might enhance the immediate relevance of PHRD projects to the GoJ, PHRD policy and management guidelines do not require such a link. It was also pointed out that JICA, as a bilateral technical assistance agency, would often necessarily have different priorities than the World Bank s lending program in the respective country. Thus, considering only JICA priorities as guiding for PHRD related decision-making could be misleading. Continued differing interpretations of what type and degree of linkages of Japanese and PHRD priorities have been agreed upon may negatively impact on the partnership between the WB and the GoJ though. Thus, both partners may benefit from clarifying agreed upon PHRD guidelines, and related expectations. 8 More specifically: $ See Appendix IV. 9 See section 4 of this report on Effectiveness. 6

13 3.4 Alternatives to PHRD for Project Preparation in Country Finding 6: At present, there are no other external funding mechanisms for project preparation easily available in Armenia. While limited Government resources could be made available, it is uncertain whether there would be the political will for such an investment. At present, PHRD is the only external funding mechanism for project preparation easily available in Armenia. Several interviewees stated that in their view the current budget situation in Armenia would theoretically allow for Government resources being made available for preparation tasks. However, they also pointed out that the government currently tended to be hesitant to invest state budget and/or credit resources into consultancies and soft issues such as studies and surveys. Thus, while they saw the possibility of some increase to GoA co-funding contributions for project preparation, they felt that it was unlikely that government resources would be made available to fully fund preparation tasks. 3.5 Need for PHRD Finding 7: The demand for PHRD grants in Armenia has been high and is expected to remain stable in the near future. Given the strong national capacities for project preparation, management, and in technical areas, Armenia s actual need for PHRD grants as HR development aids may become smaller though. The demand for PHRD grants in Armenia has been high, and at least for the foreseeable future is expected to remain high. This is partly due to the absence of other accessible external funds for project preparation, and partly due to the positive experiences of WB and GoA stakeholders with PHRD. At the same time, the objective need for future PHRD grants as specific HR development tools might become smaller, as national capacities continue to increase. Armenia already possesses strong national capacities both in project One stakeholder stated that if the Armenian economy continued to improve at the same pace as at present, the GoA might become more selective about the areas for which it would request WB loans in order to minimize external lending. This would also limit the demand for PHRD grants provided that eligibility criteria for accessing the grant would remain linked to WB operations. preparation and management, as well as in numerous technical areas. Thus, while the financial need for PHRD grants is likely to remain high, given the lack of alternative funding sources, the capacity development related need for PHRD grants in Armenia may reduce over time. 7

14 4. Effectiveness of the PHRD Grants 4.1 Preparation of Follow on Activities (QAE) Finding 8: PHRD preparation grants have been successful in completing all or most of their planned activities, and the majority of grants have resulted in the approval of a WB follow on project. All assessed projects 11 have been able to complete all or most of their planned activities within the agreed upon timelines and budgets, and have produced the deliverables outlined in the respective proposal. With very few exceptions, World Bank monitoring and completion reports of PHRD projects in Armenia rate the implementation of all project components of the reviewed projects as satisfactory. 95.9% of consulted stakeholders (WB and GoA) stated that they found the quality of completed deliverables/outputs of the PHRD projects either very satisfactory (54.2%) or satisfactory (41.7%), and the remaining 4.2% stated that they were somewhat satisfactory. This high quality of outputs was partly attributed to the good selection of international and local consultants, and partly to the appropriate design of the respective projects. In seven out of the ten reviewed cases the PHRD preparation grant resulted in an approved follow on World Bank project, and one further project is currently in the pipeline awaiting board decision. Two projects were dropped (see Had the quality of preparation deliverables not been good, we would not have succeeded in implementing the subsequent reforms. (WB Staff member 10 ) When the Utility Restructuring Project, 2002, was dropped 12, a large amount of preparation activities had already been implemented. The deliverables completed during the project are widely seen as having positively contributed to subsequent reforms in the Utility sector, and have been used in other (later) projects in the same sector such as the WB Urban Heating project, and activities under the Millennium Challenge Account. Consulted stakeholders stated that the project had been highly effective, and had even exceeded its expected results. sidebar). Most of the consulted stakeholders found that the PHRD grants positively influenced the Quality at entry of the respective follow on operation (see Exhibit 4.1 below). 10 Throughout the text, unless otherwise indicated, WB Staff member refers to both TTLs and WB country staff. 11 With exception of the Business Linkages Project (P073368). Due to the lack of available documents or other information, we were unable to assess its effectiveness. 12 Interviewed stakeholders gave two reasons why the Utility Restructuring project was dropped: a) The preparation phase had already achieved some or all of the objectives that had originally been envisaged for the follow on project, and reforms in the utility sector had progressed faster than anticipated, b) The new WB Country Assistance Strategy (CAS) FY did no longer include the same foci for utility restructuring as had been envisaged for the project. 8

15 Exhibit 4.1 Degree to which PHRD grants have improved the quality at entry of subsequent projects VERY SATISFACTORY SATISFACTORY LITTLE NO ANSWER/DON T KNOW FREQ. % FREQ. % FREQ. % FREQ. % Armenian Government Representatives (n=12) WB Staff (n=10) Country Ownership Finding 9: The fact of PHRD grants are implemented by the recipient country is widely seen as having positively influenced the degree of local ownership of grant management processes, and of project results. At the same time, ongoing World Bank support to implementing agencies is highly valued. With exception of one project 14 all assessed PHRD grants were locally led and managed. There was wide agreement among consulted stakeholders that this arrangement had helped to increase local ownership of grant implementation and results. Interviewees stated that in their view the local leadership for grant implementation increased accountability of government bodies, and enhanced transparency and efficiency of resource utilization. Several stakeholders also stated that in their view local implementing agencies tended to aim for getting better value for money, and that they were more concerned about efficiency of resource utilization than they would have been had the grant been implemented by the World Bank, or the donor. Compared to other external funding mechanisms, the PHRD is widely seen as allowing a lot more local ownership e.g. by allowing local governments to be fully in charge of the selection of project consultants. As key indicators for strong local ownership of PHRD implementation interviewees named: Local implementing agencies took leadership in developing high quality TORs for local and international consultants required for PHRD projects. The agencies ensured that the TORs focused on securing practical and relevant support suitable for the Armenian context. In most cases, implementing agencies executed the selection process for national consultants completely on their own. For international consultants, the respective PIU usually consulted with the TTL, or with another WB staff member. In several cases, the respective implementing agency engaged in ongoing in-depth discussions with the contracted consultants to ensure that chosen methodologies and approaches met the actual needs of the project. 13 As outlined in the list of consulted stakeholders in Appendix I, more than 12 GoA staff members were consulted. However, several interviews/meetings included more than one individual who jointly responded to our questions. In this case, only one questionnaire was filled out. 14 The Fourth Structural Adjustment Credit (SAC 4) in In this case, the GoA had made the request that the World Bank manage the project, as at that time the GoA did not feel to have the capacities to do so effectively. However, the successor project, SAC 5, which followed in 2002, was completely locally managed. By then, the Foreign Financing Projects Management Centre (FFPMC) had been fully established, and management capacities had been enhanced partly due to the experience gained under SAC 4. 9

16 In the Utility restructuring project, when some of the planned preparation activities were cancelled once the envisaged follow on credit was dropped, the GoA used its own funds to implement the remaining planned activities as they considered them to be important contributions to ongoing reforms of the Utility Sector. The Utility Restructuring project hired an external consultant for reviewing deliverables by other consultants, to ensure that they were of good quality, and met the criteria outlined in the TOR. In SAC 5, a cross sectoral panel including several sector ministers personally reviewed the shortlists of suggested international consultants for the project to ensure high quality of deliverables. The World Bank provided ongoing support and, where needed, coaching to the respective PIU (see also sidebar). All consulted Armenian stakeholders expressed appreciation for this support, and felt that WB TTLs and other staff had been highly supportive, while never compromising the PIUs role as the managing entity. 4.3 Capacity Building TTL support for PHRD management: Reviewing Terms of Reference developed to recruit international consultants (and in some cases of national consultants) Reviewing consultant CVs and give feedback to implementing agency Requesting and reviewing regular (monthly) progress updates and budget reports Visiting PIU in the field, and undertaking joing field visits Mitigating between PIU and Consultants / between Government and Local Staekholders in case of sensitive issues. Finding 10: There is evidence of PHRD grant initiatives having contributed to capacity development of a variety of stakeholders in Armenia. In most cases, capacity development was achieved as an indirect result of PHRD activities rather than through training or other targeted measures. All PHRD grants appear to have contributed to some extent to the capacity development of Armenian stakeholders, in particular of PIUs, other government and/or ministry staff, stakeholders in institutions and communities that were involved in the respective PHRD project, and of local consultants. 8 out of 10 (80%) of consulted WB staff, and 9 out of 12 (75%) of consulted GoA representatives stated that PHRD grants had led to a great deal or some capacity building. Some PHRD projects included targeted measures for capacity building of PIU staff (e.g. the ASIF III team received training for how to conduct community consultations with project stakeholders) or for local stakeholders through training workshops, or consultations (e.g. as part of the RESCAD 15 project, several community consultations were carried out to inform them about the project and its potential benefits for the community). However, the majority of capacity development appears to have been achieved indirectly, i.e. either as a side effect of managing the PHRD grants, or of working with international and national sector experts. This observation corresponds with the strong focus on consultancies (rather than, for example, on training) as reflected in the budget allocations for the reviewed PHRD projects (see sidebar below). 15 Rural Enterprise and Small Scale Commercial Agricultural Development Project 10

17 Some examples of the types of enhanced capacities stakeholders reported upon: Enhanced knowledge and skills of local consultants who gained new knowledge, skills, and experience by working side by side with international consultants. Working on PHRD grants and Focus on Consultancies: The eight PHRD projects reviewed in more depth allocated an average of 94.8 % of their total budget to the contracting of consultants (international and national). 16 This includes an approximate 61.9% of their overall budgets for international consultants, and 32.9% on national experts. The ratio of international and national consultants varies between different projects though. For example, the Fourth Structural Adjustment Credit (2000) used 89% of its total budget for international, and 11% for national consultants. The ASIF III project (2005) in turn used 21% for international, and 69% for local consultants. subsequent WB credit projects has also increased the local credibility of national consultants, and the likelihood of them being involved in further reform measures in the same sector. 17 Increased professional exchange and discussion among Armenian stakeholders involved in the respective content/topic area addressed by the PHRD grant through participation in PHRD initiated meetings, conferences, and round table discussions in for example - teacher preparation facilities and early childhood education institutions (PRSC I and III), utility companies (Utility Restructuring Project), Hospital owners (Health Optimization), stakeholders in communities involved in agrarian reform (RESCAD), or construction firms involved in public infrastructure (ASIF III). Expanded experience and skills of PIU and line ministry staff in managing projects using WB processes and procedures. By implementing the PHRD project, staff members were able to develop and/or deepen their knowledge of WB procurement procedures, financial management and reporting, as well as of tasks such as developing TORs for project consultants, selecting international and national consultants, and providing quality control for deliverables of different consultancies. While most PIUs had already had experience with implementing WB projects, each new PHRD grant helped to further enhance existing knowledge and skills. For example, prior to 2000 the ASIF team had already received a PHRD grant to prepare for the first phase of the ASIF project. When developing the proposal for the most recent grant (2005), the team was able to predict budget and time requirements more precisely and applied for a smaller amount than they had in the previous project. 16 Data are only approximate as we were not able to obtain final financial reports for each of the reviewed projects. In some cases, quoted percentages are therefore based on the amounts shown in the initial project proposals. 17 All project teams stated that wherever possible they tried to involve Armenian consultants if the required expertise was available locally. ASIF III was the only reviewed PHRD project though that used a larger percentage of its budget for local consultants than for internationals. However, given the differences in fees for international and national consultants, the actual number of local consultants hired was most likely equal, or even higher than that of international colleagues. We were unable to obtain reliable data on the actual number of consultants in each project. 11

18 Enhanced content/sector knowledge and skills of PIU members and line ministry staff. For example, studies and assessments carried out during the Education Quality and Relevance project preparation provided the team with new insights into gaps and shortcomings of the existing curriculum, and into needs and priorities for restructuring teacher In-service training. Similar experiences were reported from most other project teams, e.g. from the Health Optimization, and the Utility Restructuring projects. Enhanced coordination among different ministries and/or other bodies through participation in preparation related governance bodies. For example, the Utility Restructuring project established a management board that involved representatives from various ministries involved in the Utility sector. The Health optimization project established the principle of Tripartheid : Benchmarks for investments in hospitals now have to be jointly agreed upon by the key stakeholders of the envisaged hospital optimization process (hospital owner, respective health administration, and the Ministry of Health). Enhanced capacities for participatory planning and decision-making through community involvement using round table discussions, and other consultation processes as part of the project preparation, e.g. as part of the RESCAD project. Increased awareness of GoA officials with regards to the value of consultancies and similar soft investments in order to improve the quality of project preparation. None of the reviewed projects included a formal capacity assessment of the respective implementing agency/piu before submitting the PHRD proposal. However, TTLs stated that they had a good understanding of the PIU s capacities to manage the grant effectively based on previous collaboration over a period of time. Interviewed stakeholders were content with the degree to which PHRD grants had supported the strengthening of local capacities. Most interviewees stated, that in their view capacity development was not the main focus of the PHRD grants, and that they perceived enhanced capacities of local stakeholders as a very positive side effect of the preparation phases. Some added though that they would welcome if PHRD grants allowed for allocating more funds to targeted training of PIU and/or ministry staff 18, and/or if more training was available for those ministries that had not yet had any substantial experience with PHRD or WB initiatives. 4.4 Policy Development/Change Finding 11: As part of the successful preparation of WB follow on projects, PHRD supported activities that led to a broad variety of deliverables, many of which are perceived as valuable results in their own right. These include the development and/or change of sector specific policies and legislations. While the preparation and approval of a subsequent follow on WB project has been the main purpose of the PHRD grants, each preparation project included a variety of subordinated achievements, including policy changes. 76% (16 out of 21 individuals) of consulted WB staff and Armenian government representatives stated that they felt PHRD grants had substantially contributed to policy changes in the country. In many cases, these sub results of the PHRD projects are seen very valuable achievements in their own right, which positively contributed to sector reforms beyond their role for the WB follow on operation. 18 Interviewees did not specify what kind of training would be required. 12

19 The table below outlines the core types of subordinated results achieved under the reviewed PHRD grants, and some illustrative examples. Exhibit 4.2 Sub-Results of PHRD preparation grants OUTPUT EXPLANATION EXAMPLE(S) Supportive legislations, regulations, rules and procedures prepared or developed. Essential baseline data available. Scope, foci, and overall strategy of the follow on project specified. Several PHRD grants assisted in the drafting of legislative documents, or led to the development of regulations or procedures necessary to guide subsequent project implementation. Most PHRD projects included studies, surveys, assessments, or consultation processes with stakeholders that provided information on existing needs, issues, and gaps in previous programming. Collected data were required for the effective planning and implementation of the follow on credit, but also provided the respective ministry and/or PIU with better understanding of, and detailed data on their sector(s). One key aspect of most projects was to further specify what the follow on project would address, and how. This was mostly based on the findings and data gained through studies, surveys, etc. In some cases this specification included a considerable adjustment of the original project outline to reflect information and insights gained during the preparation phase. During the preparation of the Rural Enterprise and Small Scale Commercial Agricultural Development project, project consultants in cooperation with the PIU developed a secondary law (i.e. a regulation for implementation of an existing law) on Seeds and Saplings. This law was a key requirement for all seed related activities planned for the subsequent follow-on project. In addition, the law implementation guidelines are seen as a key step in the reform of the Armenian seed sector, and as an important move towards bringing the sector up to international standards. Under SAC 5, new procedures for regulating bankruptcy were developed. The PHRD project also conducted a survey among 200 local enterprises that identified delays with customs clearance of shipped goods as a major obstacle for private sector development. The results of the survey led to a change of customs legislation, introducing a new system of risk management using only sample checks on goods. The preparation grant for the Education Quality and Relevance Project for example gained detailed knowledge of existing strengths and weaknesses of the education sector, in particular on the existing curriculum, IT education, and in service teacher training. The preparation phase of the project also included a comprehensive overview study of the education sector to determine institutional, financial, and technical needs and priorities, as well as an assessment of the capacities of key institutions involved in education reform. The SAC 5 project preparation included activities in the education sector, with a focus on Higher Education. Recommendations developed during the preparation phase, based on in depth analysis of the existing situation, and on knowledge of existing best practices, prevented the government from implementing its initial plan of establishing a new agency for student loans. The assessment had found this approach to be not feasible, and recommended a more realistic interim measure instead, that suggested working with existing local credit institutes to develop a scheme for better student loans. The preparation of the Education Quality and Relevance project led to a general revision of the initial project concept, and resulted in a larger amount assigned for the project, as well as an extension of the envisaged project duration. 13

20 OUTPUT EXPLANATION EXAMPLE(S) Key project management tools developed. Follow on project promoted among relevant stakeholders. Potential risks of politically sensitive projects analyzed, and mitigation strategies developed. Relevant local capacities strengthened. Several projects used the preparation phase to develop a set of management tools required for the implementation of the follow on operation, such as detailed implementation plans, operational manuals, MIS systems, M&E frameworks, selection criteria for project beneficiaries etc. In several cases, the preparation phase was utilized to share information on the envisaged project and its potential benefits for different groups with local stakeholders, and to carry out consultations with affected target groups. Several project concepts touched upon highly sensitive issues, such as the lay off of a considerable number of staff, or of severe changes to existing institutional structures. The preparation grants contributed to strengthening relevant capacities of key stakeholders required to implement the follow on project successfully. During the preparation of the Health Optimization project, a detailed manual for the envisaged hospital optimization process (i.e. merging several hospitals into fewer, more efficient ones) was developed. The manual provided considerable details of what needed to be done, where, by whom, and in what sequence. As the optimization process was highly complex, the manual provided a valuable tool for the project team. The ASIF III project developed a methodology for community profiling and mapping, as well as a related M&E strategy to assess the impact of community investment projects. Under the Utility Restructuring Project, a database on rural communities in Armenia was elaborated. The database has subsequently been used for the preparation of activities under the Millennium Challenge Account. The preparation of the Rural Enterprise and Small Scale Commercial Agricultural Development project included several workshops with members of targeted communities. The workshop aimed at illustrating how the communities could assess their specific development needs, and how they could present them in a proposal to the envisaged follow on World Bank project in order to obtain a small grant. The Health Optimization Project: During the preparation phase, a thorough analysis of the options for, and implications of the planned hospital mergers was carried out. This study allowed the ministry to anticipate potential areas for conflict (e.g. related to staff lay offs) and criticism (e.g. related to the chosen model and scope of hospital merges). The preparation phase allowed the ministry to visibly base their plans on thorough research incorporating international best practices as well as knowledge of local conditions, and to demonstrate that various different approaches had been considered before deciding for one. Similar results were achieved under the Education Quality and Relevance Project with regards to sensitive issues related to school system optimization, and potential lay-offs of teaching staff. Preparation activities of all projects helped to strengthen knowledge, skills, and relevant experience of different local partners, including of PIU teams, local consultants, and stakeholders in involved ministries, communities, and NGOs. See also section 3.3 above. 14

21 4.5 Visibility of the Japanese Government Finding 12: While all consulted stakeholders in implementing agencies were aware of the Japanese Government s role as the PHRD donor, the absence of a Japanese embassy in Armenia has negatively affected the degree of Japan s visibility in PHRD. All consulted Armenian stakeholders in implementing agencies (PIUs, Ministries) as well as local project consultants stated that they had been aware from the onset that the Japanese government funded the PHRD preparation grant. This knowledge was largely facilitated by the fact that in Armenia the PHRD is commonly referred to as The Japanese Grant while the term PHRD is used less frequently. Stakeholders also stated that all grant application and approval documents made it clear who the PHRD donor was. One interviewee shared though that some media articles had merely referred to the World Bank having approved a new grant, without referring to Japan as the PHRD donor. 19 However, during PHRD implementation, there is next to no visibility of Japan. Most interviewees could not remember any specific activities or events that had made them aware of Japan s role beyond the general knowledge of its donor status. All consulted stakeholders, including the Japanese representative, largely attributed this lack of visibility to the absence of a Japanese embassy in Armenia. The country is served from the embassy in Moscow, which makes personal encounters or frequent exchange with Japanese representatives challenging. We heard about only two instances in which Armenian project teams had had personal contact with a Japanese representative Management of the PHRD Grants 5.1 The PHRD Grant Application and Approval Processes at Country Level Finding 13: Application and approval processes at country level appear to function effectively, without any evident bottlenecks causing delays. However, one challenge is posed by the absence of a Japanese embassy in Armenia. All consulted WB and GoA stakeholders described that grant applications were in all cases developed jointly by the respective TTL and the respective line ministry. The TTLs usually take on the administrative part of managing the actual application process and negotiations with the Japanese government. Armenian stakeholders stated that they highly appreciated this administrative support. They also emphasized though, that PHRD project concepts were mostly based on their input, ideas, and priorities. 19 This can imply that public perception in Armenia regarding Japan s role may differ from that of immediate PHRD stakeholders, but there is no further evidence either for or against this assumption. 20 The ASIF III team met with a representative of the Japanese embassy during a visit to Moscow. Also, the current representative at the Japanese embassy reported upon a meeting with another PHRD project team, but did not remember which one. 15

22 There were few reported cases of delays on side of the Armenian government in approving PHRD applications. However, both WB and GoA interviewees attributed these delays to external factors that had distracted at the time, rather than as indications of a systematic bottleneck in the decision making process. In general, approval times for PHRD preparation grants are seen as being short 21 and the application process as being straightforward and uncomplicated. In the context of PHRD project application and approval, the absence of a Japanese embassy in Armenia was raised as a challenge for both WB staff and Japanese embassy representatives who are required to be in direct contact to review new PHRD proposals. WB staff reported upon difficulties in reaching the respective embassy representative on time to allow for timely submission of a PHRD proposal, and found it challenging that personal discussions usually required a trip to Moscow. Both PIU and WB staff also stated that they were not always sure who was their assigned Japanese counterpart for PHRD issues. Staff in the Japanese embassy found the geographical distance similarly challenging, and shared that they often received PHRD proposals as well as invitations for grant signature ceremonies etc. on too short a notice to respond appropriately. The distance also prevented an ongoing field presence and closer contact with partners in Armenia. 5.2 Implementation of Grant Activities Japan When to get engaged? The Japanese embassy representative raised the question if, and to what extent Japan could/should be involved in PHRD project planning at an earlier stage. This would mean to receive PRHD project proposals on a more timely basis, but also to be engaged before an actual proposal is being submitted, i.e. during the initial identification of project concepts for the following year. Finding 14: PHRD implementation in Armenia appears to be working effectively and efficiently, without major challenges for either WB staff or Armenian government partners. There are small suggestions for changes though. According to consultations with WB staff and Armenian stakeholders, all major aspect of PHRD management in the country have been and are working very well, without posing any major challenges to either party. All interviewed WB staff members (10 out of 10), and most of the consulted government stakeholders (11 out of 12) found the recipient country s systems for PHRD management adequate. This assessment was also confirmed by the (limited) information provided in the available WB GRM reports on PHRD implementation progress and management performance. Some key areas related to the implementation of grant activities that were mentioned in conversation with WB and government stakeholders were: Roles and Responsibilities 22 for PRHD management are clear for and between all involved stakeholders. This includes WB staff in Armenia and in Washington, as well as PIU staff, and employees of the GoA line ministries, and central ministry. In several cases the clarity of roles on the ground was attributed to the fact that the respective PIU had managed PHRD grants and/or WB credits before, and had been familiar with management systems and 21 All consulted stakeholders shared this view on short PHRD approval times and quoted and average of 4-6 months as the average approval time. Unfortunately, the available project documents only gave very limited information on actual approval times though. We only found information on five of the ten reviewed projects. Approval times for these ranged between 42, 49, 90, 92, and 234 days with an average of days between proposal submission and grant approval. 22 See also detailed interview results in Table 7, Appendix V 16