Completion Report. Project Number: Loan Number: 2230-PAK October Pakistan: Rural Enterprise Modernization Project

Transcription

1 Completion Report Project Number: Loan Number: 2230-PAK October 2010 Pakistan: Rural Enterprise Modernization Project

2 CURRENCY EQUIVALENTS Currency Unit Pakistan rupee/s (PRe/PRs) At Appraisal At Project Completion 1 December June 2009 PRs1.00 = $ $ $1.00 = PRs59.72 PRs81.46 ABBREVIATIONS ADB Asian Development Bank AHAN Aik Hunar Aik Nagar BDS business development services CDWP Central Development Working Party CSP country strategy and program DMF design and monitoring framework KMS knowledge management system MDP market development preparation MoIPSI Ministry of Industries, Production, and Special Initiatives MSE micro- and small enterprises NGO nongovernment organization PCR project completion report PMU project management unit PRM Pakistan Resident Mission PSC project steering committee QCBS quality and cost based selection REMC Rural Enterprise Modernization Company REMP Rural Enterprise Modernization Project RRP report and recommendation of the President SMEDA Small and Medium Enterprise Development Authority TA technical assistance

3 NOTES (i) The fiscal year (FY) of the government of Pakistan and its agencies ends on 30 June. FY before a calendar year denotes the year in which the fiscal year ends, e.g., FY2009 ends on 30 June (ii) In this report, "$" refers to US dollars. Vice-President X. Zhao, Operations 1 Director General J. Miranda, Central and West Asia Department (CWRD) Director M. Ojiro, Advisor, Office of the Director General, CWRD Team leader Team members R. Jones, Natural Resources Economist, CWRD E. de Castro, Associate Project Analyst, CWRD In preparing any country program or strategy, financing any project, or by making any designation of or reference to a particular territory or geographic area in this document, the Asian Development Bank does not intend to make any judgments as to the legal or other status of any territory or area.

4

5 CONTENTS Page BASIC DATA ii I. PROJECT DESCRIPTION 1 A. Objectives of the Project 1 B. Project Components and Scope 2 II. EVALUATION OF DESIGN AND IMPLEMENTATION 2 A. Relevance of Design and Formulation 2 B. Project Outputs 3 C. Project Costs 5 D. Disbursements 6 E. Project Schedule 6 F. Implementation Arrangements 7 G. Conditions and Covenants 8 H. Consultant Recruitment and Procurement 8 I. Performance of Consultants, Contractors, and Suppliers 9 J. Performance of the Borrower and the Executing Agency 9 K. Performance of the Asian Development Bank 10 III. EVALUATION OF PERFORMANCE 11 A. Relevance 11 B. Effectiveness in Achieving Outcome 11 C. Efficiency in Achieving Outcome and Outputs 11 D. Preliminary Assessment of Sustainability 12 E. Impact 12 IV. OVERALL ASSESSMENT AND RECOMMENDATIONS 13 A. Overall Assessment 13 B. Lessons 13 C. Recommendations 14 APPENDIXES 1. Project Results Status of Compliance with Loan Covenants Summary of Technical Assistance Staff Requirements Implementation Schedule Project Costs Disbursements by Category Background to AHAN Company 33

6 BASIC DATA A. Loan Identification 1. Country 2. Loan Number 3. Project Title 4. Borrower 5. Executing Agency 6. Amount of Loan 7. Project Completion Report Number B. Loan Data 1. Appraisal Date Started Date Completed 2. Loan Negotiations Date Started Date Completed 3. Date of Board Approval 4. Date of Loan Agreement 5. Date of Loan Effectiveness In Loan Agreement Actual Number of Extensions 6. Closing Date In Loan Agreement Actual Number of Extensions 7. Terms of Loan Interest Rate Maturity (number of years) Grace Period (number of years) Pakistan 2230-PAK Rural Enterprise Modernization Islamic Republic of Pakistan Ministry of Industries, Production, and Special Initiatives SDR3,505,000 (equivalent to $5 million) PCR:PAK April May December December February February May May 2006 None 30 June June 2010 None 1% per annum for first 8 years and 1.5% per annum thereafter 32 8

7 iii 8. Disbursements a. Dates Initial Disbursement Final Disbursement Time Interval 13 April June months Effective Date Original Closing Date Time Interval 19 May June months b. Amount (SDR 000) Category or Subloan Original Allocation Revised Allocation Last Revised Allocation Amount Cancelled Amount Disbursed 01A Consultants International (657) 01B Consultants Domestic (364) 44 02A REMC International (576) 02B REMC Domestic (524) 03 Capacity Building (317) Vehicles & Equipment (48) 05 Recurrent Expenditures (136) Interest Charges (39) 6 07 Unallocated (539) 99 Imprest Account Total SDR 3,505 3, , Total US$ 5,241 5, , C. Project Data 1. Project Cost ($ 000) Cost Appraisal Estimate Actual Foreign Exchange Cost 2, Local Currency Cost 3,929 1,642 Total 6,500 2,124 There is an inconsistency between the RRP and the Loan Agreement. The Loan Agreement did not specify a local cost ceiling and the Loan Financial Information System did not record the local cost separately.

8 iv 2. Financing Plan ($ 000) Cost Appraisal Estimate Actual Implementation Costs Borrower Financed 1,500 1,642 ADB Financed 4, Other External Financing 0 0 Total 6,436 2,115 IDC Costs Borrower Financed 0 0 ADB Financed 64 9 Other External Financing 0 0 Total 6,500 2,124 ADB = Asian Development Bank, IDC = interest during construction 3. Cost Breakdown by Project Component ($ 000) Component Appraisal Estimate Actual 1. Market Development Preparation Facility 1,913 1, Sector Analysis and Pilots 1, Rural Enterprise Modernization Company 3,465 0 Interest 64 9 Total 6, 500 2, Project Schedule Item Appraisal Estimate Actual Date of Contract with Consultants Pilot Financial Accountant Market Researcher Design Monitoring and Evaluation Subject Matter Specialist Reallocation of Loan Proceeds Final Cancellation Consultant categories are defined in Appendix 3. February 2006 February 2006 February 2006 February 2006 June 2007 June 2007 June 2007 July 2007 March 2007 June Project Performance Report Ratings Implementation Period Development Objectives Ratings Implementation Progress From February 2006 to December 2006 Satisfactory Satisfactory From January 2007 to December 2007 Satisfactory Satisfactory From January 2008 to December 2008 Satisfactory Satisfactory From January 2009 to December 2009 Satisfactory Satisfactory From January 2010 to May 2010 Satisfactory Satisfactory

9 v D. Data on Asian Development Bank Missions Name of Mission Date No. of Persons No. of Person-Days Specialization of Members Reconnaissance 7 15 Feb a Loan Fact Finding 14 Apr 3 May a Follow-up Fact Finding 21 Jun 5 Jul a,e Inception 28 Nov 3 Dec b,d Review Apr b Review Dec b Midterm Review May b,d Project Completion Review Jul c,d a = project economist, b = rural development specialist, c = natural resources economist, d = project analyst, e = legal counsel.

10

11 I. PROJECT DESCRIPTION 1. The Government of Pakistan borrowed $5 million equivalent from the Special Funds resources of the Asian Development Bank (ADB) to implement the Rural Enterprise Modernization Project (REMP). This technical assistance (TA) loan was described as being "designed to facilitate knowledge and information acquisition, and institutional support mechanisms, with the overall aim of expanding employment opportunities, and enhancing the competitiveness of micro- and small enterprises (MSEs) in rural areas on a sustainable basis." 1 The project concept was designed to emulate the One Tambon One Product (OTOP) program of Thailand and Japan. These programs had been successful in modernizing nonfarm MSEs, thereby improving the employment prospects of the poor in those countries. 2. Pakistan's population is 68% rural, and agriculture accounts for almost 45% of total employment. Outside the agriculture sector, 46% of the rural poor are nonfarm households that do not share directly in the incomes derived from agricultural production. Nonfarm activities offer important economic opportunities for the rural poor where landlessness is high. Rural poverty remains high in Pakistan; the percentage of rural poor living below the poverty line hardly changed from 1990 (36.9%) to 1999 (35.9%), and after increasing in 2002 (38.9%), it then declined to 27% in By contrast, in 2006 the percentage of the population in urban areas living below the poverty line was 13%, indicating a large rural-urban gap in poverty rates. 3. Rural economic opportunities must expand in both the agriculture and the nonfarm sectors for Pakistan to meet its goal of reducing poverty by half in Diversifying the rural economy in nonfarm sectors, such as services, small-scale manufacturing, and value-added agricultural processing, can contribute towards this goal. 4. The project classifications were: sector industry and trade; sub-sector small and medium-scale enterprises; theme sustainable economic growth; and sub-theme promoting economic efficiency and developing rural areas. The REMP was environment category C. A. Objectives of the Project 5. The anticipated impact of the project was the strengthening and diversification of employment opportunities in the nonfarm sector, especially for the poor and vulnerable groups such as the landless, low-wage earners, and women. The expected project outcome was to enable rural and peri-urban businesses and occupational groups to access a range of business development services, including better knowledge of markets and appropriate technologies, and thus help in the further development of sustainable livelihoods. 6. It was envisaged that the project would (i) fill major gaps in information and analysis on rural nonfarm MSEs and consolidate this information in a knowledge management system (KMS); (ii) identify and initiate carefully targeted pilot interventions to support rural MSE development; and (iii) assess institutional support processes and the feasibility of establishing an autonomous not-for-profit entity to facilitate rural MSE development. The executing agency for the project was the Ministry of Industries, Production, and Special Initiatives (MoIPSI), the implementing agency was the Small and Medium Enterprise Development Authority (SMEDA), 1 ADB. January Report and Recommendation of the President to the Board of Directors: Proposed Technical Assistance Loan to the Islamic Republic of Pakistan for Rural Enterprise Modernization Project. Manila. 2 ADB. December Poverty Assessment Update: Pakistan. Pakistan Resident Mission.

12 2 and the Project Management Unit (PMU) was the Aik Hunar Aik Nagar Cell (AHAN Cell) of SMEDA. B. Project Components and Scope 7. The REMP had 3 components: (i) (ii) Market development preparation (MDP). This component was to identify areas for further research, initiate the collection of required data, and design pilot projects in selected subsectors. A feasibility study on the establishment of the need for a Rural Enterprise Modernization Company (REMC), or some other alternative, was to be conducted. This was to be followed by a feasibility study on the design of the organizational and operational framework of such a company. Sector analysis and pilot interventions. A sector analysis was to be undertaken to provide information for the implementation of a series of targeted pilot projects. The sector analysis involved developing a KMS on the rural nonfarm economy that would inform the design of the rural enterprise development agenda. The database would be used to identify where and how product subsectors tend to cluster (i.e., regional concentration). Subsector selection criteria would be determined, and then the project would prepare subsector and cluster development plans to document opportunities and constraints in the selected subsectors. Following the development of the subsector studies, pilot projects would be undertaken to gain experience and to develop a framework for the outreach and partnership arrangements with stakeholders. A series of workshops involving stakeholder participation would develop a priority list of potential pilot projects within the identified clusters. Although the precise nature of the pilot projects would be determined in the implementation phase, it was expected that between 20 and 30 possible subsector business plans for about 4 pilot interventions would be prepared. (iii) REMC. It was expected that within 18 months of the project s inception, ADB and the government would make a decision on the establishment of the REMC. Assuming a positive decision, the MDP team would define a cost-effective management structure and a strategy to ensure outreach in dispersed rural areas. The MDP team would fulfill the requirements for registering the REMC and obtaining a license to operate under Section 42 of the Companies Ordinance Once established, the REMC would extend the activities started in components 1 and 2. The database developed by the MDP team would be transferred to the company, which would be responsible for analyzing potential opportunities in selected subsectors and for implementing the pilot projects. II. EVALUATION OF DESIGN AND IMPLEMENTATION A. Relevance of Design and Formulation 8. The project design and outputs at appraisal were relevant and consistent with the government s social development objectives that targeted poverty reduction in rural areas. ADB s country strategy and program (CSP) aimed to support poverty reduction in

13 3 Pakistan through: (i) good governance; (ii) sustainable, pro-poor growth; and (iii) inclusive social development. This project was closely aligned to sustainable, pro-poor growth because it had a strong rural development focus. 9. The government developed a series of Millennium Development Goals with associated targets including halving, between 1990 and 2015, the proportion of people whose income is less than $1 per day. One means to achieve this goal was to enhance the capacity of businesses in rural and peri-urban areas to improve efficiency and achieve greater outputs and employment potential. The 2004 CSP stressed the importance of investment in rural modernization by specifically mentioning the need to develop a "holistic and integrated vision and strategies for rural modernization and frameworks for improving rural livelihoods A number of consultancy studies were commissioned to help in the design of the project. These included a sector analysis of rural development and capacity building in rural and periurban industries. The studies were of a high standard and made a positive contribution to project design. 11. An appropriate level of consultation occurred through three missions (one reconnaissance mission, one loan fact finding mission, and a follow-up mission), involving 45 person-days of ADB staff. Numerous stakeholders were consulted during project processing, including the Government of Pakistan, nongovernment organizations (NGOs), the Pakistan Poverty Alleviation Fund, SMEDA, and local government. 12. Inclusion of the REMC concept in the project design was inappropriate for two reasons. First, the project was a pilot study, and therefore it would have been preferable if the formation of a company to provide ongoing business development services (BDS) had been considered in a second phase project, once appropriate lessons had been learned. Second, the project delivers public benefits that are not provided by the private sector. This is because the markets for BDS are highly immature and the potential beneficiaries are mostly poor with a low ability to pay for these services. Thus, the concept of a self-funding BDS company in this environment was unrealistic. B. Project Outputs 13. The achievement of outputs against the targets set at appraisal is in Appendix 1. The project had five outputs; three were partly achieved and two were not achieved. 14. The deteriorating security situation in Pakistan resulted in the international consultant recruitment process collapsing, with no international consultants being employed for the duration of the project. Despite a number of advertisements, and offers made to potential candidates, a team leader was never recruited. This delayed recruitment of the remaining positions as it was planned for the team leader to be involved in the recruitment of the other consultants. As a result, many project activities that required international consultant inputs were not undertaken. These include the baseline survey, which was required to assess achievement of the project outcome and outputs. 3 ADB. September Country Strategy and Program Update : Pakistan. Manila (para. 10).

14 4 15. The outputs for each project component are described below: 1. Market development preparation facility 16. The first output to set up a process to introduce BDS in rural and peri-urban areas was partly achieved. 17. The AHAN Cell within SMEDA was established as planned and performed all necessary tasks, and regional offices were established in all four provincial headquarters Lahore, Karachi, Peshawar, and Quetta. However, the TA team could not be mobilized because consultants could not be recruited. This contributed to the lack of achievement of the strategic institutional approaches and outreach processes required of this team within 6 months of loan effectiveness. This target was also affected by a 9-month delay in the inception mission, which meant that alternative arrangements could not be put in place within a reasonable time frame. 18. The feasibility study to assess institutional options and justification for a not-for-profit REMC was not conducted. Without consulting ADB, the government converted the PMU (AHAN Cell) to a company, called Aik Hunar Aik Nagar (AHAN) Company, under Section 42 of the Companies Ordinance This was in breach of loan covenants. 19. The inability to mobilize the consultant teams meant that the required activities to achieve this output were not undertaken, and consequently project costs were reduced. AHAN Cell, and subsequently AHAN Company, attempted to carry out the tasks in-house, but could only do so in a more limited manner because of the limited financial resources available. 2(a) Sector analysis 20. The sector analysis stage had two outputs: (i) establishment of the KMS, and (ii) baseline studies and subsector and cluster development plans. These were not achieved for two reasons. First, the consultant teams were not recruited and AHAN Cell was more interested in undertaking pilot studies. Second, there was a change in implementation arrangements because the government requested that the project deliver greater and more rapid achievements with a target of at least 100 pilot projects to provide on-ground benefits to communities. SMEDA, the implementing agency, informed ADB of the change in the on-ground realities and proposed an alternative "two-pronged" approach that would involve concurrently conducting the component 2 sector analysis and pilot studies. The request was accepted by ADB during the inception mission. 4 The consequence was that the AHAN Cell became focused on implementing the government s priority pilot studies and the sector analysis activities were neglected. 21. There is no evidence that the performance target of a 50% increase in user requests for information and technologies by MSEs was met. A national market research consultant was hired to do the initial work on the KMS and this would have been further refined if the consultant team had been hired. 4 ADB Aide Memoire. Loan Inception Mission for Loan No PAK (SF): Rural Modernization Project. 29 November 3 December 2006.

15 5 2(b) Pilot projects 22. The fourth output was the pilot projects. This was partly achieved and was the major focus of AHAN activities, despite the absence of international consultant inputs. The first performance target was for 20 to 30 possible subsectors being identified for pilot interventions. AHAN greatly exceeded this target, undertaking about 650 cluster reviews, and 82 pilot projects were conducted, 45 of which were completed and 37 ongoing at the time of the project completion report (PCR) mission. Selection of the AHAN projects was based on the following criteria: (i) the number of MSEs in a geographic area; (ii) growth potential; (iii) geographic location; (iv) target group (e.g., women, landless); (v) marketability; and (vi) availability of raw materials. 23. The pilot project stage required that the preparation of business plans for the pilot interventions would be based upon gender-disaggregated data, sales turnover, updated technology, and market prospects. This was not achieved as the consulting team was not employed. AHAN contributed to this output by using in-house resources to develop business plans in the project proposal documents prepared for each subproject. 3. Establishment of REMC 24. The fifth output to establis the REMC was partly achieved. In October 2007, the government converted the PMU (AHAN Cell) into AHAN Company, but the action breached a loan covenant and ADB advised MoIPSI at mid-term review 5 that it would be ineligible to access ADB loan funds for component AHAN Company subsequently undertook operations in the knowledge that achievement of the performance targets for component 3 no longer applied to the company. However, AHAN Company s operations were consistent with the goals outlined in the project and have contributed toward achievement of some of the performance targets of component A number of the performance targets were not met because of the lack of consultants (Appendix 1), as a result of the prevailing security situation, and the withdrawal of ADB funds for component 3. However, AHAN set up the management team (Appendix 7), established regional offices, developed a set of board operating procedures and a project appraisal panel, and for each individual project developed a marketing plan and a monitoring and evaluation system. C. Project Costs 27. At appraisal, the total project cost was estimated at $6.5 million ($2.6 million in foreign exchange costs and $3.9 million equivalent in local currency costs). ADB was to provide a loan of $5 million, and the borrower $1.5 million equivalent. The actual project cost at loan closing was $2.12 million, consisting of $0.48 million from ADB and $1.64 from the borrower. The costs of the project at appraisal and physical completion are given in Appendix The low disbursement of 9.6% of the available loan was due to three factors. First, the project had significant difficulty in hiring consultants, particularly international staff. Despite a number of candidates being made offers, the international team leader was never recruited and other international positions were unfilled. Other consultants were to be recruited using quality 5 ADB Aide Memoire of the Midterm Review Mission. Loan No PAK: Rural Enterprise Modernization Project May 2008.

16 6 and cost based selection (QCBS) procedures, but this process was delayed because none of the international positions could be filled. Once the REMC (AHAN Company) was formed, the government decided to place further consultant recruitment on hold. 29. Second, a Planning Commission (PC-1) application was not prepared and approved prior to loan negotiations. SMEDA subsequently prepared a PC-1 covering all project components at a total cost of PRs390 million to be spread over three years, which was submitted to the Central Development Working Party (CDWP) for approval. Only Phase 1 of the project (comprising components 1 and 2), with an associated PRs114 million, was approved (20 May 2006), and this was to be spread over a period of 18 months from July 2006 to December SMEDA was informed that a fresh PC-1 may be submitted on conclusion of Phase 1 for the establishment of the REMC (component 3). This decision limited the ability of SMEDA and AHAN Cell to use the loan funds. 30. Third, because the loan covenant was breached when the REMC was established, the project was ineligible to access $3.464 million for component 3. A second PC-1 for the remainder of the project was not prepared. D. Disbursements 31. Of SDR3,505,000 ($5 million equivalent), only $481,893 was disbursed. Details of the annual loan utilization are given in Appendix An imprest account was established at the National Bank of Pakistan in July 2007 with an initial advance of $282,014. The fund was replenished only once, and other payments were made directly. Only expenditures relating to Phase 1 of the project were disbursed (see para. 29). In order to implement Phase 2, a new budget should have been submitted on conclusion of Phase 1, but this did not occur (see para. 30). The unutilized balance of the imprest account ($171,776) was refunded to ADB on 30 June In order to implement the two-pronged approach, a reallocation of loan proceeds was approved on 23 March E. Project Schedule 34. The project was originally envisaged to be implemented over 3 years with an estimated completion date of 31 December The project was initiated on time with the loan becoming effective ahead of schedule on 19 May 2006, while the PMU was established in Lahore in November 2005 and the project director was appointed in June Delivery of the key outputs was affected by serious implementation difficulties and the inability of SMEDA to recruit international consultants, which then caused delays in the recruitment of domestic consultants. Consequently, funds for the procurement of these services were delayed or unutilized. 36. Acceptance of the two-pronged approach resulted in changes in the project schedule. The revised project implementation schedule is in Appendix 4. First, it was decided to split the project consultants into 2 segments (i) to provide immediate support to the selected sector using individual consultants, and (ii) to complete the appropriate small and medium enterprise assessment through the recruitment of a consulting firm. Second, establishment of the REMC was rescheduled from 18 to 30 months after loan effectiveness.

17 7 37. In September 2008, the government requested a project extension with a proposed closing date of 31 December ADB refused the request on the basis that the objectives of the project were unlikely to be achieved because the level of contract awards and disbursements was very low and the proposed training and capacity building of the REMC staff had not even started. F. Implementation Arrangements 38. MoIPSI, the executing agency, was responsible for overall project facilitation, management, and monitoring. There were two implementing agencies responsible for the different project components: (i) SMEDA for components 1 and 2; and (ii) upon its incorporation as a not-for-profit company, the REMC for component A Project Steering Committee (PSC) established in September 2005 was chaired by MoIPSI, with representation from the Economic Affairs Division, Ministry of Commerce, MoIPSI, SMEDA, Khushali Bank, 7 private sector representatives, and the joint secretary as the project coordinator. The PSC met twice under the guidance of the Prime Minister. The project received significant attention and support from the government, with the Prime Minister and Ministers of Economic Affairs and MoIPSI maintaining regular contact with SMEDA and the project director. The project team supported the establishment of provincial coordination committees to facilitate engagement with provinces and districts. 40. Provincial offices in Punjab, Sindh, Khyber Paktunkhwa, and Balochistan were established, with regional coordinators being appointed. The project team made the necessary preparations for establishment of the offices, procured orders for vehicles and computers, and made logistical arrangements for full operations. 41. The REMC (AHAN Company) was established on 18 October 2007 following the government's decision to convert the PMU (AHAN Cell) into a non-profit company. AHAN Company operates as a subsidiary of Pakistan Industrial Development Corporation, which provided PRs200 million as initial funding for implementation of 100 projects. At the time of the project completion review there were 7 board of directors from the government appointed, but another 8 nominations had been submitted to the Prime Minister for approval so as to expand the size of the board. Once these new members have been appointed, the company will have a complete board and be in compliance with the Act. 42. The adoption of the two-pronged approach resulted in a number of amendments to the implementation schedule, particularly the sequencing and timing of the consultant inputs. This also required a revision of the cost tables. The changes in implementation were categorized as minor in scope and ADB approval was given in January During the midterm review mission in May 2008, a minor change in scope was proposed by the government to address food security and safety net matters. ADB requested concurrence to proceed with the proposed amendment, but despite three follow-up requests there was no response from MoIPSI. The suggested change was also outside the original scope of the project and ADB terminated the proposed initiative in August 2008.

18 8 G. Conditions and Covenants 44. The loan covenants of the Loan Agreement were considered relevant. Most covenants have been complied with (see Appendix 2 for the status of compliance) with the exception of the following: (i) Late compliance. Submission of the work plan was due on 18 May 2006 but was received on 6 July (ii) Partial compliance. Covenants that were partly complied with relate to (a) composition of the MDP team, as the consultants were not recruited; (b) SMEDA being the implementing agency not only for components 1 and 2, but then also for component 3 after the creation of AHAN Company; (c) supervision from ADB, as regular missions were not fielded; only one review mission was fielded in the first year, and the midterm review mission that should have been fielded within 18 months of the effective date was fielded after two years; (d) submission of MoIPSI s PCR, which was 9 months late. (iii) Noncompliance. The covenants for the establishment of the REMC required that a feasibility study be undertaken before its creation, but this was not done. The progress report did not use the prescribed ADB format, which had not been provided during the inception mission. (iv) Not applicable. The project did not involve involuntary resettlement or have an adverse impact on indigenous peoples or the environment (para. 58 of the RRP). H. Consultant Recruitment and Procurement 45. The project design included 8 international consultants (108 person-months) and 16 domestic consultants (218 person-months). The changed implementation schedule agreed to during the inception mission led to a revised consultant schedule for 82 person-months of international consultants and 364 person-months of domestic consultants (Appendix 3). In the revised schedule, 5 consultants were to be employed through individual consultant selection, 10 consultants through a firm using QCBS, one through single source selection, and one through consultant qualification selection or fixed budget selection. However, only four individual domestic consultants were actually recruited. 46. International consultants could not be recruited because of the deteriorating security situation. The team leader position was advertised three times, offers were made to three candidates, with each offer being declined. Recruitment of the remaining consultant positions was delayed until the team leader position could be filled. 47. The REMC decision by the government had the following implications. First, the resources that were to have been used for consultants to undertake the feasibility studies to justify the REMC were now redundant, because the company had already been formed. Second, given the breach in the loan covenant, the project was unable to fund any consultant inputs that would have been required in component 3. Third, there was no need for an updated PC-1, which would have allowed AHAN Company to access ADB resources for Phase 2.

19 9 48. There was an inconsistency between the loan agreement and the consultant schedule in the RRP. The schedule required consulting services to complete a feasibility study of institutional options and a business plan for the establishment of a REMC within 18 months of project inception. However, Schedule 3 paragraph 8 of the Loan Agreement stated no withdrawals shall be made from the Loan Account for Category 2 in the Table (page 15 of Loan Agreement) until (i) REMC has been incorporated as a not-for-profit company. The establishment of the REMC was not planned until month 24 of the project. Therefore, there was a conflict between the timing of the consulting inputs and when the funding for these services would be available. This was addressed during the inception mission by changing the consultant implementation schedule. 49. Consultants were engaged according to ADB guidelines. The market research consultant and pilot financial accountant were increased by 4 person-months and 1 personmonth, respectively. The additional time for the market research consultant was required to supervise 15 Sector Reports prepared by the regional offices and to incorporate these into a single report. The additional services required of the pilot accountant were (i) the last withdrawal application, and (ii) the liquidation of $237,446 to the imprest account. No goods were procured for the project. I. Performance of Consultants, Contractors, and Suppliers 50. The project employed four domestic consultants: subject matter specialist, pilot financial accountant, market researcher, and monitoring and evaluation specialist. AHAN was highly satisfied with the performance of the subject matter specialist and pilot financial accountant. However, the performance of both the market researcher and the monitoring and evaluation specialist was less satisfactory as AHAN was required to provide considerably more guidance and support than expected. J. Performance of the Borrower and the Executing Agency 51. The borrower established a PMU on schedule in November Establishment of the imprest account was delayed until April 2007, and initial funding was provided by the government. There was adequate provision in counterpart funds, and by project completion this was in excess of the amount stated in the loan agreement. The selection of MoIPSI as the executing agency, SMEDA as the implementing agency, and AHAN Cell within SMEDA as the PMU were logical and appropriate choices, and at project inception had the required institutional capacity to implement the project. 52. The CDWP approval of only a limited PC-1 for Phase 1 was detrimental to the disbursement of ADB funds over the course of the project. This development resulted from a divergence of views within the government between those involved in the project design and those involved in the PC-1 approval. 53. Two instances of government involvement had negative impacts on the project. First, the government request for the two-pronged approach meant that AHAN Cell focused its efforts on the government priority pilot studies and thus neglected other components of the project. Second, the decision by the government to convert the PMU AHAN Cell into the REMC in breach of a loan covenant and without consultation with ADB resulted in the project being ineligible for ADB loan funds for component 3.

20 The government instituted an organizational change of certain ministries, which influenced project management. The initial executing agency was MoIPSI, which was then split into separate ministries. The project was then administered by the Ministry of Industries and Production before being transferred to the Cabinet Division, then the Ministry of Special Initiatives, and is now back under administration by the Ministry of Industries and Production. 55. Project closing was considerably delayed through lack of action in refunding to ADB the unutilized imprest account balance of $171, The executing agency failed to fully realize the expected project outputs, and a number of the project covenants were either not complied with or complied with late. However, this was largely due to external factors such as the security situation, which affected consultant recruitment. SMEDA established the PMU and regional offices as required and provided material and institutional support. 57. The PMU AHAN Cell was constrained in its ability to deliver the project outputs because of the limited resources that were made available to it. AHAN has been able to demonstrate that it has had a positive impact and has contributed to the outcome statement of the project. It has undertaken 82 pilot projects, with over 7,000 direct and 46,000 indirect beneficiaries. It is estimated that 55% of the direct beneficiaries are women. The estimated increase in income gained by beneficiaries so far is PRs60 million (approximately $0.75 million). 58. The performance of the borrower is rated unsatisfactory, and the executing agency partly satisfactory. K. Performance of the Asian Development Bank 59. The project design recognized the inherent risks in this type of project and the need for close monitoring and supervision. Quarterly review missions in the first year of implementation were considered necessary, to be followed by project review every 6 months (para. 54 of the RRP). Assistance from the Pakistan Resident Mission (PRM) was expected with the assumption that a rural development sector specialist would be posted to PRM. The intended frequency of review did not occur: the inception mission (28 November to 2 December 2006) was 9 months after loan signing; there was only one review mission in the first year of implementation (April 2007); and the anticipated project support from PRM did not eventuate. 60. An ADB realignment in 2006 meant that there was no continuity in staff supervision from design to initial implementation. Initially, the project was designed by the South Asia Department (SARD), which had responsibility for the Pakistan portfolio. However, during the project inception phase, all Pakistan projects were transferred to the Central and West Asia Department (CWRD). The officer responsible for the design process was no longer involved in the project administration. Assignment of responsibility for the project was delayed and eventually passed to an officer who was initially unfamiliar with the problems. The inception mission was delayed and ADB was unable to field the four review missions designed for the first year. 61. The changes in the project implementation arrangements during the inception mission contributed to the poor project performance. At that time, ADB may have considered its action in accepting and approving the desired changes as responding appropriately to the client s request. However, with the benefit of hindsight, ADB should have insisted that the project be implemented according to design, because adoption of the two-pronged approach resulted in a loss in project focus and limited achievement of project outputs.

21 The project was reviewed twice in the second year of implementation (December 2007 and May 2008) as required. After the May 2008 mid-term review mission, ADB decided to close the project, which negated the need for further review missions. 63. Although PRM did not meet the initial expectation of a designated sector specialist to contribute to management of the project, it provided an appropriate level of support to the PMU through advice on training and procurement procedures and other matters. 64. The project performance ratings did not reflect the true status of the project. Ratings for development objectives and implementation progress since the midterm review would have been more appropriately stated as unsatisfactory. 65. The performance of ADB is rated as unsatisfactory. III. EVALUATION OF PERFORMANCE A. Relevance 66. At loan approval and during implementation the project is rated as relevant. The project was consistent with the CSP, particularly the objective of providing sustainable pro-poor growth. The main strategic areas under the current CSP 6 are energy, infrastructure, reforms, and urban services, with little emphasis on rural development. However, the relevance of the project regarding the Pakistan Millennium Development Goals and its associated targets remain largely unchanged. 67. The project is therefore considered to be relevant. B. Effectiveness in Achieving Outcome 68. The project undertook 82 pilot studies, completing 45 with 37 ongoing at the time of the PCR mission. According to an AHAN Company survey, 7,382 direct beneficiaries and approximately 46,000 indirect beneficiaries have benefited from the Project. 69. In order to achieve the intended outcome, a number of performance targets were required to be achieved (Appendix 1). Given that no baseline study was undertaken by the consultant team in 2006 as planned, it is not possible for AHAN to demonstrate that the performance targets have been met. These include evidence of increases in the demand for BDS in the pilot sites; changes in the sales volume of MSEs; the ability and willingness to pay for BDS in the pilot sites; increases in domestic and international transactions; and increases in microfinance services for MSEs. 70. Given the lack of evidence to measure achievement of the outcome targets, the project is considered to be ineffective. C. Efficiency in Achieving Outcome and Outputs 71. There was no formal financial and economic analysis included in the RRP because the project was viewed as a catalyst and the full economic impact was expected to arise through 6 ADB. January Pakistan: Country Partnership Strategy Manila.

22 12 multiplier effects. The direct long-term benefits were stated in the RRP (para. 56) as flowing from increased nonfarm employment and income-earning opportunities in the rural areas, particularly for wage earners, women, and poor producer groups with the potential to engage in a higher level of commercial activity. The approach adopted here to understand the potential efficiency of the project is to assess the likelihood that these stated benefits were achieved. 72. AHAN Company has identified that incomes of the 7,382 direct beneficiaries from the pilot projects has increased by PRs60 million ($0.75 million). Project costs so far are $2,124 million ($0.482 million ADB; $1.642 million government), leaving a deficit between benefits and costs of $1.374 million. The project benefits to target groups will not be a one-off effect as currently measured, and will continue to accrue in the future. 73. AHAN has achieved some economic benefits and a rating of less efficient in achieving outcomes and outputs is appropriate. D. Preliminary Assessment of Sustainability 74. A key output that contributes to the long-term sustainability of the project is the REMC, namely AHAN Company. The REMC is responsible for delivering BDS to the nonfarm sector and therefore plays an important role in achieving the project s intended outcome and impact. 75. AHAN Company cannot generate sufficient revenues from providing BDS to the industries it services. It does not charge for services as the targeted groups have a very low ability to pay. The provision of services is a public benefit, with limited scope for this to be a commercialized product until at least the target industries become more mature and have a greater ability to pay for services. Consequently AHAN is required to obtain funds from the government and donor agencies. AHAN has recently submitted a PC-1 application to the government for continued funding. 76. There is considerable uncertainty in the ability of AHAN to secure a source of long-term funding to continue providing BDS to the rural nonfarm sector. The government was enthusiastic in establishing the project to achieve this objective and provided an in-kind financial contribution, but has recently been reluctant to commit to a long-term funding plan. The design for REMC to be a self-funding body that provides public benefits in a market with limited ability to pay is misguided. Therefore, the project is rated as unsustainable. E. Impact 77. The project was focused on improving the provision of BDS and employment opportunities to the poor and vulnerable groups. There was no construction of infrastructure that would impact upon either environmental systems or would lead to issues of relocation of people affected by project activities. Consequently there are no significant environmental, socioeconomic, and socio-cultural impacts associated with the project. 78. An AHAN Company survey of project beneficiaries identified the impact of higher incomes flowing from AHAN interventions. About half the beneficiaries report that the higher incomes have allowed them to improve living conditions either through better quality housing or making improvements to existing dwellings. Over 40% report (i) they can now educate children and afford access to health services, (ii) domestic violence has declined because of the financial empowerment of women, and (iii) people with higher incomes are better able to financially cope with inflation, which is reflected in the increasing prices of household goods. Slightly over 20%

23 13 are women, who have now become self-employed by starting their own business. About 9% of AHAN project participants have not utilized the skills developed from project training. 79. There is no evidence that the project has produced the intended outputs and outcome, and is thus unlikely to have had a significant impact in terms of the stated goal of strengthening and diversifying employment opportunities in the nonfarm sector. The project has identified some direct and indirect beneficiaries, with 55% of beneficiaries being women. Therefore, the project had some limited impact, which is commendable given the limited resources made available to AHAN. Furthermore, these benefits should accrue and even increase over time if AHAN can secure funding and a more sustainable future. IV. OVERALL ASSESSMENT AND RECOMMENDATIONS A. Overall Assessment 80. The project was not implemented as conceived for a number of reasons: (i) the project implementation arrangements were changed during the inception mission to facilitate the government s requirement for immediate action and results; (ii) the deteriorating security situation caused the consultant recruitment process to collapse, meaning that the outputs dependent upon the consultant inputs could not be achieved; and (iii) the establishment of the REMC and breach of a loan covenant resulted in the project being ineligible to access component 3 loan funds. 81. The design and monitoring framework (DMF) lacked clarity, the outputs were a mixture of activity and output statements, and there were too many performance targets to objectively measure. Some targets, such as undertaking an impact analysis of the project 3 to 5 years after project inception, are inappropriate within a DMF. A performance and monitoring system was not developed to track project progress. 82. The project is rated as unsuccessful based on a review of its relevance, efficacy, efficiency, and sustainability. B. Lessons 83. The key lessons learned from the project are: (i) (ii) The project attempted to indigenize the OVOP concept that was considered successful in Thailand and Japan. However, only limited efforts were made to understand the likelihood of the success of this initiative in Pakistan. The reasons for the program s success in the other countries and whether the same positives were present in Pakistan were not assessed. Moreover, the success of such projects is often led by a bottom-up grassroots demand for the project outputs, yet this project was a top-down initiative. Future projects that rely on the transfer of foreign programs should assess the suitability of the concept before adapting it from one country to another. The security situation in Pakistan was a major factor in the failure to recruit international consultants. In countries where security concerns prevail, project design needs to account for the difficulty in recruiting and retaining suitable international consultants. Limiting the role of international consultants to a few key positions and to supervision of national consultant inputs may contribute to greater project success.

24 14 (iii) (iv) (v) (vi) (vii) (viii) (ix) The project had a high weighting of consultant inputs in the first 6 months. It is unrealistic to expect that international consultant teams can be on the ground and achieving outputs within such a short time frame in countries such as Pakistan, given the difficulties in recruitment. An alternative may be to phase in international consultant inputs later in the project to allow for a more realistic recruitment process. However, application of the streamlined business arrangement principles should provide greater project preparedness for accelerated consultant recruitment. The project implementation schedule was amended early in the project to comply with the government s request for immediate action and results via the twopronged approach. Unless there are clear deficiencies in project design, ADB should resist requests to change the design or implementation arrangements early in implementation, because the original project design should be given a chance to succeed. Corrective action can be taken later as problems become apparent. MoIPSI and the PMU had difficulty in understanding the requirements of the DMF because it lacked clarity regarding activities and output requirements. The PMU noted that, even though terms of reference had been prepared for individual consultants, the exact purpose of some of the positions and the outputs to be delivered were not clear. Review missions need to ensure that the project management fully understands the requirements of the DMF regarding achievement of activities and outputs. The officer responsible for project design should be involved in the initial stages of project management to ensure a smooth transition from processing to implementation. ADB should monitor the PC-1 approval process for Pakistan projects so as to be aware of potential problems and ensure that appropriate levels of counterpart funds are guaranteed at the time of loan negotiations. The concept of a sustainable self-funding REMC providing BDS in markets with limited ability to pay was an inappropriate design. The services provided are public benefits, which are traditionally the role of government rather than private companies. The lack of private entities currently providing these services in Pakistan reinforces this point. It would have been appropriate if management of the project had been transferred to PRM. This would have facilitated communication, supervision, and resolution of issues as they emerged. C. Recommendations 1. Project Related 84. Future monitoring. ADB should field sufficient review missions in the early part of projects to ensure that they are implemented according to the project design. In the case of the project, the 9-month delay in the inception mission meant that corrective action to implementation arrangements could not be made at an early stage. Sufficient budget resources should be made available to ensure the necessary review missions. However, the provision of four review missions in the first year of the project was an overly ambitious target. 85. Additional assistance. In view of the low investment priority given to rural development and BDS in the current CSP, there is no compelling reason for ADB to provide further