Document of The World Bank IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA JPN-26243) ON A CREDIT IN THE AMOUNT OF US$4.

Transcription

1 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Document of The World Bank Report No: ICR IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA JPN-26243) Human Development Sector Unit East Asia and Pacific Region ON A CREDIT IN THE AMOUNT OF US$4.0 MILLION TO THE SOLOMON ISLANDS FOR A HEALTH SECTOR DEVELOPMENT PROJECT June 19, 2007

2 CURRENCY EQUIVALENTS (Exchange Rate Effective June 11, 2007) Currency Unit = SBD SBD1.00 = US$0.14 US$ 1.00 = SBD 7.19 FISCAL YEAR January 1 December 31 ABBREVIATIONS AND ACRONYMS ADB AusAID CAS DHLY DO ECA FMS GDP HMIS ICR IDA IEC LIL MHMS MTDS MUP NCB NGO NMCP NRH PCC PCIU PMR PPC RAMSI SOE TOR VBDCP WHO Asian Development Bank Australian Agency for International Development Country Assistance Strategy Discounted Healthy Life Years Development Objective Europe and Central Asia Financial Management System Gross Domestic Product Health Management Information System Implementation Completion Report International Development Association Information, Education and Communication Learning and Innovation Loan Ministry of Health and Medical Services Medium Term Development Strategy Makira Ulawa Province National Competitive Bidding Non-Government Organization National Malaria Control Program National Referral Hospital Project Coordination Committee Project Coordination and Implementation Unit Project Management Report Project Preparation Committee Regional Assistance Mission to Solomon Islands Statement of Expenditures Terms of Reference Vector Borne Disease Control Program World Health Organization Vice President: James W. Adams (EAPVP) Country Director: Nigel Roberts (EACNF) Sector Manager: Fadia M. Saadah (EASHD) Project Team Leader: Lingzhi Xu (EASHD) ICR Team Leader: Lingzhi Xu (EASHD)

3 SOLOMON ISLANDS Health Sector Development Project CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design Key Factors Affecting Implementation and Outcomes Assessment of Outcomes Assessment of Risk to Development Outcome Assessment of Bank and Borrower Performance Lessons Learned Comments on Issues Raised by Borrower/Implementing Agencies/Partners Annex 1. Project Costs and Financing Annex 2. Outputs by Component Annex 3. Economic and Financial Analysis Annex 4. Bank Lending and Implementation Support/Supervision Processes Annex 5. Beneficiary Survey Results Annex 6. Stakeholder Workshop Report and Results Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Annex 9. List of Supporting Documents MAP IBRD 33482

4 A. Basic Information Country: Solomon Islands Project Name: Health Sector Development Project ID: P L/C/TF Number(s): IDA-33130,JPN ICR Date: 06/19/2007 ICR Type: Core ICR Lending Instrument: SIL Borrower: SOLOMON ISLANDS Original Total Commitment: XDR 3.0M Disbursed Amount: XDR 2.4M Environmental Category: C Implementing Agencies: Project Coordination and Implementation Unit Cofinanciers and Other External Partners: B. Key Dates Process Date Process Original Date Revised / Actual Date(s) Concept Review: 02/01/1999 Effectiveness: 03/09/ /09/2000 Appraisal: 10/05/1999 Restructuring(s): Approval: 01/06/2000 Mid-term Review: 06/15/2004 Closing: 06/30/ /31/2006 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Risk to Development Outcome: Bank Performance: Borrower Performance: Moderately Satisfactory Moderate Moderately Satisfactory Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory Quality of Supervision: Moderately Satisfactory Implementing Agency/Agencies: Moderately Satisfactory Overall Bank Overall Borrower Moderately Satisfactory Performance: Performance: Moderately Satisfactory C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Performance (if any) Potential Problem Project No at any time (Yes/No): Quality at Entry (QEA): None Rating i

5 Problem Project at any time (Yes/No): DO rating before Closing/Inactive status: Yes Moderately Satisfactory Quality of Supervision (QSA): None D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration Health Theme Code (Primary/Secondary) Health system performance Secondary Secondary Other communicable diseases Primary Primary Population and reproductive health Primary Primary Rural services and infrastructure Secondary Secondary E. Bank Staff Positions At ICR At Approval Vice President: James W. Adams Jean-Michel Severino Country Director: Nigel Roberts Klaus Rohland Sector Manager: Fadia M. Saadah Alan Ruby Project Team Leader: Lingzhi Xu Janet I. Hohnen ICR Team Leader: ICR Primary Author: Lingzhi Xu Betty Hanan F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The Project Development Objective is to assist the Government of Solomon Islands to improve health outcomes of rural communities through strengthening existing reproductive health and malaria programs, testing new approaches to reducing these problems, and through improved planning, managing and monitoring of priority health programs. Revised Project Development Objectives (as approved by original approving authority) ii

6 (a) PDO Indicator(s) Indicator Indicator 1 : Value quantitative or Qualitative) Original Target Values (from Baseline Value approval documents) Assisted births/expected births Makira 46% Guadacanal 43% National 41% Makira 60% Guadacanal 60% National 60% Formally Revised Target Values Actual Value Achieved at Completion or Target Years Makira 60% Guadacanal 60% National 56% Date achieved 01/31/ /31/ /31/2006 Although indicators related to reproductive health by the end of the project have Comments reached the target set at mid-term, the information needs to be read with caution (incl. % as despite a great deal of improvement, there are still problems with data achievement) accuracy. Indicator 2 : First Antenatal visits/expected births Value quantitative or Qualitative) Maikira 62% Guadacanal 79% National 69% Makira 90% Guadacanal 90% National 90% Makira 90% Guadacanal 90% National 90% Date achieved 01/31/ /31/ /31/2006 Although indicators related to reproductive health by the end of the project have Comments reached the target set at mid-term, the information needs to be read with caution (incl. % as despite a great deal of improvement, there are still problems with data achievement) accuracy. Indicator 3 : Outpatients with clinically defined malaria/1000 pop. Value quantitative or Qualitative) Makira 294, Guadacanal 292, National 268 Makira 213 Guadacanal 294 National 223 Makira 213 Guadacanal 290 National 293 Date achieved 01/31/ /31/ /31/2006 Although indicators related to reproductive health by the end of the project have Comments reached the target set at mid-term, the information needs to be read with caution (incl. % as despite a great deal of improvement, there are still problems with data achievement) accuracy. (b) Intermediate Outcome Indicator(s) Indicator Indicator 1 : Value (quantitative or Qualitative) Original Target Values (from Baseline Value approval documents) Slide diagnosed malaria/1000 pop. Makira 64, Guadacanal 222, National 143 Makira <127 Guadacanal <300 National <8 Formally Revised Target Values Actual Value Achieved at Completion or Target Years Miakira 63 Guadacanal 126 National 162* iii

7 Date achieved 01/31/ /31/ /31/2006 Comments *This is 2005 data as 2006 data was not available at the time of the preparation (incl. % of the ICR. achievement) G. Ratings of Project Performance in ISRs No. Date ISR Archived DO IP Actual Disbursements (USD millions) 1 06/15/2000 Satisfactory Satisfactory /19/2000 Satisfactory Satisfactory /27/2000 Satisfactory Satisfactory /04/2001 Satisfactory Satisfactory /12/2001 Satisfactory Satisfactory /25/2002 Satisfactory Satisfactory /03/2003 Unsatisfactory Unsatisfactory /17/2003 Unsatisfactory Unsatisfactory /23/2003 Unsatisfactory Unsatisfactory /25/2004 Satisfactory Satisfactory /21/2004 Satisfactory Satisfactory /04/2005 Moderately Satisfactory Moderately Satisfactory /17/2005 Moderately Satisfactory Moderately Satisfactory /20/2006 Moderately Satisfactory Moderately Satisfactory /17/2006 Moderately Satisfactory Moderately Satisfactory 3.50 H. Restructuring (if any) Not Applicable iv

8 I. Disbursement Profile v

9 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal At the time of appraisal, the Solomon Islands (SI) ranked below most of other Pacific Island countries in overall health status. Infectious diseases, especially malaria, and maternal and infant conditions were the dominant causes of mortality and morbidity. Women were a high-risk group with the maternal mortality ratio in 1999 at 549 per 100,000 live births. This problem was compounded by rapid population growth (estimated in 1999 to be 3% per annum). The increasing population, declining skill levels of provincial staff, understaffing of health personnel in provinces, reduced supervision from Honiara, unreliable operating budgets and outdated equipment had led to reduced coverage and quality of services in the provinces. As a result, people increasingly traveled to Honiara for care. It was an expensive and inefficient solution, leaving most people inadequately serviced. In addition to the supply problems of service provision, there was low demand for services due to low levels of education, especially among women, and a lack of appreciation of the need to seek prevention or curative care. The health situation in SI was further aggravated by a fiscal crisis. While the government s stated commitment to funding of health programs was strong, funds actually provided to the sector in 1998 were 40% lower than budgeted. During the later stages of project processing, SI experienced civil unrest. Fighting broke out when the Isatabu Freedom Movement began to force Malaitans out, accusing them of taking land and jobs. Around 20,000 people abandoned their homes, with many leaving the Guadacanal province. A rival militia group, the Malaitan Eagle Force, staged a coup in 200 and forced the then Prime Minister to resign. Given the situation, it was agreed with the Government that the Bank would provide funding for the public provision of essential rural health services at a time of budgetary constraint. 1.2 Original Project Development Objectives (PDO) and Key Indicators The PDO was to assist the Government to improve health outcomes of rural communities through strengthening existing reproductive health and malaria programs, testing new approaches to reducing these problems, improving planning, managing and monitoring of these and other priority health programs. Progress in achieving the PDO was to be assessed using the following key performance indicators. supervised delivery rate proportion of pregnant mothers having at least one antenatal visit proportion of women of reproductive age accepting an effective method of family planning incidence of clinically defined malaria hospital/clinic admissions for malaria (outpatients) incidence of slide diagnosed malaria deaths from malaria (all types) incidence of slide diagnosed Plasmodium falciparum use of adequately impregnated bed nets proportion of houses sprayed in the last year

10 daily case load per health worker availability of good quality provincial health program reports to stakeholders 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification While the original PDO was maintained throughout project implementation, given problems with collection of data and analysis, the mid-term review (June 2004) agreed on a simplified set of indicators to be monitored. This set of indicators was systematically monitored and reported on by supervision missions in the ISRs. In addition, it was agreed that the percentage of births attended by health personnel and the percentage of first time antenatal visits should be calculated using as the denominator expected births, rather than actual births as an accurate number of actual births was not available. The number of births was estimated on the basis of the demographic characteristics of the population in SI and fertility rates of similar populations with reliable statistics elsewhere. 1.4 Main Beneficiaries, The primary beneficiaries were to be families in rural areas facing a high burden of disease in malaria and reproductive health, and inadequate prevention and control services. The secondary beneficiaries were to be rural health staff, whose skills and functional capability would be upgraded through the project s training programs. Other beneficiaries were to be provincial and Ministry of Health managers, who were to benefit from the capacity building initiatives, which were to improve planning, resource use, and monitoring in the sector. 1.5 Original Components The project was to be implemented in up to four provinces, which were to join in stages during the first two project years. The project s components were: (i) reproductive health, (ii) malaria prevention and control, (iii) capacity building in the Ministry of Health and Medical Services (MHMS) 1 and provincial health services, and (iv) Project Coordination and support activities for the project. Component A - Improved Reproductive Health. MHMS was to improve provincial services for reproductive health, with special attention to the training of medical and nursing staff incountry and overseas, and improved clinical supervision, and upgrading of rural health facilities. A new post-basic course for nurse-midwives was to be piloted, with oversight by a representative committee on midwifery services. International assessment and monitoring guidelines for reproductive health service quality and use were to be adapted. An inter-sectoral task force with provincial representation was to provide feedback to the Ministry on reproductive health matters. The project was to fund the work of the task force, staff training, clinical supervision and technical support, upgrading of health centers, staff housing, training facilities, and necessary equipment and supplies. 1 MHMS is referred to in the rest of the document as Ministry of Health (MOH) 2

11 Component B - Improved Malaria Prevention and Control. The component was to help reduce funding gaps in the National Malaria Control Program, while improving analysis and monitoring of factors affecting the program success, and testing ways to improve its effectiveness. The project was to support a new inter-sectoral task force and fund pilot studies to improve community response and health services. Studies were to be conducted by local professionals within an international training program. Training, supervision and supplies to improve prevention and case management in health centers were to be provided. The project was to fund incremental costs for program improvement; analysis of historical information on malaria control; design and implementation of operational studies and related staff training; and the evaluation/dissemination of the findings to policy makers and health managers. Component C - Capacity Building. The project was to increase the capacity of MOH and the participating provinces in health planning, with emphasis on improvements in: (i) the Health Management Information System (HMIS), (ii) monitoring, evaluation and research, and (iii) donor coordination. The project was also to provide support for central level strengthening and for piloting new arrangements to increase outreach and community participation. The project was to finance costs associated with in-country and overseas training, staff costs and consultants associated with the HMIS, and monitoring and evaluation. Component D - Project Coordination and Management. This component supported the operation of the Project Coordination Committee (PCC), and the Project Coordination and Implementation Unit (PCIU), within the Planning Unit of MHMS, which was to provide the project support functions, including financial and procurement management, and liaison with implementing units in MHMS and the provinces. The project was to fund the staff, operating costs and limited additional refurbishment and equipment. 1.6 Revised Components The scope of the components remained the same throughout the period of implementation, but the number of participating provinces was limited to two for reasons elaborated on below. 1.7 Other significant changes Two significant changes happened after approval. The first related to the support by the project of two provinces rather than up to four. The number of participating provinces was overtaken by events, i.e. lengthy periods of civil unrest, economic crisis and inability of the Government to pay external debt, which in turn caused the suspension of disbursements from the IDA credit for two years. The second related to the decision by the Ministry of Finance in mid-2005 not to finance under the project rural civil works planned for Makira and Guadacanal provinces because these works are planned to be financed under other projects. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry Project design was appropriate and grounded in technical, institutional and social analyses. The project addressed the key elements of the Government s Medium Term Development Strategy 3

12 through strengthening provincial services in the high priority areas of reproductive health and malaria control. The project activities allowed flexibility for MOH to adjust work plans on an annual basis if grants became available. During preparation, the national programs in reproductive health and malaria prevention and control were analyzed by technically qualified experts and were considered to be appropriate. The technical analysis recognized a high standard of technical leadership in these programs within the Ministry, supported by WHO. In line with relevant international experience, the project design included enhancements to the two programs. It included specific pilots and training to test and adapt to the local conditions. The key role of strong donor coordination was rightly identified in all key documents; the documents acknowledged that funds from the Credit would supplement and not replace grant funds. They acknowledged the need for flexibility in the allocation of resources to the provinces, recognizing that certain areas were suffering from civil unrest. Relevant Lessons from health projects in comparable countries and from the Education Project in SI were taken into account in the design of the project: (i) allowances were made for inexperience and limited institutional capacity and the relative size of the operation, (ii) donor coordination was assigned as a high priority, with provision to adjust financing arrangements as other funds became available, (iii) flexible arrangements were put in place for the timing and scope of activities, and (iv) monitoring, evaluation and dissemination of lessons learned were incorporated in the project design, and supported with technical assistance. Project was prepared following extensive consultations with a range of stakeholders. It was prepared under the overall leadership of a Project Preparation Committee (PPC) established in MOH. The PPC provided active counterparts for consultants engaged in analysis and project design. Extensive consultations with other development partners were carried out to ensure that activities to be supported by IDA would complement and not duplicate those of international partners. MHMS showed ownership and commitment to the design and scope of the project. The PAD identified well the specific risks associated with the operation, but the ratings of some of the risks did not reflect adequately the levels of the risk. For example, the risk for civil unrest in potential project areas was rated Modest when there was already ethnic unrest in the Guadacanal province; indeed the PAD noted that the planned participation of the Guadacanal province in the project has been deferred due to civil unrest. Equally, the risk rating for late or inadequate counterpart funding was rated as Modest when it should have been evident at appraisal that the fiscal crisis was highly likely to result in limited availability of counterpart funding. The overall risk rating in the PAD should have been assessed as High. The QAE is rated Moderately Satisfactory. 2.2 Implementation The project was implemented during a highly volatile and difficult period, including ethnic armed conflict and social unrest. Shortly after the Credit became effective (March 2000) and during the first 3.5 years of project implementation, the country endured widespread ethnic unrest. The unrest caused a macro-economic crisis, which in turn, made health service delivery very difficult. The severe economic crisis led to the Government defaulting on its debt service obligation to the Bank, which resulted on suspension of disbursements. For over two years, the 4

13 project faced substantial challenges to sustain implementation during the suspension, reinstatement of disbursements for only two months, and then suspension again from September 2001 to October The suspension impacted on the work programs from late 2001 through Thus, implementation of the project during its early years was hampered by the ethnic disruptions, the decline in fiscal capacity of the Government and its inability to meet its debt service obligations. Through this period, the balance and pace of activities were readjusted, shifting the focus to preserving essential services in the provinces, improving the management and cost-effectiveness of service delivery, and increasing community participation. The project was implemented directly by the MHMS under the overall guidance of the PCC. The PCC met regularly to provide policy and operational guidance. The project had only one Director, the Undersecretary of Health, who was assisted by a small Project Coordination and Implementation Unit (PCIU) comprised of local contracted staff. No international consultants were hired to guide implementation. Short-term international assistance was engaged from time to time to assist with technical matters such as to review architectural designs and help prepare international competitive bidding documents, including technical specifications and bills of quantities. Despite the relatively high turnover of staff within the Ministry and in the provinces, most activities were implemented (Annex 2). Project implementation has required close attention to policy and institution building at the central level and in the two provinces supported by the project. It has benefited from effective interaction with other major health donors, particularly AusAID and the Global Fund. The implementation period can be divided into two phases: (i) immediately after credit effectiveness (March 2000) and up to the mid-term review (mid-june 2004), characterized by slow progress of activities due to ethnic conflict and macro-economic instability, and (ii) from the MTR when implementation accelerated considerably. In the four years from Credit effectiveness to the MTR, only about 23% of credit funds were disbursed. The expenditures were mainly for training (in and outside the country), PCIU salaries, and procurement of limited equipment and supplies for malaria control. No civil works had taken place prior to the MTR. This first phase was characterized by dedication from Bank staff to provide encouragement to the Ministry during the suspension and to seek alternative financing from other donors to ensure implementation of key activities. Some missions visited during the suspension, which is not typical in such situations. However, there were no missions from November 2001 until October A virtual supervision was arranged with GSI participation through a video conference organized from the Sydney WB office in May Through strong working relationships, agreement was reached for Australia to pre-finance priority activities under the project. In addition, as part of its development assistance, Australia paid the SI s debts to the development banks (ADB and WB), opening the way to the lifting of suspension and re-engagement. Health indicators deteriorated during the conflict with high levels of IMR reported and after eight years of successful control of malaria, with progressive decline in reported cases and incidence rates, an increase in these indicators were reported in Government allocation for the National Malaria Control Program was discontinued in late Implementation of the malaria program under the project was beginning to have a positive impact until the World Bank suspended disbursements. Once suspension was lifted, the performance of the Program improved a great deal in the two project provinces. 5

14 Immediately after the lifting of suspension, a high level joint World Bank-Asian Development Bank mission took place in October 2003 to conduct brief economic and sectors assessments (including of the health sector). The joint mission concluded that the security situation had improved significantly since the establishment in July 2003 of the Regional Assistance Mission to Solomon Islands (RAMSI). The improvement boosted public confidence and reinvigorated formal business activity, particularly in the capital. After four years of contraction, the economy expanded in the first nine months of 2003 particularly primary production, construction and services. The economic recovery was attributed to a resilient private sector and rapid, substantive improvement in the law and order situation consequent upon RAMSI s arrival. RAMSI also provided a significant direct demand-side stimulus to the economy; inflation in 2003 was recorded at around 8%. The MTR aide memoire (June 2004) noted that the overall macroeconomic and security situations appeared to be on a path to sustained recovery and stability. These positive developments allowed the MTR to consider full implementation of the project, which at one point had been considered for cancellation. Agreement was reached with the government during the MTR on the need to extend the Credit closing date until December 31, 2006 (original closing date was June 30, 2005). The MTR Aide Memoire reported that the health status of the population had been negatively affected by the period of tension, the interruptions of services in Guadacanal and the sudden decline in government financing of the sector. Further, it reported that the incidence of malaria and death in Guadacanal and Makira had risen sharply since the late 1990s and increased again in 2003 and Maternal mortality and infant mortality rated had also increased since the pre-tension conditions. Residual house spraying, a key malaria control activity that had worked successfully, and bed net distribution had also declined sharply during the period. The ICR rates implementation as Moderately Satisfactory. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization The results framework included a large number of key performance indicators, for which baseline data was not available. As noted in Section 6.3, at the MTR the modality for monitoring outcome indicators was redefined and the percentage of births attended by health personnel and percentage of first time antenatal visits was to be calculated using as the denominator expected births, rather than actual births. The first mission after the MTR, in March 2005, indicated that the agreed indicators in the PAD and the DCA had not been systematically collected or monitored. Targets had not been set at project launch and had not figured in project management and supervision. Given the many difficulties with civil unrest and the disbursement suspension, this oversight was understandable. However, with the Health Information System (HIS) in operation, a table of simplified project indicators was prepared in March 2005, including outcome targets. These indicators were monitored and reported on for the remainder of the project. Although the quality of the data still needs improvement, the achievements so far of the HIS should not be underestimated. For the first time, MOH has had data to enable it to make policy decisions and allocate funds according to priorities. M&E is rated Moderately Unsatisfactory. 6

15 2.4 Safeguard and Fiduciary Compliance The project [was categorized as C with regard to the safeguard policies, and it encountered no significant safeguard issues during implementation. No adverse environmental impacts were associated with the construction and rehabilitation of the limited number of health facilities supported under the project. Regarding the Bank s fiduciary requirements, the QSA7 (for FY05/06) rated the supervision of procurement as Highly Satisfactory and noted that the project could serve as a best practice for procurement supervision. Also, recognizing the size and remote location of the SI and the US$4 million size of the project, the QSA7 stressed the exemplary performance of the project team in carrying out the procurement based on service to the Borrower, resourcefulness, and pragmatism. However, looking at the performance on procurement throughout the entire period of implementation, the ICR finds that considerable delays in project implementation were as a result of slowed procurement. Delays encountered in the procurement of civil works ultimately led to the decision by the Ministry of Finance to not approve the rural civil works in the provinces. While Financial Management complied with the Bank s requirements and FM and audit reports were presented to the Bank in a timely manner during the early stages of project implementation, turnover of FM staff resulted in delays and accuracy of the FM information. Given the overall procurement and FM performance, the ICR rates the compliance with Safeguard and Fiduciary Compliance as Moderately Satisfactory. 2.5 Post-completion Operation/Next Phase The reforms introduced under the project have been integrated on the Government s strategy for health and are likely to be carried forward under a Health SWAp, currently under preparation with support from the Bank and AusAID. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation The project s DOs continue to be highly relevant to the Bank's strategy in the Solomon Islands. In terms of design and implementation, however, the Government and its two main health development partners (AusAID and the World Bank) have agreed to work towards the adoption of a sector-wide approach for the delivery of health sector development assistance in the future. 3.2 Achievement of Project Development Objectives The project has partially achieved its DOs. Although the government has continued to make progress in improving the health outcomes of rural communities by strengthening reproductive health and malaria control programs, especially in the project provinces, considerable and sustained efforts are still required to expand the progress throughout the country. The Project assisted the efforts to reduce maternal mortality and incidences of malaria through improved care in pregnancy and childbirth, increased rates of contraceptive use, reduced hospital admissions and deaths from malaria through increased use of bed nets, and community awareness of malaria control. The lessons from the project in the areas of reproductive health, malaria and health 7

16 management information have been incorporated into national health policy as articulated in the Solomon Islands National Health Strategic Plan Provincial reproductive health have been strengthened, but not to the extent planned due to the cancellation of construction of the rural civil works. The two key indicators for reproductive health (as defined after the MTR) were achieved: (i) assisted births/expected births, and (ii) first antenatal visits/expected births. Data with respect to these indicators show that the project has been effective in improving reproductive health and enhancing capacity in the use of information for health planning and management in MOH and the participating provinces. While the project included only two provinces, the project supported improvements in reproductive health outcomes in general through its midwifery training program, which trained nurses from all over the country (62 graduated prior to project closing) and who have returned to the work force in their respective provinces. (For details of outputs under the project please refer to Annex 2). The key indicator for malaria control was achieved -- outpatients with clinically defined malaria/1000 population. Provincial malaria control services were strengthened through the implementation of a comprehensive malaria strategy initiated with support of the project and now being supported with support from the Global Fund. The strategy aims at reducing clinical malaria and preventing mortality, by providing early diagnosis and prompt treatment of all suspected or confirmed cases, and at reducing malaria morbidity through feasible and sustainable vector control interventions. The project supported extensive training for lab technicians to be able to detect malaria more accurately. Malaria staff, field workers and community leaders also participated in a great number of workshops organized under the project to enhance community participation and awareness of the disease burden. Bed nets were procured and insecticides to treat bed nets. Good progress was made in building capacity in the use of information for health planning and management in MOH and participating provinces. Through the project, MOH has been successful in building institutional capacity for planning, managing and monitoring priority health programs as demonstrated by the design of the HIS and the increased level of HIS reporting from the provinces thanks to extensive training under the project. The project s objectives for institutional building, however, were necessarily narrow and did not target overall capacity building for the Ministry. For example, the project did not target policy analysis and formulation; these areas were being supported by AusAID. Even in the areas of capacity building under the project, the achievements are still fragile as the turnover of staff is high. However, it is necessary to take into account the highly volatile and difficult project context due to the armed ethnic civil conflict. Despite improvements in the past years, sector capacity is still weak, but capacity building is a process. There are several important outcomes derived from outputs under each of the components. In terms of strengthening reproductive health: First, MOH is committed to sustaining and expanding the midwifery program and is ensuring its sustainability by financing the recurrent costs of the Program with appropriate budget allocations in MOH s 2006 and 2007 budgets. Second, the Midwifery Program has been accredited by the Solomon Islands College of Higher Education and formal diplomas are being conferred to all graduates. Third, trained midwifes are required to return to their original place of work throughout the country where they can apply 8

17 their strengthened technical skills to improve the delivery of health services. Fourth, the project achieved the attainment of the reproductive health outcome indicators in participating provinces. Although still fragile, there are other important outcomes: (i) the strengthened outreach services to villages are now more focused and regular and are beginning to have a positive impact on the indicators, (ii) the pilots conducted prior to fully rolling out the Family Health Card nationwide shed light on the reasons for the low level of indicators and have enabled MOH to prioritize activities aimed at improving performance, and (iii) improved supervision and support of firstlevel health workers is playing an important role in helping improve the RH indicator outputs. It is important to stress, however, that the data should be taken cautiously because the reliability of data collected in provinces still requires considerable improvement. In terms of Malaria Control, the project has helped to strengthen a comprehensive control program in the two participating provinces, which has become applicable nation-wide with the support of the Global Fund and other development partners (see discussion above). In terms of capacity building, the high turnover of staff has constrained the impact of capacity building efforts as several of the staff trained under the project have left MOH s services and many of them have left the country seeking better opportunities. In several cases, however, trained staff moved, but remained within the system so that the sector is still benefiting from their improved skills. Although the project has been successful in building institutional capacity for planning, management, coordination, procurement, and financial management, MOH capacity is still limited and will continue to require a lot of attention in the coming years to expand its base beyond a limited number of individuals. In addition, with the closing of the PCIU, there is the danger that the capacity for procurement, and to a certain extent, financial management, which was achieved under the project will be lost. Linkages between outputs and outcomes. A number of outputs were supported in each of the sub-components (Annex 2). With the exception of activities that were discontinued in agreement with the Bank, i.e. rural civil works in the two participating provinces, the project was successful in implementing most activities, albeit with delays mainly caused by the armed ethnic civil conflict and the macro-economic/fiscal constraints. There were strong linkages in the design between outputs and outcomes, for instance: (i) the project supported the design and the implementation of the successful in-country Midwifery Program, (ii) financed extensive overseas training for midwives and nurses, including several bachelor and master programs in obstetrics and gynecology (See Exhibit 1 of Annex 7), (iii) provided medical equipment and supplies, canoes, outboard motors, and radio communication equipment to enable provincial staff to carry out their outreach activities and satellite clinics more frequently and regularly. Achievement of Development Objectives is rated Moderately Unsatisfactory. 3.3 Efficiency No NPV, ERR, or FRR were calculated a priori for the project, and no analyses are available to assess them as economic or financial results. The PAD presented supporting evidence of comparative costs of the investment choices considered. The PAD cited a cost- effectiveness study of project expenditures in terms of discounted healthy life years (DHLY) for Component A (Improved Reproductive Health) and the cost per malaria case averted for Component B (Improved Malaria Prevention and Control). The analyses reviewed the rationale for investing in 9

18 learning and development activities to improve reproductive health and reduce malaria both during the project period and beyond. It compared the cost-effectiveness of two approaches or categories: Category I: strengthening of existing health sector operations; and Category II: learning and innovation through analysis of past program performance and piloting of new or enhanced methods for reducing the priority health problems. Under the assumption of strengthening existing activities, the PAD indicated that the project would sustain the Government s achievements for the five-year duration of the project, averting maternal and infant deaths at a cost of about US$394 per DHLY gained in comparison to the approximate costs of a year of healthy life to the country of US$930 (assuming that the value of a year of healthy life in SI is approximately equal to its per capita GDP). Averting malaria was estimated to cost about US$5.80 per case, a substantial bargain in comparison to the approximate cost to the country of US$465 per case of malaria (assuming each case of malaria costs approximately one half year of healthy life). In comparison, the benefits of the learning activities for each component while being less certain were estimated to be potentially much higher -- the cost per DHLY gained (by averting maternal and infant deaths) could be reduced by 44% and the cost of averting a malaria case could be reduced by 10% to 20%. The analysis also showed that if benefits accrued under the project were retained after project completion, they would improve the efficiency and effectiveness and lower the costs of existing reproductive and malaria services. Since activities are to be continued after the Credit closing, the benefits derived are expected to continue, thus improving the efficiency and cost-effectiveness of these services. 3.4 Justification of Overall Outcome Rating Rating: The rating of Moderately Satisfactory is justified on the following basis: Throughout the implementation period, the objectives remained highly relevant to issues facing the sector in SI, particularly to the reproductive health and malaria sub-sectors. The results of the key performance (output) indicators show that the project contributed to strengthening reproductive health and that interventions for malaria control helped to strengthen a comprehensive control program in the two participating provinces. However, the strengthening of the services could have been greater if the rural civil works were undertaken under the project. As planned, the project tested new approaches to reduce reproductive health and malaria problems, and helped to strengthen planning, managing and monitoring of priority health programs more broadly. Targets on reproductive health and malaria control are potentially problematic, therefore, it is difficult to totally attribute or assess the precise impact of the project on the overall improvement in these areas. However, the results of the three outcome indicators in project areas are better than those of the country as a whole or to other comparator (non-project) provinces at the time of project completion. The civil unrest and macro-economic difficulties, and the consequent suspension of the Credit, delayed project implementation, which explains why the project assisted fewer provinces than originally foreseen, and also delayed the start of important rural civil works planned for Makira and Guadacanal. The rating takes into account the highly volatile and difficult project context due to the armed ethnic civil conflict from 1999 to 2004, the two-year suspension of disbursements, the fact that this was the first Bank-supported health project in the SI, and the administrative challenges of keeping implementation moving under extremely difficult 10

19 conditions. As noted by QAG, the Bank deserves credit for the hard work put in by the supervision Task Team and for sustaining the relatively high costs associated with supervising a small project in a small country. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development The operation had a positive effect on community involvement in reproductive health. Both participating provinces achieved the reproductive health targets prior to Credit closing. Steady gains in antenatal care and assisted births are evident, while contraceptive use approached the target. Improved supervision and support of first-level workers have been attained through use of supervision checklists, radio consultations, and supplemental resources for staff visits. The provinces achieved improved outreach by health workers through use of the family health card, following a review and assessment of the card pilot in the two provinces. Data gathered from the health card exercises shed light on the low level of some reproductive health indicators, the cultural and behavior factors at work, and how the indicators and behaviors could continue to be improved. The ICR team visited some formerly conflict-ridden areas (in Guadacanal) where the Project interventions have played an important role in catalyzing community cooperation for improved knowledge of reproductive health and use of family planning tools. Survey data, collected as part of a province-wide survey on the reasons for choice of home vs. clinic births combined with an awareness-raising tour by reproductive health nurses, indicated more positive attitudes toward assisted births and better family planning practices. (b) Institutional Change/Strengthening) Implementation has yielded lessons and information that have been incorporated into the National Health Strategic Plan for Training activities supported through the project have been instrumental not only in strengthening the performance of reproductive health programs, but in raising the morale of health staff in areas heavily affected by the conflict. Training has been conducted in and outside the country for midwives and nurses, and in integrated management of childhood illnesses. The HMIS system has provided lessons regarding the use of data and other information for planning and policymaking. Provincial-level planning and supervision systems have been strengthened as a result of participating in data collection for the HMIS. The high turnover of staff has constrained the expected impact of capacity-building efforts under the project as several of the staff trained left the services of MOH or Provincial Health Directorates. However, in several cases, the staff moved within the system or have continued to work elsewhere in the country, which therefore continues to benefit from their improved skills. The project has been successful in building institutional capacity for planning, management, coordination, procurement, and financial management. These improvements are being scaled up to help improve GOS s administrative system as a whole and the health sector in particular. 11

20 Project experiences and materials have been shared with other development partners, including during a stakeholder workshop carried out as part of the ICR mission, but they should be disseminated more widely. (c) Other Unintended Outcomes and Impacts (positive or negative) The project has contributed to helping to standardize architectural designs for rural health facilities, which can be easily amended to suit other parts of the country, depending on terrain and climate. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops Together with MOH, the ICR team organized a Stakeholders Workshop in Honiara with the participation of the Ministry of Planning; Ministry of Health; Vector Borne Disease Control Program; School of Nursing; Midwifery School; Provincial Health Directors and staff; and International partners - UNDP, UNICEF, AusAID, JICA, World Bank, and the British High Commission. The objective of the workshop was for the central and provincial authorities to present the results of the project and lessons learned from its implementation. The Bank team encouraged an open discussion and recommended participants to measure as far as feasible the outcomes and impact of project interventions, rather than concentrating only on outputs. Presentations were made for each project component sub-component, focusing on results and remaining challenges. For more details on the workshop, please refer to Annex Assessment of Risk to Development Outcome Rating: Limited. At the present time, there appears to be limited risk to sustaining the DOs for the project, given that lessons from strategies and approaches developed and tested under the project have been drawn for the development of the National Health Strategic Plans for , which are to be supported by financing through, among others, the World Bank and AusAID. However, given the fragile institutional capacity, the risk should not be underestimated. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory. Project design was appropriate and grounded in sound technical, institutional and social analyses. The project design addressed the key elements of the Government s MTDS in health. The design was modified during preparation to streamline the components, scope and strategy to facilitate provincial participation and to enable adjustments of funding in response to the plans of other donors. Thus the project activities, as approved, were expected to be influenced by the evolving activities of other donors, allowing flexibility to adjust work plans on an annual basis if and when donor grant funding became available. The national programs in reproductive health and malaria prevention and control supported by the project were analyzed by technically qualified experts and were considered to be appropriate. The Bank team promoted ownership and commitment from the outset and worked hand in hand with counterparts to ensure that foreign and local technical assistance worked closely with staff. 12

21 However, the risk assessment did not recognize the extend of the risks given that ethnic unrest had already started at the time of appraisal. (b) Quality of Supervision Rating: Moderately Satisfactory. The Bank team maintained its focus on development impact. Because of delays in implementation caused mainly by ethnic unrest, during the MTR the team agreed on a simplified set of output indicators (see relevant sections). Supervision of fiduciary tasks was carried out in a timely manner. Supervision missions could have been more regular during the last years of project implementation. Ratings of DOs and IP were generally fair, but inadequate explanations were given in ISRs to justify these ratings. There are two clear phases of project implementation/supervision. The first phase was between Credit effectiveness and the MTR (March 2000-June 2004). The second was post-mtr until December 2006 when the Credit closed. During the first period, the Bank team was proactive and resourceful, continually engaging the government and keeping the PCIU staff motivated by ensuring continuity of implementation guidance during the difficult and long periods of suspension. The Bank worked closely with the PCIU to readjust annual work programs on the basis of limitations and alternative sources of funding. The Bank was successful in catalyzing other development partners financing to pre-finance priority activities that could be later reimbursed under the Credit after the suspension was lifted. Despite the suspension financing continued for a limited number of activities during the suspension period, such as in-country and overseas training and the salaries of the PCIU staff. Without these efforts, the project could easily have collapsed as practically no counterpart funds were available for project implementation during the severe fiscal crisis. Because of the ban imposed by the UN on travel to SI, the Bank team was not able to visit the country for a period of 2 years. In May 2002 a virtual supervision was organized with the Government team visiting the Bank s office in Sydney to conduct the supervision through video with HQ. The project had only 3 TLs during its turbulent implementation period. Staff continuity was a good feature of the Bank s supervision support. However, only nine (9) supervision missions took place during the 7-year implementation period; 15 PSRs were filed in SAP; several of them reflected desk reviews of progress reports and telephone discussions with the PCIU. As noted earlier, the project was implemented during a highly volatile and difficult period, characterized by ethnic armed conflict. No missions were possible from November 2001 until October The project team ensured continuity by engaging a senior Bank consultant based in New Zealand to maintain regular contact through telephone with counterparts and helped to identify issues and provide advice on project implementation. However, more such virtual supervisions could have been organized. The quality of the supervision (as per Aide Memoires/ISRs) was mixed. During the early stages of implementation, efforts were devoted to improving the understanding of the project by counterparts and to clarify procedures and practices. The MTR Aide Memoire was comprehensive; it argued well for the importance of intensifying the Bank s efforts to bring the project back on track. During the later period of implementation, however, the documentation could have been more robust. For example, only 4 ISRs were completed, basically one a year, although there were more than 4 site visits. Although a region practice, in the view of the QAG 13