IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA & IDA-47210) CREDITS IN THE AMOUNT OF SDR 3.9 MILLION (US$ 6.0 MILLION EQUIVALENT) TO THE

Transcription

1 Public Disclosure Authorized Document of The World Bank Report No: ICR Public Disclosure Authorized Public Disclosure Authorized IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA & IDA-47210) ON CREDITS IN THE AMOUNT OF SDR 3.9 MILLION (US$ 6.0 MILLION EQUIVALENT) TO THE INDEPENDENT STATE OF SAMOA FOR A HEALTH SECTOR MANAGEMENT PROGRAM SUPPORT PROJECT AND ADDITIONAL FINANCING Public Disclosure Authorized Human Development Sector Unit Health, Nutrition, and Population East Asia & Pacific Region June 14, 2016

2 CURRENCY EQUIVALENTS (Exchange Rate Effective December 18, 2015) Currency Unit = Samoan Tala SAT$ 1.00 = US$ 0.39 US$ 1.00 = SAT$ 2.58 FISCAL YEAR (July 1-June 30) ABBREVIATIONS AND ACRONYMS Senior Global Practice Director: Practice Manager: Project Team Leader: ICR Team Leader: Timothy Evans Toomas Palu Eileen Brainne Sullivan Neesha Harnam

3

4 INDEPENDENT STATE OF SAMOA Health Sector Management Program Support Project CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design Key Factors Affecting Implementation and Outcomes Assessment of Outcomes Assessment of Risk to Development Outcome Assessment of Bank and Borrower Performance Lessons Learned Comments on Issues Raised by Borrower/Implementing Agencies/Partners Annex 1. Project Costs and Financing Annex 2. Detailed Project Components and Outputs by HSP Outcome Annex 3. Economic and Financial Analysis Annex 4. Bank Lending and Implementation Support/Supervision Processes Annex 5. Beneficiary Survey Results Annex 6. Stakeholder Workshop Report and Results Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Annex 9. List of Supporting Documents Annex 10. List of Indicators and Arrangements for Results Monitoring (PAD) Annex 11. Summary of Agreements Annex 12. SWAp Governance and Administration MAP IBRD 33472

5 A. Basic Information Country: Samoa Project Name: SAMOA HEALTH SECTOR MANAGEMENT PROGRAM SUPPORT PROJECT Project ID: P L/C/TF Number(s): IDA-44320,IDA ICR Date: 6/14/2016 ICR Type: Core ICR Lending Instrument: SIL Borrower: Original Total Commitment: SDR 1.90M Revised Amount: SDR 3.90M Environmental Category: B Disbursed Amount: GOVERNMENT OF SAMOA SDR 3.90M (Includes Additional Financing of SDR 2.00M) Implementing Agencies: Ministry of Health (MOH) and National Health Service (NHS) Cofinanciers and Other External Partners: Ministry of Finance and Trade (MFAT; New Zealand) and Department of Foreign Affairs and Trade (DFAT; Australia), UNFPA, UNICEF and WHO. Pool partners were those providing financing (MFAT, DFAT, IDA, GOS); together with UNFPA, UNICEF and WHO they are referred to as development partners (DPs) in the Joint Partnership Agreement (JPA). B. Key Dates Process Date Process Original Date Revised / Actual Date(s) Concept Review: 05/22/2007 Effectiveness: 10/21/ /21/2008 Appraisal: 04/09/2008 Restructuring(s): 12/03/2013 Approval: 06/19/2008 Mid-term Review: 04/06/ /14/2011 Closing: 12/31/ /18/2015 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Risk to Development Outcome: Bank Performance: Borrower Performance: Moderately Unsatisfactory Moderate Moderately Unsatisfactory Moderately Unsatisfactory i

6 C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Unsatisfactory Government: Moderately Satisfactory Quality of Supervision: Moderately Implementing Moderately Unsatisfactory Agency/Agencies: Unsatisfactory Overall Bank Performance: Moderately Unsatisfactory Overall Borrower Performance: Moderately Unsatisfactory C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Performance (if any) Potential Problem Project at any time (Yes/No): Problem Project at any time (Yes/No): DO rating before Closing/Inactive status: No Yes Moderately Unsatisfactory Quality at Entry (QEA): Quality of Supervision (QSA): None None Rating D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration Health Theme Code (as % of total Bank financing) Child health Health system performance Injuries and non-communicable diseases Other communicable diseases Population and reproductive health E. Bank Staff Positions At ICR At Approval Vice President: Keith Hansen James Adams Country Director: Franz Drees-Gross Nigel Roberts Practice Manager/Manager: Toomas Palu ii Fadia M. Saadah Project Team Leader: Eileen Brainne Sullivan Muhammad Ali Pate ICR Team Leader: Neesha Harnam

7 ICR Primary Author: Neesha Harnam F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) This ICR uses the PDO stated in Schedule 1 of the Financing Agreement, which reads: The objective of the Project is to improve the Recipient's effectiveness in managing and implementing the Program based on the use of results from sector performance monitoring. This is identical to the first-tier objective of the Project Appraisal Document (PAD); the PAD also includes the objective of the Health Sector Plan (HSP) as a secondtier objective, namely The HSP s aim in the medium-term (FY ) is to improve access to, and utilization of effective, efficient and quality health services to improve the health status of the Samoan population. Revised Project Development Objectives (as approved by original approving authority) No revisions were made to the PDOs. (a) PDO Indicator(s) Indicator Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years KPI #1 (SWAp): Percent health sector budgets and disbursements conform to policy objectives and HSP priority areas Value quantitative or Beginning of SWAp 90% >90% Qualitative) Date achieved 10/21/2008 Year 4 onward 12/18/2015 Comments (incl. % achievement) ACHIEVED. The SWAp aimed to introduce a mechanism for the government to better align its resources with priorities. This indicator was tracked using the proportion of budget expenditure by the various program activities. KPI #2 (SWAp): Share of annual outpatient visits by poorest quintile of population (indicator of equity of access HIES) Achieved according Value to proxy data on quantitative or N/A No target set vaccinations and Qualitative) antenatal care Date achieved 10/21/ /18/2015 iii

8 Indicator Comments (incl. % achievement) Baseline Value Original Target Values (from approval documents) iv Formally Revised Target Values Actual Value Achieved at Completion or Target Years ACHIEVED. While targets were not set, evidence suggests that improvements were seen in terms of equity of access. The March 2009 summit was supposed to establish the baseline and target for the rest of the program, but due to a lack of data this was not done. However, proxy data on vaccinations and antenatal care show improvements based on DHS data by quintile. The share of population receiving all basic vaccinations generally improved from 2009 to 2014 from 20.0 to 48.7 (lowest); 15.1 to 51.7 (second); 29.3 to 40.5 (middle); 31.2 to 47.1 (fourth); 29.5 to 45.3 (highest). Similarly, the percentage of women receiving antenatal care from a health care provider from 2009 to 2014 generally improved from 86.5 to 89.1 (lowest); 91.6 to 91.8 (second); 95.1 to 92.3 (middle); 93.9 to 96.7 (fourth); 99.1 to 98.2 (highest). KPI #3 (HSP): Prevalence of diabetes Value quantitative or 21.5% No target set 45.8% Qualitative) Date achieved 2002 (STEPS Survey) 2013 (STEPS Survey) NOT ACHIEVED against HSP target value of 25-50% decrease. Figures Comments reported represent prevalence among adults years old. While it is possible (incl. % that increased access (see KPI #2) may be responsible for the increase in achievement) prevalence this is likely only a partial explanation for the observed increase. KPI #4 (HSP): Perinatal mortality rate (22 weeks gestation to <7 days after birth per 1000 live births/year WHO definition) Value quantitative or Qualitative) 7 per 1000 live births No target set 8 per 1000 live births Date achieved (DHS) (DHS) PARTIALLY ACHIEVED. DHS 2014 definition is 28 weeks gestation to <7 days after birth per 1000 live births/year, and not the WHO definition. The baseline figure may have been artificially low, as a survey of all public health Comments facilities in 2008 place the perinatal mortality rate at 23.9 per 1000 live births (in (incl. % line with data obtained from the census), which suggests improvement over time. achievement) IMR reduced from 20.4/1000 (2006 census) to 15.4/1000 (2011 census); U5MR improved from 24.4/1000 to 19.4/1000 (2011 census). Maternal mortality also declined based on DHS data from 40.2/100,000 (2009) to 26/100,000 (2014). KPI #5 (HSP): Adolescent birth rate (Annual births to women yrs) Value quantitative or Qualitative) No target set Date achieved (Census) (Census) Comments (incl. % achievement) NOT ACHIEVED against the HSP target for 2018 of 30. Defined as annual births to women years old.

9 Indicator Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years KPI #6 (HSP): Incidence of water- and food borne infections Value quantitative or 670 No target set 135 Qualitative) Date achieved 2009 (PATIS; Cases of clinically suspected typhoid) 2014 (PATIS; Cases of clinically suspected typhoid) Comments (incl. % achievement) PARTIALLY ACHIEVED. While there was some concern that typhoid cases reported in 2014 may not reflect the total number of cases due to timing of surveillance reports, a steady and substantial decline has been observed over the years of the Project. It is also worth noting that not much improvement was seen among children with diarrhea: 68% (DHS 2009) to 62% (DHS 2014) while the proportion of untreated children with diarrhea increased from 3% (DHS 2009) to 7% (DHS 2014). KPI #7 (HSP): Injuries in children (under 15 years old) Value 237 quantitative or No target set (population 70,489) Qualitative) 2007/2008; Date achieved population data is from 2006 census Comments (incl. % achievement) 282 (population 71,890) 2012/2013; population data is from 2011 census NOT ACHIEVED. Numerator reflects number of hospital admissions reported for children under 13. Denominator reflects population of children under 15. Slight increase seen on a percentage basis. 2010/2011 data had 267 admissions. KPI #8 (HSP/ICR Indicator): Percentage who currently smoke tobacco Value quantitative or Qualitative) 40.4% No target set 27.1% Date achieved STEPS STEPS (2002) (2013) Comments (incl. % achievement) PARTIALLY ACHIEVED against the HSP target for 2018 of 20.2%. While the target has not fully been achieved, tobacco control progress to date has been notable. KPI #9 (HSP/ICR Indicator): Percentage of those who have drunk (alcohol) in the past 12 months Value quantitative or Qualitative) 29.3% No target set 16.9% Date achieved STEPS (2002) STEPS (2013) Comments (incl. % ACHIEVED/EXCEEDED against the HSP target for 2018 of 26.5%. achievement) KPI #10 (HSP/ICR Indicator): Percentage of people with high levels of physical activity v

10 Indicator Value quantitative or Qualitative) Date achieved Comments (incl. % achievement) Baseline Value Original Target Values (from approval documents) vi Formally Revised Target Values Actual Value Achieved at Completion or Target Years 32.6% No target set 61.1% STEPS STEPS (2002) (2013) ACHIEVED. A high level of physical activity is defined as >3000 MET minutes/week. While specific targets were not set, available evidence from the STEPS survey suggests an improvement over time. (b) Intermediate Outcome Indicator(s) Indicator Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years IOI #1 (SWAp): MTEF and related Procurement Plan updated and adjusted based on recommendations from sector reviews. Value quantitative or No MTEF Complete Complete Qualitative) Date achieved /18/ /18/2015 Comments (incl. % achievement) PARTIALLY ACHIEVED. MTEF 2009 and 2012 covered three-year periods, and the MTEF and related procurement plan for is currently under preparation. However, the MTEF has not been annually updated since IOI #2 (SWAp): Key Sector Partners corporate Plans and Government investments aligned with HSP priorities Value quantitative or Qualitative) Not aligned Complete Complete; alignment seen Date achieved 2008 Year 2 onward 12/18/2015 Comments (incl. % achievement) ACHIEVED. Alignment efforts are undertaken through the Health Advisory Committee which meets regularly. In addition, the MOH has instituted a combined plan that incorporates all investments in the sector. IOI #3 (SWAp): Percentage of SWAp Program funds released according to agreed schedule Value quantitative or 0% 100% 100% Qualitative) Date achieved 9/25/2008 Year 2 onward 12/18/2015 Comments (incl. % achievement) ACHIEVED. Funds were released based on commitments by the government, though there were some delays in disbursement during the course of the Project. It is worth noting that the SWAp incurred budget overruns, which were covered by DPs and Government towards the end of the FY IOI #4 (SWAp): Number of reported drug stock-outs by facility Value Unknown <10% of baseline 2%

11 Indicator Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years quantitative or Qualitative) Date achieved 2008 Year 3 onward March 2012 ACHIEVED as 2% is negligible. While there is still no data on stockouts at the Comments national level, data from March 2012 notes a 2% stockout rate in rural areas and (incl. % 95% availability of essential medicines at all public health facilities at any one achievement) time. IOI #5 (SWAp): Health expenditure as percentage of government budget expenditure Value quantitative or 15% >15% 16.1% Qualitative) Date achieved 9/25/ /18/ /30/2015 Comments (incl. % achievement) ACHIEVED. IOI #6 (SWAp): Stakeholders participation in program planning and implementation reviews Value quantitative or Qualitative) None Full Full and regular consultation Date achieved 9/25/2008 Year 3 onward 10/30/2015 ACHIEVED. Stakeholder participation was evident in Annual Summits and Comments Joint Reviews as documented in an Analysis of Review Missions (Annex 7 of (incl. % the Evaluation of Samoa Health Sector Management Program conducted in achievement) 2013). IOI #7 (SWAp): Disaggregation of data by sex, age and domicile enhances planning for services Surveys Value disaggregated quantitative or None Full (STEPS 2013, Qualitative) DHS 2014) Date achieved 9/25/2008 Year 3 onward 10/30/2015 Comments (incl. % achievement) PARTIALLY ACHIEVED. STEPS 2013 and DHS 2014 surveys were disaggregated, though it is unclear that this has enhanced planning. IOI #8 (SWAp): MOH Financial Audits submitted on time and action plan agreed for resolving outstanding issues Value quantitative or Unknown Complete Complete Qualitative) Date achieved 2008 Year 1 onward 10/30/2015 Comments (incl. % achievement) ACHIEVED. vii

12 Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years IOI #9 (SWAp): DHS and other statistical reports completed within stated timeframe and made public Value quantitative or Qualitative) Yes with delays, available in public domain Complete Yes and with less delays than baseline and made available in public domain Date achieved 9/25/2008 Year 1 onward 10/30/2015 Comments (incl. % achievement) ACHIEVED. IOI #10 (HSP): People aged 18 years and over overweight or obese Value Overweight: 85.62% Overweight: 89.1% quantitative or No target set Obese: 56.0% Obese: 63.1% Qualitative) Date achieved (STEPS) (STEPS) Comments (incl. % achievement) NOT ACHIEVED against HSP target of: Overweight: 63.9%; Obese: 42% (25% reduction from baseline). In addition, the share of the population meeting the recommended intake for fruit and vegetable consumption (5+ servings/day) decreased from 8% of males and 3% of females (DHS 2009) to 2% and 1% (DHS 2014), respectively. IOI #11 (HSP/Core): Percentage of children under 1 year received at least one dose of measles vaccine Value quantitative or Qualitative) 63.00% No target set Comparable data not available Date achieved 2009 (DHS) ACHIEVED against HSP target for children 0-29 months of 90%. This is similar to a core indicator on children immunized. The Health Sector Plan Midyear Comments (incl. % achievement) Review notes that about 85% of children under one-year-old have received immunization against measles in More recent comparable data was not available due to changes in the national vaccine program. However, annual immunization rates from the UNICEF/WHO joint reporting form suggests improvement over time from 49% (2008/09) to 91% (2013/14). IOI #12 (HSP): Improved medical waste management Value New HCWM Plan not yet quantitative or implemented Qualitative) No target set Improved Date achieved /18/2015 viii

13 Indicator Comments (incl. % achievement) Baseline Value Original Target Values (from approval documents) ix Formally Revised Target Values Actual Value Achieved at Completion or Target Years PARTIALLY ACHIEVED. While a HCWM Plan was in place in 2008, a revised Plan was signed in June Investments in HCWM infrastructure (incinerators, medical waste bins) and training (waste management and the importance of HCWM) were made, though training participation was weak. Regular supervision missions took place to ensure compliance with the HCWM Plan, though it appeared that compliance and funding was limited. IOI #13 (HSP): Primary care utilization by gender, age, domicile Value No information quantitative or No information available. No target set available Qualitative) Date achieved /18/2015 Comments (incl. % achievement) UNKNOWN. IOI #14 (HSP/Core): Antenatal care coverage for three visits Value 58.40% 72.90% quantitative or No target set (4+ visits) (4+ visits) Qualitative) Date achieved (DHS) (DHS) ACHIEVED/EXCEEDED against HSP target of 70% (4+ visits). Indicator in the Comments (incl. % achievement) RF originally read antenatal care coverage at least one visit, same as one of the core indicators. This was revised to three visits in later ISRs, but the DHS reports 4+ visits which is reported here. Median age of visit was also earlier from 6 months to 5.9 months (2009 to 2014 DHS). IOI #15 (HSP): Proportion of Rheumatic Heart Disease patients complying with treatment Value quantitative or 84% No target set 80% Qualitative) Date achieved ACHIEVED. While no target was set, evidence suggests improved compliance Comments (incl. % achievement) that was sustained over time. Data from 2006 and earlier suggests compliance was below 50%. Screening efforts began in 2008, and the proportion of RHD patients complying with treatment has been maintained at >80% since then, reaching a peak of 86% in IOI #16 (HSP): Staff mix and distribution according to national standards The NHS issued a Workforce Development Plan Value (April 2013) which quantitative or N/A No target set identifies the needs Qualitative) and gaps to be filled to meet quality standards. Date achieved /18/2015

14 Indicator Baseline Value Original Target Values (from approval documents) x Formally Revised Target Values Actual Value Achieved at Completion or Target Years PARTIALLY ACHIEVED. The national standards have been articulated in the Comments Workforce Development Plan. Based on the number of annual practicing (incl. % certificates issued, there has been an increase in the health professional to achievement) population ratios since 2008, particularly for nurses though the ratio for doctors has declined slightly. IOI #17 (HSP): Evidence of performance monitoring leading to policy and regulatory action to improve health services Value quantitative or Qualitative) N/A No target set Many frameworks and policies developed but limited evidence of improvement in health services. Date achieved PARTIALLY ACHIEVED. The Monitoring and Evaluation Framework was developed in 2011, and all health professions have developed standards (2013/2014). A key achievement was the development of an interim HRH database focused on identifying staff gaps by facility and qualification; however, this database is not yet available for use at NHS. Additionally, the Tobacco Comments Control Policy 2010 enforced implementation of and public compliance of the (incl. % Tobacco Control Act While there were General Prevention and Health achievement) Promotion Policies developed focusing on Health Information Systems (HIS) development and policy, these were not fully exploited to inform policy and develop regulatory action. In terms of performance monitoring, limited data on health outcomes (including NCDs), was collected and reported outside of national surveys, limiting the evidence base available in decision-making. IOI #18 (HSP): Demonstrated outcomes of training plan by component Value quantitative or Qualitative) N/A No target set Outcomes demonstrated in Component 2 (HIS and coding capacity) and Component 3 (DHS, M&E, HRH) Date achieved PARTIALLY ACHIEVED. DHS trainings have been instrumental in taking the Comments (incl. % achievement) DHS from initial stages to report writing. Trainees of the M&E study tour developed the M&E Framework. Health information management training has equipped staff to improve HIS. HRH-related trainings resulted in the HRH country profile and establishment of an HRH taskforce. IOI #19 (Core): Health personnel receiving training (Number) Value quantitative or Qualitative) 0 No target set 2125 (MOH Accounting Documents) Date achieved 10/21/ Dec 2015

15 Indicator Baseline Value Original Target Values (from approval documents) xi Formally Revised Target Values Actual Value Achieved at Completion or Target Years A sizeable number of health personnel received training. Training included attachments in urology and orthopedics, and in health management. This allowed Comments (incl. % achievement) successful utilization of equipment procured under the SWAp, improvement of documents produced under MOH (Operational Manual, review of protocols and standards), and the development of leadership skills. Training included graphic design (2 people), improving their ability to produce and design multimedia materials for the health sector. IOI #20 (Core): Health facilities constructed, renovated, and/or equipped (Number) Value quantitative or 0 No target set 13 Qualitative) Date achieved 5/31/ /18/2015 Comments (incl. % achievement) IOI #21 (Core): People with access to a basic package of health, nutrition, or population services (Number) Value quantitative or No target set Qualitative) Date achieved Comments (incl. % Dropped from ISR #7 (July 22, 2012) onward. achievement) IOI #22 (Core): Children receiving a dose of vitamin A (Number) Value quantitative or No target set Qualitative) Date achieved Comments (incl. % Dropped from ISR #7 (July 22, 2012) onward. achievement) G. Ratings of Project Performance in ISRs No. Date ISR Archived DO IP Actual Disbursements (USD millions) 1 10/28/2008 Satisfactory Satisfactory /11/2009 Satisfactory Moderately Satisfactory /07/2010 Satisfactory Moderately Satisfactory /12/2010 Satisfactory Moderately Satisfactory /20/2011 Satisfactory Moderately Satisfactory /03/2012 Moderately Satisfactory Moderately Satisfactory 2.17

16 No. Actual Date ISR DO IP Disbursements Archived (USD millions) 7 07/22/2012 Moderately Satisfactory Moderately Satisfactory /15/2013 Moderately Unsatisfactory Moderately Satisfactory /23/2013 Moderately Unsatisfactory Moderately Satisfactory /19/2013 Moderately Satisfactory Moderately Satisfactory /21/2014 Moderately Satisfactory Moderately Satisfactory /17/2014 Moderately Satisfactory Moderately Satisfactory /06/2015 Moderately Satisfactory Moderately Satisfactory /29/2015 Moderately Satisfactory Moderately Satisfactory /14/2015 Moderately Unsatisfactory Moderately Satisfactory 5.87 H. Restructuring (if any) Restructuring Date(s) Board Approved PDO Change ISR Ratings at Restructuring DO 05/20/2010 S MS /02/2013 MS MS 2.91 IP Amount Disbursed at Reason for Restructuring & Key Restructuring Changes Made in USD millions Additional financing (AF) for an amount of US$3.0 million equivalent was granted for posttsunami activities within the health sector and reflected in the Program of Work (see Annex 2). No other changes were made. Level 2 restructuring. No cost extension of the Project closing date by two years to December 18, Amendments to the Financing Agreements are dated December 3, 2013 xii

17 I. Disbursement Profile xiii

18 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. At the time of appraisal, Samoa s economic reforms, launched in 1993, had resulted in macroeconomic stabilization and comprehensive structural changes with real Growth Domestic Product (GDP) growth of over 3 percent realized in 2007 and steadily rising per capita incomes. However, the fiscal space of the Government and its ability to increase expenditures were constrained, with total revenues and grants expected to decline over the period to from 37.7 percent to 36.6 percent of GDP. 2. Over the past several decades the health status of Samoa s population saw significant improvement, largely through the provision of basic health services. The country had high emigration rates among the population of working age, resulting in a relatively high dependency ratio mitigated by high remittance levels back to the country. For a number of years, Samoa had struggled with the classic epidemiological transition. The rising burden of NCDs, found increasingly among younger adults, posed serious risks to long term growth prospects. Although most infectious diseases were under reasonable control, acute respiratory infections and rheumatic fever remained significant causes of morbidity and, in the latter case, resulted in a high and growing demand on hospital services. Infectious diseases (with outbreaks of measles and dengue seen in recent years) and maternity-related illnesses accounted for over 30 percent of hospital admissions in 2003/2004, yet did not receive the appropriate level of funding. While Government of Samoa (GOS) policy favored cost-effective preventive services, the bulk of public resources were still directed towards less cost-effective hospital services. In early 2007, the Government developed a new Health Sector Plan (HSP) focused on dealing with: (i) rising non-communicable diseases; (ii) reproductive, maternal and child health; (iii) emerging and re-emerging infectious diseases; and (iv) prevention of injuries, given the significant share of the health burden associated with injury and poisoning among children. In addressing these issues, the Health Sector Plan ( ) identified six priority areas and strategies: (i) health promotion and primordial prevention; (ii) quality health service delivery; (iii) governance, human resources for health and health systems; (iv) partnership commitment; (v) health financing; and (vi) donor assistance. 3. As part of an overall major public sector governance reform program, the GOS began implementation of a purchaser-provider split within the health sector with the passing of the Ministry of Health (MOH) Act 2006 and the National Health Service (NHS) Act The NHS Act that gave it the mandate for service delivery also created a separate Executive Board to oversee NHS operations aside from the MOH 1. Under these arrangements: (i) the MOH was responsible for regulatory oversight of the health sector, including operational budgets of State health entities and human resources, monitoring of health system performance as well as primordial health promotion and prevention services; and (ii) NHS was the major Government agency responsible for the provision of public health care services, including secondary prevention, primary health care (PHC) and hospital services, though the Ministry of Finance (MOF) retained direct control over the budget of NHS. 1 The CEO of MOH sits on the board of the NHS. 1

19 4. Based on the HSP and in accordance with the shift towards a sector-wide approach (SWAp) across the whole of Government, the GOS requested key development partners (the New Zealand Agency for International Development (NZAID; later MFAT), the Australian Agency for International Development (AusAID; later DFAT) and the World Bank) to establish a health SWAp Program as a basis for assistance to implement the first five-year period of the HSP (FY ). Further, the GOS was promoting a more coordinated approach as a means for increasing the effectiveness of development assistance. This approach is particularly important given the small island state context where there is limited capacity in an environment vulnerable to external shocks and natural disasters. A summary of the SWAp governance and administration is provided in Annex 12. The GOS had prior experiences with the Education Sector Program II (established in 2006) and the Water Sector Support Program ( ), both of which had SWAp aspects. 5. There was strong rationale for the World Bank s involvement in the HSP. First, the Bank had previous experience in working in Samoa, having provided 15 credits (US$87.8 million) between 1974 and 2008 for support to various sectors, including health. The preceding World Bank Project (the Samoa Health Sector Project (P064926)) closed at the end of 2006 and had been rated Moderately Satisfactory. Second, the Bank s global and regional expertise in design and implementation of sectoral programs could be adapted to the Samoan context. Third, by leveraging IDA s support together with other DPs (NZAID and AusAID), the proposed Program represented a new way of engagement with the health sector in the country and of building institutional capacity. Fourth, the Bank engagement in Samoa at both macro as well as sector levels provided an avenue for linking health sector development efforts within a whole-ofgovernment context. Finally, the Bank s non-lending knowledge and analytical products effectively complemented the Program context. 1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) 6. This ICR focuses on the PDO stated in Schedule 1 of the Financing Agreement (FA) dated July 23, 2008, which reads: The objective of the Project is to improve the Recipient's effectiveness in managing and implementing the Program based on the use of results from sector performance monitoring (FA, p. 5). It is worth noting that in the PAD, a two-tiered 2 system of development objectives was employed, the first of which corresponds with that of the FA. The second-tier PDO reflects the objective of the HSP, 3 and states that The HSP s aim in the medium-term (FY ) is to improve access to, and utilization of effective, efficient and quality health services to improve the health status of the Samoan population (PAD, p. 5). The Joint Partnership Agreement (JPA) identifies two objectives (institutional and development objectives) which correspond to the first and second-tier PDOs of the PAD. 2 A pre-review of the Project was conducted by the Independent Evaluation Group on 8/24/2015, who distinguished between the first- (Health Sector Program) and second-tier (SWAp Process) PDOs. The same tiered approach is used in discussed the PDOs here, though the tiers described in the ICR refer to the PDOs as listed in the PAD. 3 While the rationale for a SWAp was explicitly expressed in the HSP, the areas of focus in the HSP were different than the PDO. The HSP notes that the SWAp for quality health for Samoa will focus on: (i) genuine and effective partnerships; (ii) donor participation, coordination and contribution; (iii) consistent collaborative planning, monitoring and evaluation arrangements; (iv) human resource development for health (HRH) and skills transfer; and (v) a Samoan perspective of health. 2

20 7. A complete list of indicators (KPIs and IOIs) used throughout the Project is presented in the ICR Data Sheet and summarized here. Key indicators for assessment of the SWAp (Part B of the results framework in the PAD) are as follows: (i) Percent health sector budgets and disbursements conform to policy objectives and HSP priority areas; and (ii) Share of annual outpatient visits by poorest quintile of population (indicator of equity of access HIES). Areas of focus and specific KPIs included for assessment of the HSP (Part A of the results framework in the PAD) were: (i) Control of non-communicable diseases (Prevalence of diabetes); (ii) Improved maternal and child health (Perinatal mortality rate); (iii) Universal access to reproductive health services (Adolescent birth rate); (iv) Control of communicable diseases (Incidence of water- and food borne infections); (v) Injury prevention and management (injuries in children under 15 years). The complete results framework as approved and presented in the PAD is provided in Annex Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 8. The PDOs were not revised. 1.4 Main Beneficiaries 9. The PAD does not specify a primary target group, though the economic evaluation notes that a main thrust of the Program is to support institutional development for better policy planning, the benefits of which are spread over a range of beneficiaries and therefore difficult to quantify. The second-tier PDO listed in the PAD specifies that the target population of the project is the Samoan population. Given the link between the first- and second-tiers, the beneficiaries were considered to be the Samoan population. 1.5 Original Components 10. A detailed description of the three components is available in Annex 2 and is outlined here. The Project components listed in the PAD were more comprehensive than those listed in Schedule 1 of the Original FA. Nonetheless, these differences were not found to be substantive. A summary of the Project Components is as follows: Component 1 (Health Promotion and Prevention) aimed to support the transformation of the health sector from a narrower medical orientation towards a health and wellness model. Central to MOH s efforts to reorient the sector was health promotion. Component 2 (Enhancement of Quality Health Care Service Delivery) aimed to support improvement of the various dimensions of the quality of health care, particularly at the primary care level. Component 3 (Strengthening Policy, Monitoring and Regulatory Oversight of the Health System) aimed to support the MOH with its role of coordination, policy development, monitoring and regulation in the health sector. More specifically, this component aimed to support implementation of the Monitoring and Evaluation (M&E) system to monitor the Program and SWAp performance and the incremental operating costs for Program implementation. 1.6 Revised Components 3

21 11. Project components were not revised though, as part of the SWAp redevelopment 4, a further outcome area was added in 2011, namely improved risk management and response to disasters, emergencies and climate change. While no formal amendments were made to the HSP and the Project was not restructured to reflect this outcome area, its addition 5 is reflected in subsequent plans and programs of work. 1.7 Other significant changes 12. A summary of Agreements is provided in Annex 11. Amendments to the Financing Agreements, Cr WS and Cr WS (December 3, 2013), extended the closing date of the project by two years at no cost. The Partnership Agreements were also amended to extend the end date of the partnership. In addition, following the tsunami that occurred in 2009, the Project allocated SWAp resources (obtained from an additional US$3.0 million equivalent of World Bank funding) to post-tsunami activities within the health sector. Additional funds were provided through the Additional Financing Agreement (May 20, 2010) and reflected in the change in the Program of Work (POW). In particular, the additional proportion of costs 6 (from 43 percent in to 67 percent in ) allocated to Component 2 partly reflects the incorporation of post-tsunami activities. These activities are described in the February 2010 Aide Memoire, and listed in Annex 2 of this ICR. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 13. Soundness of Background Analysis: Project preparation was supported by a grant of $500,000 from the Japan International Cooperation Agency (TF90307) which was largely spent on consultant services. The background analysis conducted was strong in that it provided a good overview of the health issues in Samoa using available evidence on the burden of disease, but there was room for improvement. The Project would have benefited from background analysis that also included an inventory of tools and policies in place and from using this information to refine the Project indicators. For example, IOI #12 of improved waste management system lacked a target describing what constitutes improvement. More broadly, the achievement of the PDO was predicated on the assumption that the necessary tools and policies and an adequate monitoring and evaluation system were in place, which proved not to be the case as the Project progressed. Thus, while the background analysis correctly identified the health issues in Samoa, more in-depth analysis would have been beneficial where the PDO and related indicators were concerned. Finally, in terms of approach, alternatives such as traditional project support and Development Policy Lending were considered prior to deciding on the sector-wide approach. 14. Assessment of Project Design: It is important to note that the Project was not a traditional SWAp, as MFAT served as Coordinating Development Partner for most of the SWAp implementation period, though this role was later rotated to DFAT. The Project was aligned with the HSP in approach (SWAp) and scope (overlap in components). The development of HSP had 4 Further details regarding the redevelopment are available in Section It should be noted that the addition of an outcome area focused on disasters, emergencies and climate change represented an innovative and proactive approach. 6 Evaluation of Samoa Health Sector Management Program (Health SWAp). Philip Davies, AusAID HRF. May 21,

22 been a participatory process with the DPs providing input. The three components of the Project broadly overlapped with the first three strategies of the HSP, and the SWAp could be said to align with the last three strategies 7. Though Project design followed good practices in SWAp design at the time, the design did not adequately address the implications of policy changes such as the MOH-NHS split, or the link between the results chain and activities as detailed below. 15. A key challenge of the Project was with regard to timing. The concept note review for the SWAp took place in 2007 with consultations taking place beforehand, shortly after the split between MOH and NHS. Project design incorporated several important lessons learned, including (i) the importance of government ownership to the success of SWAps; and (ii) the need for higher than average supervision. However, government ownership within the health sector initially proved to be difficult given the MOH-NHS split, which, while mentioned in the context of reforms within the health sector (PAD, p. 23), was not explicitly addressed in terms of implementation arrangements. In particular, while the implementation arrangements delineate responsibilities of the two agencies, their ability to successfully carry out their duties in the context of the split was not discussed. Closer or more appropriate supervision (tailored to needs as they arose) may have been able to mitigate these issues to some extent, particularly if the skill mix required for supervision had been considered among DPs (see Section 5.1(b)). 16. While the PAD included a results chain linking components to outcomes (PAD, Annex 9, Figure 1), it appeared to assume that the requisite programs, policies and systems were in place at the outset, which turned out not to be the case. Project design could have benefited from an improved results chain, as the link between HSP objectives and strategies, and the components and constituent activities were unclear as noted by an evaluation of the SWAp conducted in It appears that the absence of a clear results chain hampered achievement of the Project going forward, as discussed under Section Assessment of Quality at Entry: The PAD incorporated several lessons learned from two prior projects in Samoa as well as lessons from other SWAps in the region. A lesson learned from the Samoa Health Sector Management Project (P064926) that would have been useful to include was that of clear and measurable project development objectives, including the use of appropriate indicators and early establishment of baselines. SWAp indicators were process indicators, and only a subset of outcome indicators were regularly reported on as part of the Results Monitoring Arrangements presented in the PAD as shown in Annex 10. Monitoring and Evaluation (M&E) design was weak and the results chain could have been strengthened. While the team involved in Project preparation proactively held a Quality Enhancement Review early in the Project cycle (March 5, 2008), documents available at the time did not provide the Review Panel with sufficient information in some areas. For example, a Results Framework was absent. The recommendations of the Panel appear to have been taken into consideration to some extent (e.g. the number of Health Summits was reduced to one instead of two as originally proposed), other issues highlighted by 7 HSP six priority areas and strategies are: (i) health promotion and primordial prevention; (ii) quality health service delivery; (iii) governance, human resources for health and health systems; (iv) partnership commitment; (v) health financing; and (vi) donor assistance. 8 Evaluation of Samoa Health Sector Management Program (Health SWAp). Philip Davies, AusAID HRF. May 21,

23 the Reviewers appear to have been only partially addressed. For example, the Panel noted the need to establish a clear link between the health system/status in Samoa and interventions proposed in the HSP, and cautioned against possible procurement issues, recommending against being too conservative. Largely due to these shortcomings, Bank performance in ensuring quality at entry was rated Moderately Unsatisfactory (Section 5.1). 18. The Project represented the first health SWAp in the country, and overall risk was considered Moderate. There was high-level government commitment to the Project, and the SWAp modality was seen as a natural next step for the country given its success in Project implementation and the presence of multiple development partners. In general, the assessment of risks and mitigation measures were appropriate. Where the split is concerned, the PAD notes that, conflict between health sector entities over evolving responsibilities and accountabilities within the sector and a consequent inability for sector entities to maintain an open and participatory sector dialogue (p. 13) was a high risk, and identified suitable mitigation measures. In hindsight, these mitigation measures may have been inadequate as understanding of responsibilities and accountabilities of the MOH and NHS were still being developed after the split, as well as the dual role of MOH as implementing agency of two components and coordinator of DP inputs to the sector. 2.2 Implementation 19. Implementation progress was initially slow (only about 40 percent of planned funding had been expended by the end of September 2011 or three years after effectiveness), and the Project faced challenges on several fronts. Several major events occurred over the course of the Project, namely an H1N1 epidemic (2009), a tsunami (2009), and Cyclone Evan (2012), presenting a difficult environment for implementation and absorbing capacity of the health sector. The flexibility of the SWAp allowed it to quickly provide additional funds (US$3.0 million equivalent in IDA funds) in response to the tsunami. While ISR #7 (June 2012) noted that the GOS was likely to achieve the PDOs if implementation continued at the pace observed in the past six months, PDO achievement was downgraded to MU a few months later (January 2013) due to delays under Component 2 and disbursement, and a lack of progress on Project indicators. Over-programming of the POW and a resulting shortfall in funds was also seen at that time. The POW was subsequently revised and the Project received a two-year no cost extension for completion of remaining infrastructure works. Progress was seen in the completion of non-infrastructure activities by October 2013, resulting in a rating change to MS (October 2013). However, further challenges (procurement, monitoring and evaluation, overall project management) resulted in a downgrade of PDO rating back to MU prior to Project closing (ISR #15; November 2015). Key factors affecting implementation are described below: 20. Program of Work: Development of the POW has been criticized in prior reviews of the SWAp for having been a bottom-up instead of a top-down process. In this context, it is important to note that the bottom-up approach taken may have been successful if accompanied by a clear process of prioritization. The POW was developed based on solicitation of proposals for activities to be completed within the SWAp envelope, with activities falling broadly within the six HSP outcome areas 9 considered eligible for financing. Activities were then prioritized, with component 9 HSP outcome areas are: (i) Improved Healthy Living through Health Promotion and Primordial Prevention; (ii) Improved Prevention, Control, and Management of Chronic and NCDs; (iii) Improved Prevention, Control, and 6

24 focal points involved in the prioritization process. Many activities unable to be funded in one year were rolled over to the next. The POW thus evolved into an increasingly large and varied set of activities over the years. In general, evidence of systematic prioritization and selection of activities (or specific mix of activities) emphasizing use of data from sector performance monitoring was not clearly seen. While the Medium-Term Expenditure Framework (MTEF) was conducted and did provide some evidence regarding resource allocation, it was not regularly updated. It was also unclear as to why some activities were selected and others were not, and why, when overruns occurred, some activities were dropped while others were retained. In some cases, people were informed that an activity would be funded through the SWAp only to be told later that the activity was being dropped. Similarly, there was limited evidence of meaningful policy dialogue between DPs and GOS. Given these challenges, management of the SWAp went from an initial focus on overall strategy and achievement of outcomes, to a focus on outputs and activities. The risk that the SWAp would deliver a wide array of inputs and outputs to the sector without the desired impact on health outcomes was noted by March 2012 (Aide Memoire). 21. A significant and unexpected event was the redevelopment of the Tupua Tamasese Meaole (TTM) Hospital into a teaching hospital supported under the Government of the People s Republic of China assistance program (2009/2010). This resulted in additional delays, as efforts were made to ensure that works were not duplicated with those conducted under the SWAp. Further, despite the stated emphasis of the HSP on promotion, prevention and primary care services, there appears to have been a particular disconnect between resource allocation and these priority areas, in part due to the need to develop the facilities at the new TTM Hospital. 22. Limited capacity, high turnover and weak communication: The SWAp was challenged by limited capacity, which had implications in many aspects of the Project. First, the limited capacity of the MOH meant that it was a challenge to address and remedy weaknesses that arose (ISR #9, June 2013). The strong emphasis on implementation of SWAp activities and the short timeframe in which activities needed to be carried out (with the exception of the first POW, subsequent POWs were on an annual basis) meant that there was limited capacity remaining for M&E, which was handled by a single person. Individuals were also required to perform activities beyond their area of expertise. For example, part of the reason for cost overruns was the costing of infrastructure activities by technical specialists in health instead of those with a more appropriate background e.g. engineering. Poor communication further complicated these issues, as unsynchronized inputs were provided to the architect from the different agencies and as personnel involved changed. The ICR mission found that on occasion, architects involved were not clear of the budgetary envelope when designing a building. Further, while the frequency of the Health Summit and Joint Review meetings was defined and the need for additional supervision was noted in the PAD, implementation was challenged by frequent turnover both on the part of the GOS and DPs. An independent evaluation of the SWAp which reviewed participation in missions described the monitoring process by DPs as hampered by revolving door staffing (Phillip Davies, May 2013). Management of Communicable Diseases and Infectious Diseases; (iv) Improved Sexual and Reproductive Health; (v) Improved Maternal/Child Health; (vi) Improved Health Systems, Governance and Administration. A seventh area was later added; namely (vii) Improved Risk Management and Response to Disasters, Emergencies and Climate Change. 7

25 23. Program redevelopment in place of mid-term review: The midterm review was recast as a program redevelopment, 10 which may have limited the opportunity to assess shortcomings of the SWAp in a candid manner and to course-correct. The redevelopment led to simplification in some processes (e.g. procurement) and provided additional clarification surrounding roles and responsibilities. In addition, the redevelopment resulted in the revision of the format of the POW, with activities grouped under the seven HSP Outcome areas which, in turn, could then be mapped onto the three SWAp components. While a mid-term review of the HSP was conducted, this only took place in An independent evaluation of the SWAp was also conducted in 2013, and it appears that recommendations made, such as to focus on specific priority projects (Philip Davies, p. 34) were taken into consideration, at least based on activities completed following the review. 2.3 Monitoring and Evaluation Design, Implementation and Utilization 24. Challenges in Project implementation were closely linked to weaknesses in monitoring and evaluation. Monitoring and evaluation of the Project faced challenges on two fronts: (i) development of M&E and HIS systems; and (ii) assessment of key indicators in the results framework over the course of the Project. These are further described below. 25. Design: The Results Framework of the PAD consisted of 25 indicators in total (Part A and Part B), with Part A consisting of a subset of overall HSP results indicators and Part B consisting of SWAp process indicators. Only a subset of 11 indicators (largely SWAp process indicators) were monitored as part of the results agreement, which may have resulted in a weaker emphasis on achievement of health outcomes than was desirable. An evaluation of the SWAp conducted in May 2013 noted that some indicators used were poorly defined, lack targets or benchmarks and/or are likely to be difficult to collect and interpret (Philip Davies). Further, while the use of results from sector performance monitoring was emphasized in the PDO, health outcomes indicators that were part of the results framework were those collected from large and infrequent surveys. Finally, even among the HSP indicators with targets, some targets may have been out of reach, such as a percent reduction in diabetes prevalence. Underpinning these challenges was the lack of a robust HIS or M&E system that collected reliable data on a regular basis though support to the implementation of an M&E system was foreseen in the Project design under Component Several aspects of M&E specified in the PAD lacked clarity. In particular, the PAD notes that sector M&E reports would be discussed during the Health Summits, the Joint Review meetings and with MOF on a quarterly basis as part of the government s budget monitoring process (p. 45). However, the PAD does not specify what constitutes a sector M&E report, and the ICR mission did not find evidence of discussions driven by sector M&E reports. Similarly, the PAD notes that the SWAp would develop a mechanism to encourage the use of the information collected in dayto-day decision-making within the different agencies and stakeholders, but does not provide further details on this proposed mechanism. 27. Implementation: Technical assistance for HIS fell through in the initial stages of the SWAp, and had knock-on adverse effects on data collection and analysis going forward. The 10 Neither the GoS Mid-Term Review nor the pool partners Mid-term Evaluation identified in the JPA took place. However, a Mid-Term Review of the HSP was conducted in 2013 by an external consultant (Joan McFarlane). 8

26 general need to improve M&E efforts was echoed over the course of the SWAp through multiple reports and documents (a total of four consultancy reports were developed in 2013/2014 focusing on HIS). An M&E manual focusing on Health Sector Performance was developed by the MOH in 2009, but did not provide baselines or targets for any of the indicators mentioned while the M&E framework only became available in Annex 1 of the M&E manual lists indicators used to monitor Health Sector Performance and makes it apparent that there was a need for improvement or development of institutional systems in order to assess the indicators. By September 2009, only three of the 25 indicators (Part A and Part B) had baseline data available. While it is not unusual for baseline data to be missing at the beginning of an operation, the lack of baseline data, targets, and definitions continued over the course of the Project. These reflected broader M&E weaknesses that occurred despite efforts such as the establishment of a Health Sector Information and Communication Technology (ICT) Taskforce in 2010 and a separate taskforce established through a cabinet directive in The final Aide Memoire of the Project (November 2015) notes that the SWAp should have focused more intently on HIS in the early years, including staff development in this area. M&E also presented challenges outside the GOS. The review conducted by DFAT and the Health Resource Facility (HRF) noted weaknesses in record-keeping among development partners, resulting in multiple, inconsistent and sometimes contradictory data being recorded. 28. Utilization: In the absence of a robust M&E system, utilization of data in decision-making does not appear to have routinely taken place. While results from large-scale surveys provided vital information on the health status of the country, such data was only available every few years, with a lack of data on sector performance over the course of the SWAp. Instead, it appears that the use of multiple forms and registries (some of which may have been outdated) continued over the course of the SWAp, with weak feedback loops and limited dissemination of data. An emphasis on achievement of results in the form of improved health outcomes, and a clear link between plans, expenditures and results was largely lacking. An assessment of M&E commissioned by the World Bank noted that the Health Sector Plan results framework was urgently in need of critical review to ensure logical program management links and progression from inputs, to outputs to outcomes to impact (on health status) (Nick Davis, 2013). While there were many useful outputs that were aligned with the priority areas of the HSP and that were driven by the SWAp modality, there is limited evidence that they led to improved outcomes, or to the achievement of the desired impact (Section 3.2). In retrospect, it may have been prudent to focus on fewer, regularly measureable indicators, and to emphasize the achievements of those outcomes over the course of the SWAp. 2.4 Safeguard and Fiduciary Compliance 29. Safeguards: At Project closing, the Overall Safeguard rating was Moderately Unsatisfactory. The only safeguard triggered by the Project was the Environmental Assessment (OP/BP/GP 4.01) as investments made through the SWAp were anticipated to increase health care waste. The implementation of a Health Care Waste Management (HCWM) Plan was included as a covenant in the Financing Agreement and as an intermediate outcome indicator on improved medical waste management in the Results Framework. A revised HCWM Plan was signed in In addition to IDA Safeguards Policies, the MOH undertook its own Environmental Impact Assessment which was submitted only after the SWAp extension was granted in 2013, and required revisions as it did not originally include a social impact assessment, focusing on physical and biological impacts only. This assessment focused on the impact of the three buildings at TTM Hospital, namely the PHC building, the Orthotic/Prosthetic workshop, and the Pharmaceutical 9

27 Warehouse. Regular supervision from the HCWM was provided over the course of the SWAp, but several issues remained unresolved by the end of the SWAp. First, there were insufficient attention to funding recurrent costs and maintenance to provide the needed inputs to improve HCWM. Funds were also not allocated for the replacement of incinerators due to depreciation. Second, segregation bins were found to be of insufficient quantity and quality while plastic bags were unavailable in sites visited. In general, staff continue to need further training surrounding the concept of proper waste segregation. While certain districts have implemented the recommendations surrounding HCWM, the ICR mission found that HCWM bins were accessible to the general population, unsecured, and mixed in terms of content, though the MOH has expressed commitment to resolving outstanding issues. 30. Fiduciary: The POW was frequently over-programmed and resulted in cost overruns despite activities being dropped. An Aide Memoire from November 2012 noted that several items had detailed designs and costings exceeding SWAp funds by US$9.4 million. This may have been reflective of a lack of knowledge surrounding the technical specifications necessary for an activity, resulting in an underestimate of costs (ISR #8, January 2013). At the same time, funds were not being disbursed as seen by high levels of unutilized funds. The 2013/2014 audit identified the purchase of faulty fixed assets using donor funds, though this was remedied by the MOH under the warranty for equipment. At the instigation of the DPs, a maintenance plan was developed for the sector, given the high investment in medical equipment and infrastructure made using SWAp funds. The Financial Management Implementation Review Report of November 2015 notes that it is unlikely that remaining works will be completed by June 2016, and while completion by September 2016 is likely, a more realistic expectation is a completion by December Financial covenants were complied with, and the Financial Management was considered Moderately Satisfactory at the time of Project closing. 31. Procurement: Procurement activities consumed the bulk of time in the early years of the SWAp, and encountered difficulties over the course of the Project. The possibility of procurement delays seriously undermining the SWAp was noted in the PAD (p. 9). The World Bank s procurement procedures were used, and were consistent with government procedures. Despite the consistency, there was a perception that the use of two sets of procedures (GoS Tenders Board and WB), while preferable over the use of multiple donor-specific guidelines, resulted in delays for reviews and approvals, largely due to limited procurement capacity. Component heads were responsible for development of technical procurement documents, but lacked the capacity and experience to perform this function well. This weakness was seen in the inaccuracy of submitted proposals with regard to specifications and related cost estimates; professional review of specifications and solicitation of bids revealed that actual costs were substantially higher. As a result, activities meant to be funded under the SWAp had to be reduced, though it remains unclear as to how activities were subsequently dropped from the priority list. 32. As major infrastructure items (PHC Center, Malietoa Tanumafili II (MTII) Hospital renovations and Pharmaceutical Warehouse, and the Orthotics and Prosthetics Workshop) under the SWAp were coordinated by MOH and utilized by NHS, consultations surrounding design of the building included inputs from both agencies. Despite these consultations, there were reports of unsynchronized inputs from the two agencies, which subsequently resulted in delays and cost overruns. While major infrastructure items were approved as part of the POW of the first eighteen 10

28 months of the Project, at the time of Project closing (which included a two-year, no-cost extension) only the Orthotics and Prosthetics Workshop had been completed though the contract for the renovation (but not supervision of renovations) of the PHC Center had been signed. Procurement efforts are no longer being handled by an accredited specialist following the departure of the MOH Procurement Specialist in late 2014, though DFAT has been providing assistance with support from their Infrastructure Specialist. Renovation of the MTII Hospital Radiology Department remains incomplete and while the ICR mission was informed that tasks are on track and scheduled for completion by June 2016, it remains uncertain if completion by that date is possible. Further, there continues to be some concern regarding the quality of the infrastructure (see Annex 3). 2.5 Post-completion Operation/Next Phase 33. The World Bank exited the SWAp on December 18, 2015 and no follow-up operations in the health sector are planned at this time beyond technical assistance. The other DPs will continue to provide support to the MOH, and to oversee progress of the remaining infrastructure program which is currently planned for completion by June 2016 with the end warranty period in June Efforts such as the Health Program Advisory Committee (HPAC) 11 are likely to continue going forward, but further fragmentation of the health sector (from two to three agencies) is a risk given plans for the establishment of the Health Promotion Foundation. Nonetheless, this risk may not materialize if a strong emphasis is placed on impact assessment and cost-effective interventions. The absence of robust monitoring and evaluation systems continues to remain a challenge, though WHO, ADB, and DFAT have expressed their commitment to provide support in the development of HIS. 34. Where infrastructure is concerned, the NHS has developed a Maintenance Plan but it is uncertain as to whether sufficient resources will be provided to maintenance efforts, particularly where preventive maintenance is concerned. From 2009/2010 to 2013/2014, approximately 0.24 percent (or SAT$500,000) of spending on infrastructure and equipment was spent on maintenance, half of which came from DPs (Philip Davies, 2013). While the 2014/2015 NHS budget allocation provides ~1 percent of its budget or SAT$746,200 for maintenance, a greater amount will need to be spent in this area going forward. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 35. Objectives: The SWAp focused on the Health Sector Program, which is aligned with the priorities and strategies articulated in the Health Sector Plan The current Country Assistance Strategy (CAS, ), which represents the first individual strategy for Samoa, emphasizes the importance of improving resilience going forward. The CAS notes that vital to efforts to improve resilience is strengthened planning services and improved public expenditure management, which are in line with a SWAp. Finally, the Government of Samoa s Strategy for the Development of Samoa (SDS) emphasizes strengthening economic resilience and encouraging inclusive growth through increasing investment in the productive sectors of the economy, and commits to continued investment in the social sectors, including in health as part of efforts to achieve a healthy Samoa. 11 Formerly known as the Health Program Steering Committee. 11

29 36. Design: The SWAp represented a key milestone for the GOS where the health sector was concerned. At the same time, other elements that may have contributed to the successful achievement of the Project were missing, such as an in-depth assessment of M&E capacity, and a clear, evidence-based, results chain. The absence of regularly updated epidemiological evidence made the identification of priority areas and successful interventions challenging, and consequently, the selection of activities under the POW may not have been as strategic and focused on the highest-impact interventions possible. Further, it is important to note that the use of an annually updated POW may not have been the most appropriate instrument for monitoring the sector and focusing on key policy issues as it lacked a holistic perspective and did not form part of the MTEF on a rolling basis as noted in the PAD (p. 44). 37. Implementation: The flexibility in Project design proved to be a strength when the country faced challenges such as the tsunami (2009), for which additional financing was provided. The SWAp has also been successful in getting different stakeholders across the health sector around the table as part of the implementation process, which is particularly important given the limited capacity of the MOH and NHS and the relatively heavy reliance on international development cooperation flows as seen in many small island states. While there was a shift in priorities that occurred following the redevelopment of the TTM Hospital, efforts continued to be made on health promotion and primordial prevention, which are important areas considering the burden of disease in the country. 38. In view of the information provided above, relevance is rated High for Objectives and Substantial for Design and Implementation. 3.2 Achievement of Project Development Objectives 39. The PDO was to improve the Recipient's effectiveness in managing and implementing the Program based on the use of results from sector performance monitoring. The complete list of indicators is provided in the Data Sheet while a list of activities and outputs completed under the project is provided in Annex 2; a partial description of indicators and activities is provided here to assess Project achievements against its PDO. 40. Achievement of the first sub-objective focused on improved effectiveness of management was considered Substantial largely based on Part B of the Results Framework (SWAp indicators). From the IOIs, it was inferred that aspects of management that the SWAp sought to improve was efficiency in (i) resource allocation (IOI #1 and IOI #2), use of human and physical resources; and (ii) governance (IOI #3-IOI #5), which were found to have demonstrated Modest and Substantial improvements over the course of the SWAp. (i) With regard to efficiency in resource allocation, the project improved alignment between health sector finance and HSP priority areas to over 90 percent (KPI #1). Alignment with HSP priorities was also seen in Key Sector Partners corporate plans and government investments (IOI #2). HSP priority areas were those listed in the HSP, and were relatively broad. While health expenditure as a share of government expenditure increased over the course of the SWAp (IOI #5), it is unclear as to whether there was improved efficiency in government spending. A Public Expenditure Review (PER) focused on notes that 80 percent of the increase in domestic resources in that period have gone toward increased personnel 12

30 spending. In particular, it emphasizes an increase in overtime payments and nursing staff was seen, though it appears the increase was largely among more experienced nurses despite the HSP noting gaps in lower levels (such as auxiliary nurses). It also notes increased funds for primary care and treatment services but a decline in resources for prevention. The PER further notes that increased spending on noncommunicable diseases (NCDs) is pushing out spending in other areas with greater benefit, namely primordial prevention. Implementation of policies and plans according to agreed priorities was seen through the development of an MTEF (IOI #1), but annual updates have not been performed since Also unclear is the extent to which funds were allocated in response to new information, if any, beyond that of large surveys or studies. However, responsiveness was seen with regard to natural disasters such as the tsunami and Cyclone Evan, where funds were provided to aid in the response thus ensuring minimal impact in terms of health consequences. (ii) Greater efficiency in use of human and physical resources was measured over the course of the SWAp based on the share of SWAp Program funds released according to agreed schedule (IOI #3) and the decline in reported drug stock-outs by facility (IOI #4). Some delays in disbursement of SWAp Program funds were observed, as well as cost overruns (see Section 3.3). While drug stock-outs do not appear to be a problem, stock-outs of reagents for laboratory testing appear to be a challenge. Despite these shortcomings, evidence (described below) suggest that overall, resources are being used more efficiently. In terms of human resources, the NHS completed a Human Resources for Health (HRH) Workforce Development Plan, highlighting gaps and needs for the coming years (e.g. mental health specialists, pathology, radiology), with some effort already made to address observed gaps (IOI #16). Through the SWAp, relevant staff were provided with opportunities to develop and retool their skills to support the country s needs (see Annex 2), which allowed some tasks to be completed in-house at the MOH and NHS instead of being outsourced (e.g. graphic design). However, costs continue to be incurred due to a lack of General Practitioners (GPs) in the public sector, requiring contracting of private sector GPs to meet the needs of the public sector programs. Further, the PER notes from , the health sector has seen increases in average salaries (by 26 percent) and in employment (by 30 percent), raising questions about sustainability and affordability. Finally, there continues to be gaps in human resources for health in outlying areas. In terms of physical resources, the SWAp financed the purchase of biomedical equipment for the National Hospital, 12 which was accompanied by a maintenance strategy developed by NHS and training of an in-country Biomedical Specialist, aimed at reducing the cost of overseas treatment, particularly for diagnosis. However, some weaknesses have been observed where the infrastructure investments of the SWAp are concerned, suggesting room for improvement where efficiency of physical resources is concerned (see Section 3.3). Going forward, commitment to improved efficiency has been demonstrated through the National Hospital in the reduced allocation for overseas treatment from 11 million Tala (2014/2015) to 4 million Tala (2.5 million Tala original allocation and 1.5 million Tala supplementary allocation) 13 (2015/2016). 41. Improvements in governance was similarly seen in many areas: 12 DFAT financed USD$3 million dollars outside the SWAp for additional biomedical equipment. 13 The total budget allocation for the Health Sector in 2015/2016 is SAT$ 79.3 million. 13

31 (i) First, the SWAp improved multisectoral engagement where the achievement of health outcomes was concerned. Oversight of the SWAp was provided by the HPAC, which was chaired by the MOF, and included the private sector, nongovernmental organizations (NGOs), and development partners. The SWAp allowed for cross-sectoral collaboration which has been strengthened over the years with improved working relationships with the Ministries of Education (on nutrition and NCDs), Women, Social and Community Development (various aspects of gender-health and gender-financing), Agriculture (on improving nutrition and use of local healthy produce to promote wellbeing), Commerce and Trade (taxation on tobacco, sugary drinks), and the private sector (to promote health). In addition to the HPAC, the project saw more ownership from the Parliament through the Samoa Parliamentary Advocacy Group for Healthy Living (SPAGHL). (ii) Where health sector governance was concerned, the relationship between MOH and NHS has improved over the years with an increased awareness of the role each should play. There has been improvement in health sector management (IOI #8), though the NHS has been hampered by inadequate tools and inability to manage certain functions which remain under MOH decision making or management. The completion of the Demographic and Health Survey (DHS) (IOI #9) in 2009 and 2014 are major achievements made under the Project. In addition, the PER notes that while in 2006 almost the entire healthcare budget was classified as tertiary spending, this has changed over time as the government shifts to a community-level, primarycare based model though tertiary expenditure has been maintained in real terms and continues to constitute the largest component of health expenditure. A major weakness of the SWAp was the inability to develop an HIS system, further contributing to a lack of data where consistent sector performance monitoring, and relatedly, planning, are concerned. An evaluation in 2013 notes no discernible use of data in planning or policy cycles (Mark Spohr). These weaknesses have been described in Section 2.3. It is worth noting that there is some evidence that results of large studies (NHA, DHS, STEPS Noncommunicable Disease Risk Factor survey) were used in resource allocation and decision-making. For example, the STEPS 2013 survey highlighted the lack of progress on NCDs, thus providing an impetus for the implementation of WHO s Package of Essential NCD interventions (PEN) which is currently being rolled out. Prior to that, the DHS 2009 highlighted the need to improve reproductive health, after which UNFPA was brought in to provide assistance. Recent increases in the birth rate (KPI #5) have also resulted in the involvement of MOF in the population agenda. While compliance among schools in areas such as nutrition and diet (which has shown improvement from 2012 to 2015) appear to have been closely monitored through the Annual School Monitoring Reports, evidence of the direct impact of these policies on the health of schoolchildren is limited. In general, monitoring efforts were irregular, and regular sector performance monitoring does not appear to have taken place over the course of the SWAp. 42. The second sub-objective focused on effectiveness of implementation was considered Modest. This reflected (i) how well the HSP was implemented based on the results of sector performance monitoring; and (ii) how well the HSP performed, which is particularly important given the second-tier PDO (as reflected in the PAD) and as reflected by Part A of the Results Framework (HSP indicators). Attribution to the SWAp where achievements of the HSP is concerned is challenging, as the Project did not exceed more than 14 percent of the GOS budget 14

32 over the period it was implemented, and a substantial investment was made outside the SWAp in the redevelopment of the TTM Hospital. Nonetheless, two key aspects are highlighted here: (i) Translation of plans and policies into action: The SWAp enabled the creation and the institutionalization of the Policy Unit, which in turn developed a number of important policies and strategies including the National Food and Nutrition Policy, National Child and Adolescent Health Policy, NCD Policy, the Health Promotion Act 2014 and Food Act While the management-related outputs over the course of the SWAp were large and varied, there is limited evidence surrounding their utilization, particularly with regard to improvement in health outcomes. Assessment of management training funded under the SWAp revealed that 67 percent of MOH/NHS staff were not aware of policies in place, highlighting a clear need to improve dissemination and knowledge of these policies so that they can inform implementation of health service delivery. An exception was where tobacco-related policy was concerned, which included a clear results chain and resulted in improved outcomes (see KPI #8, IOI #17). (ii) Achievements of HSP by priority area: Achievements of the HSP by priority area (described as four crucial areas of health challenges and based on the Health Sector Situational Analysis conducted in May 2006) was mixed, with worsening outcomes particularly noticeable for NCDs. While the HSP lacked targets for many indicators, it is unlikely that worsening outcomes were desired. Achievements of the HSP based on a selected number of indicators are summarized below (Table 1). Table 1. Achievements of the HSP HSP Area of Focus Status Control of NCDs KPI#3 Prevalence of diabetes Not achieved IOI#10 People aged 18 years and over overweight or obese Not achieved KPI#10 Percentage of people with high levels of physical activity Achieved Improved MCH KPI#4 Perinatal mortality rate Partially achieved IOI#11 Percentage of children under 1 year who received at least one dose of measles vaccine Achieved IOI#15 Proportion of rheumatic heart disease patients complying with treatment Achieved Universal access to reproductive health services KPI#5 Adolescent birth rate Not achieved IOI#14 Antenatal care coverage for at least one visit Achieved/Exceeded Control of communicable diseases KPI#6 Incidence of water and foodborne infections Partially achieved Injury prevention and management KPI#7 Injuries in children (under 15 years old) Not achieved 43. Assessment of the drivers behind the lack of achievement of some areas of the HSP was difficult due to a lack of information. Primary care utilization by gender, age and domicile (IOI #13) remained unknown at the end of the Project, making it challenging to determine if the increase 15

33 in diabetes prevalence and overweight/obesity could be due to a change in the population accessing services. Similarly, data on the share of annual outpatient visits by the poorest quintile of population (KPI #2) is lacking. Evidence from the DHS from 2009 and 2014 provides comparable data, and proxy data suggests that improvements in accessibility were observed among the lower income quintiles (described in the Data Sheet under KPI #2), which may partly explain the increase in diabetes prevalence and overweight/obesity (KPI #3; IOI #10). This explanation is partial at best, and the picture that remains is one where health outcomes for several indicators worsened over time. 44. A summary of Efficacy ratings is provided in Table 2 below. Both parts of the PDO are weighed equally, and the overall rating obtained is Modest. Table 2. Summary of Efficacy Ratings 14 PDO Rating Improved effectiveness of management - Substantial Efficiency (i) Efficiency in resource allocation Substantial (ii) Efficiency in use of human and physical resources Modest Governance (iii) Multisectoral engagement High (iv) Health sector governance Modest Improved effectiveness of implementation - Modest (i) Translation of plans and policies into action Moderate (Substantial/Modest) (ii) Achievements of HSP by priority area Modest PDO Achievement Rating MODEST 3.3 Efficiency 45. The PAD does not include a full economic analysis of the Project, but instead highlights expected cost-effectiveness of interventions for preventing and treating diabetes and rheumatic fever. The PAD also highlights benefits associated with: (i) early diagnosis and treatment; (ii) productivity gains; (iii) health outcomes and quality of life; and (v) a reduction in risks and costs associated with disease outbreaks and other public health emergencies. The economic and financial analysis of the ICR (Annex 3) thus focuses on presenting the evidence surrounding these issues. With the exception of rheumatic heart disease, systematic improvements in early diagnosis and treatment were not observed over the course of the SWAp, in part due to the emergence of the TTM Hospital which skewed resources toward tertiary care. Where diabetes is concerned, evidence of productivity gains is weak, with increased prevalence and severity observed in recent years (Annex 3, Box 1). However, the efforts of the DFC at the Orthotics and Prosthetics Workshop has resulted in some productivity gains in a small share of the diabetes population. Health outcomes have demonstrated mixed results to date, and noncommunicable diseases remain an area of concern with limited evidence of early detection though there is some evidence of behavior change (Section 3.2). The response to natural disasters (tsunami and Cyclone Evan) can 14 Possible ratings are Negligible, Modest, Substantial, and High. An additional category of Moderate was added to represent achievement that was considered to fall between those two categories. 16

34 be said to be positive, as these disasters did not result in disease outbreaks despite widespread devastation. While a public health emergency was seen in the form of an H1N1 epidemic, this occurred early in the SWAp (2009). In addition, efficiency in resource allocation, and use of human and physical resources have been discussed previously, and were rated Substantial and Modest, respectively. 46. An assessment of outcomes in relation to cost is necessary in assessing Efficiency; these are detailed in Annex 3 and discussed briefly here. Major project expenditures were those of infrastructure and equipment, which saw procurement delays and exceeding cost estimates. Challenges in the form of cost overruns, delays and/or unsynchronized inputs, were observed where the Nurses Hostel, PHC Centre, Orthotics and Prosthetics Workshop, Pharmaceutical Warehouse, and renovations to the MTII Hospital in Savaii were concerned, with each major infrastructure investment associated with a distinct set of challenges. An example of challenges faced is provided in Annex 3, Box 2. Despite a two-year extension, there continues to be outstanding infrastructure investments. Based on the mixed results where health outcomes are concerned and the inefficiencies associated with the major investments, Efficiency is found to be Modest. 3.4 Justification of Overall Outcome Rating Rating: The overall rating of Moderately Unsatisfactory reflects the Substantial rating for Relevance and Modest ratings for Efficacy and Efficiency. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 47. A strength of the SWAp was its ability to leverage funding from other sources. The Orthotics and Prosthetics Workshop funded under the SWAp received further assistance from DFAT through Motivation Australia, and have managed to use their new location to also provide a Diabetic Foot Clinic (DFC). The Workshop has gone above and beyond in their role where DFC services are concerned, and have coordinated the provision of additional health services (such as eye appointments and dietician appointments) for diabetes patients. Further, the DFC have made concerted efforts to ensure appointments are made outside of school hours to ensure that schoolchildren (usually girls) do not miss school to accompany patients to their appointments. 48. The private sector has also become involved in efforts to improve health outcomes. An example of these efforts is the provision of exercise classes at several private companies in Samoa using guidelines provided by the GOS. The village community organizations have also participated in these efforts, with the total number of workplaces enrolled in the Healthy Workplace Program Monitoring at 55 and the total number of village community organizations at 213 as of 2012, the most recent year for which data is available. (b) Institutional Change/Strengthening 49. Several aspects of institutional strengthening were seen through the SWAp. The SWAp began soon after the MOH-NHS split took place, at a time when both entities were trying to come to terms with their new roles. The relationship between the two improved over the course of the SWAp, and evidence for the institutionalization of a sector-wide process was seen to have taken place in the form of improved dialogue and coordinated policy development efforts. These efforts 17

35 were facilitated through many avenues, such as through the HPAC, through the administration of surveys where collaboration with the Bureau of Statistics was seen, the occurrence of regular meetings between the NHS and MOH on NCDs and the collaboration of NHS and MOH on Village Health Fairs. In addition, the SWAp has succeeded in bringing together the various stakeholders in the health sector around the table, ranging from the private sector to civil society organizations. The value of these common platforms is recognized by the GOS, and it appears that they are committed to maintaining these notable accomplishments going forward. 50. The SWAp also strengthened the management of the sector through the development of many policies, protocols and frameworks and through capacity building (see Annex 2 and Section 3.2). Training was provided for 2125 people (IOI #19), and newly-acquired skills were utilized in many areas, such as where development of training materials for health promotion were concerned. These training activities also benefited other sectors, such as the Memorandum of Understanding between MOH and Ministry of Agriculture and Forestry (MAF) for the sharing of knowledge and skills in health, nutrition and food production activities. (c) Other Unintended Outcomes and Impacts (positive or negative) N/A 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops N/A 4. Assessment of Risk to Development Outcome Rating: Moderate 51. The risk to development outcome is considered Moderate. The Project strengthened overall capacity and resulted in improved multisectoral engagement and health sector coordination, but these achievements translated to limited improvements in health outcomes during the life of the SWAp. Further, it is uncertain as to how these improvements will take place over time. While the results of large scale surveys such as the DHS and STEPS and tools such as the MTEF and National Health Accounts (NHA) have been utilized in decision-making and resource allocation, the use of results from sector performance monitoring continue to be lacking. Finally, while the relationship between MOH and NHS has improved since the beginning of the SWAp, there continues to be fragmentation within the sector, and there is a risk of further fragmentation in the near future with the setup of the Health Promotion Foundation. The Bank has exited the SWAp and will remain engaged in a limited capacity through the provision of technical assistance. WHO, ADB, and DFAT will contribute to the improvement of HIS going forward, and other development partners (notably MFAT and DFAT) remain committed to overseeing completion of infrastructure investments made under the SWAp, and to improved health outcomes in the country. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Unsatisfactory 52. Quality at entry was adversely affected by weaknesses in M&E and in the results chain. SWAp consultations took place soon after the health sector reform that saw the split between MOH and NHS. The SWAp modality represented a new way of engagement for the country, and was 18

36 seen as an improvement over the previous Project approach. The preceding Project, the Samoa Health Sector Project (P064926) had closed in 2006 rated Moderately Satisfactory, and as part of implementation arrangements had provided in-country support for 1-2 years. To assist with implementation of a new modality, the PAD provided substantial detail regarding implementation arrangements (see Annex 13), and clearly stated the goals of the annual meetings (Health Summit and Joint Review) and the role of the Coordinating Development Partner. Similar clarity was provided in terms of procurement and financial management, though such clarity was lacking for M&E. Indeed, M&E design contained several weaknesses as described in Section 2.3. A results focus was also absent a coherent results chain, appropriate indicators with accompanying definitions 15, baseline, and targets were missing. While the areas of focus of the SWAp were relevant to the needs of the country, the background analysis conducted as part of Project preparation could also have been strengthened, as could several other aspects of Quality at Entry (Section 2.1). In summary, Bank performance in ensuring quality at entry was Moderately Unsatisfactory. (b) Quality of Supervision Rating: Moderately Unsatisfactory 53. Despite shortcomings in quality at entry, the Bank team provided close supervision over the course of the SWAp, though the decision to conduct a redevelopment in place of the MTR represented a major lost opportunity. The Task Team conducted monthly videoconferences and weekly audioconferences with the SWAp Coordination Unit (SCU) and relevant partners at the beginning of the Project. In response to slow progress at the beginning of the SWAp, an agreement was made for increased supervision, as detailed in the Aide Memoire from December This additional supervision came in the form of two operational support missions (in addition to the two/year already scheduled), and the provision of in-country presence for several months. There does not appear to have been discussions on the skill-mix needed (in terms of supervision or incountry presence) or the specific technical roles of the DPs, which, in retrospect, may have been particularly important as this was not a traditional SWAp (Section 2.1). There were a total of four Task Team Leaders over the course of the SWAp, and changes in staffing among the DPs were also seen. While there was always a harmonious relationship between the DPs, there were weaknesses in record-keeping. When coupled with the frequent changes in staffing, this may have meant that monitoring of the results framework was not done as closely as it could have been, and along with it, the opportunity to focus on outcomes instead of outputs was lost. 54. A decision was made to conduct a redevelopment 16 of the SWAp in place of an MTR to readjust the SWAp given challenges faced related to implementation, with delays in procurement serving as the major issue. The redevelopment confirmed that the development objectives, performance indicators and implementation arrangements of the Program would remain unchanged. However, it resulted in the strengthening of the HPAC, expansion of the SCU to become a Sector Coordination Unit (renamed as the Health Sector Coordination, Resourcing, and Monitoring Division), modification of the PoW templates, and changes to procurement thresholds 15 The Final Aide Memoire (November 2015) notes that no additional support from the SWAp was provided to develop a data definition dictionary to provide clarification on indicators being collected for SWAp review. 16 Further details regarding the redevelopment are available in Annex III of the Follow-up Letter to the November 2010 Health Redevelopment Mission. 19

37 which addressed some challenges that the SWAp had encountered. Issues associated with quality at entry, such as the weaknesses in M&E and the weak results chain were not addressed. The decision to conduct a redevelopment in place of an MTR thus represented a further lost opportunity, as the MTR would likely have confronted the weaknesses to date and allowed for a restructuring to take place if needed. The Bank s systems were used for monitoring of fiduciary and safeguard policies, and these were followed closely. In addition, close supervision and proactive management allowed for timely response through the SWAp following the tsunami (2009) and Cyclone Evan (2012). In view of the achievements and lost opportunities described above, the quality of supervision provided was considered Moderately Unsatisfactory. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Unsatisfactory. This is based on ratings of Moderately Unsatisfactory for Quality at Entry, Quality of Supervision, and Overall Outcome. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory 55. The SWAP saw strong country leadership and commitment as evidenced by chairing of the Health Summit and Joint Review meetings by the MOF ACEO for the Aid Coordination and Debt Management Division. The GOS provided all planned contributions to the financing envelope despite the occurrence of natural disasters. Evidence, where available, was taken into consideration when providing input into the design of the SWAp. In particular, the NHA 1999 had highlighted the lack of investment in health promotion and primordial prevention while the STEPS 2002 had highlighted the increasing burden of NCDs; these areas were subsequently identified as priority areas in the SWAp. When challenges in meeting deadlines occurred, the Prime Minister got personally involved in order to mandate action from MOH and NHS in order for SWAp deadlines to be met and outstanding issues to be resolved. In general, government performance was found to be Satisfactory. However, taking the rating of implementing agencies performance into consideration (as they constitute part of the GOS), government performance is rated Moderately Satisfactory. (b) Implementing Agency or Agencies Performance Rating: Moderately Unsatisfactory 56. The SWAp had two implementing agencies, namely the MOH and NHS. While the MOH was tasked with overall coordination and oversight of the Project and Program implementation, NHS was responsible for the implementation of Component 2. As previously noted, the MOH Act 2016 resulted in a split in functions between the MOH (policy, regulatory, oversight) and NHS (service delivery) immediately prior to the Project, which resulted in tensions, and ambiguity regarding roles and responsibilities at the beginning of the SWAp. MOH managed all financing under the SWAp, including financing allocated to the NHS for Component 2. The relationship between the two entities has improved over the course of the SWAp, though there continues to be room for further improvement and synchronization. One area of weakness was procurement, as detailed in Section 2.4. Challenges in procurement impacted efficiency of the Project, as described in Section 3.3. The impact of the SWAp on health outcomes remains mixed at project closing, and it is uncertain to what extent the Project allocated resources to highest-priority, highest-impact interventions or whether a specific mix of activities was selected in response to the epidemiological 20

38 burden of the country. Given weaknesses in procurement and efficiency, and the mixed health outcomes observed at closing, performance of the implementing agencies was found to be Moderately Unsatisfactory. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Unsatisfactory. This is based on ratings of Moderately Satisfactory for Government Performance, Moderately Unsatisfactory for Implementing Agencies Performance, and Moderately Unsatisfactory for Overall Outcome. 6. Lessons Learned 57. The Project may have benefited from the use of incentives linked to achievement of outcomes in the form of results-based financing. Disbursement-linked indicators (DLIs) could potentially have been a useful approach in the context of this SWAp. The use of DLIs may also have incentivized improvements in M&E and HIS as progress would need to have been documented in order to access funds. DLIs would need to be developed appropriately given the presence of substantial non-swap funds in the health sector. 58. The use of a clear results chain linking activities to outcomes, a theory of change, or the MTEF may have allowed the GOS and DPs to be more selective about the activities funded under the SWAp and emphasized the importance of M&E where individual activities were concerned. Such an approach would also have driven a clear prioritization process for the POW. As part of efforts linking activities to outcomes, templates could also have been developed early on emphasizing measurement of impact of proposed activities and identification of the link from activity to outcome. 59. A phased approach such as an Adjustment Program Loan (APL) may have also been beneficial, where the first phase focused on the development of key policies, programs and systems, and the second phase sought to improve the utilization of these systems for the improvement of health outcomes. Such an instrument would have narrowed the focus of the Project and placed greater emphasis on achievement of results. An example of an APL-type operation is the Tanzania s Health Sector Development Program Project (Phase I) and Second Health Sector Development Project (Phase II), which also employed a SWAp and aimed to support the Health Sector Development Program over the period Recurrent problems should be investigated in greater detail. Infrastructure activities faced many challenges, and a closer look may have identified the source of these problems early on, allowing them to be rectified as soon as possible and preventing recurrence of similar problems down the line. An alternative approach would have been to include a clear escalation pathway for resolution of issues during the SWAp. 61. The lack of an MTR represented a lost opportunity for candid assessment of the SWAp and to course-correct. It could also have provided the necessary evidence for the redevelopment of the SWAp, ensuring that the redevelopment exercise represented a comprehensive response to problems identified to date and possible restructuring if needed. Weaknesses in the quality of record-keeping among DPs and gaps in data where M&E efforts were concerned which occurred 21

39 despite a focus on M&E within Component 3 may have been noticed then and a more rigorous approach to data documentation and collection could have taken place. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies Comments from the Borrower/implementing agencies are included in Annex 7. (b) Cofinanciers Comments provided by the cofinanciers (DFAT and MFAT) are included in Annex 8. (c) Other partners and stakeholders No comments were received from other partners and stakeholders. 22

40 Annex 1. Project Costs and Financing (a) Project Cost by Component (in US$ Million equivalent) Components Appraisal Confirmed Cost Estimate 17 (US$ millions) Appraisal Likely Cost Estimate (US$ millions) Latest Cost Estimate (Dec 31, 2015) (US$ millions) Component Not Specified 2.50 Component Not Specified Component Not Specified 7.48 Total Cost (b) Financing Source of Funds Indicative Financing (US$) 19 Committed Funds (US$) 20 Amount Disbursed (US$) 20 Percentage Disbursed Borrower (GOS) 1,500,000 1,828,156 1,828, % World Bank 3,000,000 5,869,648 (1) 5,869, % DFAT 7,200,000 15,600,000 15,600, % MFAT 12,600,000 11,768,012 (2) 11,768, % Interest Received 468,045 Total 24,300,000 35,065,816 35,533,861 (1) Based on the November 2015 Aide Memoire. Total World Bank credit funds allocated was SDR 3.9 million (US$6.0 million equivalent). Table shows disbursements/commitments in US$ equivalent (November 2015) for Credit of US$2,912,494, and Credit of US$2,957,154. (2) MFAT funds allocated is NZD16.0 million, table shows US$ equivalent. 17 This amount included commitments from MFAT and DFAT for the first 2 years of Program implementation only ( ). In addition, both MFAT and DFAT expressed that subject to approval they were likely to contribute additional funds for the subsequent 3 years ( ) (MFAT - US$4.8 m equivalent and DFAT US$8.10 m equivalent). With the additional funding, the resource envelope was expected to be US$24.3 m (at current exchange rates) at the time of appraisal. 18 This includes taxes and duties estimated at US$0.8 million. 19 This amount as shown in the PAD. 20 This includes additional funding of US$3 million paid directly by DFAT for medical equipment. 23

41 Annex 2. Detailed Project Components and Outputs by HSP Outcome The Program Components 21 and Outputs as listed in the PAD were as follows: Component 1: Health Promotion and Prevention Planned outputs under Component 1 were as follows: Subcomponent (i) development and implementation of health promotion strategies including, inter alia, provision of Sub-Grants to support the implementation of health promotion activities consistent with priorities set out in the Program Subcomponent (ii) supporting the prevention of non-communicable diseases through behavior change promotion and screening Subcomponent (iii) supporting activities aimed at reducing anemia among children Subcomponent (iv) strengthening infectious disease surveillance, screening and prophylaxis for post-streptococcal rheumatic heart disease and reducing the prevalence of vector-borne diseases such as dengue and filariasis Subcomponent (v) implementation of sexual and reproductive health interventions Subcomponent (vi) carrying out an injury prevention program Subcomponent (vii) implementation of the Health Care Waste Management Plan and the handling of bio-safety hazards. Component 2: Enhancement of Quality Health Care Service Delivery Planned outputs under Component 2 were as follows: Subcomponent (i) dissemination and training of staff on clinical standards and protocols for disease management adapted to the Samoan context Subcomponent (ii) upgrading the skills of health workers Subcomponent (iii) supporting improvements to reproductive, maternal and child health services Subcomponent (iv) improvement in primary health care and community health outreach through integrated health services Subcomponent (v) improvement in the forecasting of pharmaceutical requirements, procurement, storage, distribution and use through adoption of an upgraded warehouse and inventory management system Subcomponent (vi) improvement in physical disability services through activities and investments aimed at improving the quality of life of people with physical disabilities Subcomponent (vii) supporting improvements to oral health and dentistry Subcomponent (viii) supporting improvements to blood safety collection and storage Subcomponent (ix) improvement in mental health services through refurbishing/upgrading the mental health block of the hospital to ensure safety and security of mental health patients and staff [dropped given priority placed on the Orthotics and Prosthetics Workshop] Subcomponent (x) carrying out of small infrastructural improvements and maintenance, and provision of medical equipment to health care facilities. 21 Indicative resource allocation for each component is provided in Annex 5 of the PAD and was subject to review at the Annual Health Summits. 24

42 Component 3: Strengthening Policy, Monitoring and Regulatory Oversight of the Health System The outputs were as follows: Subcomponent (i) implementing a human resources development strategy for the health sector Subcomponent (ii) supporting the monitoring and evaluation of the implementation of the Program and the sector at large through use of improved information systems Subcomponent (iii) supporting the implementation of a public-private partnership framework for service delivery Subcomponent (iv) establishing standards for quality assurance of pharmaceuticals and enhancement of the quality assurance and regulatory function for the sector Subcomponent (v) strengthening the capacity of the SWAp Coordination Unit to coordinate and carry out its Program administration and monitoring responsibilities. 25

43 The SWAp financed the following activities (activities in bold were those financed as part of the post-tsunami response): Component 1: Health Promotion & Prevention National Health Promotion Program Improve Typhoid Investigation National Campaign to Promote Fruit & Vegetables Reduce Prevalence of Dengue and Filariasis Cases Anemia Reduction in 0-5 Age Group Reducing Tobacco Smoking Prevalence Cervical (Pap Smear) Screening Program Strengthening Secondary Prevention - NKFS Strengthen Comprehensive Screening program Injury Prevention Program Population Risk Screening for Diabetes Infant and young child Feeding Program Health Care Waste Management Post Tsunami Anti-Alcohol Campaign Component 2: Enhance Quality Health Care Service Delivery Health Sector Warehouse for pharmaceutical & medical supplies Primary Health Care Services delivery for Apia Urban area Physical Disability Services improvement EPI catch up Campaign Improve Diagnosis and Management of Pediatric Cardiac Disease Imaging Equipment Refurbishment Program Development of BA Science Program Improving transportation systems for outreach Improve Mental Health Unit Service Delivery National Blood Transfusion Service Program Improve Health Care Equipment for Patient Care Develop capacity to implement evidence based primary prevention program in Oral Health Extension of NKFS Dialysis Unit MTII Hospital Improvement of Infrastructure H1N1 Ventilators/supplies Procurement of Medical Equipment and Supplies for Health Sector Strengthening Secondary Prevention - NKFS Pandemic Disaster Reflection and ICHS Leadership Workshops Component 3: Strengthen MOH Policy, Monitoring and Regulatory Strengthening of ICT System Development of Protocols for health care services Human Resources for Health Sector Database 26

44 Credentialing System for Nurses & Midwives Annual Analysis of Health Sector Situation Responsive Human Resource Strategy for nurses & midwives Revenue Generation and Health Financing Standards for Quality Assurance of pharmaceutical supplies Enhancement of Quality Assurance Regulatory Function SWAp Unit Operational Costs Project Management Training Extension of Diabetic Clinic WHO Regional Committee Meeting MTEF/Governance Training 27

45 Final outputs are organized by HSP Building Block following Mid-term Redevelopment, and were as follows: 1. Governance and Leadership Strengthening Health Promotion Council Activities led by SPAGHL TA to develop Regulations for MOH Act 2006 TA to develop Regulations for Health Professional Council TA to review the Health Sector Plan TA for Institutional Framework to enhance QA and monitoring TA to undertake Marketing Strategy to enforce the WHO Code of Marketing breast milk TA to develop the National Tobacco Control Policy TA to develop the General Health Prevention Policy TA to develop the Health Promotion Policy TA to develop the Non-Communicable Disease Policy TA to develop Priority Strategies for Anaemia TA to review Primary Health Care Strategies TA to undertake Food Safety and Food Preparation Audit TA to conduct Cleanliness Audit of Toilets for WinLA Organisations TA to develop the Health Sector History based on the Six Building Blocks of the Health System Health Promoting School Standards developed and published TA to review Food and Nutrition Policy Develop standards and protocols for the Prevention and Control of Dengue and Filarisis TA to conduct Food Premises Cleanliness Audit TA to Develop the Cervical Screening Program Policies, Protocols & Guidelines Sanitation Standards developed and implemented TA to develop Standards & Protocols for Vector Control Programs (Post Tsunami) Support HPF Legislation support Scoping Visit to VicHealth and consultations Health Promoting School Standards published, stakeholder trainings conducted etc. Annual Health Forums for two years Exhibition for Health Sector Archives and exhibition implemented Advocacy and Communal Engagement (50 th Anniversary) TA to improve Asset Management at NHS and to develop NHS Assets and Facility Management Plan 2. Health Service Delivery Support for the establishment of the Health Promotion Foundation 28

46 Supporting the HPF Manager position (ACEO level) which has already been established and implementing its core functions Tobacco Control and Smoking Prevention Programs Billboards (140) for tobacco control prevention program produced and installed Pilot RHD screening: support for TA to undertaken screening and procurement of mobile RHD van Improve collection and segregation of health care wastes HWC Outsourcing of Health Care Professional Services (support for the hiring of the consultant specialist, 3 GPs and a nurse specialist) Monitoring of Water Treatment System for affected villages (Post Tsunami) Reducing tobacco smoking prevalence amongst school children Anaemia Reduction in the 0-5 age group national program health sector Reduce prevalence of Dengue and Filarisis cases in the health sector. Improve Typhoid investigation program for sector partners National Campaign to Promote Fruit and Vegetables for community based organizations Injury Prevention Program with National Campaign, evaluation plan with sectors Strengthening Secondary Prevention Program NKFS and clinical partners as well as strengthening the comprehensive health screening program for health sector Increasing child survival through infant and young child feeding sector Improvement of health care waste management (HCWM) EPI Catch up Campaign Savaii and health sector Improve Diagnosis and management of Pediatric Cardiac Disease for NHS and Health Sector Partners Post Tsunami H1N1 Vaccination Campaign Firm to design and supervise Construction of MOH Nurses Hostel and Renovations to the MOH Centre for Credentialing of Nurses, Midwifery and Allied Health Professionals Firm to design and supervise the construction of the Primary Health Care Centre, Orthotics and Prosthetics Workshop, Pharmaceutical Warehouse and renovations to the MTII Hospital in Savaii 3. Human Resources for Health (HRH) / Human Workforce Health Leadership and Management Development Program implemented for the health sector Improving Nursing & Midwifery Workforce for the Revitalisation of PHC within and / or reflective of total health care context of Samoa. UTS Contract. TA to improve procurement processes at NHS and staff trained IHR training for Health Quarantine staff Workshops on clinical procedures conducted HCW Generation Source established (surveillance) Training of Database 29

47 Undertake workshop on selected clinical procedures TA to conduct ECG Clinical Training Undertake workshop on selected clinical procedures ECG Clinical Training Two weeks workshop for Improved Best Practice in Neonatal TA to train Nursing Auxiliary TA to undertake staff training on Clinical procedures ie ECG Operations, Reading and interpretation Support and facilitate NHS workforce planning CISCO local training for ICT staff TA to undertake Renal Training for NKFS Nurses NHS SWAp Manager for NHS Comp 2. Component 1 & 3 Assistants (Principal Level) National Procurement Specialist Additional professional developments as listed in the training matrix 4. Health Information System TA for HIS Policy & Planning (Phase 1) Develop and enhance Health Sector HIS Surveillance IT Network Connection to Nursing School Building Geographical Information System / Geographical Positioning System (GIS/GPS) Demographic and Health Survey 2009 STEPS Survey including mental health sub-survey 5. Health Financing TA to develop and review the Midterm Expenditure Framework 6. Medical Products and Technology Mobile Clinic for RHD for screening. Portable mobile X-Ray units for Imaging Division X-Ray Mobile vans to be used at RDHs and in emergency situations Portable echo machine (Post Tsunami) Procurement of mammography machine for early detection of breast diseases and related trainings. Portable echo cardiogram for RHD screening (Post tsunami). Static Clinics for Dental Units at the 3 Primary Schools: St Mary s, Marist and Apia constructed. Forklift for warehouse and Orthotics Buildings procured Improve pharmaceutical stock control storage (cooler and bar coding system) Design and supervision of Nursing Hostel and simulation laboratory / renovation of nursing school building (T&G) 30

48 Construction of Nursing hostel and simulation laboratory / renovation of nursing school building Procurement of Bio-medical equipment Consultations and operations related to Biomedical Equipment support was provided Mobility tools for physical disability unit Design and Supervision of the HNS PHC, Warehouse and Orthotic Buildings Construction of NHS, PHC Warehouse and Orthotic buildings Improve Pharmaceutical Stock control and storage (cooler and bar coding system) Design and supervision of MTII Hospital Construction of MTII Hospital facilities Hospital equipment procured for MTII Hospital and other RDH (Rural District Hospitals) in Savaii Additional medical equipment for NHS New Hospital Phase A Vehicle procured to improve emergency and relief management services for NHS Emergency standby generator for emergency support to MTII hospital purchased Operational Costs for the National Dental Epidemiology Survey Vector Control Equipment: Antibody tests/stool jar for affected villages (Post Tsunami) Food Safety Equipment for Environmental Health Inspection Public Health Laboratory Equipment for Food and Water Health Sector Warehouse for all Pharmaceutical Products Establishment of National Blood Transfusion Service and implementation NHS and Samoa Red Cross this activity is half completed, supply of refrigerators for storage and promotion of outreach program to encourage blood donors conducted. 31

49 Annex 3. Economic and Financial Analysis 1. The PAD does not include a full economic analysis of the Project, but instead highlights the expected cost-effectiveness of interventions for preventing and treating diabetes and rheumatic fever. The PAD further emphasizes that the Project would result in several achievements as described in the Table below, though a lack of data prevents assessment of some areas. Expected Result 1. Earlier detection and response to illness and reduced reliance on most costly, tertiary care services at late stage of illness 2. Improved customer satisfaction 3. Resource savings (for households and government) due to improved resource allocation, gains in internal efficiency 4. Reduction in risks and costs associated with disease outbreaks and public health emergencies Actual Result Evidence is lacking that early detection and response efforts have systematically increased over the course of the SWAp. A major reason for this was the emergence of the teaching hospital (TTM Hospital) which skewed resources toward tertiary care. While it is possible that it will allow for increased in-country treatment going forward, and relatedly, cost savings, evidence is unavailable at this time for a comprehensive assessment of cost-savings, and the limited evidence to date suggests increased costs in general, with lost opportunities for improved cost-effectiveness where diabetes is concerned (see Box 1). A notable exception is rheumatic heart disease (RHD), where increased screening and treatment efforts have been demonstrated over the life of the SWAp, and translated into reductions in incidence of acute rheumatic fever from 29 per 100,000 in 2011 to 7 per 100,000 by 2015 while compliance to treatment (IM Penicillin) among those with RHD has been maintained at 80% or higher since Other issues include unavailable results of analysis from specimens collected (e.g. the Men s Health Fair), and a lack of availability of reagents for testing (e.g. for chlamydia) though syndromic management is conducted. Baseline information is not available for a comparison of trends over time. MOH reports that the proportion of clients satisfied with health services was 60% in Resource allocation was assessed under PDO Sub-objective #1, and was found to be Substantial. Improved health sector governance was assessed under PDO Sub-objective #2 and was noted to be Modest. It is unclear that there has been a reduced reliance on tertiary care services, suggesting a potential lost opportunity for resource savings. Further, new infrastructure investments for tertiary care services are expected to incur high recurrent costs while there continues to be a need to maintain district health services. No disease outbreaks were observed in the period following the tsunami (2009) and Cyclone Evan (2012). An H1N1 outbreak occurred in 2009 but this was in the early stages of the SWAp. While immunization rates have generally improved, STIs 32

50 5. Reduction in time lost due to illness and improved productivity 6. Improved health outcomes and quality of life continue to remain an area of concern with chlamydia rates unchanged at 28% (NHS Lab, 2009 and 2015). Immunization rates (IOI #11) and typhoid rates (KPI #6) have demonstrated a decline over the course of the SWAp, though not much improvement has been seen in treatment of diarrhea (KPI #6). No new cases of HIV have been reported since Evidence on this front has been mixed, as illustrated using evidence from diabetes (see Box 1). Health outcomes have demonstrated mixed results to date, and noncommunicable disease remain an area of concern with limited evidence of early detection though there is some evidence of behavior change (see Section 3.2). 33

51 Box 1. Diabetes Management in Samoa Data from the STEPS Survey suggests a large increase in diabetes in Samoa, from a prevalence of 21.5% (2002) to 45.8% (2013) among adults between 25 and 64 years old. Evidence from the National Kidney Foundation of Samoa (NKFS) suggests a substantial increase in late-stage treatment in recent years. The local patient population increased from 28 (2008) to 116 (2015) and the Dialysis Registry has increased from 50 (2010) to 495 (2015). 22 The dropout rate is high at around 60% (2015), with only around 10% of those registered for treatment attending regular consultations for diabetes management. It is worth noting that the NKFS does not have a nutritionist on-site and thus consultations provided may be limited in scope. In addition to the increase in prevalence, the average age of the patient population is reportedly declining, suggesting additional costs and lost productivity. As of 2010/2011, the cost of dialysis was estimated at US$38,686 per patient per year (The Economic Costs of Noncommunicable Diseases in the Pacific Islands, World Bank, 2012). The mortality rate is high, with approximately 25% of patients dying in the most recent financial year. Further, while some funding for the NKFS is provided every year in the budget, it is unclear that spending is efficient at the cost of $38,686 per patient per year and a 25% mortality rate. In contrast, the Orthotics and Prosthetics Workshop (construction of which was funded by the SWAp) provides a diabetes foot clinic (DFC) through funding received by Motivation Australia. The Orthotics and Prosthetics Workshop staff have taken on a role as a one-stop center for their 42 clients to date (June 18 to December 1, 2015), and have made efforts to coordinate their visits to various specialists. Their dropout rate is less than 5% with an average of six follow-up appointments provided after the initial assessment, and evidence suggests approximately 80% of patients have been able to resume their normal duties upon receiving treatment. 23 Staff who received training abroad in Orthotics and Prosthetics (funded by Motivation Australia, an NGO) have been in place since January 2016, but have not been able to utilize their newly-acquired skills due to a lack of equipment. Construction of the Workshop was completed in April 2015 (with the builders warranty set to expire in April 2016), but continues to lack Orthotic and Prosthetic-making equipment, which was originally supposed to be funded by the SWAp though this was canceled. Instead, funding for equipment will now be provided by Motivation Australia with substantial delay. Thus, while the services rendered in terms of DFC are cost-efficient, the lack of equipment has not allowed for maximum cost-efficiency of the workshop. Funding for the DFC operated through the Workshop is not provided through the annual budget despite evidence of cost-efficiency. 2. An assessment of the outcomes described above in relation to cost is necessary in assessing Efficiency. Major project expenditures were those of infrastructure and equipment, which saw substantial delays and frequent overruns of cost estimates. Despite a two-year extension for completion of infrastructure, it had yet to be completed by December The estimates for figures obtained from interviews conducted during the ICR mission. 23 Figures for the drop-out rate and resumption of normal duties obtained from interviews conducted during the ICR mission. 34

52 completion of infrastructure activities were underestimated in the POW, resulting in a major shortfall in October A comparison of estimated and actual costs shows that the resulting budget shortfall totaling SAT $15.5 million was due to quite substantial cost overruns, with the largest overrun of SAT $5.8 million for construction of PHC, Pharmaceutical Warehouse, and Orthotics Center. In the case of nonmedical equipment for the new hospital (TTM Hospital), no funds were originally allocated for this purpose, and an additional request of SAT $3.8 million was made (Aide Memoire, November 2012). 3. Each of the major procurement items, namely the Nurses Hostel, PHC Centre, Orthotics and Prosthetics Workshop, Pharmaceutical Warehouse, renovations to the MTII Hospital in Savaii, and the purchase of equipment for the new TTM Hospital experienced challenges. An illustrative example of challenges faced using the MTII Hospital is provided in Box 2 below. The experience described in Box 2 was not an isolated incident, and in addition to increased costs, likely resulted in lost opportunities. Box 2. MTII Hospital Renovations Improvements to the MTII Hospital were initially anticipated to be SAT$1.5 million, with actual costs originally estimated to be closer to SAT$4 million (Aide Memoire, November 2012). This discrepancy was likely the result of costing conducted by technical experts instead of those more familiar with construction, which appears to have occurred in other instances of infrastructure development over the course of the SWAp. The ICR mission found that there was a lack of clarity regarding the budget as consultations with MOH and NHS took place. MFAT confirmed their willingness to provide additional funds for the renovation and extension of the Hospital. Final costs at the end of the SWAp were SAT $5.8 million (MOH Accountant), reflecting further overruns from the originally estimated amount of SAT $4 million. The Hospital was originally supposed to house an X-ray machine from TTM Hospital in Apia. However, that machine broke down in March Several months later (in late 2014), the design works for the Hospital was approved and signed. A new machine was ordered by NHS soon after the machine broke down, though the MOH was only advised of this new machine in May 2015, at which point the construction work was halfway through. It is unclear when the MOH and architect were informed of the change in equipment size from the old machine to the new machine. Subsequently, the NHS requested use of the CT Scan room to house the machine while the X-ray room was being extended to accommodate the size of the new machine, which was much larger than the old machine. This necessitated a redesign, and translated to some variation to the original construction costs which was not approved by the Tenders Board. The NHS then requested the long-term (instead of temporary) use of the CT Scan room as it could accommodate the X-ray machine, but this request was denied on the grounds that the room was needed for the CT Scan machine. The CT Scan machine subsequently broke down. Following that (and approximately one week before completion of construction), the NHS requested reconfiguration to the original design, but this could not be done as the redesign had already started. The renovated building was handed over to the MOH in December At the time of the ICR mission and more than two years after the original end date for the SWAp, both the X-ray and 35

53 CT Scan spaces remain closed off and unutilized. While the new X-ray machine arrived some time ago, it remains unused. The mobile X-ray machine continues to be used in place of the new machine. 4. Financial sustainability also requires consideration of recurrent costs. These have been described in Section 2.5 of the ICR. While financially sustainable with ~1% of the NHS budget for 2014/2015 going toward maintenance costs, it is thought that a greater amount will be necessary going forward, particularly given the additional infrastructure investments made in recent years. 36

54 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Names Title Unit Responsibility/ Specialty Lending Muhammad Ali Pate Senior Public Health Specialist EASHD Team Leader Aparnaa Somanathan Senior Economist GHNDR Team Member A. Juliana Williams Senior Program Assistant Senior Program EASHD Assistant Reem Hafez Consultant GHNDR Team Member Betty Hanan Consultant GSP06 Team Member Ian P. Morris Consultant GHN02 Team Member Melinda Good Chief Counsel LEGES Legal Sheila Braka Musiime Counsel LEG Legal David Michael Chandler Sr. Financial Management Specialist EAPDE Financial Management Stephen Paul Hartung Financial Management Specialist GGODR Financial Management Cristiano Costa e Silva Nunes Senior Procurement Specialist GGODR Procurement Evelyne Villatoro Senior Procurement Specialist GGODR Procurement Supervision/ICR Eileen Brainne Sullivan Senior Operations Officer GHN02 Team Leader Eva Jarawan Lead Health Specialist EASHH Team Member Aparnaa Somanathan Senior Economist GHNDR Team Member Neesha Harnam Health Specialist GHN04 Team Member Nicholas Davis M&E Specialist EASHD M&E A. Juliana Williams Senior Program Assistant Senior Program EASHD Assistant Sabrina Terry Program Assistant GHN02 Program Assistant Cornelio Quintao De Carvalho Team Assistant EACDF Team Assistant Ian P. Morris Consultant GHN02 Team Member Betty Hanan Consultant GSP06 Team Member Bengt Jacobson Consultant EASHD Team Member Nicolas Drossas Consultant EASNS Team Member David Bruce Whitehead Financial Management Specialist GGO20 Financial Management Stephen Paul Hartung Financial Management Specialist GGODR Team Member Cristiano Costa e Silva Nunes Senior Procurement Specialist GGODR Procurement Evelyn Villatoro Senior Procurement Specialist GGODR Procurement Jinan Shi Senior Procurement Specialist GGODR Procurement Eric Leonard Blackburn Procurement Specialist GGO08 Procurement Manuela Da Cruz Procurement Specialist GGO08 Procurement Joyce Miriam Denise Witana Procurement Specialist GGODR Procurement Jonas Garcia Bautista Safeguards Specialist GENDR Safeguards L. Panneer Selvam Safeguards Specialist GSU03 Safeguards Mark Hodges Safeguards Specialist IPN Safeguards 37

55 (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY FY Total: Supervision/ICR FY FY FY FY FY FY FY FY Total:

56 Annex 5. Beneficiary Survey Results N/A 39

57 Annex 6. Stakeholder Workshop Report and Results N/A 40

58 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Comments from the Government of Samoa on the draft ICR are provided below. Datasheet Implementing Agencies: Refer to PAD MoF is the Executing Agency and MoH is the Responsible Agency. These should be reflected in the document. Cofinanciers and Other External Partners: The SWAp DPs were noted as Pool Partners because their funds were pooled into a Special Purposes Account (SPA) for the Health SWAp Program. That reference became irrelevant as the years progressed with the maturity of the sectorwide approach modality in the health sector. Results Framework Analysis: IOI #1: If the period of review is from then the Procurement Plan for the Project was achieved because that Plan was updated throughout the years even with the last Technical World Bank mission in IOI #3: and Government towards the end of the FY IOI #5: Government s commitment to the health agenda and Health Sector Program should be acknowledged herein. IOI #20: The Health Sector Infrastructure Project is about 80% complete with the National Pharmaceutical Warehouse and PHC Centre to be completed by July Despite the delays, the completed infrastructure works (Nurses, Midwifery and Allied Credentialing Centre, Nurses Hostel, National Orthotics & Prosthetics Workshop and the Renovated & Extended MTII Hospital) continue to provide quality facilities for staff and visiting clients. The two medical equipment packages also provided under the Health SWAp Program, comprehensively equipped the new TTM Hospital on Upolu, the referral MTII Hospital in Savaii and Rural District Hospitals around the country. These achievements should be noted in this ICR for the value and difference made in service delivery given the new facilities and much needed medical equipment. The latter which was notably acknowledged by the NHS Biomedical Team in government s ICR. Section 2.2 Implementation Para. 23: Strategies were grouped under the 7 HSP Outcome areas noted above so that they strategically respond to outcomes and related indicators. That was in conformance to government s planning framework which was adopted across-government sectors. Section 2.3 Monitoring and Evaluation Design, Implementation, and Utilization Para. 26: The Sector M&E Reports are as follows: (1) HSP Annual Reviews; (2) Health Program Advisory Committee quarterly progress reports; (3) Bi-annual PMRs (Progress Management Reports) and updates of health related MDGs provided to MoF over the years. These sector M&E reports are discussed at various national (eg. Cabinet Development Committee meetings, Quarterly Sector Coordinators Forums) and sector forums and serve to provide sector direction and evidence based verifications. 41

59 Section 2.4 Safeguard and Fiduciary Compliance Para. 31: POWs were discussed and approved at the HAC meetings. All activities dropped from the POWs or priority list were well discussed with related sector agencies. Thus activities which were dropped were replaced by other priorities over time eg. some NHS activities were dropped given the priority given to the infrastructure works and medical equipment packages. The latter were much influenced by what the Govt of the PRC could support with the new teaching hospital. Following procedures, MoH was officially advised of such changes from the NHS Board or GM levels. Para. 32: Procurement activities for the Program continued to be undertaken by MoH through the Health Sector Coordination Division and DG s Office with the Legal Consultant. Section 2.5 Post-completion Operation/Next Phase Para. 33: How is that? [in reference to further fragmentation of the health sector] Section 4. Assessment of Risk to Development Outcome Para. 51: The emphasis made on the relationship between MoH and NHS is overly stated when it s more professional than anything else. This was also noted throughout the life of the SWAp, that the DPs emphasis on this relationship has done more harm than any good and that they also contributed to fragmented approaches and procedures because of the disrespect of government s approved governance and communication frameworks. There are various documented evidence of such nature. Please explain the risk of further fragmentation with the establishment of the HPF? Section 5. Assessment of Bank and Borrower Performance Para. 53: Is this an incomplete sentence? [in reference to the sentence There were a total of four Task Team Leaders over the course of the SWAp, and changes in staffing among the DPs were also seen. ] Para. 54: The HAC was established with the SWAp since It s role was strengthened and redefined as a policy advisory committee for the sector. Please refer to Redevelopment document provided for that historical achievement. The SCU was also re-defined with the establishment of the SWAp Program in By 2012 it was re-established and re-named as the Health Sector Coordination, Resourcing and Monitoring Division within MoH. It was the first sector coordination division acknowledged by PSC for all 14 sectors of government and that s a sector achievement in itself. Para. 55: If this is related to the delays in infrastructure completion then it should be noted herein. Para. 56: Correction. MoH was the Responsible Agency for the Health SWAp. It s implementation roles is in the management, monitoring, fiduciary, procurement and legal aspects of development assistance. IAs (Implementing agencies) were the service providers (eg. NHS, NKFS, NUS, etc) and sector partners like MWCSD, etc. Again the relationship between the two agencies is overly stated in this review when it should not be the case. It has thus missed the opportunities to note the various sector collaboration and synergies which has resulted in various sector achievements over the years of the SWAp Program. 42

60 Section 6. Lessons Learned Para. 57: This is part of the health reforms which was not fully realized during the term of the SWAp and is currently being worked out by MoH in the form of Purchaser-Provider arrangements. Annex 2. Detailed Project Components and Outputs by HSP Outcome Component 2: This was taken out of the SWAp Program given priority placed on O&P Workshop etc. Annex 3. Economic and Financial Analysis Box 2, Para. 1: To a large extend, the discrepancies came from initial unrealistic estimates made by the sector indicating lack of proactive analysis on the cost and functionalities of services needed to be renovated and extended. Additional requests for variations to design initially approved, also contributed to the estimated cost initially proposed for MTII for eg. Box 2, Para. 2: The chronological order of this event was provided to MoF and DPs and the mission should have had access to the documentation. The reference here is misleading and should be corrected. For the sake of this exercise, the design works for MTII was approved and signed upon by latter end of MoH was only advised of the new machine by May By then the construction works was half way through and contract variation was obvious by then, which NHS had officially noted with MoH of their being responsible for such. 43

61 The Borrower s ICR is provided below. Summary Activity information Activity Title (i) Program Development Objective (PDO) Health SWAp Program Government of Samoa To improve the effectiveness of the Government of Samoa in managing and implementing the Health Sector Plan using results from sector performance monitoring. (ii) Health Sector Program s aim in the medium-term (FY ) Joint Partnership Arrangement (JPA) To improve access to, and utilization of effective, efficient and quality health services to improve the health status of the Samoan population. Start and end date (original) August 8, 2008 December 31, 2013 Variation No. 1 December 16, 2013 December 18, 2015 Total cost USD$33m or SAT$81.4m Reporting period August 8, 2008 June 2015 Completion report preparation Prepared by: Others involved or consulted Date report submitted : Ministry of Health Health Sector 44

62 Executive Summary Report 1. Background and context: The Health Sector Plan ( ) provides the strategic directions for Samoa s Health Sector. The Plan outlines the need to strengthen health systems to address existing and emerging challenges to the health of the population. The overarching goal (or mission) of the Plan is to regulate and provide quality, accountable, and sustainable health services through people working in partnership. The Plan identifies six priority areas and strategies: (i) health promotion and primordial prevention; (ii) quality health service delivery; (iii) governance, human resources for health and health systems; (iv) partnership commitment; (v) health financing; and (vi) donor assistance. It also emphasizes the need to improve the quality dimensions of health programs, including enhancing the quality of health staff and staff numbers to reduce currently evident capacity constraints. To monitor and evaluate the success of the Program in meeting its objectives, agreement was reached between the health sector and the development partners on: (a) a set of five strategic sector performance indicators; and (b) a set of intermediate indicators for each component of the Program. The five strategic sector performance indicators included the: (i) control of noncommunicable diseases; (ii) child health, morbidity and mortality; (iii) reproductive and maternal health; (iv) control of communicable diseases; and (v) injury prevention. The full details of these program performance indicators are the core elements of the Health SWAP in Samoa. These are (i) clear sector policy and strategy as articulated in the Samoa Development Strategy, the Health Sector Plan , and corresponding 3-year health entity Corporate Plans; (ii) the health sector expenditure program(mtef) with prioritized activities aligned with the HSP over the period FY (government and development partner funds); (iii) a common performance management system for the sector with common-program indicators and mechanisms for donor coordination and partnership arrangements; (iv) processes for harmonization of budgeting, financial management, procurement, and reporting; and (v) a consultative mechanism for dialogue among public and private health sector providers, clients, and civil society led by Government. Initially the Health SWAp was designed to have three broad components, which were directly aligned with the PDO. These were: Component 1: Health Promotion and Prevention Component 2: Quality Health Care Service Delivery; and Component 3: Strengthen Policy and Regulatory Oversight of the Health System By 2010 the Government of Samoa along with its Development Partners jointly agreed to consider a more outcome based approach to the Health SWAp. The redevelopment in essence allowed the Health Sector to rethink and readjust the design of the Program as a step forward in ensuring maximum synergies and participation is realized between related activities and programs that were driven to meet a common broad outcome. Sector ownership was key to allow the capacity within to lead, manage development and to utilize government systems which were in place, build on the partnerships developed over time to bring about mutual accountability and more effective 45

63 harmonization of analysis, funding and reporting; reduce the fragmentation of aid, increase value for money and to strengthen sector linkages. The transition was made effective and simplified due to the fact that prior to the Paris Declaration in 2005, Sāmoa had already begun a continuing program of reforms amongst which was the key objective to achieve development effectiveness. In fact the analysis of development assistance options in the Pacific by then, had indicated that Samoa was well placed to adopt a greater harmonization of donor policies and procedures. Government had therefore by the time the Health SWAp became effective had implemented and institutionalized two other sector SWAp Programs Water and Education and the re-design phase of the Health SWAp was to further strengthen aid management for better development results at national and sectorial levels. 2. Assessment against the Results Framework for Performance Monitoring: Program performance in terms of health status, linked with broader national and international health development goals (e.g MDGs) The indicator results for the MDGs indicated that progress was made in reducing child mortality, increased vaccinations, reduced maternal mortality, increase births attended by skilled health personnel, increase ANC coverage, increase knowledge of HIV/AIDS, slight decrease in hypertension prevalence. Challenges included a reduced rate of contraceptive use, increased adolescent birth rate, and increase of diabetes. Refer to Appendix 1 for full results. Intermediate health sector outcomes The results framework was made up of 6 objectives; 1 SWAp institutional development objective and 5 intermediate objectives. They are defined as to overall improve the effectiveness of the Government of Samoa in managing and implementing the HSP based on the use of results from sector performance monitoring. This is supposed to reflect through: (1) Policies and plans implemented according to agreed priorities; (2) Greater efficiency in use of resources; (3) National ownership and commitment to health sector program; (4) Results of performance monitoring used in shaping the program implementation, and; (5) Improved health sector governance. Each of these objectives are measured by 2 indicators. Based on the indicator results for these 6 objectives, the table below summarizes the progress for the Health SWAp Program. The overall institutional development goal, Objective 3, and Objective 4 are areas where progress is a challenge. See Appendix 2 for a list of the SWAp results by indicator. Intermediate Indicator Results Grouped by Objective Objective Baseline YR1 YR2 YR3 YR4 YR5 SWAp Inst. n/a n/a n/a n/a n/a n/a Dev. Objective Objective 1 Complete Complete Complete Complete Complete n/a Objective 2 50% Complete Complete Complete Complete n/a Objective 3 n/a 50% 50% 50% 50% 50% Objective 4 n/a unchanged unchanged unchanged unchanged unchanged Objective 5 n/a Complete Complete Complete Complete Complete (ongoing) n/a= no data available to report 46

64 Monitoring performance of the health SWAp process itself Outcome indicators were used to evaluate the performance of the Health SWAp in terms of achieving priority objectives of the Health Sector Plan, however it did not consider the revised Sector priorities that were identified during the lifespan of the SWAp Program. Baselines were supposed to be established in the first year and performance reviewed on key indicators as part of the Health Summits. Some of the baselines are not applicable to its respected project outcome indicators. 3. Risk Management: Throughout the implementation of the activities under the three components, there have been challenging risks encountered in implementing of the SWAp Program as well as its outcomes. Below are some notable examples under the three components: Table 1 Risk Management and Mitigation Measures Risks Breakdown of sector partnerships derailing the Program s implementation Capacity limitations delay Program implementation Procurement delays result in slowed Program implementation Risk Rating Mitigation and Adaptation Measures M Broad consensus on core principles and Program design by all key partners. Strong GoS Aid Coordination mechanism in MoF. Notable Sector Coordination platform managed by MoF for the 14 Sectors which ensures quarterly. HPAC meetings regularized. Utilisation of Govt systems, management and implementation. Continuous participatory engagement and consultations especially with end users through design and construction meetings, public health sector forums and HPAC meetings, to ensure ownership. Financial management risk with changes in health priorities, misuse of funds, use on ad-hoc activities that were not approved. S Mobilize existing latent in-country technical expertise. Support for capacity building at all levels of the health system, including through technical assistance, pre an in-service training, systems and procedures development. Efforts to retain build capacity in the Sector. H Use/retain built capacity for procurement. Procurement assessment and capacity action plan. Early procurement planning and preparation of procurement documents for first 18 months of Program implementation. Hiring of Procurement Specialist for NHS to expedite and facilitate the procuring of NHS activities. Provided support for capacity building. 47

65 Conflict between health sector entities over evolving responsibilities and accountabilities within the sector entities maintain an open and participatory sector dialogue. Weakened commitment from PSC to support health sector reforms especially in capacity building. H The health sector reform is part of an overall public sector reform program with strong central agency and political support. Formal arrangements between DPs and MOH and other health entities provide for open sectoral dialogue and sharing of information. Government s Official Communication Process reaffirmed and reinforced at all levels. H High degree of commitment from PSC to approve sectorial organisational changes for eg. HSCRM Division by 2012 and positions. MOH continues to dialogue with the PSC on critical positions requiring the strengthening of work undertaken by the Health Sector Program. Prolonged feedback/endorsement on issues related to Program positions such as the Finance Manager, Projects. 4. Monitoring and Evaluation Matters / Issues: The assessment of the M&E Manual also carried out by the consultant Mr Phillip Davis by June 2013, noted that the M&E was identified as a priority from the beginning of the Health Sector Program. In the PAD, an M&E framework was suggested with two key measuring focal points: program outcome indicators to monitor health sector performance; and SWAp processes monitoring indicators to assess institutional developments outcomes, previously discussed. However, implementing M&E components in the monitoring framework of the Program were very challenging. There was no formal review of quality of Results Framework implementation, therefore there are no empirical data to measure the impact of implementation efforts. This section reflects on the quality and implementation of the results framework developed as per SWAP activities. First, the results framework for SWAp will be presented and summarized. Please refer to Appendix 2 for SWAp Monitoring Framework. Second, a brief summary of the outcome and indicator results for the Health Sector Monitoring and Evaluation (developed to supplement SWAp processes) will be presented for the context. Third the progress made in the implementation of health sector and SWAp monitoring and evaluation will be discussed. Finally, we will provide lessons learnt from implementation, challenges and recommendations. A. Component 1- Health Promotion and Prevention Table 2 below summarizes the indicators as per SWAP processes for Component 1. Data for waste management in particular is not currently available. However, MOH is currently working with partners to solidify implementation of M&E procedures for this indicator. 48

66 Prevalence of Diabetes Adolescent Birthrate years (annual) Injuries in Children <15 Table 2. Component 1 Indicators Indicator Baseline Project Outcome Indicators Freq. and Reporting 22.30% 22.30% 49.70% Yr 2, Yr 5, annual cases 42.6 birth per 1000 women aged yrs. Intermediate Outcome Indicators People years overweight and obese Percentage of children under 1 yr. received at least one dose of measles vaccine Improved medical waste management 85.6% and 56% Forthcoming April 30th Instruments STEPS, Diabetes register 44 per per years DHS 2009, 2014 Responsibility for Collection MOH, Diabetes Foundation SBS, DHS, MOH Annual PATIS NHS, MOH 85.6% and 56.0% 89.1% and 63.1% Yr 2 and 5 STEPS MOH 45% 55.70% 62% 67% 85% 98% Annual EPI/NHS, DHS B. Component 2- Enhancement of Quality Health Care Service Delivery Annual Waste Management Audit Report NHS, MOH NHS, MOH Table 3 below summarizes the indicators as per SWAP processes for Component 2. It should be noted that data for primary care, antenatal care visits and rheumatic heart disease treatment compliance have years of missing data. 49

67 Table 3. Component 2 Indicators Indicator Baseline Project Outcome Indicators Perinatal Mortality (22 wks gestation to <7 days) Adolescent Birthrate years (annual) 23.90% 7 per 1000 pregnancies of 7 mo. Or more 42.6 birth per 1000 women aged yrs. Intermediate Outcome Indicators Primary Care Utilization (gender, age, domicile) Antenatal care coverage for at least 4 visits Proportion of Rheumatic Heart Disease patients complying with treatment 12/12 months Forthcoming April 30th 61% (min. 3 visits) 8 per 1000 pregnancies of 7 mo. Or more Freq. and Reporting Instruments Annual DHS 2009, per per years DHS 2009, 2014 Annual CHNIS, PATIS 58.40% 64% 73% CHNIS, PATIS 84% 86% Annual Rheumatic fever register Responsibility for Collection SBS, DHS, MOH SBS, DHS, MOH NHS, MOH NHS, MOH NHS 50

68 C. Component 3- Strengthening Policy, Monitoring and Regulatory Oversight of the Health System Table 4. Indicators as per SWAP processes for Component 3 Indicator Baseline Freq. and Reporting Intermediate Outcome Indicators Staff mix and distribution according to national standards Evidence of performance monitoring leading to policy and regulatory action to improve health services demonstrated outcomes of training plan by component Vacancies Data 2008 Registered Dental Practitioners 17 Doctors Medical Practitioners 103 Doctors Registered Pharmacists - 15 Source: Ministry of Health - Registrar : Interim database for HRH for the health sector in place as a starting point in identifying staff gaps by facility and qualification. Tobacco Control Policy 2010 developed to enforce the implementation of and public compliance to the Tobacco Control Act : Tobacco regulations have allowed the Ministry to issue penalties for non-compliance. The development of the General Prevention Policy and Health Promotion Policy: Strengthening of ICT System trainings-led to the development of the health information systems and HIS policy : Develop capacity to implement evidence based primary prevention program in Oral Health (Fiji) : Improved Diagnosis and Management of Paediatric cardiac disease (New Zealand) : Imaging Equipment Refurbishment Program (Radiology) n/a See Appendix See Appendix See Appendix See Appendix See Appendix Instruments Annual MOH Annual Annual Review report and POW Review report and POW Responsibility for Collection MOH NHS and MOH 51

69 4 (a) Summary of Health Sector M&E Framework A framework for monitoring sector performance was developed as part of the SWAP preparation. This framework contains seven (7) long-term health outcomes, each with a core set of indicators. Indicators for measuring performance are at 3 levels; national/sector level, institutional level (ministry level), and operational level. The manual provides a list of core health indicators identified by the Health Sector. A minimum set of core indicators have been used for reporting health performance on a 6 monthly, annual and 2-5 year basis. These indicators and outcomes were operationalized in 2011 through the Monitoring & Evaluation Operational Manual 2011, which was a sector wide framework for monitoring performance. Additionally, this framework was incorporated in the Health Sector Plan for under Key Strategy 3 which seeks to strengthen health information development and management through surveillance, research, and health intelligence. The 7 outcomes are as follows. Table 5 shows a brief summary of indicators for each outcome area of the results framework: Table 5. Results Framework Summary Outcome 1 Outcome 2 Outcome 3 Outcome 4 Outcome 5 Outcome 6 Outcome 7 Prevalence of Alcohol Drinkers Prevalence of Current Smokers Prevalence of People Who Are Physically Active Proportion of the Population Who Eat 5+ Servings of Fruit and Vegetables a Day Prevalence of Diabetes Prevalence of Hypertension Prevalence of Overweight and Obesity Prevalence of Rheumatic Heart Disease Prevalence of Cancer Prevalence of STI's Prevalence of Notifiable and Vaccine Preventable Diseases Incidence and Death Rate Assoc. with TB Acute Respiratory Infections Among Children Aged <5 Years Prevalence of HIV Life Expectancy at Birth Total Fertility Rate Adolescent Birth Rate Contraceptive Prevalence Rate Maternal Mortality Rate Percentage of Births Attended By Skilled Health Personnel Infant Mortality Rate Under 5 Mortality Percentage of Infants Exclusively Breastfed Waiting Time for Emergencies, Triaging & General Outpatients Health Facilities and Providers That Are Accredited and Certified Health Facilities Compliance with Legislation, Policies, Protocols, and Standards Proportion of Clients Satisfied With Health Services Health Facilities Utilization Rates Climate Change Indicators (in development) Compliance with Disaster and Emergency Response Plans & Legislation Percent of Health Professionals with Competencies to Respond to Disasters 52

70 Table 5. Results Framework Summary Outcome 1 Outcome 2 Outcome 3 Outcome 4 Outcome 5 Outcome 6 Outcome 7 Injuries in Children <15 Number of Attempted Suicides Prevalence of Cervical Cancer in Women Aged 20 Years and Over Antenatal Coverage Proportion of 1 Year Old Children Immunized Against Measles Proportion of Fully Immunized Children Immunized Percent of Health Personnel Aged 55 Years and Over Health Professional to Population Rates Total Health Expenditure Government Expenditure on Health 5. Governance and Management Matters and Issues: 5.1 Time period in designing and planning of the program for implementation was one year and was too rushed for such a comprehensive Program. Some shortcomings in terms of the sector s readiness and capacity of the SWAp Unit, may have been overlooked. 5.2 The establishment of the Health Sector Coordination, Resourcing and Monitoring Division (HSCRM) by 2012 has allowed the MOH to oversee most aspects of the Health Sector Program (which the Health SWAp is part of) which has reduced transaction costs for government. It has also permitted a more holistic outfit with most development programs being coordinated from the Division to ensure synergies and avoid duplication of efforts and waste of resources. 5.3 Adoption of the SWAp modality has led to improvements in GoS capacity to plan and manage complex health sector projects. This has provided the continuous support from Government Central Agencies MOF as the Executing Agency of the Program, the AG s Office and the Development Partners namely DFAT, NZAP and the World Bank - has been a success throughout the lifespan of the SWAp Program. The significant development assistance provided for the Health SWAp in fact gave oomph and substance to the implementation of the HSP. It is also noteworthy to mention that through the SWAp modality, GoS has had previous experience with SWAp (in water and sanitation and education sectors) and Ministry of Finance appears to have had good capacity to serve as executive agency. The use of a single system to record and report SWAp-related finances, while initially demanding, has been more efficient than complying with varying individual project requirements. 53

71 There have also been rigorous procurement processes combining those of the GoS Tenders Board and the World Bank have been applied throughout the life of the SWAp. While those processes have created challenges and brought about delays in implementation, it is reasonable to assume that value for money and capacity building ought to have been achieved in all significant purchases of goods and services. 6. Lessons Learnt: 6.1 SWAp modality was and continues to be effective in harmonizing development activities for the sector and harmonizing assistance from donors to effectively realizing key strategic areas of the HSP. There have been excellent opportunity for professional development in specific areas given the investments in new medical equipment (eg. for Biomedical Team) and health infrastructure (eg. PMT). 6.2 The HSCRM Division sees the need to continuously review the Master Work Plan to be inclusive of activities supported under the local budget and development assistance such as WHO, UNFPA, Global Fund, SPC, SWAp etc to ensure holistic planning and reporting. This process has strengthened sectoral and across sector approach and place emphasis on synchronizing indicators and liked-activities. 6.3 The Program provided an effective change from Outcome-based Program rather than Component-based focused Program as it was originally designed. The Master Work Plan that guided the implementation of the Program s activities follows the 7 health outcomes in which directly links to the HSP The establishment of Outcome Based Groups for each health outcome to meet on a regular basis to plan, monitor and coordinate the health sector response or Program Implementation Plan; is the way forward. 6.4 The whole M&E System including personnel and framework should have been in place first before the Program was implemented. That would have ensured that specialized M&E person(s) were hired to monitor the implementation of SWAp activities as part of the whole M&E Framework and synchronize all the information to ensure that it is in line with Program objectives. Understanding and awareness of health workers on Health Sector developments (e.g M&E Framework, MTEF, etc.) 7. Overall judgement and critical matters and issues: The SWAp Program has enabled the country and the government as a whole to overcome inefficiencies, lack of government ownership, and a number of other problems that were constraining the impact of international support to our country. This Program and the approach it has taken place for the Health Sector embraced many of the principles of harmonization and alignment that were later endorsed by the Paris Declaration on Aid Effectiveness in 2005 and subsequent international meetings. This Program has become a locally owned program for the Health Sector in a comprehensive and coordinated manner. In saying this, the Program has been able to move towards the use of country systems. Acknowledgement on the successes of the Health SWAp program at CDC level by the 54

72 Prime Minister and the Forum Combat Peer Review for Samoa November 2013; have been some of the main highlights. In fact it has been acknowledged over its lifetime that the program provided the main modality in implementing the HSP ( ). In other words, the Health SWAp Program provided oomph or substance that gave life to the HSP. In respect of programme outcomes of the Program, it is visible that the SWAp has delivered benefits in a number of important areas especially with the services, goods and civil works provided for the sector. Despite these, the Program appears to have fallen short of expectations in a number of program outcome areas. Some of the more significant examples are: Failure to develop and implement a comprehensive health information system and currently no HIS Policy which has had knock-on adverse impacts on the ability to acquire and analyse health sector activity data for M&E purposes. Slow progress with the three key capital works projects pharmaceutical warehouse; orthortics and prosthetics facility; renovation and extension of the MTII Hospital and the renovations of the Primary Health Care Centre. Delays in establishing improved cervical and breast cancer screening programmes. Aside from that, fiscal analysis has confirmed that there has been fiscal space for the Program given its limited demands for incremental recurrent costs (ie. SWAp Unit Operational Costs), which are estimated at less than 1.5percent of the health budget during the five years of the Program. Sustainability of the Program is also being enhanced by improved composition of recurrent budget allocations, including through increased allocations for maintenance and critical operation costs for service delivery, particularly in rural areas and for rural outreach. The Government's strong ownership of the HSP and the policy and investment priorities embedded therein are critical to its overall sustainability. The participatory bottom up planning process has increased health stakeholder ownership of the Program. The technical soundness of policy and investment choices included in the MTEF affect the sustainability of overall health sector financing. Specifically the focus on both primary and secondary prevention to contain costs, develop staff capacities, address key skill constraints and strengthen core government systems as part of Program implementation will mean that the future costs of the health sector in dealing with disease and disability will be substantially reduced by the Program's investments. Those which should enhance sustainability of health sector financing. Consolidating and strengthening approaches for capacity development, including (i) rationalization of the selection and oversight of technical assistance (to ensure it contributes to building local capacity and training of local staff); and (ii) establishing innovative arrangements for capacity building, including use of national policy, planning, budgeting, and reporting procedures to the extent possible. All such developments which has ensured that the health sector strategically aligns its planning and progress reporting according to the Government of Samoa s Strategic Planning and Policy Framework. 8. Final actual expenditure against budget: Total Budget = SAT$81.4 million Total Expenditure as at June 30, 2015: SAT$61.4 million 55

73 Total Balance (Works and Goods) = SAT$20 million 56

74 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Australia (DFAT) and New Zealand (MFAT) response on the Implementation Completion and Results Report (IDA & IDA for the Health Sector Management Program Support Project and Additional Financing We are pleased to have worked in partnership with the Government of Samoa and the World Bank to implement this project, commonly known as the Health Sector Wide Approach (SWAP). The World Bank has produced a comprehensive Independent Completion Report (ICR). We welcome the opportunity to comment. We note that the evaluation is an independent Bank process and that the overall rating of moderately unsatisfactory is based on the Bank s review and rating criteria. We were pleased to attach Australia and New Zealand s own independent consultant to participate in this evaluation and we appreciate the flexibility shown by the World Bank to accommodate this. New Zealand and Australia initially funded this initiative from 2008 to 2013, and then agreed to a no-cost extension until December 2016 to ensure the completion of key infrastructure. The World Bank discontinued its involvement in the Project on December We urge that upmost effort be taken to ensure that the remaining infrastructure is completed to high quality. We are pleased that the ICR highlights the strong ownership by the Government of Samoa of this project. This ownership is evident by the establishment and ongoing role of the Health Programme Advisory Committee which continues to work across the sector and the flexibility of the programme to respond to the natural disasters which occurred during the program. The SWAP will have a positive legacy that will include: Improved policies and processes Improved infrastructure including a new Pharmaceutical Warehouse; Orthotics and Prosthetics facility; renovated MTII hospital and Primary Health Care Centre; and The purchase and upgrade of equipment for the hospitals and outreach. Australia and New Zealand continue to support the health sector through a range of activities, and both agencies are currently undertaking design of future assistance to support improved health outcomes in Samoa. The lessons learned from the Health SWAp are welcome and will be important in informing future engagement in the health sector by Australia and New Zealand. We note some of the key lessons for all parties highlighted in the ICR: More attention was required to clarify the roles and responsibilities of each health sector agency at the outset of the program. Despite this being recognised as high risk prior to the program commencing, it was not adequately addressed by all parties at that time. 57

75 More attention was required to address recurrent costs and asset maintenance associated with new initiatives. For example the Health Care Waste Management system. We recognise that this issue will require close attention as new health infrastructure is completed and is no longer within their defects liability period. Procurement challenges were encountered over the course of the Project. This resulted in delays and cost overruns and a requirement for ongoing capacity building. The lack of appropriate and measurable indicators and the lack of early establishment of baseline information impeded timely problem solving. This in turn made it difficult to draw on evidence to inform programming and resource allocation. Despite the stated emphasis of the Health Sector Plan on promotion, prevention and primary care services, resource allocation to these areas did not match ambition. The focus of the SWAP concentrated on the capital resources provided by development partners and how these were being utilised rather than on the total quantum of effort and resources available to the sector. This is evident in that the Medium Term Expenditure Framework (MTEF) had not been updated since Recent efforts by the sector to map the full range of development assistance to health sector agencies and how this assistance supports the achievement of their plans is an indication that this issue has been recognised and is in the process of being addressed. Looking ahead Australia and New Zealand note the importance of continuing to work in partnership with the Government of Samoa to achieve Samoa s vision of a healthier Samoa. In doing so, we note the importance of cooperation, coordination and mutual accountability and the need for regular dialogue, including at a senior leadership level. 58

76 Annex 9. List of Supporting Documents 1. Project Appraisal Document, May 29, Financing Agreement (C4432-WS), July 23, Amendment to Financing Agreement, Credit 4432-WS, January 28, Amendment to Financing Agreements Credit 4432-WS and 4721-WS, December 3, Joint Partnership Agreement, July 23, Amendment to the Partnership Agreement, Credit 4721-WS and Credit 4432-WS, February 5, Implementation Status and Results Reports No Aide Memoires The Economic Costs of Noncommunicable Diseases in the Pacific Islands: A Rapid Stocktake of the situation in Samoa, Tonga and Vanuatu, November Samoa Public Expenditure Review Notes - Taking stock of expenditure trends from FY06-FY12, March National Kidney Foundation Annual Report, Evaluation of Samoa Health Sector Management Programme (Health SWAp), Philip Davies, May Summary of Recommendations from previous reports on Health Sector M&E and HIS in Samoa, Joan Marfarlane, January Assessment of Monitoring and Evaluation for the Samoa Health Sector Program, Nick Davis, June Health Sector Plan, , Mid-Term Review Report, Ministry of Health,

77 Annex 10. List of Indicators and Arrangements for Results Monitoring (PAD) PART A: Health Sector Program Indicators Program Development Impact Indicators Use of the information Objective The proposed health sector Control of non-communicable diseases Outcome indicators will be used to program s aim in the Prevalence of diabetes evaluate the performance of the medium-term ( ) Improved maternal and child health health sector program in terms of would be to improve access Perinatal mortality rate achieving priority objectives of the to, and utilization of Universal access to reproductive health HSP. effective, efficient and quality services health services to improve the Adolescent birth rate Baselines will be established in the health status of the Samoan Control of communicable diseases first year and performance population. Incidence of water- and food borne reviewed on key indicators as part infections of the Health Summits. Injury prevention and management 24 Injuries in children (under 15 years) Intermediate Results Results Indicators for Each Component One per Component Health Promotion and Component One: These are proxy indicators for Primordial Prevention: Supporting the transformation People aged 18yrs and over overweight or obese behavior change in the Samoan population in line with objectives of the health sector from a medical model towards wellness orientation and Percentage of children under 1 year received at least one dose of measles vaccine of the HSP and the program. health promotion Improved medical waste management Quality Health Services: Improving the quality of health services through strengthened human resources, standards, supplies, equipment and infrastructure Improving Policy, Monitoring and Regulatory Oversight of the Health Sector: Support the MOH in its policy development, coordination and regulation of the sector. Component Two: Primary care utilization by gender, age, domicile Antenatal care coverage for at least one visit Proportion of Rheumatic Heart Disease patients complying with treatment Component Three : Staff mix and distribution according to national standards Evidence of performance monitoring leading to policy and regulatory action to improve health services These are proxy indicators for improvement in quality of health delivery These are proxy indicators of improved health sector stewardship. 24 To include S. typhi 60

78 Program Development Objective Impact Indicators Demonstrated outcomes of training plan by component Use of the information 61

79 PART B: Health SWAp Process Monitoring Indicators Expected Outcome Program Indicator Use Health SWAp development objective: To improve the effectiveness of the Percent health sector budgets and disbursements conform to policy objectives and HSP priority areas 25 Government of Samoa in managing and implementing the HSP based on the use of results from sector performance monitoring Share of annual outpatient visits by poorest quintile of population (indicator of equity of access HIES) Intermediate objective 1: Policies and plans implemented according to agreed priorities Intermediate objective 2: Greater efficiency in use of resources Intermediate objective 3: National ownership and commitment to health sector program Intermediate objective 4: Results of performance monitoring used in shaping the program implementation Intermediate objective 5: Improved health sector governance MTEF and related Procurement Plan updated and adjusted based on recommendations from sector reviews Key Sector Partners corporate Plans and Government investments aligned with HSP priorities. Percentage of SWAp Program funds released according to agreed schedule Number of reported drug stock-outs by facility 26 Health expenditure as percentage of govt. expenditure Stakeholders participation in program planning and implementation reviews Disaggregation of data by sex, age and domicile enhances planning for services MOH Financial Audits submitted on time and action plan agreed for resolving outstanding issues. DHS and other statistical reports completed within Improved public expenditure management, allocating resources aligned to the priorities. Improved execution of budget by programs. Reduced duplication and better deployment of limited resources National ownership of the development process. Linking program results with budget allocation in iterative process Mitigate the risk of corruption. Improving accountability 25 Proportion of budget expenditure on public health, primary care, hospital services and administration. 26 Any drug out of stock for more than one week at any health facility. 62

80 Expected Outcome Program Indicator Use stated timeframe and made public 63

81

82 Arrangements for Results Monitoring Intermediary SWAp Outcome Indicators Percent health sector budgets and disbursements conform to policy objectives and HSP priority areas Target Values Baseline YR1 YR2 YR3 YR4 YR5 SWAp begins 27 70% 90% Biennial Share of annual outpatient visits by the poorest quintile of the population MTEF and related Procurement Plan updated and adjusted based on recommendations from sector reviews To be determined by HIES 28 Complete Complete Complete Complete Biennial Annual Key Sector Partners corporate plans and government investment aligned with HSP priorities Complete Complete Complete Complete Annual 27 SWAp will introduce mechanism for Government to better align its resources with priorities. This indicator could be tracked using proportion of budget expenditure by the various program activities. 28 HIES will be completed by end The indicator is intended to measure equity and may be revised after the first HIES. At the March 2009 Summit baseline and target will be established for the rest of the Program. 65

83 Intermediary SWAp Outcome Indicators Percentage of SWAp Program funds released according to agreed schedule Number of reported drug stock-outs by facility Target Values Baseline YR1 YR2 YR3 YR4 YR5 Health allocation as percentage of govt. budget expenditure 70% 100% 100% 100% Annual Stakeholders participation in program planning and implementation reviews Disaggregation of data by sex, age and domicile enhances planning for services 15% 15% <20% of baseline 15% <10% of baseline >15% <10% of baseline >15% Annual Annual MOH Financial Audits submitted on time and action plan agreed for resolving outstanding issues. None Full Full Full Full Biennial DHS and other statistical reports completed within stated timeframe and made public None Partial Full Annual Audits Complete Complete Complete Complete Annual 66

84 Intermediary SWAp Outcome Indicators Target Values Baseline YR1 YR2 YR3 YR4 YR5 Complete June 30, 2009 Complete Complete Complete 67

85 Annex 11. Summary of Agreements Date Name/No Summary July 23, 2008 Financing Agreement (C4432-WS) Original financing agreement SDR 1.9 million January 28, 2009 Amendment to Financing Agreement, Credit 4432-WS Extends dates for development of TORs to conduct DHS from June 2009 to December 2009 April 13, 2010 Recommendations of Statutory Provides additional financing in the amount of SDR 2 Committee for Credit 4721-WS (Closing Package) million (100% allocated to goods, works, subgrants, consultant services, training and interim operating costs) May 20, 2010 December 3, 2013 Supplemental Letter ref. Performance for Credit 4721-WS (Closing Package) Amendments to Financing Agreements, Cr WS and 4721-WS Provides Results Framework for the Project, and notes that A major contribution expected from the project is to further strengthen the mechanism of consultation among health sector partners to set priorities for resource allocation monitoring and making adjustments in the Program based on the results. Extends closing date to Dec 18, Financing reallocated so that it finances goods, works, subgrants, consultant services, training and interim operating costs for the Project. In original FA, $1,286,500 had been unallocated. February 5, 2014 Amendment to the Partnership Agreement Extends closing date to Dec 18,

86 Annex 12. SWAp Governance and Administration The following table, developed from information contained in the Joint Partnership Arrangement between Pool Partnership and Samoa for C4432-WS 29 (July 23, 2008) includes a summary of the governance and administration of the SWAp: Organization/Agency Ministry of Finance Ministry of Health Health Program Steering Committee SWAp Coordination Unit in the MOH Role Functions as the Executing Agency. MOF along with the MOH will have responsibility for management of the financial contributions of the DPs. Overall responsibility for Program administration, financial management and procurement. It will also have overall coordinating role for the implementation of the Program. Policy oversight and oversee Program steering and implementation. The HPSC will comprise high level GoS officials (MOH, MOF, NHS, Ministry of Women Community and Social Development, and MESC), coordinating representatives from the DPs, a representative of other health institutions, a representative of the private sector, and a representative of NGOs. The HPSC will provide coordination among local stakeholders and external dialogue and coordination with DPs. The HPSC will meet on a quarterly basis or more frequently if the need arises. The SCU will be the Secretariat for the HPSC. Overall coordinating role for the Program. The SCU, led by the SWAp Coordinator, will be responsible for day-to-day Program administration, procurement and financial management. Each activity will be managed by an Output Manager, consistent with GoS budget and accountability structure. The SCU is responsible for: i. Procurement and financial management under the Program. ii. Coordinating the preparation of the Annual Program of Work (APW) and budget forecasts. POWs will be prepared in cooperation with the Component Focal Persons and will follow the MTEF process. 29 The agreement was between GOS and AusAID, NZAID, IDA, UNFPA, UNICEF, and WHO. 69

87 Coordinating Development Partners (CDPs), i.e. representatives of the Development Partners iii. Coordinating the preparation of quarterly and annual Program Management Reports (PMRs). PMRs will be prepared in consultation with component focal persons. Each PMR will contain information in relation to procurement, financial management, a summary of implementation of activities (by component), identification of problems and/or issues, recommendations as to how they may be solved, and an update of the rolling annual procurement plan (APP). iv. Coordinating in collaboration with MOH, MOF, and the HPSC, the organization of the annual Health Summits (in March) and Joint Reviews (in September). v. Provide Secretariat support to the HPSC. vi. Prepare the Financial Monitoring Reports (FMRs) for the Program. Appointed to support coordination among the Development Partners and the GoS and act as a focal point of communication between all Partners. The CDP role will be shared between one bilateral development partner and one multilateral development partner at any one time and rotated every eighteen months. The JPA also specified the focal person for each of the components: Component 1 will be the ACEO for Health Promotion and Prevention in MOH Component 2 will be the General Manager of NHS Component 3 will be the ACEO for Strategic Development and Planning in the MOH 70

88 71