Document of The World Bank

Transcription

1 Document of The World Bank FOR OFFICIAL USE ONLY Report No: IMPLEMENTATION COMPLETION REPORT (SCL TF-29286) ON A LOAN/CREDIT/GRANT IN THE AMOUNT OF US$ MILLION TO THE INDONESIA FOR A ID-FIFTH HEALTH PROJECT April 18, 2005 Health, Nutrition, and Population Sector Unit Human Development Department East Asia and Pacific Region This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization.

2 CURRENCY EQUIVALENTS (Exchange Rate Effective April 2005) Currency Unit = Indonesia Rupiah (Rp) Rp. 1 million = US$ 200 US$ 1 = Rp 5,000 FISCAL YEAR Government of Indonesia: April 1 - March 31 ABBREVIATIONS AND ACRONYMS Academies - Nurse-training schools (normal course) ADB - Asian Development Bank Bapelkes - Regional or provincial training center (in-service) CAS - Country Assistance Strategy CJ - Central Java province CK - Central Kalimantan province CPS - Central Project Secretariat DI - Deployment innovations DO - Development objective DSP - Personnel planning software EU - European Union GOI - Government of Indonesia HPV - Fifth Health Project HR - Human resources HWS - Health Workforce and Services Project (PHP III) IBI - Indonesia Midwives Association ICB - International Competitive Bidding ICR - Implementation Completion Report IDI - Indonesia Medical Association IND - Indonesia Kanwil - Provincial Office of MOH LG - Local government LRC - Learning Resource Center MOF - Ministry of Finance MOH - Ministry of Health MTR - Mid-Term Review NBF - Not Bank-financed NCB - National Competitive Bidding NOL - No Objection Letters OSCA - Objectives Structured Clinical Assessment PAD - Project Appraisal Document PHP - Provincial Health Project (PHP III is same as DWS) PM - Project management

3 Poltekkes - PPNI - Puskesmas - QA - Repelita - SIMKA - SPK - SS - Unit swadana - Health polytechnic schools (pre-service) Indonesia Nurses Association Health center Quality Assurance GOI 5-year Plan Health Personnel Information System Nurse-training schools (accelerated course) South Sulawesi province Autonomous facility Vice President: Country Director Sector Manager Task Team Leader/Task Manager: Jemal-ud-din Kassum, EAPVP Andrew Steer, EACIF Fadia Saadah, EASHD Puti Marzoeki, EASHD

4 INDONESIA ID-FIFTH HEALTH PROJECT CONTENTS Page No. 1. Project Data 1 2. Principal Performance Ratings 1 3. Assessment of Development Objective and Design, and of Quality at Entry 2 4. Achievement of Objective and Outputs 4 5. Major Factors Affecting Implementation and Outcome Sustainability Bank and Borrower Performance Lessons Learned Partner Comments Additional Information 28 Annex 1. Key Performance Indicators/Log Frame Matrix 29 Annex 2. Project Costs and Financing 31 Annex 3. Economic Costs and Benefits 35 Annex 4. Bank Inputs 37 Annex 5. Ratings for Achievement of Objectives/Outputs of Components 39 Annex 6. Ratings of Bank and Borrower Performance 40 Annex 7. List of Supporting Documents 41

5 Project ID: P Team Leader: Puti Marzoeki Project Name: ID-FIFTH HEALTH PROJECT TL Unit: EASHD ICR Type: Core ICR Report Date: May 29, Project Data Name: ID-FIFTH HEALTH PROJECT L/C/TF Number: SCL-43740; TF Country/Department: INDONESIA Region: East Asia and Pacific Region Sector/subsector: Health (85%); Sub-national government administration (8%); Central government administration (7%) Theme: Health system performance (P); Rural services and infrastructure (S); Decentralization (S) KEY DATES Original Revised/Actual PCD: 06/19/1997 Effective: Appraisal: 03/26/1998 MTR: Approval: 07/14/1998 Closing: Borrower/Implementing Agency: Other Partners: GOI/MOH STAFF Current At Appraisal Vice President: Jemal-ud-din Kassum Jean Michel Severino Country Director: Andrew D. Steer Dennis de Tray Sector Manager: Fadia Saadah Maureen Law Team Leader at ICR: Puti Marzoeki ICR Primary Author: James Knowles 2. Principal Performance Ratings (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HL=Highly Likely, L=Likely, UN=Unlikely, HUN=Highly Unlikely, HU=Highly Unsatisfactory, H=High, SU=Substantial, M=Modest, N=Negligible) Outcome: Sustainability: Institutional Development Impact: Bank Performance: Borrower Performance: U UN M U U QAG (if available) Quality at Entry: Project at Risk at Any Time: ICR U

6 3. Assessment of Development Objective and Design, and of Quality at Entry 3.1 Original Objective: The Project s development objectives (DO) were to achieve greater efficiency of utilization and equity of distribution of health personnel, increase the skills of health professionals, and improve the quality of health professional practices. Instrumental objectives to achieve these goals were to strengthen the decentralization of planning, deployment and management of personnel, and the role of the private sector; to strengthen the licensing of health personnel, and support professional associations and quality assurance; to improve the quality of pre-service and in-service training and strengthen the accreditation of training institutions. The Project s objectives were clear, realistic and important for the country/sector and were consistent with the Bank s Country Assistance Strategy (CAS), which included the following objectives: Greater decentralization through increased autonomy and accountability Increasing efficiency and productivity in the public and private sectors Enhancing equitable development The Project was also designed to respond to major sector issues affecting health professionals as identified at the time by the MOH planning unit. These issues included: Low productivity on average of the health work force Skewed rural-urban distribution of health personnel and very low utilization of public health services by the poor Rapidly growing private sector in the absence of effective quality assurance mechanisms Uneven management and planning skills and inadequately developed information systems in the public sector below the central level and in the private sector more generally An increased demand for better-quality health services, due to increasing incomes and changing epidemiological requirements Need to re-orient the public sector away from service delivery in general to focus more on regulation, standard-setting, public health, and service provision to the poor and other under-served groups The Project also supported the Government s strategy for addressing health manpower issues, developed in connection with preparation for Repelita VII (i.e., the Government s 5-year plan). This strategy was based on the following principles: Decentralize planning and management of health professionals from the central to the provincial and local levels Shift planning from a target-driven to a market-driven and facility needs-based approach Encourage the growth of demand-responsive private and quasi-private providers to serve urban and higher-income groups, re-focusing government services on public health and on under-served groups Undertake policy and regulatory changes and redirect resources to increase the quality of training institutions and programs Adopt policies, incentives, and public sector programs to address distribution and equity deficiencies Introduce finance, budgeting and management innovations to promote greater efficiency, quality and equity of services Adopt training, programmatic and financing policies to encourage greater private sector participation in producing and utilizing human resources in health

7 Designing a project to assist the Government in implementing this strategy was challenging for several reasons. First, there was evidence from previous Bank health projects that health manpower problems needed to be addressed through decentralization and required simultaneous efforts on both the demand and supply sides. Second, although there were numerous problems to address, the 1997/98 economic crisis began during the Project s design and heightened the need for effective measures that would contribute to increased efficiency and more effective deployment of the professional health work force in the public and private sectors. There was an urgent need for public funds to focus on basic quality care and services for the poor while strengthening the private sector to cover expanding demand for quality services in the future. Third, although there was some recently completed and well-regarded economic and sector work on health manpower issues in Indonesia, baseline data were lacking in many areas, and there was no tried and tested formula for addressing health manpower issues in a setting as complex and rapidly changing as Indonesia s. Lastly, the Project involved important policy reforms (i.e., decentralization of many health manpower functions, piloting of innovations in health manpower incentives, development of new licensing and accreditation mechanisms) whose successful implementation required broad-based support. During design, agreement was reached with the appropriate central units about the functions they were prepared to devolve to the provincial and district levels, as well as the appropriate timing of these changes. To keep the Project manageable and in light of the constraints noted above it was decided to focus the Project on a limited geographical area (three provinces), on the pre-service training of nurses and midwives (i.e., pre-service medical doctor training was not supported by the Project), on demand-side financing initiatives that were clearly focused on health manpower utilization and training (i.e., excluding broader health insurance or managed care initiatives), and without any health service components apart from improving health manpower. 3.2 Revised Objective: The Project s objectives were not revised during implementation. However, several Project amendments canceled some loan funds (see Table 1) and extended the project completion date by four months. 3.3 Original Components: As originally designed, the Project included two distinct groups of project components, one at the province level and one at the central level. At the province level, the project components included: (A) improving the efficiency of utilization and equity of distribution of health professionals (B) improving the quality of health professionals practices (C) improving quality of health professional training (D) province project management. At the central level, the project components included: (E) building central support for health professionals and (F) project management. These project components appeared to be a reasonable response to the issues and constraints noted above, were appropriate to the capacity of the implementing agency and took into account relevant lessons learned in the 15 previous Bank health sector projects since The limited financial management capacity at the province level was to be addressed through training provided by the center during the early stages of implementation. 3.4 Revised Components: The Project s components were revised by a Loan Agreement amendment on 6 August 2002 following recommendations made during the Mid-Term Review. Project management (PM) at both the province and central levels was consolidated into a single Component D (formerly central PM was included in a separate Component F, which was dropped by this amendment). Component E was revised to include a greater emphasis on human resource policy development. These changes did not amount to a restructuring of the Project, i.e., the changes were not major

8 3.5 Quality at Entry: The Project s quality at entry is rated unsatisfactory. Although the project design included some positive elements, it lacked an effective monitoring and evaluation framework and plan and was also conceptually unsound inasmuch as it implicitly assumed that large Project investments in human resources would automatically translate into better quality services. The project s benefits were also skewed in favor of the non-poor (i.e., Government health workers). However, some positive design elements included consistency of the Project s objectives with the CAS and with the Government s own priorities and strategy in the health manpower area. The Project design also effectively built on prior Bank lending experience in the health sector and on recently completed economic and sector analyses. Attempts were made to keep the Project simple and focused and to achieve agreement on key policy reforms prior to implementation. Environmental and social safeguards were considered not to apply to this Project. An important weakness at entry was the absence of an appropriate monitoring and evaluation framework and plan. The Project DO listed in the Project Appraisal Document (PAD) Project Design Summary does not completely match the project DO described in the text of the PAD, and some of the performance indicators listed for the Project s DO are also inappropriate. As discussed below, the listing of outputs in the Project Design Summary (as well as the choice of output indicators in some cases) was also deficient. The Project Implementation Plan covering the first 18 months of Project activities did not include any monitoring and evaluation plan, and no baseline or target values were provided for any of the performance indicators. There was no strategy in the project design to measure the project s effects or ultimate impact. Indicators and targets were subsequently replaced in the Loan Amendment of 6 August 2002 (following the Mid-Term Review) by a shortened list of 7 performance indicators without targets. Regular project monitoring and reporting began in mid-2002, limiting feedback to Project staff (and to the Bank) about the Project s progress to its final two years. Project design was appropriately realistic in recognizing that progress in implementing recommendations to re-deploy personnel would require incentives (the Deployment Innovations). It was also realistic in recognizing that incentives were necessary to motivate re-deployed staff to provide services to the poor (for example, the targeted vouchers that were piloted in some Project districts). However, the project design was naïve in believing that the substantial investments in human resources (for example, the on-site and off-site training, including substantial QA training) would automatically translate into better-quality services. The project design also under-estimated the obstacles to using public funds to contract with private providers. A final problem with the project design is that the benefits were skewed to the non-poor. There was no analysis of the likely beneficiaries of the substantial human resources investments made by the project. For example, non-poor fellowship recipients (3,640 employees of the public health system) received a large share of the Project s benefits. A broader group of non-poor public health workers received per diems for attending the many workshops and short-term training courses supported by the Project, while central staff received support for preparing numerous guidelines and modules. There was no clear mechanism in the Project design for re-directing these benefits to the poor. An alternative project design might have provided a substantial increase in purchasing power to the poor and to populations residing in remote areas and supplemented this with performance-based incentives linked to the quality of care actually provided in poor and remote areas. This would have created a demand on the part of providers for more effective training investments. 4. Achievement of Objective and Outputs 4.1 Outcome/achievement of objective: - 4 -

9 Achievement of the Project s DOs and outputs is rated unsatisfactory, based on an unsatisfactory rating for two of the three Project DOs, as discussed below. The Project faced some major constraints during its first three years that affected its implementation, including the aftermath of the 1997/98 economic crisis and the major Government-wide decentralization that was implemented in Under the decentralization, districts suddenly became the front-line providers of health services in an atmosphere of substantial uncertainty and confusion. Many former central and province-level personnel were transferred to other posts, and these unanticipated personnel changes posed many problems during the Project s implementation. Although the Project was designed to support a continuing process of decentralization, it did not anticipate the extent and rapidity with which many important functions related to health manpower were transferred to districts. It was only at the mid-term review that adjustments were made to the project design to match the substantial changes that occurred in government organization since the Project began. This delay may have been, in part, due to the uncertainty around decentralization that continues to be an issue. Another factor may have been attempts by the center and the provinces to maintain control over functions that had in fact been devolved to the district level. At the same time, it is important to note that serious problems with project implementation were brought to the attention of the Government in Bank-prepared Aide Memoires as early as 1999 (well in advance of the January 2001 decentralization), and many of these problems were not resolved (for example, weak project management in SS and delays in procuring needed technical assistance). Some of these issues were easy to solve and MOH could have addressed some of them (e.g. project management issues) but there was a lot of delay in taking action from the centeral level at MOH. Project implementation improved following the Mid-Term Review in 2001 and was rated by the Bank as having been satisfactory during the Project s last year of implementation. However, the subsequent improvements in implementation performance were not enough to make up for the lost time during the Project s early years since achievement of the Project s DOs would have required early and successful achievement of several key outputs, for example, the pilot Deployment Innovations (DI). Consequently, the Project was not able to achieve its DOs. The late achievement of some Project outputs also reduced prospects for sustainability, as discussed in Section 6.1. Despite an overall unsatisfactory rating in terms of the achievement of the Project s DOs, the Project managed to complete a number of important activities, and some of these have been subsequently incorporated into an ongoing World Bank financed project the Health Workforce and Services (Provincial Health III) Project, while others are expected to be supported by other donors or LGs (see Section 6.2). The Project has also engendered significant institutional development (see Section 4.5). A background study prepared for the Project s Final Evaluation notes the following problems in assessing its effectiveness in achieving its DOs, including the absence of clear indicators and targets for DO-level indicators, a dearth of project-related information prior to 2002, and the Project s focus on 20 pilot districts following the 2001 Government-wide decentralization and the Project s Mid-Term Review. With these limitations in mind, the Project s achievement of each of its DOs is assessed as follows: Objective 1: Achieve greater efficiency of utilization and greater equity in the distribution of health personnel Achievement of this DO is rated unsatisfactory. A lot of project activity focused on improving HR planning and management at the province level. At project completion, most districts reported that they were using a systematic approach to HR planning and management based on Project-provided software (SIMKA and DSP). However, the available evidence suggests that these reports overstate the degree to which province and district-level HR planning and management practices improved during the Project. The reported HR - 5 -

10 plans were of uneven quality, and it is unclear how many were actually implemented (only 50% reportedly identified funding sources). Much of the training focused on how to use the project-provided software rather than on developing the capacity to analyze the information they produced or on developing advocacy skills that could be used to effect needed policy changes on the basis of that information. The Project s problems in obtaining reliable and timely information, even from the 20 pilot districts, suggests that the information systems were not functioning very well even during the Project s final two years. Staff incentive schemes and DIs were implemented very late in the Project, and most were still at an embryonic stage at project completion. Some of the incentive schemes were not clearly linked to improved staff performance (for example, schemes piloted in Central Java that used formulas based on qualifications and seniority to allocate user fee revenue among staff). Local Government (LG) support for the incentive schemes and DIs was reported to be very limited (in part because delayed implementation made it difficult to establish their benefits to local leaders). There is no systematic evidence that the DIs resulted in a strong link between staff deployment and workload or that they led to a re-deployment of staff in favor of remote and under-served areas (except possibly in the case of some midwives in Central Kalimantan). Objective 2: Increase the skills of health professionals in Project provinces Achievement of this DO is rated satisfactory. All training institutions in the three provinces were improved and accredited. Sister School Programs were successfully established in all three provinces (although after delays). All SPKs (short-course nurse-training schools) were successfully converted to academies (normal-course nurse-training schools) except for two private SPKs in Central Java. However, questions were raised during the Final Evaluation about whether their upgraded status reflects improved teacher qualifications, facilities or equipment. Most of the training provided by the Project was competency-based. Learning resource centers were established in all three provinces (although their utilization rates were low at project completion). In addition to institutional improvements, a large number of public health staff (3,640) received fellowships from the Project to support long-term training, mostly to improve clinical skills. As a result, it is clear that the individual capacity of a significant proportion of the health workforce in the three provinces was improved. Preference appeared to be given to staff from remote locations in awarding fellowships (except in South Sulawesi), and Central Java (CJ) even developed a special training program for students from poor areas. What is less clear is whether the long-term training responded to actual HR needs as well as the extent to which these needs had been successfully met at project completion. Objective 3: Improve the quality of health professionals practice Achievement of this DO is rated unsatisfactory. Most districts claim that they have systems in place to license health professionals. However, these systems were in place prior to the Project in most districts, and it is unclear what is involved in obtaining and retaining a license and what percentage of a district s workforce is currently licensed. It is also unclear that the licensing systems currently in place make any significant contribution to the quality of health professionals practice. In addition to licensing systems, the Project contributed to some strengthening of professional associations for doctors, nurses and midwives. Many health professionals registered with their professional associations in the three Project provinces. However, the role of the professional associations in contributing to the improved quality of the services provided by their members remains unclear. A substantial amount of quality assurance (QA) training was provided by the Project. One or more staff of most puskesmas in the 20 pilot districts (85%) received QA training, with almost one-half of the puskesmas - 6 -

11 reporting that at least one doctor, nurse and midwife received QA training. Most puskesmas (80%) reported that they were operating QA programs at the end of Many districts also reported that they had established QA teams. However, it is unclear whether these QA training activities had any effect on the actual quality (including the health effectiveness) of the services provided (the Mid Term Review notes that the major change produced by the QA training related to client satisfaction, i.e., service with a smile). There is no evidence that the QA training had any effect on service utilization rates. Since many public providers also have private practices, it is possible that the QA training might also have improved the quality of services in the private sector. Unfortunately, there is no information available on how private practices were affected by the Project. The Mid-Term Review noted that some trainees felt the QA training was more theoretical than practical. There is also concern that the reported number of hospitals operating QA programs was already decreasing toward the end of the Project. 4.2 Outputs by components: I. Province Components: A. Improve Efficiency of Utilization and Equity of Distribution of Health Professionals The outputs of this component are rated unsatisfactory, consistent with the unsatisfactory rating of the outputs of its two sub-components and with that of the corresponding DO (#1). A.1. Planning and Management The outputs of this sub-component are: 1) enhanced province capacity for human resource (HR) planning and management, 2) simplified and rapid information systems and analysis with integrated information on staffing and program performance (SIMKA), and (3) an authorized staffing level system (DSP) for use in recruiting and allocating staff according to demonstrated local demand for services. Performance in achieving these outputs is rated unsatisfactory. Some improvement in province-level HR planning and management occurred during the last two years of the Project as indicated by an increased percentage of districts reported to have a HR planning document (except for CK, which registered a constant rate of 50%). However, the planning documents were of greatly varying quality. An increasing number of staff reportedly received short-term leadership and management training. However, there is very limited information on the types and quality of the training received, the types of persons trained and on whether the training contributed to improved HR planning and management skills. In addition, some fellowships were awarded to province and district-level staff to study HR planning and management. Efforts to develop province-level HR planning and management capacity proceeded more slowly than expected and were also more narrowly focused than expected on training in the use of the SIMKA and DSP software. The latter were successfully developed by the Project, and province/district personnel were successfully trained in their use (particularly in CJ). However, the Bank s April 2002 Review Mission noted that SIMKA was not operational in the remaining two Project provinces (because districts were not providing the necessary information to provinces). Although districts and health centers have received training in the use of DSP tools to assess staff needs and have applied these tools in many cases at the puskesmas level, the information provided has not been sufficient to alter staff allocations in such a manner as to improve equitable access (although some transfers of personnel were achieved in urban areas). A.2. Innovations to Support Allocative Efficiency and Productivity The outputs of this sub-component are: 1) improved incentives for more efficient and equitable deployment - 7 -

12 and performance in rural areas, 2) self-sustaining private practitioners (midwives, nurses, medical doctors), 3) support for Unit Swadana (autonomous facilities), and 4) DIs at the district and province levels. Performance in achieving these outputs is rated unsatisfactory. Significant progress was achieved in the first and fourth outputs, but only after substantial delays. A system of performance-based staff incentives was successfully developed and piloted in CJ, while successful DIs were designed and piloted in both Central Kalimantan (CK) and in South Sulawesi (SS). Development of the DIs took more time than expected due to lack of capacity at the district level, late and in some cases inadequate TA provided by the center, uncertainty in connection with the decentralization process, and the challenges posed by the changed role of higher administrative levels from previously directive to supportive. Although the DIs piloted by the districts have provided valuable experience and lessons learned, they have remained as pilots and have had little impact on local government staff deployment policies (mainly because of their delayed implementation but also due to the absence of advocacy skills among HR staff). Little or no progress was registered in the second and third outputs. For example, interventions to strengthen the role of private providers were not implemented at all, apart from the efforts to improve the regulatory framework (discussed below). Most Unit Swadana proposals were not approved because they appeared to concentrate on investments to strengthen business viability without giving sufficient attention to the Government s equity, efficiency and quality objectives. Swadana initiatives were also not strongly supported by some LGs out of concern that they would lose control over user fee revenues and by some government agencies out of concern that they would lead to increased fees. However, at least one pilot Puskesmas Swadana was successfully established through the Project in CK. A background report prepared for the MTR indicates that 13 hospitals and 49 puskesmas were converted to Unit Swadana. The ICR mission was not able to assess on what these numbers are based. B. Improve Quality of Health Professionals Practices The outputs of this component are rated unsatisfactory, consistent with the rating of its three sub-components and with that of its corresponding DO (#3). B.1. Strengthen the Regulatory System The outputs of this sub-component are: 1) an improved regulatory and licensing system for health professionals (medical doctors, nurses and midwives initially) in conjunction with health professional organizations that would be piloted in the three Project provinces and 2) an accreditation system for training institutions and continuing education courses that would be pilot tested in the three Project provinces in conjunction with health professional organizations. Performance in achieving these outputs is rated unsatisfactory. Project efforts to deliver the first output encountered many obstacles. An improved regulatory and licensing system was successfully developed and piloted in one Project province (CJ), but it has not yet been implemented (and its future status is uncertain, given the newly approved law on Medical Councils). Most districts reported that a licensing system for health professionals was in place at the time of project completion (100% of districts in CJ and CK and 60% in SS). However, these are the systems that existed prior to the Project in many cases. Licensing is still based largely on fulfillment of various administrative requirements instead of on the basis of an assessment of professional skills. The second output was partially achieved. A system for accrediting training institutions was developed, and all training institutions in the three pilot provinces (4 pre-service Poltekkes and 4 in-service Bapelkes) - 8 -

13 achieved accreditation by the end of The accreditation instrument developed under the Project was legalized through a Ministerial Decree mandating its use at the national level. CJ also produced guidelines for continuing education and is reportedly establishing a Quality Council to accredit health facilities. However, little progress was made by professional associations in developing programs of continuing education linked to professional certification, although this process is reportedly ongoing. B.2. Strengthen Professional Associations The output of this sub-component is strengthened province-level offices of professional associations for doctors, dentists, pharmacists, midwives and nurses to develop and support in-service training, registration, certification, networking and professional communications for members and to assist in the accreditation of training institutions. Performance in achieving this output is rated unsatisfactory. Provincial offices were established in all three Project provinces and in most pilot districts, and many members were registered by all five associations. Unfortunately, there is no reliable information on the number of health professionals eligible to register in each province or district, so coverage cannot be accurately assessed. At project completion it was still difficult to get accurate data on registered members for the project performance indicators due to inconsistencies in the data reported. B.3. Promote Quality Assurance for Health Services The outputs of this sub-component are: 1) further development of a quality assurance system for puskesmas in the Project provinces as initiated under Health Project IV, and 2) quality assurance systems introduced into district hospitals, based on the Total Quality Management concept. Performance in achieving these outputs is rated unsatisfactory. It is clear that a substantial amount of QA training was provided through the Project, and an increasing percentage of puskesmas are reported to be implementing QA. However, there is ambiguity in the performance indicators used to measure achievement of these outputs. Specifically, it is unclear what is meant by the Percentage of staff trained in quality assurance. There appears to have been double counting of some staff who attended more than one QA training course, and it is unclear that attendance at only one QA course signals any meaningful level of proficiency. Similarly, it is unclear what is meant by a facility implementing QA. No independent assessment was done to evaluate the impact of the QA training on the quality of service delivery or on consumer satisfaction. C. Improve Quality of Health Professional Training The outputs of this component are rated marginally satisfactory, consistent with the rating of its two sub-components and with that of its corresponding DO (#2). C.1. Improve Training Methods and Programs The outputs of this sub-component are: 1) competency-based training and case-based training methodologies developed and linked to quality assurance programs and to the province context, 2) learning resource centers established in selected district offices, 3) Sister School relationships established linking foreign and local training institutions, 4) distance learning initiatives and support for professional networks, and (5) training-related monitoring and evaluation studies. Many health staff (including teachers at these training institutions) received fellowships for long-term training under this sub-component

14 Performance in achieving these outputs is rated marginally satisfactory. Several of the outputs were achieved under this Sub-component, although with considerable delays in some cases. The biggest achievement was the improvement of pre-service training at the Poltekkes, three of which (out of a total of 4 in the three Project provinces) participated in the Sister School Program that was finally implemented after delays. Competency-based training and case-based training methodologies were developed and implemented in all of the training institutions in the three provinces (although the Project s Final Evaluation reports that Problem-Based Learning is still used more widely in CJ). Learning resource centers (LRC) were established in all three provinces, but their utilization has been low. Only one province (CJ) developed a marketing strategy and business plan for its LRC. Many health staff also received foreign and domestic training under this Sub-component (as discussed below). Progress in establishing distance learning initiatives was limited. Some early progress was made to develop a distance-learning program for midwives and nurses, but there was disagreement between the Pusdiklat (Center for Workforce Training) and Pusdiknakes (Center for Workforce Education) over which institution has the mandate to develop distance learning and over how to build clinical practice into the system developed initially by Pusdiklat. C.2. Strengthen Training Institutions The output of this sub-component is improved pre-service and in-service training programs, training institutions (regional, provincial and district-level), teachers, and materials used in training nurses and midwives. Many health staff (including teachers at these training institutions) received fellowships for long-term training under this Sub-component. Performance in achieving this output is rated marginally satisfactory. All of the training institutions (pre-service and in-service) in the three Project provinces were substantially strengthened during the Project, in part through assistance received under the Sister School Program. Project-supported improvements included: 1) updated curriculums, 2) plans for future upgrading of faculties, 3) equipment, books, journals and other learning materials, 4) workshops and studies, and 5) signed memoranda of understanding between training institutions and clinical training sites in district hospitals and in puskesmas. Substantial long-term training was provided by the Project. A total of 3,640 fellowships for long-term training (domestic and overseas) were awarded, mostly to health center, hospital, and district health office staff to upgrade their clinical skills. According to a Training Plan developed during loan negotiation, fellowships could be used both to improve the clinical skills of health workers and to upgrade the qualifications of training institution faculty. However, most fellowships were awarded to health workers to upgrade their clinical skills. During the MTR, it was decided to re-orient the long-term training to strengthen human resource planning, public health and policy development capacity at the district level. It was also decided to award more fellowships to poor candidates and to candidates from remote areas. Nurses in remote health centers were subsequently given preference in the award of fellowships in CJ and CK, but not in SS. CJ in particular developed a special training program to send students from poor families (40 initially) to midwifery schools. The expectation is that these students will return to work in their home villages after completing their studies. A consultant evaluation of the Project s fellowship program (a tracer study) found that most fellowship recipients (99%) successfully completed their studies and returned to their previous provinces of residence

15 However, many fellowship recipients (particularly those who went for overseas training) subsequently changed jobs, and some even transferred to central positions. The evaluation noted weaknesses in fellowship management D. Province Project Management The output of this component is province-level project management and administration. Performance in achieving this output is rated unsatisfactory. Although project management performance was reported to be satisfactory from the beginning in one Project province (CK), problems were consistently reported in two others (SS and CJ). In SS, for example, it was reported that the PM team was unresponsive to the Central Project Secretariat (CPS), to the Bank s procurement auditors, and to a Bank team during a project review visit. In CJ, the problems noted included weak leadership and low provincial commitment to the Project. The CPS delayed addressing the management problems in SS for more than a year. II. Central Components E. Build Central Support for Health Professionals The outputs of this component are rated unsatisfactory, consistent with the rating of its two sub-components. E1. Strengthen Planning Capacity and Technical Support The outputs of this sub-component are: 1) enhanced central capacity in human resources planning, 2) an efficient, user-friendly and integrated data management system for health professionals, and 3) technical support to provinces in implementing the Project. Performance in achieving these outputs is rated unsatisfactory. This component was restructured during the MTR because central-level activities were mostly limited to developing modules and guidelines for which there was uncertain demand at the district level (since following decentralization the center no longer had the power to instruct districts to adopt them). At the same time, decentralization required adjustments to existing workforce policies in which the center could play an important role. Accordingly, the central role was restructured to focus on policy development and on the provision of technical assistance requested by districts. Despite this change, no meaningful changes in workforce policy had been achieved by the Project at completion. E2. Improve the Professional Environment The outputs of this sub-component are: 1) systems and procedures for regulation, licensing and certification, 2) strengthened capability of national professional associations to communicate with members, to develop professional cohesiveness and to support national quality initiatives. Performance in achieving these outputs is rated unsatisfactory. Central efforts to enact a statute regulating medical and dental practice encountered political opposition at the national level (for example, to the establishment of special medical courts to review allegations of malpractice). When Government efforts stalled, the law was introduced in Parliament and approved after project completion (October 2004). Before the law can be implemented, however, it will be necessary to establish both a national Medical Council and a national Ethics Council

16 The second output was achieved to varying degrees by each professional organization. National professional associations were substantially strengthened by assistance received from the Project. For example, the medical doctor association (IDI) implemented an information system at its web site, where it also provides free services, a health journal and continuing education. However, IDI still has problems collecting dues from its members. In contrast, the successful midwife association (IBI) has no problems collecting dues from its members. Progress in developing the nurses association (PPNI) was hampered by the absence of full-time staff during the Project s first few years. One encouraging development is that the five professional associations have established a Joint Secretariat and are currently working together on common problems at the national and local levels. The professional organizations are responsible for non-formal continuing education (public training institutions are responsible for formal continuing education). Although continuing education is not yet required to maintain a license (there is no re-certification system in place), the professional organizations are developing examinations that might eventually be used to re-certify their members professional standards. F. Project Management The output of this component is central-level project management (PM) and administration. Performance on this component is rated unsatisfactory. There were improvements in central PM following the MTR (although Project records for 2002 and 2003 indicate that less than 50% of planned activities were implemented and only about 40% of budgeted funds were disbursed). Problems in PM included: Failure to report on Project progress or to develop an effective reporting system during the first three years of the Project Inadequate staffing of the CPS Delays in procuring needed technical assistance Inadequate monitoring of provincial and district activities Failure to develop and achieve effective annual work plans Failure to document achievement of several project outputs through independent assessments Failure to follow up audit findings promptly Failure to provide sufficient training to province staff on Bank procurement procedures Delayed implementation of actions requested in the MTR The main responsibility of the CPS was for PM while a Technical Team was responsible for providing technical guidance to the Project. However, the Technical Team did not function during the first half of the Project, and the CPS had to cover project technical issues as well. This added responsibility adversely affected their PM performance. At MTR, the Technical Team was strengthened by appointing a focal point for each of the Project s four centers. A team of international consultants was recruited, with one consultant assigned to each center. Following completion of the international consultant contract, a team of national consultants took over. These changes contributed to improved PM and technical performance during the post-mtr period, but it was too late to compensate for earlier delays. 4.3 Net Present Value/Economic rate of return: The economic analysis in the PAD included a cost-benefit analysis providing an illustrative rate of return from the project. The economic analysis was based on strong assumptions about the Project s health impact, i.e., that after 5 years the project would eliminate 5% of the observed difference between the healthy years of life in the project area and that observed in OECD countries for a group of nine priority

17 disease groups and 2.5% of the difference for remaining disease groups. Under these assumptions, the IRR for the overall Project was estimated to be 45% (99% in Central Java, 28% in South Sulawezi and 4% in Central Kalimantan). Unfortunately, the absence of suitable data makes it impossible to re-estimate the Project s IRR or even to assess the reasonableness of several of the key assumptions used in preparing the original estimates. However, the fact that the Project s implementation was substantially delayed and that it was actually implemented in only 20 pilot districts (compared to about 70 total districts in the three Project provinces), suggests that a revised estimate of the IRR would be considerably lower than that reported in the PAD. 4.4 Financial rate of return: The PAD included an analysis of fiscal impact that included projections under three alternative scenarios of the Project s recurrent cost burden in each province at the completion of the Project. This indicator is relevant in assessing the Project s financial sustainability. The alternative scenarios differed in the assumed changes in each province s health budget during the 5-year period of the Project. For example, the best-case scenario assumed a 20% decline in the first year, no change in the second year, and a recovery to 5% annual growth thereafter. Under these assumptions, Project-related recurrent costs as a percentage of province health budgets were projected to be 1.6% in Central Java, 8% in Central Kalimantan, and 3% in South Sulawezi. By comparison, under the worst-case scenario (declines in province health budgets of 30% and 20% respectively in the first and second years, a 5% decline in the third year, no change in the fourth year, and a recovery to 5% growth in the Project s fifth year), the corresponding projected recurrent cost burdens would be 3% in Central Java, 13% in Central Kalimantan and 5% in South Sulawezi. Unfortunately, there are no comprehensive data available on consolidated Government health expenditure in the three Project provinces following the 2001 decentralization that could be used to assess actual fiscal experience during the period of project implementation ( ). 4.5 Institutional development impact: The Project is rated to have had a modest institutional development impact on the following institutions: district health offices and district governments more broadly, training institutions and professional associations. Impact on district-level health facilities is less clear. The Project has not contributed significantly to the creation of strong central leadership behind sector reform and has not created effective mechanisms for addressing HR issues in a post-decentralized setting. Districts have benefited greatly from the training and experience gained in formulating proposals for and implementing the DIs. Effective ties have been developed in several pilot districts between District Health Offices and both other LG offices and Local Parliaments. For example, in some districts, the Project-supported SIMKA software has been used by other district offices to manage information on their personnel. These new relationships are particularly important in the context of Indonesia s decentralization of many governmental functions to the district level. Training institutions have also been substantially upgraded and strengthened by the Project, as evidenced by the fact that all training institutions in the three Project provinces have achieved accreditation. Improved working relationships have also been established between pre-service training institutions (Poltekkes) and clinical training sites (both district hospitals and puskesmas). The professional associations have also been strengthened through the Project. It is reported that they have been transformed from political organizations to effective professional associations and are now working together to address common problems. District-level health facilities (district hospitals and puskesmas) have been strengthened by the QA and