IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-77370) ON A LOAN IN THE AMOUNT OF US$77.82 MILLION TO THE REPUBLIC OF INDONESIA FOR A

Transcription

1 Public Disclosure Authorized Document of The World Bank Report No: ICR Public Disclosure Authorized Public Disclosure Authorized IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-77370) ON A LOAN IN THE AMOUNT OF US$77.82 MILLION TO THE REPUBLIC OF INDONESIA FOR A HEALTH PROFESSIONAL EDUCATION QUALITY PROJECT (HPEQ) Public Disclosure Authorized Health, Nutrition and Population Global Practice Indonesia Country Management Unit East Asia and Pacific Region June 15, 2015

2 CURRENCY EQUIVALENTS (Exchange Rate Effective June 15, 2015) Currency Unit = Indonesian Rupiah US$1.00 = IDR 13, IDR1.00 = US$ FISCAL YEAR January 1 December 31 ABBREVIATIONS AND ACRONYMS AIPKI BAN-PT BAPPENAS BHE CBC CBT CPCU CPS DGHE FAP GDP GoI HEI HELT HRH HWS ICT IMC IDR IFLS ISR KKI KPI LAM-PTKes LASA LPUK-NAKES Asosiasi Institusi Pendidikan Kedokteran Indonesia (Association of Indonesian Medical Schools) Badan Akreditasi Nasional Perguruan Tinggi (National Accreditation Body for Higher Education) Badan Perencanaan Pembangunan Nasional (Ministry of National Development Planning) Board of Higher Education Competency-based curriculum Competence-based testing Central Project Coordination Unit Country Partnership Strategy Directorate General of Higher Education Financial Assistance Package Gross Domestic Product Government of Indonesia Higher Education Institutions Higher Education Long Term Human resources for health Health Work Force and Services Project Information, Communications and Technology Indonesian Medical Council (see KKI) Indonesian Rupiah Indonesia Family Life Survey Implementation Status and Results Konsil Kedokteran Indonesia (Indonesia Medical Council) Key performance indicators Lembaga Akreditasi Mandiri Pendidikan Tinggi Kesehatan Directorate of Learning and Student Affairs Lembaga Pengembangan Uji Kompetensi Tenaga Kesegatan MCQ MDG MoF MoEC MoH MoNE MoRTHE NAA NACEHhealthPro NCBE OSCE OOP PAD PBL PD-DiktiKes PDO PHC PUSKESMAS PUSDIKNAKES SATKER TOR UNCEN UNDANA Multiple Choice Questions Millennium Development Goal Ministry of Financing Ministry of Education and Culture ( ) Ministry of Health Ministry of National Education (before 2011) Ministry of Research, Technology and Higher Education (2014 now) National Accreditation Agency (then LAM-PTKES) National Agency of Competence Examination for Health Professions National Competency-based Examination Objective Structured Clinical Examination Out-of-pocket Project Appraisal Document Problem Based Learning Database for Health Higher Education Project Development Objectives Primary health care Pusat Kesehatan Masyarakat (Community Health Center) Pusat Pendidikan Tenaga Kesehatan (Center for Health Workforce Education) Satuan Kerja (DGHE s Working Unit) Terms of References Cendrawasih University Nusa Cendana University Vice President: Country Director: HNP GP Director: Practice Manager: Project Team Leader: ICR Team Leader/Primary Author: Axel van Trotsenburg Rodrigo A. Chaves Olusoji O. Adeyi Toomas Palu Puti Marzoeki Edson Correia Araujo i

3 Republic of Indonesia Health Professional Education Quality Project CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design Key Factors Affecting Implementation and Outcomes Assessment of Outcomes Assessment of Risk to Development Outcome Assessment of Bank and Borrower Performance Lessons Learned Comments on Issues Raised by Borrower/Implementing Agencies/Partners Annex 1. Project Costs and Financing Annex 2. Outputs by Component Annex 3. Economic and Financial Analysis Annex 4. Bank Lending and Implementation Support/Supervision Processes Annex 5. Beneficiary Survey Results Annex 6. Stakeholder Workshop Report and Results Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Annex 9. List of Supporting Documents MAP ii

4

5 A. Basic Information Country: Indonesia Project Name: Health Professional Education Quality Project Project ID: P L/C/TF Number(s): IBRD ICR Date: 06/15/2015 ICR Type: Core ICR Lending Instrument: SIL Borrower: Original Total Commitment: Revised Amount: USD 77.82M Environmental Category: C REPUBLIC OF INDONESIA USD 77.82M Disbursed Amount: USD 67.18M Implementing Agencies: Ministry of National Education, Directorate General of Higher Education Cofinanciers and Other External Partners: B. Key Dates Process Date Process Original Date Revised / Actual Date(s) Concept Review: 11/25/2008 Effectiveness: 12/30/ /09/2009 Appraisal: 06/01/2009 Restructuring(s): 02/28/ /16/2014 Approval: 09/24/2009 Mid-term Review: 10/29/ /29/2012 Closing: 12/31/ /31/2014 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Risk to Development Outcome: Bank Performance: Borrower Performance: Satisfactory Moderate Satisfactory Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Satisfactory Government: Satisfactory Quality of Supervision: Satisfactory Implementing Agency/Agencies: Satisfactory Overall Bank Performance: Satisfactory Overall Borrower Performance: Satisfactory iii

6 C.3 Quality at Entry and Implementation Performance Indicators Implementation Performance Indicators QAG Assessments (if any) Rating Potential Problem Project at any time (Yes/No): Problem Project at any time (Yes/No): DO rating before Closing/Inactive status: D. Sector and Theme Codes No Quality at Entry (QEA): None Yes Satisfactory Sector Code (as % of total Bank financing) Quality of Supervision (QSA): Original None Health Tertiary education Actual Theme Code (as % of total Bank financing) Health system performance Other human development E. Bank Staff Positions At ICR At Approval Vice President: Axel van Trotsenburg James M. Adams Country Director: Rodrigo A. Chaves Joachim von Amsberg Practice Manager/Manager: Toomas Palu Juan Pablo Uribe Project Team Leader: Puti Marzoeki Puti Marzoeki ICR Team Leader: ICR Primary Author: Edson Correia Araujo Edson Correia Araujo F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The specific Project Development Objectives are to strengthen quality assurance policies governing the education of health professionals in Indonesia. This will be achieved by: 1) rationalizing and assuring competency-focused accreditation of public and private health professional training institutions; 2) developing national competency standards and testing procedures for certification and licensing of health professionals; and 3) building institutional capacity to employ results-based grants for encouraging the use of accreditation and certification standards in the development of medical school quality. iv

7 Revised Project Development Objectives (as approved by original approving authority) (a) PDO Indicator(s) Indicator Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years Indicator 1 : The establishment of an independent National Accreditation Agency (NAA) Value BAN-PT-IMC Joint Establishment of the quantitative or Y Commission Charter of the NAA Qualitative) Date achieved 12/31/ /31/ /31/2014 Comments Achieved. NAA (created as LAMP-PTKes) was established and legally ratified by the (incl. % MoEC Decree in October achievement) Indicator 2 : The establishment of an independent National Agency for Competency Examination of Health Professionals (NACEHealthPro) Value Establishment of the quantitative or NA Charter of the Y Qualitative) NACEHealthPro Date achieved 12/31/ /31/ /20/2013 Achieved. The legal framework for the agency (created as LPUK-NAKES) was Comments established in 12/20/2013. The agency had exercised its function, as a task force under (incl. % the CPCU, to develop the methodology for competence examination since early achievement) implementation. The percentage of health professional schools (medical, dentistry, nursing, and Indicator 3 : midwifery) that have gone through the accreditation process and have publicized the results Value quantitative or Qualitative) zero medicine (100%), dentistry (100%), nursing (48%), midwifery (18%). medicine (29, 42%), dentistry (10, 42%), medicine (21), nursing (52, dentistry (9), nursing 10%), (82), midwifery (56). midwifery (33, Total 168 schools 10%). Total 124 schools Date achieved 12/31/ /31/ /31/ /31/2014 Comments (incl. % achievement) Surpassed - the total number of schools accredited exceeded the target. The number of accredited medical and dentistry schools was (slightly) lower than the target and exceeded the target for nursing and midwifery. Baseline data only included percentages. Indicator 4 : The percentage of graduates of health professional schools (medicine, dentistry, nursing, and midwifery) passing national competency testing at the first attempt Value quantitative or Qualitative) medicine (71.67%), dentistry (81.69%), nursing (0%), midwifery (0%) medicine (84%), dentistry (90%), nursing (65%), midwifery (65%) medicine (72.47%), dentistry (92.31%), nursing (57.81%), midwifery (64.65%) Date achieved 12/31/ /31/ /15/2014 v

8 Comments (incl. % achievement) Indicator 5 : Achieved for dentistry and marginally under the target for midwifery. Target not achieved for medicine and nursing. However, for nursing and midwifery the baseline values were zero. The mean test score of graduates from the Financial Assistance Package (FAP) recipient schools who have taken the National Competence Test Value quantitative or Qualitative) Date achieved 12/31/ /31/2014 Comments (incl. % achievement) There was no target defined. The mean test scores of non-fap recipients was at baseline and at project closing (14% increase against 11% increase among grantees). (b) Intermediate Outcome Indicator(s) Indicator Indicator 1 : Value (quantitative or Qualitative) Value (quantitative or Qualitative) Original Target Values (from Baseline Value approval documents) Completion status of preparatory activities for the establishment of the NAA NA NAA has independent and adequate budget to conduct accreditation and has access to adequate numbers of suitably trained assessors Formally Revised Target Values Y Actual Value Achieved at Completion or Target Years Date achieved 12/31/ /31/ /15/2014 Comments (incl. % achievement) Achieved. Indicator 2 : Standard of Competencies and Standard of Education for the four health professions are available. medicine (standards released by the IMC in 2006), dentistry (standards nursing (both released by the IMC in standards completed 2006), nursing (draft of both standards available), midwifery (standard of competencies released by MoH in Draft standard of education available) and legalized), midwifery (standard of education completed and legalized) added: three professions (nutrition, pharmacy and public health) Date achieved 12/31/ /31/ /31/ /31/2013 Y vi

9 Comments (incl. % achievement) Indicator 3 : Value (quantitative or Qualitative) Achieved. The standards of competencies were achieved for pharmacy and public health disciplines. Both have conducted the national competency examination trial. Standards not completed for nutrition, expected to conduct trial in Accreditation instrument for the four health professional schools are ready for use. medicine (international peer review of instrument), medicine (draft available) dentistry (piloting and dissemination), Y nursing (piloting and dissemination), midwifery (piloting and dissemination) Date achieved 12/31/ /31/ /15/2014 Comments (incl. % Achieved. achievement) Indicator 4 : Number of trained assessors Value (quantitative or Qualitative) 0 medicine (86), dentistry (39), nursing (105), midwifery (105). added: 40/profession for nutrition, public health and pharmacy medicine (132), dentistry (34), nursing (123), midwifery (70), public health (17), nutrition (16), pharmacy (41) Date achieved 12/31/ /31/ /31/ /31/2014 Comments Surpassed for medicine, nursing and pharmacy. The target was not achieved for (incl. % dentistry (slightly under the target), midwifery, public health and nutrition. achievement) Indicator 5 : Introduction of CBT and OSCE for NCE Value (quantitative or Qualitative) N CBT and OSCE for all four professions. CBT for all four professions. OSCE for medicine and dentistry, CBT try out for the added professions, OSCE preparation only CBT for medicine, dentistry, and nurse. Paper based test for midwifery and diploma of nursing (DIII). OSCE for medicine and dentistry, and under preparation for for pharmacy. pharmacy. Date achieved 12/31/ /31/ /31/ /31/2014 Comments Achieved. OSCE was added formally to the medicine and dentistry NCE in CBT (incl. % had been implemented for medicine and dentistry and for nursing. Paper based test was achievement) implemented for midwifery a diploma of nursing (DIII). Not achieved for nutrition. Indicator 6 : Value (quantitative Number of National OSCE trainers 0 72/profession 72/profession (medicine and medicine (4,950), dentistry (84) vii

10 or Qualitative) dentistry only). 36/Pharmacy. Date achieved 12/31/ /31/ /31/ /31/2014 Comments (incl. % achievement) Indicator 7 : Value (quantitative or Qualitative) Surpassed. The total number of OSCE trainers exceeded the target for medical and dentistry. Target no achieved yet for pharmacy. Number of National MCQ Item Writers 0 1,044/profession added: 36/profession for nutrition, public health and pharmacy medicine (254), dentistry (650), nursing (828), midwifery (675), pharmacy (219), public health (141) and nutrition (144). Date achieved 12/31/ /31/ /31/ /31/2014 Comments (incl. % achievement) Target not achieved for the four initial professions (medicine, dentistry, nursing and midwifery). Surpassed for the three additional professions (nutrition, public health and pharmacy). Indicator 8 : Number of National OSCE Item Writers Value (quantitative or Qualitative) 0 72/profession 72/profession (medicine and dentistry), 36/pharmacy. medicine (221), nurse (48), dentistry (650), pharmacy (39). Date achieved 12/31/ /31/ /31/ /31/2014 Comments Surpassed. The total number of OSCE item writers exceeded the targets for all (incl. % professions. achievement) Indicator 9 : Number of MCQ Item Writers and reviewers Value (quantitative or Qualitative) NA 1,044/profession added: 36/profession for nutrition, public health and pharmacy medicine (216), dentistry (650), nursing (828), midwifery (675), pharmacy (78), public health (84) and nutrition (132). Date achieved 12/31/ /31/ /31/ /31/2014 Comments (incl. % achievement) Target not achieved for the four initial professions (medicine, dentistry, nursing and midwifery). Surpassed for the three additional professions (nutrition, public health and pharmacy). Indicator 10 : Number of OSCE Value (quantitative or Qualitative) 0 1,044/profession 1,044/professio n (medicine and dentistry only), 36/pharmacy medicine (86), dentistry(63), pharmacy (39). Date achieved 12/31/ /31/ /31/ /31/2014 Comments Target not achieved for the medicine and dentistry. Surpassed for pharmacy. (incl. % viii

11 achievement) Indicator 11 : Number or percentage of medical schools receiving finance support to strengthen the program Value (quantitative or Qualitative) Date achieved 12/31/ /31/ /15/2014 Comments (incl. % achievement) Achieved. G. Ratings of Project Performance in ISRs No. Date ISR Actual Disbursements DO IP Archived (USD millions) 1 04/12/2010 Satisfactory Satisfactory /27/2010 Satisfactory Satisfactory /04/2011 Satisfactory Satisfactory /28/2012 Satisfactory Satisfactory /08/2013 Moderately Unsatisfactory Moderately Satisfactory /24/2013 Moderately Unsatisfactory Moderately Unsatisfactory /21/2014 Moderately Unsatisfactory Moderately Unsatisfactory /08/2014 Moderately Satisfactory Moderately Satisfactory /24/2014 Satisfactory Moderately Satisfactory H. Restructuring (if any) Restructuring Date(s) Board Approved PDO Change ISR Ratings at Restructuring DO 02/28/2013 N MU MS IP Amount Disbursed at Reason for Restructuring & Key Restructuring Changes Made in USD millions Upon request of the Government of Indonesia, the following changes were made: (i) expand the activities to include three additional professions (pharmacy, nutrition and public health) and two additional medical schools (UNCEN and UNDANA); (ii) reallocate the proceeds of the Loan (redistribution of funds across components to accommodate changes); and (iii) update the reference to the national regulation on procurement for the clarifications relating to National Competitive Bidding Procedures ix

12 Restructuring Date(s) Board Approved PDO Change ISR Ratings at Restructuring DO 09/16/2014 N MS MS IP Amount Disbursed at Reason for Restructuring & Key Restructuring Changes Made in USD millions (to accommodate inclusion of two universities UNCEN and UNDANA) Following the Government of Indonesia request, the following change was made: reallocation of funds from component 3 (FAP) to component 1 (training, workshops, incremental operating costs, research expenditures, consultants' services and goods under component 1) and component 4 (training, workshops, incremental operating costs, research expenditures, consultants' services and goods under component 4) I. Disbursement Profile x

13 1. Project Context, Development Objectives and Design 1. The Health Professional Education Quality (HPEQ) was approved on September 24, 2009 in the amount of US$77.82 million and became effective on December 09, Further financial support in the amount of US$8.9 million was provided by the Government of Indonesia (GoI). 1.1 Context at Appraisal 2. Background. At the time of project preparation, Indonesia ranked as a lower-middle income country, with a gross domestic product (GDP) per capita of US$2,272. Indonesia s economic growth was 4.6% while the percentage of the population living in poverty (less than US$2 a day) was 17.8% (fall of 4.7% in the period of ) 1. Indonesia is a geographically dispersed country, spread over 17,000 islands with 34 provinces and approximately 500 districts. It has enormous variety within its borders, ranging from densely populated urban cities, in the island of Java, to sparsely populated rural and remote islands of Nusa Tenggara Timur and Maluku. Since 2001, the provision of health care services was decentralized with regional governments having full discretion over how the health services were distributed and budgets are allocated. The central government role was restricted to the distribution of financing and regulation of the health care sector. 3. Main issues in the health sector. In 2009, the total health spending accounted for only 2.8% of the GDP, with a per capita health expenditure of US$64.2, which was significantly lower than the average in the East Asia and Pacific countries (4.9%), excluding Japan, Korea, Singapore and Australia. The private sector was the dominant source of health care financing with 49% of the total health expenditure paid through household out-of-pocket (OOP) payments, higher than the average of 37.8% in the region. The public health spending accounted for 36.1% of the total health spending, much lower than the average of 51.9% in the neighboring countries (e.g. Malaysia, 59%) At the time of project appraisal there were three major health insurance schemes in the country, namely, Jamkesmas, Jamsostek and Askes. Jamkesmas was a government financed health insurance program for the poor and the near poor, covering about a third of the population. The beneficiaries were identified by a combination of means testing and local government eligibility criteria. The Jamkesmas faced some important challenges, particularly related to beneficiary entitlement awareness (reports suggest coverage rates among the poor households of only 47.6%) and leakages (evidence available suggests leakages of about 52.4 %,) Indonesia had, and still has, a mixed public-private provision of health care services. Basic primary health care was provided by the public sector via a network of Puskesmas (Pusat Kesehatan Masyarakat, or health center at the sub-district level ), which serve a catchment area at the sub-district level of about 25,000 to 30,000 individuals, and by the private sector via private clinics and individual health professional private practices. Each Puskesmas was expected to 1 Data source: 2 Data source: 3 Harimurti et.al. (2013). Nuts and Bolts of Jamkesmas. Indonesia s Government Financed Health Coverage Program. The World Bank, Washington DC, January

14 provide outpatient care, health prevention and health promotion services, to function as a gatekeeper to the health care system, and to ensure continuity of care. Nevertheless, a third of the Puskesmas also provided inpatient services. Secondary care was provided by both public and private health care providers. Approximately half of the secondary-care hospitals and a third of the hospital beds (estimated 163,000 in 2008) were private. 4 Despite these numbers, more than half of the inpatient visits in 2009 occurred in the public sector (57%). 6. At the time of project appraisal the Indonesian health sector faced major human resources (HRH) challenges. Although the total availability of health personnel was not low, there were issues regarding HRH distribution, skill mix and quality of health care personnel. The production of new physicians had grown steadily with a peak in the 2008/2010 period when the production was approximately 9,000 new physicians/year. 5 In terms of distribution of health professionals, there were imbalances across provinces and across rural and urban areas. 6 HRH quality was a growing concern in Indonesia during project preparation. The main source of evidence was the 2007 Indonesia Family Life Survey (IFLS) vignettes, which measured the diagnostic and treatment ability among doctors, nurses and midwives. The IFLS found a low percentage of correct responses to vignette questions: 45% for antenatal care, 62% for child curative care, and 57% for adult curative care. 7. The trends in the quality of the services provided by the health care professionals was associated with the fast expansion in the number of private schools. At the time of project preparation 57% of the medical schools in Indonesia were private and over half of 7,000 doctors graduated from private schools. The expansion of schools, also observed in other professions, notably in nursing and midwifery, was not accompanied by improvements in quality standards. One-third of medical schools were not accredited and only a quarter received the highest accreditation standard given by the Indonesia Directorate General of Higher Education (DGHE). According to the Association of Indonesian Medical Schools (AIPKI), in 2007 only 50% of students passed the national based test examination with a passing score of 45 out of At the time of project preparation there were already some initiatives that placed the foundation for establishing quality assurance system for health professional education. 7 The Medical Practice Act (2004) supported the establishment of the Indonesia Medical Council (KKI) which, in 2006, produced the standards of competencies for doctors and the standards for medical education. These were the basis for the DGHE of the Ministry of National Education (MoNE) to require all medical schools to implement a competency-based curriculum (CBC) along with the adoption of problem-based learning and the integration of various medical disciplines. However, given the different capacity of the schools, at the time of project development, the CBC was implemented in an unstandardized manner across schools, which resulted in a large variation in the quality of education across medical schools. 4 BDEHR (2010). HRH Registration MoH: Jakarta. 5 Data source: 6 Anderson et al. (2014). The production, distribution, and performance of physicians, nurses, and midwives in Indonesia: an update. HNP GP Discussion Paper 91324, World Bank Group, Washington, DC. 7 The National Education System Act (2003), Medical Practice Act (2004) and the Lecturers Act (2006) provided the legal basis for improving the quality of Indonesian doctors. 2

15 9. The Medical Practice Act (2004) also established the accreditation of medical and dentistry schools as a mandate to measure quality of education. The accreditation function was under the responsibility of the National Accreditation Body for Higher Education (BAN-PT) under MoNE. BAN-PT faced challenges as an independent accreditation body given its financial dependence on MoNE, and in terms of developing specific instruments and processes more consistent with the characteristics of the medical and dentistry education. From June 2007 all graduates of medical schools were required to take the National Competency-Based Examination (NCBE) before obtaining their certification. 10. The regulatory framework to ensure quality of nurses and midwives education was much less developed than for doctors and dentists at the time of project preparation. The accreditation of nursing and midwifery schools was undertaken by both the Center for Health Workforce Education of the Ministry of Health (PUSDIKNAKES, MoH) and by the BAN-PT without a common approach nor criterion and there was no formal entity certifying the quality of graduating midwives and nurses. After finalizing the undergraduate program, graduates were allowed to practice without going through a nationally standardized competence testing process. 11. Government strategy. The project objectives and interventions were highly relevant as it supported the strengthening of the quality assurance system of health professional education, one of the priorities of the government's health sector medium-term development plan. They were also consistent with the Higher Education Long Term Strategy (HELTS) stating that quality assurance should be internally driven and institutionalized; and that quality improvement should aim at producing quality outputs and outcomes as part of public accountability, while BAN-PT, professional associations and other independent agencies could play a key role in conducting an objective external control based on standards. 12. Rationale for Bank s assistance. The project was aligned with the World Bank Country Partnership Strategy (CPS) , which supported the reform of the education sector from early childhood education to higher education and teacher upgrading. The project was also aligned with the CPS objectives in the health sector which aimed to improve quality, coverage, and utilization of health services, especially for the poorest 40%. The CPS identified quality and access to health services determinants of the slow progress towards the attainment of key Millennium Development Goals (MDGs), specifically maternal mortality rates (which was among the highest in the region). Broadly, the project was aligned with the sectoral core engagement component, which focused, among other things, on improving accountability, effectiveness of the schools and on strengthening human resource capabilities through medical/health education Since early 1990s the Bank provides financial assistance to build capacity of higher education institutions in Indonesia. As noted in the PAD, these experiences have been successful and the government allocated own budget to support and institutionalize them. HPEQ also followed up the Health Work Force and Services Project (HWS), closed on December The HWS included a sub-component on enhancing the quality of medical education, more specifically it supported the MoNE to: i) increase its institutional capacity to organize and manage 8 World Bank (2008). Indonesia Country Strategy and Program

16 medical education; ii) improve the quality of formal medical education; and iii) enhance the learning and teaching environment for both undergraduate and postgraduate medical education Original Project Development Objectives (PDOs) and Key Indicators 14. The specific PDO was to strengthen quality assurance policies governing the education of health professionals in Indonesia. This was to be achieved by: i) rationalizing and assuring competency-focused accreditation of public and private health professional training institutions; ii) developing national competency standards and testing procedures for certification and licensing of health professionals; and iii) building institutional capacity to employ results-based grants for encouraging the use of accreditation and certification standards in the development of medical school quality. 15. Progress towards achieving the PDOs were to be measured against the following key performance indicators (KPIs): The establishment of an independent National Accreditation Agency (NAA); The establishment of an independent National Agency for Competency Examination of Health Professionals (NACEHealthPro); The percentage of health professional schools (medical dentistry, nursing, and midwifery) that have gone through the accreditation process and have publicized the results; The percentage of graduates of health professional schools (medical, dentistry, nursing, and midwifery) passing national competency testing at the first attempt; The mean test score of graduates from the Financial Assistance Package (FAP) recipient schools who have taken the National Competence Test. 16. The progress of the project implementation were to be measured against the following intermediate indicators: Completion status of preparatory activities for the establishment of the NAA; Standard of Competencies and Standard of Education for the four health professions are available; Accreditation instrument for the four health professional schools are ready for use; Number of trained assessors; Introduction of Competence-Based Testing (CBT) and Objective Structured Clinical Examination (OSCE) for National Competency-Based Examination (NBCE); Number of National OSCE trainers; Number of National Multiple Choice Questions (MCQ) Item Writers; Number of National OSCE Item Writers; Number of MCQ Item Writers and reviewers; 9 HWS ICR, June

17 Number of OSCE Item Writers and reviewers; Number or percentage of medical schools receiving finance support to strengthen the program. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 17. The PDO was not modified during the project implementation period. Nevertheless, the results indicators were revised to reflect the changes made during the 2013 project restructuring. This revision led to the expansion of the project through the incorporation of three professions (pharmacist, nutritionist and public health specialist) in components 1 and 2 and two medical schools (UNCEN and UNDANA) in component Main Beneficiaries 18. The primary target groups were the Indonesian MoNE, the Indonesian Ministry of Health (MoH), schools of the seven professions included in the project (medical, dentistry, nursing, midwifery, nutrition, pharmacy, and public health) and the students of these programs. The ultimate beneficiaries of the project were the population of Indonesia who will benefit from better quality of health care that is now provided by certified health professionals trained by accredited schools. Finally, the Indonesian higher education system as a whole is expected to benefit from the quality assurance system developed under the project, which can serve as a model for other professions. 1.5 Original Components 19. Component 1: Strengthening Policies and Procedures for School Accreditation (US$7.184 million). This component was designed to support the GoI in improving the accreditation system of medical, dentistry, nursing, and midwifery schools and make it comparable to internationally recognized systems. The main objective was to create an independent accreditation body (the NAA) to conduct the accreditation of health higher education institutions in Indonesia. The subcomponents were: i) development of strategic framework, policies and procedures for accreditation; ii) development of standards of health education programs and standards of competencies; iii) development of accreditation instruments; iv) development of a pool of assessors; v) establishment of an accountability system for accreditation of health higher education institutions; and vi) data management to support the accreditation system. The component expenditure categories included organizing workshops, trainings, benchmarking the accreditation instruments, international and local technical assistance, conducting legal assistance studies and surveys, acquiring office equipment, IT, audio visual equipment and furniture. 20. Component 2: Certification of Graduates Using a National Competency-based Examination (US$ million) by: i) establishing an independent national competence examination agency (the NACEHhealthPro); ii) improving the methodology and management of the national competency-based examination; and by iii) developing an item bank networking system to support the national competence examination. The project aimed to support the NACEHhealthPro in establishing CBT and OSCE facilities in 12 medical schools that were expected to function as regional centers. The component expenditure categories included 5

18 information technology, and audiovisual, computer software, skills laboratory, office equipment and furniture, contracting international and local technical assistance, international benchmarking of competence standards, and organizing workshops and trainings. 21. Component 3: Results-based Financial Assistance Package (FAP) for Medical Schools (US$61.4 million). This component aimed to support selected medical schools to obtain the needed resources to improve medical education quality and capacity-building to achieve the national accreditation standards. The key principles guiding the FAPs allocation were: i) resultsbased allocation of resources; ii) fair competition among medical schools according to their capacity; and iii) partnership between leading and less advanced medical schools to build the capacity of the latter according to their specific needs. Medical schools were divided into three FAP schemes: Scheme A - FAP to support leading medical schools to build their international reputation and to strengthen Indonesia s global competitiveness in the area; Scheme B - FAP to support weak capacity new medical schools in achieving the medical education standards mandated by the Indonesian Medical Council (KKI) through partnerships with a leading medical school; and Scheme C - FAP to support moderate capacity medical schools in achieving the medical education standards mandated by the KKI. 22. The Board of Higher Education (BHE) was in charge of the FAP competitive selection process. This included establishing the guidelines for the FAP recipient selection and proposal approval process. BHE was also responsible for overseeing the FAP implementation, including the preparation of a FAP manual to guide the implementation of the grants. Only one selection process was expected during the lifetime of the project which had to be conducted during the first year of the project. Selected schools (grantees) had to implement the proposed program within three years. 23. FAP resources could be used to finance interventions in the following areas: i) improving the implementation of the competence-based curriculum (CBC) (which included: establishing studentcentered learning, early significant clinical exposure, adjusting student evaluation to be consistent with the CBC, and periodic review of the curriculum to ensure achievement of competencies); ii) strengthening teaching, training and learning facilities (which included: modernizing and strengthening libraries, computer centers to allow e-learning through e-libraries, and establishing electronic connectivity and high-speed internet facilities to allow networking among the medical schools); iii) development of the medical faculty (which included: support for recruitment systems and the training of clinical instructors, the training of examination item writers, the training of Problem Based Learning (PBL) problem writers, and the training of PBL tutors); iv) strengthening the Medical Education Unit (which included: staff recruitment system and physical/office facilities improvement, and staff capacity building through short- and long-term in-country and overseas training); and v) establishing a data management capacity (which included: building capacity to manage a database on medical education, data analysis and reporting for education planning and development, institution decision making and accreditation). 24. Eligible expenditures under the FAP included: workshops, teaching and laboratory equipment, degree and non-degree training, scholarships for poor students from underserved areas (maximum 10 students per school), information technology, technical assistance, minor building renovation and enhancing library collections. Medical schools under Scheme A could not allocate more than 20% of the total package to purchase goods, while those under Scheme B and C, the maximum allowed to purchase goods was set to 60%. 6

19 25. Component 4: Project Management (US$5.239 million). This was expected to fund the establishment of the Central Project Coordination Unit (CPCU) at the DGHE. Project resources were expected to finance incremental operating costs, project management consultant, office equipment, furniture and project monitoring and evaluation. 1.6 Revised Components 26. On December 20, 2012, the GoI requested a Level II restructuring of the HPEQ. This aimed to: i) expand the project activities to include three additional professions; and ii) include two universities under the FAP scheme. This request was approved on February 28, 2013 and the project components were changed as follows: Components 1 and 2 were expanded to incorporate three new professions: pharmacist, nutritionist, and public health specialist. This included financing the system for quality assurance of these three professions through building of stakeholder capacity and commitment, preparation of academic papers, formulation of standards and accreditation of instruments, development of examination blue prints and other items, and piloting the system; Component 3 was expanded to include two universities in the FAP scheme: Nusa Cendana University (UNDANA) in East Nusa Tenggara, and Cendrawasih University (UNCEN) in Papua. The two universities were not successful in the FAP competitive grant selection. However, given the strategic roles both universities played in meeting the needs for basic health services in Eastern Indonesia, the government requested a specially designed scheme for providing FAP grants to both universities (this was called the affirmative FAP ). 27. In the light of the above restructuring there was a reallocation of funds to accommodate the inclusion of three professions and two universities listed above. There were also changes in the procurement plan to accommodate the participation of the two new universities. 1.7 Other significant changes 28. The GoI requested two reallocations of funds across components, as follows: The first, as described above, on February 28, 2013, aimed to accommodate the inclusion of the three professions and the two universities. This request included reallocation of loan proceeds from component 3 (reduction of 6.7%) to components 1 (26% increase) and 2 (16% increase) see Annex 1, section c; The second, on September 16, 2014, aimed to: a) fund activities under component 1 in order to meet the KPI for this component (number of accredited schools); and b) fund activities under component 4 in order to complete the various evaluation and learning activities leading to project closing in December These reallocations resulted in transfer of proceeds of the loan from component 3 (3% reduction) to component 1 (9.5% increase) and to component 4 (17% increase) see Annex 1, section c. 7

20 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 29. Project Preparation. The project design was in line with national priorities and the World Bank CPS ( ) for Indonesia. Project preparation was relatively short, it took only eight months from the identification in late September 2008 to appraisal in mid June The project was effective on December 9, Project preparation included analytical work, including a stakeholder analysis to map key actors, their interests and potential course of actions. Preparation also involved a quality enhancement review (QER), which took place in May 2009 and provided recommendations to improve project s design. The QER panel endorsed the project objectives and design, with the following recommendations: i) simplify project design by reducing the number of KPIs; ii) reconsideration of the feasibility of components 1 and 2 implementation during the proposed timeline and caution with the use of KPIs that depends on policy changes through legislative processes; iii) adjustments in the school selection and matching criteria under the FAP component 3 (to allow schools to select their partners); and iv) reconsideration of the feasibility of the proposed FAP grant in driving institutional changes in the light of the institutional incentives and capacity to absorb the investment. 30. Design and Quality at Entry. The project built on previous successful experiences in improving quality of higher education in Indonesia through Bank operations and on the priorities identified under the HWS Project. 10 PDOs were defined with precision and KPI and intermediate indicators generally helped to assess project achievements. Major limitation in terms of design was the fact that grants were allocated only to medicals schools, representing 71% of the project s funds, but PDOs aimed at quality assurance for seven health professions. 31. The emphasis on improving medical schools capacity to meet the accreditation standards was justified because: i) medical schools had already an incipient system of accreditation prior to the project approval, consequently it made sense to allocate more resources to strengthen this system; ii) strengthening medical education through accreditation and competence-based examination would incentivize other professions to follow medical profession standards given the role and influence of the profession in the provision of health care services; iii) the investments made by medical schools on equipment, computer stations, etc., to conduct examinations (OSCE and CBT) would benefit other professions (by sharing mannequins for OSCE and using computer stations for CBT). 11 Additionally, important to point that the initial GoI request for a lending program focused only on the improvement of medical education. Midwifery and nursing were added during the identification mission, given their importance in achieving the national goals of maternal mortality rate (MMR) and infant mortality rate (IMR) reduction. But due to the anticipated challenge of conducting a competitive grant program involving large number of schools (around 700 nursing and 700 midwifery), it was decided to limit the block grant component (FAP) to medicine, to gain experience before expanding to other programs. 10 Previous Bank-funded operations include the Development of Undergraduate Education (closed in September 2002) and the Quality of Undergraduate Education (closed in March 2004) Projects. The Health Workforce and Services (HWS) project was a World Bank-financed project implemented by MoH and MoNE from Although HPEQ is funded through public resources, the GoI agreed to include the private medical schools in competitive selection for the FAP grant. This was a recognition that medical graduates regardless of their school of origin, would provide services to the entire population indeed 46% of the original block grant recipient were private schools. 8

21 2.2 Implementation 32. The project was able to implement all the envisaged activities and achieved the large majority of the output targets. The initial implementation was faced with some challenges at different levels, which are explained below and in more details in Section At government level: The main implementation challenges were: i) the changing of the ministry structure (from MoNE to MoEC); ii) the delays in the transfer of resources from Ministry of Financing (MoF) to MoEC due to delays in the approval of government s budget (this resulted that few activities could be implemented between the 6 th and 7 th implementation support missions); and iii) the lengthy process to finalize and build consensus around the legal framework for the functioning of the two agencies (NAA and NACEHhealthPro). Additionally, changes in the legal landscape during project implementation with the approval of new acts, the Higher Education Act and the Medical Education Act, required additional efforts to establish NAA and NACEHhealthPro. Due to these delays in the initial years of the project, during the mid-term review the project implementation progress was downgraded from moderately satisfactory to moderately unsatisfactory. 34. At implementing agency level: The main implementation challenge was the DGHE s limited capacity to supervise and implement activities due to scarce human resources. Despite this limitation, the DGHE was able to implement and monitor the project satisfactorily (see M&E section). 35. At university level: The capacity to implement the grants varied widely among schools. This was reflected in the heterogeneous levels of achievements across schools as well as different disbursement performance. Schools in A scheme, for example, generally had access to other sources of funds, local and international, and added to the restrictions and time limitation to use the FAP grant, some schools did not use resources available. 36. Despite these initial challenges, during the last 2 years the project implementation was largely on track to achieve its PDOs. At the time of project closing, most of the envisaged activities had been carried out and the majority of the project funds had been disbursed (86.26%). These are considerable achievements given the initial implementation delays, the complexity of the project (multiple stakeholders often with conflicting interests) and the several changes in the ministry structure during implementation period. These achievements are attributable to the constant DGHE oversight and actions to maintain stakeholder engagement and progress of implementation (e.g., creation of task force to develop standards of education and implement NCBE). The remaining balance (13.67%) was not used and was cancelled at project closing. 37. The project had two restructurings, as mentioned in paragraphs above. Both included reallocations of funds across components and only the first restructuring involved the revision of the project s components (inclusion of three professions). 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 38. M&E design is moderately satisfactory. The project was monitored using five KPIs and eleven intermediate indicators. Overall, the selected KPIs and intermediate indicators were defined 9

22 with enough precision to allow the measurement of implementation progress to achieve the PDOs. Specific issues related to the indicators are: Most of the indicators did not have baseline data which is understandable as the agencies and standards for accreditation and competence examination were not in place at the beginning of the project; Target values were provided in the PAD for most of the intermediate indicators and KPIs except for KPI 5: Mean test score of graduates from FAP recipient schools who take the National Competence Test. The ICR team understand this implies a comparison between grantees and non-grantees schools, but there was no benchmark or target associated with this indicator; The feasibility of component 1 KPI Percentage of health professional schools that have gone through accreditation process and publicize results was questioned during the project preparation (QER) due to the complexity of the institutional changes necessary to implement this component within the project timeline. And indeed there was a revision of the initial target due to the delays in establishing the accreditation agency (LAM-PTKes); Likewise, component 2 KPI Establishment of an independent National Agency for Competency Examination of Health Professionals (NACEHealthPro) was also linked to policy actions which depended on lengthy negotiations and bureaucratic process. 39. M&E implementation and utilization was satisfactory. Throughout the project the CPCU was able to provide, during the Bank s team implementation support missions, information about the project implementation progress. This information was collected through several instruments developed to assess project progress (see paragraph 40). This was sufficient to assess project s overall performance as documented in the Aide Memoires and Implementation Status and Results (ISR) reports. The M&E framework in place prioritized actions to monitor identified risks and to respond to complaints. These actions ranged from periodic reviews to continuous monitoring and review depending on the nature of the risk. 40. Several monitoring instruments were developed to assess the progress of project components. These included administrative data and field visits. For components 1 and 2 the Database for Health Higher Education (PD-DiktiKes) was used to monitor the number of schools going through the accreditation process and the number of students undertaking the CBT and their passing scores. The CPCU also made use of Information, Communications and Technology (ICT) to improve communications and coordination across different stakeholders and to monitor complains and improve governance. For example, the CPCU made available a website ( to record complaints and follow up. For component 3 (FAP), the CPCU maintained two strategies: i) regular assessment of the quality of terms of references (TORs) submitted by website from each recipient school; and ii) regularly sent teams of experts to monitor the technical and managerial performance of the FAP recipient schools. These teams were composed of one expert in higher education from the BHE and one medical education specialist from the AIPKI, and one procurement and one financial management specialists. 10

23 2.4 Safeguard and Fiduciary Compliance 41. Environmental Safeguard. The project was classified into safeguard category C, endorsed by the World Bank s safeguard review team. The FAP manual included a provision that any civil work to be undertaken through the FAP program was limited to the rehabilitation of existing lecture rooms and laboratories. 42. Social and Indigenous People Safeguards. Project was classified into category C on condition that the scholarship program under the FAP grant component was not going to actively seek scholarship candidates. During implementation, some medical schools requested to have some active components in their recruitment processes. In response, the Bank team included a social safeguard specialist in the implementation support mission to conduct a specific review of the scholarship program implementation and assess its compliance with Bank s safeguard policies. Based on this review an implementation framework for the scholarship program was developed to ensure that the program implementation was in compliance with relevant safeguard policies. This framework was then included in an updated version of the FAP implementation manual, which was revised and approved by the Bank safeguard team. At the end, the scholarship program for the poor under HPEQ was not extensively used because the DGHE launched a national fellowship program targeting the poor (called bidik misi) with significant amount of funding and for longer period of time than HPEQ scholarship. 43. Financial management. During the initial implementation support missions the financial management was rated satisfactorily as the selection of the FAP recipients was completed on time. During the 3 rd implementation support mission it was detected that the capacity to implement the FAP varied widely among the FAP recipients especially in terms of financial and procurement management. As a result, the supervision team requested the CPCU to map existing implementation capacity of the grantees and to provide support based on each specific need. During the subsequent implementation support missions the rating was downgraded to moderately satisfactory due to the following challenges: i) the recruitment of individual consultants for financial management was delayed as the CPCU could not identify individuals fulfilling the required qualifications; ii) the submission of financial management reports was delayed; iii) the performance of the financial management was inadequate; iv) the audit follow-up was delayed; and v) the disbursement discrepancy between the Bank and project records was uncorrected. 44. Procurement. Procurement ratings ranged from satisfactory at the onset of the project implementation to moderately satisfactory throughout the project implementation. Initially, the Bank team found that the capacity to implement the procurement packages varied widely across FAP recipient schools and, as a result, there were delays in completing procurement packages. Following the assessments done during the implementation support missions the following specific recommendations were made: i) recruit individual consultant to replace the existing project management consultant; and ii) improve the monitoring of procurement process in each school and proactively communicate with the schools to expedite the procurement process. 45. Midway through the project implementation there were some improvements in the management of procurement activities. The CPCU managed to recruit the procurement consultant, the team completed the mapping of FAP recipient capacity and provided technical assistance to 11

24 weaker FAP recipients. The disbursement rate increased significantly indicating that the universities had successfully overcome initial challenges. The final disbursement rate (86.26%) did not negatively affect the achievement of project objectives as the disbursement gap was mainly due to rupiah depreciation and efficiency gains in the procurement of goods. The mapping of the school capacity to implement the FAP conducted by the CPCU and its proactive engagement in managing the project were key in addressing the challenges occurring during the project implementation. 2.5 Post-completion Operation/Next Phase 46. HPEQ established the pathway to strengthen quality of higher education through accreditation and competence-based examination. The legal framework, processes, instruments, manuals, training materials and modules required to implement accreditation and competence-base examination can be of use for more health professions (HPEQ initially covered 7, but the MoH has signaled there is scope to apply for 22 health related occupations) as well as for other sectors. The MoRTHE has indicated that HPEQ approach will be used as a model for other professions, started already with engineering. 47. HPEQ success has triggered challenges for the future. The main one being the need to align health professionals roles and functions within service delivery with CBC. This from one side will ensure employability of graduating health professionals, and from the other will improve health service delivery by increase the supply of health professionals with enhanced competences and skills. The MoH has introduced the PHC competences, though this has not yet been linked with the new CBC for any of the seven professions included in HPEQ. 12 Another challenge is the expansion of quality assurance mechanisms to a wider pool of health professionals, beyond the seven categories included. And even when considering only the professions included in HPEQ the process to scale up the accreditation and examination for the total universe of schools and graduates will require major efforts, as for example: i) develop more accreditation instruments, expand the training of facilitators, assessors and validators, expand the item bank and the number of item bank writers; ii) ensure financial and operational sustainability of the two agencies; iii) keep close monitoring of the effects of the new systems on health professionals labor market in Indonesia and the impacts on population s access to care and health outcomes. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 48. Relevance of Objectives. The PDOs were highly relevant to the country s sectoral need of strengthening the quality assurance system of health professional education, one of the priorities of the government's health sector medium term development plan. The new quality assurance system is particularly important in the context of rapid expansion of health 12 The competencies of medical doctors are based on the standards of competencies released by the IMC in 2006 and revised in 2012, while the standards of competencies for the other professions were developed by the respective professional association during HPEQ. 12

25 professionals schools in Indonesia, particularly in the private sector. Newly established schools, both public and private, face challenges to achieve and sustain standards. In 2010, one-third of medical schools were not accredited and only one-quarter had received the highest accreditation level. The project was also aligned with the World Bank Indonesia CPS , which supported the reform of the education sector Relevance of Design. Design of the HPEQ is considered substantial. The project built on priorities identified from previous sector work and project preparation benefited from Bank s cross-sectoral expertise (health and education). Design was supported by rigorous analytical work, which helped to identify risks, opportunities and mitigation actions as well as build partnerships and foster ownership of project activities and objectives among key stakeholders. HPEQ design also benefited from a QER, which included experts from health and education sectors as peerreviewers. 50. PDO s were defined with precision and KPI and intermediate indicators generally helped to assess project achievements. There were very few shortcomings in the design, namely: i) the selection of the KPIs KPIs 1 and 2 were linked to policy actions which depended on legislative and lengthy bureaucratic process; and KPIs 4 and 5 were related to students CBT results which will take longer maturation period to have more pronounced impacts; ii) the allocation of resources across components although PDOs aimed at 7 professions, FAP grants were allocated only to medical schools and it accounted for 71% of project funds at appraisal; iii) the affirmative FAP was not successful both universities had low rates of disbursement and no significant progress was achieved in terms of capacity building to improve quality of medical education in these institutions. However, important to point out, the two universities were included after project restructuring and not at appraisal. 51. Relevance of Implementation. Implementation is considered substantial. Most of project activities were successfully completed at the project closing date. This include the achievement of almost all PDOs and intermediate outcomes. Disbursement rates varied across components, but overall disbursement was 86.26%. There were some issues with the financial and procurement management capacity of FAP recipients, which caused delays in the initial years of the FAP implementation. The most important aspect of the project implementation that led to the achievement of almost all PDO targets was the strong commitment and sense of ownership fostered among main stakeholders (government, professionals associations, schools associations, students, etc.). This was crucial to build consensus around the accreditation and competence-based examination systems and around the role and functioning of the two agencies created under HPEQ (NAA and NACEHealthPro). This is an important aspect of HPEQ that needs to be highlighted as accreditation and certification are highly controversial processes and HPEQ managed to overcome conflicting views and interests to establish the new systems and institutions. 3.2 Achievement of Project Development Objectives 52. The ICR team rates the achievement of PDOs as substantial. The reasons for this rating are explained below and are supported by the data provided in the ICR datasheet and in the Annex Anderson et al. (2014). 13

26 The PDO aimed to strengthen quality assurance policies governing the education of health professionals in Indonesia. This was to be achieved by: i) rationalizing and assuring competencyfocused accreditation of public and private health professional training institutions; ii) developing national competency standards and testing procedures for certification and licensing of health professionals; and iii) building institutional capacity to employ results-based grants for encouraging the use of accreditation and certification standards in the development of medical school quality. The achievements of the KPIs, for each intermediate PDO, are explained below: i) Rationalizing and assuring competency-focused accreditation of public and private health professional training institutions. 53. With regard to the first KPI - The establishment of an independent National Accreditation Agency (NAA). The NAA was established in October 2014 and formally recognized as an independent entity authorized to conduct the accreditation of health higher education institutions in Indonesia. The agency was created under the name of LAMP-PTKes (Lembaga Akreditasi Mandiri Pendidikan Tinggi Kesehatan). Therefore, the first outcome indicator, related to component 1, was fully achieved. 54. With regard to the third KPI - The percentage of health professional schools (medical dentistry, nursing, and midwifery) that have gone through the accreditation process and have publicized the results. At the time of project closing 173 schools completed the accreditation process using specifically designed instruments and accreditation process, against a KPI target of 124 schools. Therefore, the third outcome indicator, related to component 1, had the target exceeded by 39.5%. Although total number of schools/study programs accredited exceed the target, there was some variation in the achievement across different professions. Medical and dentistry schools were slightly below the target (21 medical schools and 10 dentistry schools were accredited, against the target of 29 and 10 schools, respectively) and nursing and midwifery exceeded the target significantly (81 nursing schools and 62 midwifery schools accredited, against the target of 52 and 33, respectively). 14 For 2015, LAM-PTKes expects to accredit 788 schools. 55. The new accreditation system has the following characteristics: i) the accreditation process includes both desk review of the schools self-assessment and site visits. LAM-PTKes adopted the principle of continuous quality improvement, in which LAM-PTKes identifies weaknesses in compliance to the standards of education, provide feedback and suggestion to improve and comes back to reassess after an agreed period; ii) the selection of assessors for LAM-PTKes starts with candidates submitting their application followed by several tests. Once selected, assessors attend training provided by experts from BAN-PT and LAM-PTKes. The assignment of assessors to schools is conducted by the Accreditation Directorate of LAM-PTKes and follows the core principle of avoiding conflict of interest; iii) it follows the standard that requires that students be involved in the curriculum management process (consistent with the European standards of excellence for student engagement; and iv) it is centralized, LAM-PTKes conducts accreditation in the entire country (for the seven health programs included in HPEQ). ii) Developing national competency standards and testing procedures for certification and licensing of health professionals. 14 At the time of the ICR mission (February 2015) additional four medical schools were undergoing accreditation. 14

27 56. With regard to the second KPI - The establishment of an independent National Agency for Competency Examination of Health Professionals (NACEHealthPro), The National Agency for Competency Examination was legalized as independent agency on December 2013 through the Ministry of Justice and Human Rights Decree (No. AHU-291). The agency was created under the name of LPUK-NAKES (Lembaga Pengembangan Uji Kompetensi Tenaga Kesehatan). Under the new legal framework the authority to implement the competence test is given to the National Committee for Competence Examination (for medicine and dentistry) and to the Indonesia Health Workforce Council (MTKI) (for the remaining five professions). LPUK-NAKES has full authority to develop and ensure the quality of testing, which includes: provide training for examination item development, item bank management, examination management and standard setting, and running examination trials. Therefore, the second outcome indicator, related to component 2, was fully achieved. 57. With regard to the fourth KPI - The percentage of graduates of health professional schools (medical, dentistry, nursing, and midwifery) passing national competency testing at the first attempt, the results are mixed. The KPI was not achieved for medical students (actual 76% against the target of 84%) and for nursing (actual 58% for bachelors nurses and 47.81% for diploma nurses - DIII, against the target of 65% for both bachelors and diploma nurses - DIII). 15 The target was achieved for dentistry (target of 83% and actual value equal to 88%) and for midwives (target of 65% and actual equal to 64.65%). iii) Building institutional capacity to employ results-based grants for encouraging the use of accreditation and certification standards in the development of medical school quality 58. With regard to the fifth KPI - The mean test score of graduates from the FAP recipient schools who have taken the National Competence Test, results are not fully conclusive given the absence of a concrete target for this indicator. However, the mean competency examination scores of FAP recipients showed an increase of around 10% from baseline (60.88 in 2010) to project closing (67.07 by the end of 2014). Additionally, when comparing FAP recipients with non-fap recipients the mean score of FAP recipient graduates was higher than of those of non-fap recipients during the entire period of project implementation. Moreover, this difference increased from the baseline (3.96 in 2010) to project closing (5.76 in 2014). 3.3 Efficiency 59. The project efficiency is considered substantial. The reasons for this rate are: i) HPEQ achieved most of the PDO targets without making use of the total project funds with no prejudice to the project implementation; ii) the policy and regulatory reforms introduced, the creation and functioning of the new agencies and the methodology for developing the various standards of competences, accreditation instruments and competence test examination, were all triggered by HPEQ. 15 DIII is a three year nursing diploma. 15

28 60. Disbursement rates varied across components, but overall disbursement at the time of project closing was 86.26%. This disbursement level did not affect the achievement of the PDO targets. The final disbursement level was influenced by the rupiah depreciation and efficiency gains in the procurement of goods (FAP recipients benefited from discounts in the procurement of computers for CBT and mannequins for OSCE and, therefore, spending was lower than the initially planned). 16 As previously noted, the majority of project funds was allocated to component 3 (FAP), even though this component only directly affected one outcome indicator. However, in terms of efficiency, this was justified because: i) medical schools had already an incipient system of accreditation prior to the project approval, so it was more cost-effective to focus on them; ii) the investments made by medical schools on equipment, computer stations, and mannequins for OSCE, benefited other professions (by sharing equipment). 61. Although limited, the existing evidence has demonstrated that a better educated health workforce improves quality of care and also saves lives. What is completely absent is the empirical assessment of the economic costs and benefits of accreditation and certification processes. A full economic analysis (such as cost-benefit analysis) of the HPEQ impacts is difficult and unlikely to realistically capture full project impacts. Firstly because the main expected impact of the project is the improvement in quality of health services through improved quality of health professionals. But although there is a clear causality between improved quality of care and competences of health professionals, the exact impact may not be possible to be measured and may vary across cadres, levels of care, providers characteristics and broader systems of HRH management and incentives. Secondly, the impacts of HPEQ will take time to be observed in service provision, when the newly graduated health professionals start practicing and implementing competences learned during training and tested. Finally, significant part of the HPEQ benefits will come to individual health professionals who will likely have more opportunities within the local labor market (and regional, international labor markets) as now they graduate from accredited schools and have their knowledge assessed through standardized examination. The same applies to schools, in the sense that in the future accreditation status will influence students choices of schools to enroll The cost-benefit analysis presented in the annex 3 takes a rather conservative approach in terms of impacts and timeline for HPEQ results. It links the (improved) quality of training to improved quality of care over a practitioner s career, and hence population health outcomes (in terms of infant mortality rate and maternal mortality rate). It then monetizes the gains and compares them to project costs. The discounted total benefits of the project, estimated in productive life years gained, is estimated at US$ million which is significantly higher than the total value of the project costs (US$77.82 million). This results in a benefit-cost ratio equal to US$10.77, which means that for each US$1 invested through the project there is an expected return of US$10.77 (only considering the period of five years after project closing). Although significantly high, this result are based on conservative assumptions adopted and likely underestimates the total project benefits. 16 Rupiah depreciation from US$ 1 = IDR 10,300 at project preparation to US$ 1 = IDR 12,275 at project closing. 17 GoI already selects civil servants applying a rule that exclude graduates from low performers schools. 16

29 3.4 Justification of Overall Outcome Rating Rating: Satisfactory. 63. As discussed above, the project design was considered highly relevant due to its fine alignment with government priorities and Bank s strategy (CPS) and it addressed important health sector challenge (quality of health professionals). The design also benefited from analytical work and recommendations from the QER that helped to identify risks, opportunities and mitigation measures. PDOs were defined with enough precision that allowed to measure their implementation progress and project achievements. The activities funded under the four components helped to achieve HPEQ PDOs satisfactorily, most KPIs were achieved or surpassed. Therefore, the overall project rating is satisfactory. Project Achievement of PDO Efficiency Overall Rating Relevance (efficacy) High Substantial Substantial Satisfactory 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development. 64. Although HPEQ did not have any specific focus on gender, it is expected that its achievements will benefit women considerably. The reason is that a large share of the health workforce in Indonesia is female. In 2010, for example, percentage of contract female doctors in the country was around 60% while the contract female dentists was 81%. 18 This share for nursing and midwifery is likely to be higher as these are female dominated professions worldwide. Therefore, HPEQ will likely to impact on female employability in Indonesia. 65. The affirmative FAP included two universities (UNCEN and UNDANA) from poor regions of the country (East Nusa Tenggara, and Papua). HPEQ also provided, under component 3, an affirmation program by providing bachelor degree scholarship to poor medical student candidates; 152 scholarships were granted under this program. (b) Institutional Change/Strengthening 66. HPEQ s major contribution is the establishment of the legal and regulatory frameworks for school accreditation and the competence-based examination for health professionals. Annex 9 lists all regulations that were put in place to guarantee the functioning of the two independent agencies and the implementation of the accreditation and competence-based examination. The close participation of key stakeholders and the sense of ownership built during project implementation helped to strengthen relationships among universities, professional associations, schools associations, and different ministries. 18 Anderson et al. (2014), 17

30 67. HPEQ also improved capacity at DGHE/MoRTHE to develop accreditation systems for other professions. The minister already signalized that the system developed under LAM-PTKes and LPUK-NAKES will be a model for other fields. Another important contribution is the strengthening of MoRTHE and MoH collaboration, which was less harmonious before HPEQ. HPEQ also contributed to create a long-term cooperation among universities matched in the FAP. During the ICR field visits it was noted that some universities had planned activities beyond project timeframe and to be funded through own resources. 68. The ICT system implemented under HPEQ supported implementing agency in the project management and M&E throughout implementation. This system will serve as the information management system for the two new agencies (LAM-PTKes and LPUK-NAKES). The CPCU also implemented a web-based knowledge portal to disseminate HPEQ products (documents, publications, etc.), policy reforms and other materials related to health professionals education. This is expected to be a channel for collaboration and debate among health professionals. 19 (c) Other Unintended Outcomes and Impacts (positive or negative) 69. During the ICR visit the following unintended outcomes were noted: Students reported that some schools focused more on providing short term training for students to take the NCBE instead of implementing curriculum changes based on the test requirements/contents; Related to the above, some schools seem to have been selecting students that will take the NCBE based on their try-out results. This may be influenced by the fact that NCBE results is part of the accreditation process; HPEQ also improved GoI capacity to develop and implement quality assurance systems for higher education. The GoI has already signaled that the system developed under LAM-PTKes and LPUK-NAKES would be a model for other fields; HPEQ also supported the creation of HPEQ Student. This network includes students from the seven professions from all universities in Indonesia. HPEQ Student aims to engage students in health professional education policy formulation and to foster inter-professional collaboration across health disciplines; HPEQ helped to establish a network of health professionals associations, health professional schools association, government entities, students and broader civil society, committed to long term support for quality improvements in the education of health professional in Indonesia. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops Not applicable

31 4. Assessment of Risk to Development Outcome Rating: Moderate. 70. There are moderate risks to the sustainability of outcomes, these are: Financial: Besides the fact that the two agencies (LAM-PTKes and LPUK-NAKES) have developed their business plans, there are issues regarding how they will be financed in the medium and long run: - This is particularly the case for the LAM-PTKes as the new accreditation arrangement requires schools to bear the costs of accreditation. Under the previous model (BAN- PT) accreditation was subsidized by the government, free of costs to schools. Under the new model, the unit cost for accreditation is estimated at IDR for bachelor level and IDR for vocational, master, doctoral and specialized level. 20 The GoI has guaranteed resources for the accreditation of 400 schools in 2015 (out of 788 expected to be accredited during this year). Professional associations and association of education institutions had also made financial commitments to LAM- PTKes. But these funds are not stable and there is a need to secure government support for the initial years of the agency operation (beyond 2015); - For LPUK-NAKES the funding sources are: contributions from members (professional associations), fees on services provided to members (CBT try out, training of assessors, etc.), and government subsidies (no specified amount). The funding mechanism for LPUK-NAKES is more straightforward as most of the schools recovers the cost of the CBT from the students, either directly (through specific fees) or indirectly (including the costs in the tuition fees). Technical: Both agencies face challenges to recruit full and part-time professionals with qualification in health education. Particularly for LAM-PTKes given the need to recruit facilitators, assessors and validators; Institutional: There is a need to ensure the alignment of competences and curriculum to accreditation and examination systems. Schools seem to have incentives to adopt opportunistic behavior by training students specifically for the CBT and OSCE as opposed to strengthening their curriculum (adopting CBC). Additionally, it is not clear whether the government recognition of LAM-PTKes means that the agency is the only entity in charge of the accreditation system or whether there is scope for other institutions to play a role in this market ; Political: Change in government leadership may affect political commitment and support to the agencies and, consequently, the financial support. Additionally, agencies will need to maintain the engagement of the different stakeholders beyond HPEQ closing. During project implementation, stakeholders engagement was facilitated by the project capacity to mobilize funds for travel, venue rental and other logistics necessary to carry out meetings and 20 Equal to US$6, and US$5, respectively (1 US$ = 13, IDR on May 7, 2015). 19

32 consultations. Additionally, the MoNE is under restructuring (to become MoRTHE) and this may affect the housing of initiatives developed under the project within the ministry; Finally, there is a need to ensure alignment between health services delivery strategies and health workers competences. Ensuring alignment with government initiatives (e.g., PHC competences) will increase the relevance of the new accreditation and examination systems to the extent that they can help to support government health sector strategies (e.g., in expanding PHC services). 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Satisfactory 71. The project design built on priorities identified from earlier sector work and objectives were in line with national priorities. More specifically, the Bank accompanied the GOI s interest in improving the quality of education as depicted in the medium term development plan for The project preparation benefited from Bank s expertise from both health and education sectors, and was rigorously supported by analytical work (particularly the stakeholder analysis), which helped to map and integrate key stakeholders (governmental and non-governmental organizations) based on their interest and capacity to influence and oversee project activities. 72. The early reviews and analytical work helped to improve project design. For instance, the stakeholder analysis identified risks, opportunities and mitigation measures. This included identifying key stakeholders views and interests regarding the accreditation and certification procedures. The project benefited from the stakeholder analysis as, unquestionably, the project achievements were a direct result of the ownership and commitment from key stakeholders. The preparation period also benefited from a QER. The QER panel endorsed the project objectives and design, and provided recommendations (see paragraph 30). Some of the issues raised during the QER were observed during project implementation, as for example: i) the delays in the legislative process to approve regulations. This affected the achievement of the KPIs within the proposed period of time; and ii) the balancing between the size of the grants with the institutional capacity and incentive to use the grants was anticipated during the QER. During the ICR visit the team noted that the incentives as well as the capacity to use FAP resources varied significantly across FAP recipients. Overall, these shortcomings did not affect the achievement of the projects objectives, at worst case they just delayed the time of achievement. Taking into account the complexity of the project as well as what was achieved, these delays did not compromise the quality of the project design. (b) Quality of Supervision Rating: Satisfactory 20

33 73. HPEQ had two task team leaders (TTL) during the entire project. TTLs were based in Jakarta during the entire project implementation allowing continuity of support to the government and close monitoring of project activities. For example, the Bank team actively sought advice from fiduciary and safeguards specialists to support implementation. The team also proactively identified consultants, international and local experts, to assess the implementation of the component 3 partnership scheme and the implementation of components 1 and 2. These initiatives provided timely technical inputs to the development of policies and operational instruments for accreditation and CBT. 74. There were regular implementation support missions, which allowed the team to provide timely inputs to the government and up-to-date information to Bank management. The findings of each visit, nine in total (two per year), were conveyed in a structured manner in the ISRs, including specific sections with detailed reports of the progress towards achieving the PDOs, project status, implementation shortcomings and courses of action. 75. The Bank team timely intensified efforts to address challenges as they began to surface. For example: the delays in channeling government resources to the DGHE; the lengthy process to put in place the legal basis for the functioning of the two agencies (for accreditation and CBT); the complex management of the collaboration across different government units and non-government organizations. Efforts to address these challenges were done through extensive reviews of the implementation of project activities (field visits), frequent meetings with key stakeholders, proactively facilitating restructuring and reallocation of project funds across components to address project s needs. Additionally, after the implementation progress was reduced to moderately satisfactory the Bank team started to have interim implementation support missions between regular (implementation support) missions. (c) Justification of Rating for Overall Bank Performance Rating: Satisfactory 76. The ICR team rates the overall Bank performance as satisfactory given the satisfactory rating for both quality at entry and supervision. 5.2 Borrower Performance (a) Government Performance Rating: Satisfactory 77. The Government demonstrated strong commitment and full ownership throughout the project. This is reflected in the engagement of different ministries in the process of setting the legal basis for the accreditation and competence test systems and respective agencies (see Annex 9). For example, the MoEC and MoH joint regulation determined that the national competency examination for health professions would refer to the methodology designed by LPUK-NAKES. For the accreditation system, the MoEC declared that LAM-PTKes is in charge of conducting accreditation of health professionals schools in the country and allocated resources to cover accreditation costs of 400 schools in

34 78. Despite some delays in setting up legal framework (mainly due to MoEC internal processes and lengthy consultations with stakeholders), these were defined and implemented within the project implementation period. Additionally, to minimize the impacts of the delays the GoI implemented two task forces to exercise the functions of the two agencies until the legal frameworks were in place. This allowed to carry out the supporting activities necessary to implement the new accreditation and competence test systems. 79. Main challenges faced by MoEC during the implementation period were: i) changing ministry structure (from MoNE to MoEC and finally to MoRTHE) 21 ; ii) delays in the transfer of resources from MoF to MoEC due to delays in the government budget approval; and iii) lengthy process to finalize and build consensus around the legal framework for the functioning of the two agencies (LPUK-NAKES and LAM-PTKes). 80. Government effectively overcame the above mentioned challenges and the ICR team rates the GoI performance as satisfactory. The main reasons were: the mitigation measures taken to guarantee the progress of project implementation given the unexpected delays, and the support and strong commitment from different ministries (MoEC, MoH, Ministry of National Development Planning (BAPPENAS), and MoF) to the reform and to effectively managing complex and often conflicting interests among stakeholders. Additionally, the establishment of the two bodies resulted in transferring of power from the government to a non-government entity, which is not usual for a country like Indonesia. (b) Implementing Agency or Agencies Performance Rating: Satisfactory 81. The executing agency for HPEQ was the MoEC while the implementing unit was the DGHE. Although not an implementing agency, the MoH participated in all policy discussions and played an important role in setting up of the legal framework for accreditation and examination systems. The CPCU was established within the office of the Directorate of Learning and Student Affairs (LASA). The project director and project manager were the Director General of Higher Education and the Director of Learning and Student Affairs, respectively. A Project Steering Committee was established with representatives from BAPPENAS, MoEC, MoH and MoF and was very functional during project implementation. At university level (component 3), Project Implementation Units (PIU) were established, prior to FAP contract being signed, to support implementation and administration of the project and rectors had the ultimate responsibility for project implementation. 82. At university level, the capacity to execute component 3 activities was an issue. The capacity to implement the FAP varied widely among schools. In particular, the performance of the partnerships between the schools A and B schemes varied widely due to the limited administrative and technical capacity to implement the proposed activities. Furthermore, the lengthy process to review TORs (by the CPCU) and the incomplete implementation manual influenced negatively the ability of the FAP recipients to implement the planned activities on time. 21 MoNE before 2011, then MoEC in the period of , then MoRTHE from

35 83. The CPCU and the DGHE s have performed remarkably well in pushing for the regulations to establish the two bodies. Despite the significant bureaucratic hurdles within MoEC and often conflicting views of key stakeholders. The CPCU and the DGHE played an important role in advocating, lobbying, organizing numerous meetings/workshops, and going through numerous redrafting of the regulations to satisfy all stakeholders and yet still maintain the quality and commitment to the HPEQ goals. (c) Justification of Rating for Overall Borrower Performance Rating: Satisfactory 84. The ICR team rates the overall borrower performance as satisfactory. This is because rating for borrower performance and implementing unit performance were both satisfactory for the reasons discussed above. 6. Lessons Learned 85. HPEQ was an unique project given its importance, cross sectoral nature, complexity and impacts. The lessons learned from HPEQ are useful from both operational perspective (design and implementation of Bank s project) as well as technical (in terms of providing a platform for higher education quality assurance systems for other professions in Indonesia and for other countries). 22 The key lessons learned are: Project design was successful in engaging multiple stakeholders and creating a sense of ownership of the project. It created a momentum to strengthen stakeholders commitment to improve quality of health professionals education; Strong collaboration between government and non-government entities was essential to conduct the reform process. Commitment and active involvement of the professional associations and the associations of health professional schools were determinant to achieve HPEQ PDOs. These associations were the founders of LAM-PTKes and LPUK- NAKES and they were also the main contributors to the preparation of: i) the academic papers for the accreditation and national examination systems; ii) the standards of education; iii) the standards of competencies; and iv) the accreditation instruments; Incentives for schools were very different across regions, accreditation level and ownership (public and private). The size of the FAP must be better balanced in future similar operations to create more appropriate incentives for participation and to improve disbursement; The process to set accreditation and competence-base examination are lengthy and depend on the existing legal and institutional framework. Additionally, the political economy of transferring the authority (accreditation) from the government to non-government entities 22 Accordingly to WHO more than half the countries of the world appear to lack a credible, transparent and comprehensive accreditation system (WHO, Transforming and Scaling Up Health Professional Education and Training. Policy Brief on Accreditation of Institutions for Health Professional Education. World Health Organization, Geneva). 23

36 is challenging and requires a very long and difficult change process. In future similar operations, consideration should be given on the selection of KPIs to take into account lengthy political and legal processes; For the FAP the comparison of mean test between FAP recipients and non-fap recipients are not useful to monitor implementation and achievement as there are too many confounding factors influencing the results; 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies See Annex 7. (b) Cofinanciers Not applicable. (c) Other partners and stakeholders Not applicable. 24

37 Annex 1. Project Costs and Financing (a) Project Cost by Component (in US$ Million equivalent) Component Component 1- - Training and workshops, incremental operating costs, research expenditure, consultant services and goods under Part 1 of the Project Component 2 - Training and workshops, incremental operating costs, research expenditure, consultant services and goods under Part 2 of the Project Component 3 - FAPs for sub-projects under Part 3 of the Project Component 4 -Training and workshops, incremental operating costs, consultant services and goods under Part 4 of the Project (b) Financing Appraisal estimate (US$ millions) Actual/Latest Estimate (US$ millions) Percentage of Appraisal 7,184 7, % 12,899 14, % 61,400 44, % 5,239 5, % Total 86,722 71, % Source of funds International Bank for Reconstruction and development Government of Indonesia & Higher Education Institutions Appraisal estimate (US$ millions) Actual/Latest Estimate (US$ millions) Percentage of Appraisal 77,822 67, % 8,900 4, % Total 86,722 71, % (c) Reallocation by Category of Expenditure (in million US$) Category of expenditure Component 1: Training and workshops, incremental operating costs, research expenditure, consultant services and goods under Part 1 of the Project Component 2 -Training and workshops, incremental operating costs, research expenditure, consultant services and goods under Part 2 of the Project Component 3-FAPs for sub-projects under Part 3 of the Project Component 4-Training and workshops, incremental operating costs, consultant services and goods under Part 4 of the Project Appraisal 2009 Reallocation 2012 Reallocation 2014 Actual (disbursed) * ,584 8,335 9,127 7,629 12,099 14,047 14,047 12,470 55,000 51,301 49,784 46,695 4,139 4,139 4,864 4,679 Total 77,822 77,822 77,822 71,473 * According to the government ICR. 25

38 Annex 2. Outputs by Component Indicator # Indicator's name according to the PAD Target value Actual end line value Result Component 1: Strengthening Policies and Procedures for School Accreditation KPI 1 KPI 3 Intermediate Outcome 1 Intermediate Outcome 3 The establishment of an independent National Accreditation Agency (NAA). The percentage of health professional schools (medical dentistry, nursing, and midwifery) that have gone through the accreditation process and have publicized the results. Completion status of preparatory activities for the establishment of the NAA Accreditation instrument for the four health professional schools are ready for use. Establishment of the Charter of the NAA medicine (29, 42%), dentistry (10, 42%), Nursing (52, 10%), Midwifery (33, 10%). Total 124 schools. a NAA has independent and adequate budget to conduct accreditation and has access to adequate numbers of suitably trained assessors medicine (international peer review of instrument), dentistry (piloting and dissemination), nursing (piloting and dissemination), Midwifery (piloting and dissemination) Through the MoEC Decree No 291/P, October , the independent accreditation agency was formally recognized as an independent entity authorized to conduct the accreditation of health higher education (under the name of Lembaga Akreditasi Mandiri Pendidikan Tinggi Kesehata LAMP-TKES). medicine (21), dentistry (9), nursing (82), midwifery (56). Total 168 schools Y Y Achieved Surpassed Achieved Achieved 26

39 Component 2: Certification of Graduates Using a National Competency-based Examination KPI 2 KPI 4 Intermediate Outcome 2 Intermediate Outcome 4 Intermediate Outcome 5 The establishment of an independent National Agency for Competency Examination of Health Professionals (NACEHealthPro). The percentage of graduates of health professional schools (medicine, dentistry, nursing, and midwifery) passing national competency testing at the first attempt. Standard of Competencies and Standard of Education for the four health professions are available. Number of trained assessors Introduction of CBT and OSCE for NCE Establishment of the Charter of the NACEHealthPro medicine (84%), dentistry (90%), nursing (65%), midwifery (65%) nursing (both standards completed and legalized), midwifery (standard of education completed and legalized) medicine (86), dentistry (39), nursing (105), midwifery (105), nutrition (40), public health (40), pharmacy (40). b CBT for all four professions. OSCE for medicine and dentistry, OSCE try out for midwifery and nursing. CBT try out for the added professions, OSCE preparation only for pharmacy. a The National Agency for Competency Examination (Lembaga Pengembangan Uji Kompetensi Tenaga Kesegatan - LPUK-NAKES) was legalized as independent agency on December through the Ministry of Justice and Human Rights Decree No. AHU-291 medicine (72.47%), dentistry (92.31%), nursing (57.81%), midwifery (64.65%) Y medicine (132), dentistry (34), nursing (123), midwifery (70), public health (17), nutrition (16), pharmacy (41) CBT for medicine, dentistry, and nursing Paper based test for midwifery and diploma of nursing (DIII). CBT try out for nutrition and pharmacy. OSCE for medicine and dentistry, and under preparation for pharmacy. Achieved Achieved for dentistry and midwifery. Not achieved for medicine and nursing Achieved Surpassed for medicine, nursing and pharmacy. Not achieved for dentistry, midwifery, public health and nutrition. Achieved except for nutrition 27

40 Intermediate Outcome 6 Intermediate Outcome 7 Intermediate Outcome 8 Intermediate Outcome 9 Intermediate Outcome 10 Number of National OSCE trainers Number of National MCQ Item Writers Number of National OSCE Item Writers Number of MCQ Item Writers and reviewers Number of OSCE Item Writers and Reviewers. 72/profession (medicine and dentistry only). 36/pharmacy. a 1,044 /initial professions), and added 36/profession for nutrition, public health and pharmacy. b 72/profession (medicine and dentistry), 36/pharmacy. a 1,044 /initial professions), and added 36/profession for nutrition, public health and pharmacy. b 1,044/profession (medicine and dentistry only), 36/pharmacy. c medicine (4950), dentistry (84) medicine (254), dentistry (650), nursing (828), midwifery (675), pharmacy (219), public health (141) and nutrition (144). medicine (221), dentistry (650), pharmacy (39). medicine (216), dentistry (650), nursing (828), midwifery (675), pharmacy (78), public health (84) and nutrition (132). medicine (86), dentistry (63), pharmacy (39). Surpassed for medicine and dentistry. Not achieved for pharmacy. Not achieved for the four initial professions. Surpassed for the three added professions Surpassed Not achieved for the four initial professions. Surpassed for the three additional professions. Not achieved for medicine and dentistry. Surpassed for pharmacy. Component 3: Results-based Financial Assistance Package (FAP) for Medical Schools KPI 5 Intermediate Outcome 11 The mean test score of graduates from the Financial Assistance Package (FAP) recipient schools who have taken the National Competence Test. Number or percentage of medical schools receiving finance support to strengthen the program a Revised targets; b The targets for the three new professions were added during MTR; Target not specified (The mean test scores of non-fap at project closing) Achieved 28

41 Annex 3. Economic and Financial Analysis 1. HPEQ focused on strengthening quality assurance systems for health professional education in Indonesia. This was done through: i) rationalizing and strengthening the accreditation of public and private medical, dental, nursing, and midwifery schools; 2) implementing competency-based certification; and 3) providing results-oriented resources to assist health education institutions meet these challenges. The project activities focused on the establishment of an independent body for schools accreditation, the LAMP-TKes, and an independent national agency for competency examination, the LPUK-NAKES. Project funded the implementation of grants for medical schools to strengthen the quality of medical education in Indonesia. 2. A functioning quality assurance system for health professional education is a way to minimize the asymmetry of information inherent to health care markets. Consumers may not be able to determine the quality of services provided by the health workforce, therefore accreditation and competence examination are mechanisms to assure consumers that the inputs into the health production function (in this case, labor) are of sufficient high quality. 23 Accreditation, for example, is expected to reduce variations in the clinical practice, eliminate inappropriate care, and control costs The need to improve the competence and knowledge of the Indonesia s health workforce was a consensus at time of project preparation. The 2007 Indonesia Family Life Survey (IFLS) provided the evidence based for that. The IFLS included vignettes to measure the diagnostic and treatment capacity among doctors, nurses and midwives. Results showed a low percentage of correct responses to vignette questions for antenatal care (45%), for child curative care (62%), and for adult curative care (57%). 25 Additionally, evidence from Barber and colleagues (2007) demonstrated that low-quality of care in Indonesia was to a large extent results of poor quality of training among health professionals There is evidence demonstrating the links between schools accreditation, health professional certification and health outcomes. Silber and colleagues (2002) found that certification of anesthesiologists in the Unites States is significantly associated with a 13% reduction in mortality, after controlling for a number of observable variables. 27 A systematic review conducted by Alkhenizana & Shaw (2011) has shown a consistent positive association between accreditation programs and clinical outcomes. 28 Another study, by Clark and colleagues (1998), found positive effects of providers education on child health outcomes related to asthma. 29 In developing country context, Peabody and colleagues (2008) found that national level accreditation influenced quality of care in the Philippines. 18 Barber and Gertler (2009) estimated that a one standard deviation increase in quality (of providers) reduces the prevalence of child stunting by six percentage points in Indonesia. 30 The authors measured quality of care by the practitioners answers to the vignettes questions. 23 Nicholson and Propper (2008). Nicholson S and Propper C (2012). Medical Workforce. Handbook of Health Economics, Vol. 2: Peabody et al. (2008). Should we have confidence if a physician is accredited? A Study of the Relative Impacts of Accreditation and Insurance Payments on Quality of Care in the Philippines. Soc Sci Med August ; 67(4): Anderson et al. (2014). 26 Barber et al. (2007). Differences in Access to High-Quality Outpatient Care in Indonesia. Health Affairs, 26(3): w352-w Silber et al. (2002). Anesthesiologist Board Certification and Patient Outcomes. Anesthesiology, 96: Alkhenizana and Shaw (2011). Impact of Accreditation on the Quality of Healthcare Services: a Systematic Review of the Literature. Ann Saudi Med Jul-Aug; 31(4): Clark et al. (1998). Impact of Education for Physicians on Patient Outcomes. Pediatrics, 101: Barber and Gertler (2009). Health workers, quality of care, and child health: simulating the relationships between increases in health staffing and child length, 91(2):

42 5. Although limited, the existing evidence has demonstrated that a better educated health workforce improves quality of care and also saves lives. What is completely absent is the empirical assessment of the economic costs and benefits of accreditation and certification processes. A full economic analysis (such as cost-benefit analysis) of the HPEQ impacts is unlikely to capture the full project impacts. Firstly, because the main expected impact of the project is the improvement in the quality of health services through improved quality of health professionals. However, despite the fact that there is a clear causality between improved quality of care and competences of health professionals, the exact impact may not be possible to be measured and may vary across cadres, levels of care, providers characteristics and broader systems of HRH management and incentives. Secondly, the impacts of HPEQ on service delivery will take time to be observed. This will be felt when the newly graduated health professionals start to practice and to implement the competences learned during training and tested through CBT. Finally, significant part of the HPEQ benefits will benefit individual health professionals who will likely have more opportunities within the local (as well as regional and international) health care labor market as now they graduate from accredited schools and have their knowledge assessed through the CBT. The same applies to schools, in the sense that in the future accreditation status will influence students choices of schools Having these limitations in perspective, the ICR economic analysis adopts a conservative approach. The approach is conservative in three dimensions: i) it suggests modest gains in the quality of care provided by a certified health practitioner trained by an accredited institution; and ii) it only considers dimensions of health improvements that are relatively easily translated into monetary terms; iii) the analysis covers an horizon of five years only (while project benefits are permanent). The analysis links the (improved) quality of training to improved quality of care over a practitioner s career, and hence population health outcomes. It then monetizes the health outcomes gains and compares them to project costs. 7. The impact of the project can be modeled as a series of interventions that affect the following health indicators applicable to the beneficiaries of the project: infant mortality rate and maternal mortality rate. The next steps is to measure the impact of HPEQ outcomes on these two indicators, value these impacts in monetary units and compare them to HPEQ costs. 8. Valuing gains from reduced mortality is a long standing debate in the health economics literature. Major issues are the ethical and equity debate around the task of how to value a life saved or improved. The available literature provide some basis for the monetary estimation of these gains. Alderman and Berhman (2004), for example, estimates the savings of US$1,250 for saving an infant s life through a measles campaign. 32 An alternative approach is to measure the impacts, and the monetary benefits, in terms of productive life years gained due to reduced mortality. This is done by calculating the number of years gained as a result of project interventions and calculating the economic benefit of these years. Given the nature of the interventions and the likelihood of having great impact among children, hence large gains in terms of future productive life years, the economic evaluation adopts this approach to measure the economic benefits of the project. 9. The analysis uses population health and demographic indicators (table A3.1) and apply the following assumptions to estimate the (economic) benefits of reducing child and maternal mortality, as follows: 31 GoI already selects civil servants applying a rule that exclude graduates from low performers schools. 32 Alderman and Behrman (2004). Estimated Economic Benefits of Reducing Low Birth Weight in Low-Income Countries. World Bank, Washington, DC. 30

43 - For children, the productive years are assumed to range from 13 to 71 (Indonesia s life expectancy at birth). This means a child s productive life years will only start to count when a child is 13 years old. It is also assumed the average year of a saved child is 2 years old; - For pregnant women, is assumed that their average age is 20 years old and the maximum age for being active in the labor force is 72.9 (which is the women life expectancy at birth in Indonesia); - The GNI per capita is used to value each productive life years gained. Indonesia GNI per capita in 2013 was US$3,580; - The project benefits in terms of productive life years gained are discounted with a 3% discounting rate which is standard in economic evaluations. 33 Table A3.1: Population health and demographic indicators Population under five 16,380,000* Number of pregnant women (national) 2,876,700** GNI per capita (US$, 2013) 3, Life expectancy at birth (women) 72,9 Life expectancy at birth 71 Infant Mortality Ratio (per 1,000) 29 Maternal Mortality Ratio (per 100,000) 190 Population growth rate (2013) 1.2% SOURCES: World Bank, WDI. * Indonesia DHS, 2012 and Population Reference Bureau, ** Indonesia DHS, Table A3.2 display the results of the analysis. The discounted total benefits of the project, estimated in productive life years gained, is estimated in US$838.9 million which is significantly higher than the total value of the project costs (US$77.82 million). The benefit-cost ratio is estimated in US$10.77, this means that for each US$1 invested through the project there is an expected return of US$10.77, only considering the period of five years after project closing. Although significantly high, this result is based on rather conservative assumptions adopted and likely underestimates the total project benefits. For example, health care costs (for health systems and households) saved due to reduced morbidity and mortality are not taken into account, efficiency gains are not included, and the effect of project interventions on health outcomes are considerably low (from 0.3% to 0.7% reduction in mortality rates for infants and pregnant women) and, most important, the timeline adopted to measure impacts is rather short. Finally, the analysis does not take into account the (individual and social) impacts on the Indonesia labor market. 33 Drummond et al. (2005). Methods for the Economic Evaluation of Health Care Programmes. Paperback. 31

44 Table A3.2: Cost-Benefit Analysis Results Child Health Benefit Number of Children under five Total 16,380,000 16,576,560 16,775,479 16,976,784 17,180,506 Saved Children under five 1,425 1,923 2,432 2,954 3,488 12,222 Gained productive life-years per child under five (Present Value) Total Gained productive life-years (Present value) 28,869 37,808 46,419 54,711 62, ,501 Economics Gains relate to improved child health (US$ million, Present value) Maternal Health Benefit Number of pregnant women 2,876,700 2,911,220 2,946,155 2,981,509 3,017,287 Saved women form maternal death Gained productive life-years per saved women (Present value) Total Gained productive life-years (Present value) Economics Gains relate to improved maternal health (US$ million, Present value) Total Health Benefit Total Gained productive life-years (Present value) 29, , , , , ,160 Economic gains related to improved child and maternal health (US$, million, Present value) Total Costs Total Costs (Nominal, US$ million) BENEFITS/COSTS RATIO

45 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Names Title Unit Responsibility/Specialty At Preparation Puti Marzoeki Claudia Rokx Pandu Harimurti Jed Friedman Yogana Prasta Susiana Iskandar Ratna Kesuma Jamil Salmi Andreas Blom Christopher Smith Novira Asra Imad Saleh Paulus Bagus Tjahjanto Andrew Daniel Sembel Dayu Nirma Amurwanti Susan Stout Pierre Jean Gordon Page Rosalia Sciortino Ratih Hardjono Tetty Rachmawati Khadrian Adrima Estie Suryatna Sr. Health Specialist Lead Health Specialist Health Specialist Sr. Economist Operations Advisor Sr. Education Specialist Operations Officer Tertiary Education Coordinator Sr. Economist Consultant Financial Management Specialist Sr. Procurement Specialist Procurement Specialist Environmental Specialist Operations Officer Consultant Consultant Consultant Consultant Consultant Consultant Consultant Team Assistant GHNDR GHNDR GHNDR DECPI EACIF GEDDR GEDDR GEDDR GEDDR GEDDR GGODR GGODR GGODR GENDR GSURR GHNDR GHNDR GHNDR GHNDR GHNDR GHNDR GHNDR GHNDR Task team leader Health cluster leader Medical education Economic analysis Operations advisor Education governance School grants Peer reviewer Peer reviewer Peer reviewer Financial management Procurement Procurement Environment safeguard Governance Monitoring and evaluation Accreditation Examination Stakeholder Analysis Communication Communication Project costing Team assistant At implementation Puti Marzoeki Pandu Harimurti Yogana Prasta Siwage Dharma Negara Novira Asra Aswin Hidayat Budi Permana Angelia Budi Nurwihapsari Dayu Nirma Amurwanti Achmad Affandi Nasution Isono Sadoko Pierre Jean Diane Brown Gordon Page Satrio S. Brodjonegoro Shita Listyadewi Sr. Health Specialist Health Specialist Operations Advisor Sr. Education Specialist Sr. Financial Management Specialist Consultant Procurement Specialist Procurement Specialist Operations Officer Consultant Consultant Consultant Consultant Consultant Consultant Consultant GHNDR GHNDR EACIF GEDDR GGODR GGODR GGODR GGODR EACIF GSURR GSURR GHNDR GHNDR GHNDR GHNDR GHNDR Task team leader Task team leader Operations advisor Education governance Financial management Financial management Procurement Procurement Governance Governance Social Safeguard Accreditation (medicine) Accreditation (nursing) Examination FAP Implementation reporting 33

46 Christina Sukmawat Program Assistant GHNDR Program assistant At Completion Puti Marzoeki Edson Correia Araujo Diana Iuliana Pirjol Victoriano Arias Christina Sukmawati Sr. Health Specialist Sr. Economist Consultant Program Assistant Program Assistant GHNDR GHNDR GHNDR GHNDR GHNDR Task team leader ICR Primary Author ICR team member Program assistant Program assistant (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage US$ 000 (including travel and No. of staff weeks consultant costs) Lending FY FY Total: Supervision FY FY FY FY FY Total:

47 Annex 5. Beneficiary Survey Results Not applicable. 35

48 Annex 6. Stakeholder Workshop Report and Results Not applicable. 36

49 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR 1. HPEQ project was unique as overall project implementation was attached to and followed the regulations of DGHE s Working Unit (Satuan Kerja - SATKER). This model was intended to facilitate early integration of the project to the programs into the SATKER for maintaining sustainability of the programs after project completion. On the other hand, by integrating the project into the SATKER, HPEQ was vulnerable to the dynamics and changes of government planning and budgeting policies. Challenges in Project Implementation 2. The achievement of PDO indicators/kpi targets in 2012 and 2013 was affected by the implementation of new acts: the Higher Education act No.12/2012 and the Medical Education Act No.20/2013. Enactment of these two acts was not predicted during project preparation and had significant impact on project implementation, particularly on the timing and the extra effort required to establish LAM-PTKES and LPUK-Nakes. As the result, the target of establishing the two bodies could only be met towards the end of the project. 3. During project mid-term review, some KPI targets and intermediate outcome indicators were revised to become more realistic. Measurement of the indicators were re-negotiated and agreed. The changes were explained in the supplemental letter of revised performance indicators from the Bank and was reflected in the revised Project Management Manual. 4. The implementing agency had to be agile in coping with the ambiguity and uncertainty of government decisions and has adopted action through learning principles. In addressing the challenge, the implementing agency has developed an effective monitoring and evaluation (M&E) system by designing specific methods and instruments for each component and applying structured data recording and reporting system for strengthening evidence based-internal quality assurance of the project. For the FAP program, the M&E system was developed in collaboration with the Board of Higher Education using best practices from previous school grant programs financed by the World Bank, such as DUE, QUE, and I-MHERE. The formative approach through nurturing/mentoring as a part of the M&E strategy was very useful for the School Project Implementing Units in managing the FAP program. The implementing agency has also developed a risk mitigation framework to guide the development of the M&E strategy for each project component. 5. The implementing agency and the School Project Implementing Unit has continued to experience problems with the annual audits by the Finance and Development Supervisory Board Ministry of Finance (Badan Pengawasan Keuangan dan Pembangunan/BPKP). The problems were mainly related to unstandardized evaluation by the auditors (especially regional auditors) in interpreting project guidelines. The GoI recommends the World Bank to evaluate the BPKP audit mechanisms to ensure they are standardized. 37

50 To what extent have the objectives been met / results been achieved 6. From GoI and stakeholders perspective, HPEQ project has hugely contributed to national priorities, particularly by facilitating and accelerating the formulation of the Medical Education Act No.20/2013 to guide the fundamental reform of medical education in Indonesia. This Act has endorsed Ministry of Health and the Parliament to publish the Health Workforce Act No.36/2014 and the Nursing Act No.38/ LAM-PTKes and LPUK-Nakes were key actors in changing the health education paradigm by integrating the education and health systems, facilitating stakeholders engagement, and interprofessional collaboration. The values driven by LAM-PTKes and LPUK-Nakes were fundamental for conflict management among the health professional society. Other disciplines can learn from the lessons of the health professional society in establishing LAM-PTKes and LPUK-Nakes. 8. The implementing agency has taken advantage of the momentum and opportunity created by the project for developing several innovative programs aimed for supporting the achievement of KPI target and at the same time, adding values to HPEQ programs. The innovative programs were also essential for developing the strategy for maintaining the sustainability of project outcomes. These programs included: the teaching hospital program, the qualification framework for health disciplines, the medical specialist study program, various policy formulation such as the joint decree on competence examination and on public university hospital, intermediate regulations of Medical Education Act No.20/2013, and the program for empowering students and young health professionals. 9. For ensuring effective publication of HPEQ outputs, the implementing agency has also created a social marketing division with a task to conduct evidence-based studies to support the publication of HPEQ program and outputs through official social media. This effort was also intended to stimulate the government, the stakeholders and the society in general to notice health education as an important issue. 10. The implementing agency with the approval of the government has an open access policy for the use of HPEQ data/outputs for further scientific research as evidenced by the use of HPEQ data by three masters and one doctorate program students. What will be the follow-up to sustain results post project 11. Consistent commitment from the government, the health professional society, young generations and the parliament is a key factor for maintaining the sustainability of HPEQ program. Government Institutionalization Arrangements 38

51 12. The Directorate of Academic and Student Affairs has formulated the KPI for quality improvement of higher education following some of HPEQ project s KPI, particularly the number of accredited study programs and the number of graduates passing the national competence examination. Furthermore, the new structure of MORTHE will include a directorate for accreditation and competence examination. Institutionalization Arrangement for the Associations of Health Higher Education and Professional Organizations 13. HPEQ project has provided much lesson in capacity building to strengthen both organizations. The methodology for developing various standards has been adopted by the organizations involved in HPEQ for other purposes. The methodology has also been adopted by other disciplines not only to develop standards, but also to develop the accreditation instruments, and the blue print for competence examination. 14. HPEQ project has also empowered the associations of health higher education and professional organizations at the sub-national level. These organizations have been active sub-nationally to disseminate updated policies and information regarding health education, especially those related to accreditation and competence examination. The organizations have also become more independent financially in providing technical support activities, and have improved capacity in formulating financial planning and unit cost for future programs. Recognition from External Stakeholders 15. The accreditation system developed by LAM-PTKes and competence examination system developed by LPUK-Nakes have been endorsed by international experts, such as the Liaison Committee on Medical Education (LCME) and the National Board of Medical Examiners (NBME). In addition, during the project MTR World Bank international expert has stated that the national examination system established for medicine was impressive and could be considered world class. Such international recognition is crucial for maintaining the sustainability of LAM-PTKes and LPUK-Nakes. Empowering Young Health Professionals and Health Students 16. GoI and health professional societies acknowledged the important participation and contribution of the young health professionals and health students as key stakeholders. Both groups have actively and importantly contributed in shaping health professional education policy making, particularly the implementation of inter-professional collaboration and inter-professional education, as important values embraced by the LAM-PTKes and LPUK-Nakes. HPEQ Project supported student participation through the development of (i) health student alliance (HPEQ Student), which has been 39

52 acknowledged as a model for the national program of student organization, and (ii) the Indonesian Young Health Professional Society (IYHPS). At the end of the project, HPEQ Students has committed to continue the program through the Indonesian Health Student Organizations Alliance (IHSOA), while the IYHPS members have committed to develop a professional and independent organization as a legal entity soon to be acknowledged by the Ministry of Justice and Human Rights. The empowerment of IYHPS and HPEQ Students is a key strategy for maintaining HPEQ sustainability. ICT System as an Asset for Sustainability 17. ICT as the backbone of project management has supported the implementing agency in conducting paperless office management system and in taking advantage of tele-media to reduce meeting costs. In its early implementation, HPEQ has also developed a blue print of the IT system as the information management system platform for LAM-PTKes and LPUK-Nakes. 18. By the end of the project, the implementing agency has developed the Knowledge Management System (KMS) regarded as an asset for the health professional community and government. KMS HPEQ is a web-based knowledge portal containing HPEQ products, recent policies and related references on health education. From project management perspective, KMS HPEQ is a form of accountability through systematic documentation and publication. KMS will also accommodate communication and opinion channeling among health professional community post HPEQ project. It is a user friendly interface easily accessed through 40

53 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Not applicable. 41

54 Annex 9. List of Supporting Documents Component 1: Strengthening Policies and Procedures for School Accreditation i. Higher Education Act No. UU No.12/2012 ii. iii. iv. Ministerial Decree No.49/2014 on National Standard for Higher Education (SN-Dikti) Ministerial Decree No.50/2014 on Quality Assurance System Of Higher Education (SPM-Dikti) Ministerial Decree No. No.87/2014 on Accreditation for Study Programs And Institutions v. Ministerial Decree No.291/P/2014 on Ratification of LAM PTKes Establishment vi. Ministerial Decree of Law and Human Rights No. AHU 30.AH.01.07/2014 on Ratification of LAM-PTKes as a Legal Entity Component 2: Certification of Graduates Using a National Competency-based Examination i. Higher Education Act No. UU No.12/2012 ii. iii. iv. Joint Ministerial Decree MoEC and MoH on Competence Examination for DIII Midwifery, DIII Nurse and Ners Ministerial Decree on National Committee for Competence Examination of DIII Midwifery, DIII Nurse and Ners for year 2014 Ministerial Decree No.36/2014 on Competence Examination for Medical and Dentistry Students v. DGHE Decree No. 27/2014 on Committee for Competence Examination of Medical Students vi. DGHE Decree No. 35/2014 on Committee for Competence Examination of Dentistry Students 42

55 MAP 43