EXPLORING THE IMPACT OF PERFORMANCE-BASED FINANCING ON HEALTH WORKERS PERFORMANCE IN BENIN

Transcription

1 EXPLORING THE IMPACT OF PERFORMANCE-BASED FINANCING ON HEALTH WORKERS PERFORMANCE IN BENIN Mylene Lagarde, Samantha Burn, Lionel Lawin, Kefilath Bello, Jean-Paul Dossou, Patrick Makoutode, Beatrice Sourou Goufodji, Christophe Lemiere, Maud Juquois September 2015

2 Exploring the impact of Performance-Based Financing on Health Workers Performance in Benin Mylene Lagarde a Samantha Burn c Lionel Lawin b Kefilath Bello b Jean-Paul Dossou b Patrick Makoutode b Beatrice Sourou Goufodji b Christophe Lemiere d Maud Juquois d a London School of Hygiene and Tropical Medicine, London, UK b Centre de Recherche en Reproduction Humaine et en Démographie, Cotonou, Bénin c Independent consultant, Washington DC, USA d Health, Nutrition and Population Global Practice, The World Bank, Washington DC, USA Abstract: Training, motivating and retaining human resources is crucial for the improvement of health outcomes, especially in low and middle-income countries (LMICs) where human resources availability and management have been recognized as one of the key health system s barriers to the achievement of the Millennium Development Goals. In recognition of the limitations of current financial incentives and/or remuneration levels, Performance- Based Financing (PBF) mechanisms have been introduced in many LMICs in recent years. This project took advantage of an existing field experiment funded by the World Bank (WB) that introduced PBF for some randomly selected primary care facilities in Benin in 8 pilot districts since March While the impact evaluation of the WB project focuses mainly on the effects of the financial incentives on the uptake of health services and health outcomes in the population, the current project complements this evaluation by collecting detailed information of individual health care provider performance and motivations, in order to test the impact and causal pathways through which financial incentives operate. The objective of this mixed-methods survey is to explore the impact of PBF on health worker performance as well as to propose some explanations. It examines the impact of PBF on different dimensions of health worker performance, including clinical productivity (i.e. absenteeism, average time per patient, reduction of slack, etc.), quality of care and responsiveness to patients as well as reasons for change, namely the improvement of cando skills and knowledge, working environment and/or will-do attitude (motivation/effort) or contextual factors (level of management autonomy). We surveyed 434 qualified health care workers (doctors, nurses, midwives) in 135 primary care facilities located in three different groups of facilities: (1) those receiving quarterly financial bonuses linked to the facility performance, (2) those receiving an equivalent amount of financial resources independently of their own performance and (3) some pure control ones, receiving no additional resources. We collected information on health workers' characteristics using a standard questionnaire, and measured several dimensions of the performance of health workers thanks to direct observations, patient exit interviews and a time-and-motion survey. ii

3 Results from the study suggest that the PBF intervention in Benin improved some aspects of the performance of health workers: positive impact on quality of care and responsiveness towards patients but has no significant impact on clinical productivity. There was no difference on the volume of activities in facilities that received the conditional bonus compared to those that received additional resources unconditionally. However, there was a positive impact of the bonuses on a variety of measures of quality of care provided by the staff (observed and reported by patients), which provides some evidence of the successful emphasis on quality of the PBF intervention in Benin. The absence of impact on clinical productivity can be explained by the strong focus of the PBF scheme in Benin on the quality of care. The absence of negative impact on the number of patients seen also suggests that HWs were operating below their production constraint. Comparing facilities that received the PBF bonuses to the facilities in the pure control group, we found relatively small and inconsistent effects of PBF. The PBF facilities were not associated with higher volume of activities, but there was a slight reduction in costs associated with PBF. There was no evidence of a difference in the resources of facilities available to facilities in the PBF group compared to the pure control group. For some of these outcomes, this lack of results might be partly due to the lack of power to detect differences in outcomes measured at the facility level. Besides, the lack of equivalence of the pure control group makes it harder to control for all potential factors that may explain differences (or lack thereof). Further analyses will use matching techniques to reduce differences between PBF and pure control facilities. Key words: Results-based financing, Performance-based financing, human resources for health quality of care, clinical productivity, health provider satisfaction and motivation, financial incentives, Benin Disclaimer: The findings, interpretations and conclusions expressed in the paper are entirely those of the authors, and do not represent the views of the World Bank, its Executive Directors, or the countries they represent. Correspondence details: Maud Juquois, Health, Nutrition and Population Global Practice, The World Bank Group, Washington DC, USA. Tel: ; mjuquois@worldbank.org iii

4 TABLE OF CONTENTS ACKOWLEDGEMENTS... vi LIST OF ABBREVIATIONS... vii 1. INTRODUCTION BACKGROUND THE PERFORMANCE-BASED FINANCING EXPERIMENT IN BENIN ANALYTICAL FRAMEWORK AND STUDY METHODS ANALYTICAL FRAMEWORK FOR EXPLORING HEALTH WORKERS PERFORMANCE STUDY DESIGN QUANTITATIVE TOOLS QUALITATIVE INTERVIEWS ECONOMETRIC APPROACH RESULTS DESCRIPTIVE STATISTICS Health care facilities Health workers Patient observations and interviews IMPACT OF PBF ON THE DIFFERENT DIMENSIONS OF PERFORMANCE Absence of impact on productivity and presence at work Positive effect of PBF on the quality of care Greater patients satisfaction and HWs responsiveness CAUSUAL PATHWAYS TO EXPLAIN PBF IMPACT ON PERFORMANCE Absence of significant impact of the can-do factors No significant effect of PBF on the will-do Contextual factors Summary of results SOME INSIGHTS FROM THE QUALITATIVE DATA POSITIVE EFFECTS OF PBF CHALLENGES ASSOCIATED WITH THE PBF PILOT SCHEME THE LIMITS OF PBF? CONCLUSIONS iv

5 REFERENCES APPENDICES DESCRIPTIVE STATISTICS TABLES ANALYSIS RESULTS COMPARISON OF PBF VS. PBF CONTROL FACILITIES Impact on productivity Impact on quality of care Impact on patient centredness Impact on cost to patients Impact on charging practices Causal pathway: Facility resources Causal pathway: monitoring and supervision Causal pathway: staff motivation ANALYSIS RESULTS COMPARISON OF PBF VS. PURE CONTROL FACILITIES Impact on productivity Impact on quality of care Impact on patient centredness Impact on patient satisfaction Impact on cost to patients Impact on charging practices Causal pathway: Facility resources Causal pathway: monitoring and supervision Causal pathway: Staff motivation v

6 ACKOWLEDGEMENTS This report is a component of a multi-country study assessing the linkage between Results- Based Financing and Human Resources for Health. The authors would like to thank the Health Results Innovation Trust Fund, funded by the governments of Norway and the United Kingdom and managed by the World Bank, for providing financial support. We would like to acknowledge the Ministry of Health of Benin for its support and cooperation in carrying out this study, in particular the Health System Strengthening Coordination Unit. We also acknowledge all health workers and patients who participated in the survey. The authors thank the Centre de Recherche en Reproduction Humaine et en Démographie for collecting the survey data. The report benefits substantially from the peer review conducted by Damien De Walque, Paul Jacob Robyn, and Shunsuke Mabuchi. We are grateful for the strategic guidance from Nicole Klingen and Trina Haque, Practice Managers, and Olusoji Adeyi, Director, Global Health Practice of The World Bank Group. vi

7 LIST OF ABBREVIATIONS ANC ARV CSA CSC DCOs FP HCA HIV HW INSAE ISBA LMICs LSHTM MoH NGO PBF PNC PNLS RDT SSA TB TMS WB Antenatal Care Antiretroviral Centre de Santé d Arrondissement Centre de Santé de Commune Direct Clinical Observations Family Planning Health Care Assistant Human Immunodeficiency Virus Health Worker Institut National de la Statistique et de l Analyse Économique (National Institute of Statistics and Economic Analysis) Institut des Sciences Biomédicales Appliquées (Institute of Biomedical Science) Low- and Middle-Income Countries London School of Hygiene and Tropical Medicine Ministry of Health Non-Governmental Organization Performance-Based Financing Postnatal Care Programme National de Lutte contre le Sida (National Program Against AIDS) Rapid Diagnostic Test Sub Saharan Africa Tuberculosis Time and Motion Survey World Bank vii

8 1. INTRODUCTION 1.1. BACKGROUND 1. Benin has made progress but not fast enough to reach the Millennium Development Goals related to child health and maternal and neonatal health. In 2014, Benin had achieved one of the highest rates for assisted deliveries across Sub Saharan African countries (SSA), with 87% of women delivering in health care facilities (MICS 2014). However, this average rate hides some inequalities: only 55.8% of women in the poorest quintile deliver in health facilities vs. 97.4% for the richest quintile (Institut National de la Statistique et de l Analyse Économique (INSAE), Programme National de Lutte contre le Sida (PNLS) et al. 2006). In addition, when comparing maternal mortality rates and rates of assisted delivery across SSA countries, Benin would be expected to have a lower maternal mortality (estimated maternal mortality rate decreased only from 490 per 100,000 live births to 340 in 2013). In a study carried out in the four biggest referral hospitals in the country, researchers found that 60% of maternal deaths were caused by low quality of care and could have been avoided (Saizonou, Godin et al. 2006). A comprehensive assessment of the health care system (World Bank 2009) underlined that poor quality of care was explained not only by a lack of resources but also by a low accountability of institutions and health workers (43% absenteeism rate in 2011, drug pilfering, illegal payments, moonlighting,..). In recognition of the limitations of current financial incentives to align the behaviors of health workers with the interests of patients, the government of Benin supported the implementation of a pilot scheme providing financial incentives to health workers for increased performance (World Bank 2010). In a nutshell, the Performance-Based Financing (PBF) pilot in Benin provides bonus payments to primary care facilities and hospitals based on the provision of various types of services and the quality of those services. PBF payments go directly to facilities and are used partly to pay for staff bonuses and partly to buy equipment and drugs. 2. Similar PBF schemes have been introduced in many low- and middle-income countries (LMIC) in recent years. While promising results have emerged about the ability of PBF to improve health utilization and, potentially, health outcomes, there has been much less indication of the causal pathways through which they may have operated, and the impact of PBF on providers behaviors have largely remained a black box. This is partly because so far the evaluations of PBF have fallen short of assessing the wider range of effects it can have at the micro level, in particular on the health workforce. As a result, whether these financial incentives work (partially) or not, policy-makers do not know why, and cannot modify these schemes or design alternative interventions to obtain better results. This study was funded by the World Bank to address this gap and look specifically at the impact of the PBF program in Benin on a wide range of outcomes measured at the health worker level. 1

9 1.2. THE PERFORMANCE-BASED FINANCING EXPERIMENT IN BENIN Design of the field experiment 3. Since March 2012, a PBF pilot has been implemented in eight health districts (out of the existing 34). In these districts, all public and not-for-profit facilities are included in the PBF pilot and are part of an experiment. Quarterly results of the PBF pilot are presented in the web based application: including documentation about the program. To evaluate the effects of PBF, the government of Benin agreed to let the World Bank (WB) carry out a field experiment in eight pilot districts (see map below). A combination of two interventions is tested through the Impact Evaluation: a. RBF conditional rewards (credits linked to results achieved by health centers) versus unconditional rewards (credits not linked to results achieved, see more below). b. Management autonomy versus no management autonomy. 4. In these eight districts, all public and private not-for-profit health centers were randomized to one of two treatment groups: Therefore, there are currently five groups in the IE: Management autonomy treatment (84 HF) Management autonomy control (88 HF) RBF treatment (85 FS) T1 40 HF T3 45HF RBF control (additional financing) (87 FS) T2 44 HF T4 43HF No intervention No intervention - - C 46 HF However, during implementation, managerial autonomy in the control and intervention facilities remains very limited, the difference between the 2 arms is very thin. It has been difficult to implement the intervention arm «managerial autonomy», even at a limited stage, when not fully endorse by the authorities at district and national levels. That is the reason, for this specific mid-line survey focusing on HRH performance, it was decided to consider only the PBF treatment and control groups. Facilities in all groups were subjected to the same level of monitoring and data verification. 2

10 Figure 1 : Eight pilot districts of the PBF field experiment in Benin The pilot districts are: (1) Banikoara, (2) Kouandé-Ouassa-Péhunco-Kérou, (3) Covè-Ouinhi-Zangnanado, (4) Bohicon-Zakpota-Zogbodomey, (5) Lokossa-Athiémè, (6) Adjohoun-Bonou-Dangbo, (7) Porto-Novo-Aguégués-Sèmè-Podji, (8) Ouidah-Kpomassè-Tori-Bossito. 6. After the field experiment was designed, a third group of facilities was identified to serve as a pure control group to test the joined effect of the additional resources and the performance monitoring. Those facilities were located in neighboring districts of the eight pilot districts, at a minimum distance of 15km from the district boundaries to avoid any contamination effect of the PBF intervention. 7. A baseline study (facilities and households surveys) was conducted in Data from this survey demonstrates that the IE groups were balanced before implementation of the intervention (see balance tables in Appendices). For the 2015 mixed-methods survey, only a sample of health facilities from the baseline study were surveyed (as detailed in section 2.2 about study design). Payment calculation in the PBF group 8. Similar to the seminal PBF program in Rwanda (Basinga, Gertler et al. 2011) and many other PBF schemes implemented in SSA countries (Fritsche, Soeters et al. 2014), the 3

11 PBF scheme in Benin pays for a large number (28) of maternal and child healthcare services conditioned on an overall facility quality assessment score. The formula used for payment to facility i in quarter t is in the PBF intervention arm is: Pay RBF it = ( j P j U jit ) Q it with 0 Q it 1 where P j is the payment per service unit j (e.g. institutional delivery or child preventive care visit), U jit is the number of patients who have used service j in facility i in period t, and Q it is the overall quality index of facility i in period. 9. The 28 service indicators (U jit ) and associated payment rates (P j ) are listed in Table 1. Some of the visit indicators are validated only if some specific aspects of content of care are followed. Table 1: Service rewarded by the PBF scheme and their unit price Unit Service indicators price (FCFA) Maternal Health indicators First Antenatal care visit (in first trimester)* 3,500 First Antenatal care visit (in first trimester) for a woman identified as poor * 2,800 Fourth Antenatal care visit* 3,000 Fourth Antenatal care visit for a woman identified as poor* 3,000 Assisted delivery in the facility 7,500 Assisted delivery in the facility for a woman identified as poor 6,000 First postnatal visit (7-10 days after delivery)** 3,500 Third postnatal visit (42-45 days after delivery)** 1,750 Family planning visit (new long-term contraceptive user) 6,300 Family planning visit (new short-term contraceptive user) 1,750 Malaria cases detected with RDT and treated in pregnant women 655 Acute malaria cases detected with RDT and treated in pregnant women 5,365 Pregnant women diagnosed HIV positive and put under ARV treatment 15,750 Appropriate and urgent referral to hospital for delivery*** 3,150 Child health indicators Child (11-59 months) growth monitoring/preventive care visit 420 Malaria cases detected with RDT and treated in children under 5y 330 Acute malaria cases detected with RDT and treated in children under 5y 5,890 New BCG vaccinations 875 Children who have received the PENTA 3 vaccine 700 Children who completed vaccinations on time 3,000 HIV positive children initiated to ARV treatment in the last month 19,250 4

12 Other curative services indicators New curative consultation (child and adult) 350 New curative consultation (child and adult) for patients identified as poor 1,750 HIV positive patients under ARV treatment 8,750 IST cases diagnosed and treated 700 TB cases detected 14,350 TB cases detected and fully treated 15,000 Appropriate referral to hospital*** 1,050 * including VAT, mosquito net and micronutrients given; and sulfadoxine for 4 th antenatal visit. ** women must have received completed postnatal check-up. *** referrals must be confirmed by hospital that patient was treated and that referral was necessary 10. The bonus paid to each facility is adjusted based on the overall facility quality Q it. This quality index Qit is a function of structural and process measures of quality. Structural measures are the extent to which the facility has the adequate and necessary resources (infrastructure, equipment, drugs, medical supplies and personnel) to be able to deliver health care services. Process measures are the clinical content of care provided for specific services (e.g. proper examinations during consultations observed). The formula for the quality index is: Q it = k ω k S kit with k ω k = 1 where S kit is the share of indicators for service k that are met by facility i in period t, and ω k is the weight for service k. If a facility has perfect structural and process quality, then all the S kit take on value 1 and the overall quality index is equal to 1. In this case, the facility is paid the maximum possible bonus for the services provided. Table 2 details the services that are included in the quality index, their weights and the relative importance of structural and process indicators in the computation of the score for each service k. 5

13 Table 2 : Areas of quality and their weights in the PBF quality score Areas of quality Number of indicators Weight 1 General administration % 2 Monitoring, evaluation and information system % 3 Hygiene, environment and sterilization % 4 Curative care (outpatient and inpatient) % 5 Maternal care 5 8.5% 6 Prenatal care % 7 Family Planning 9 4.0% 8 Immunization and children services % 9 HIV/Aids % 10 Tuberculosis and Leprosy 4 5.7% 11 Laboratory 6 3.5% 12 Minor surgery 5 2.0% 13 Pharmacy management 8 7.7% 14 Financial management % Total % 11. Since Q it enters the payment formula as a multiplicative factor bounded between 0 and 1, it effectively almost systematically lowers the payment for the output indicators. Indeed, the facility receives the full bonus payment when the index equals one which only happens if the facility meets all of the 124 quality criteria included in the calculation of Q it. By contrast, if the quality index is less than one, PBF payments are discounted for all services provided at the facility. In other words, facilities are penalised for not meeting the desired level of quality, they are not rewarded for good quality. 6

14 Payment calculation in the PBF control group 12. The idea of the PBF control treatment was to provide additional financial resources to facilities, independently of their performance, that would on average be the same as the additional resources obtained by PBF facilities. Based on its main characteristics (number of staff, size of catchment area, rural or urban location), each facility i in the PBF control group was matched to an average facility k made up of facilities in the PBF treatment arm that have similar characteristics 1. Ideally each facility i would receive Pay kt RBF, the average bonus paid in quarter t to the sub-group k of facilities from the PBF treatment group. 13. However, due to the requirements of the funder, the design had to include some form of link to the performance of facilities. The following formula was therefore used for payment in quarter t to facility i in the control group, matched to average facility k in the PBF intervention arm: Pay C ikt = min (( j P j RBF U jit ), Pay kt ) where j P j U jit is the bonus resulting from providing all the services j, as defined in the PBF treatment group 2 with no adjustment for quality. In effect, only a low-performing facility would receive a bonus Pay C ikt that would be different from the average additional resources initially planned. Verification process 14. Facilities submit monthly activity reports (U jit ) and quarterly requests for payment to the central PBF Unit at the MoH, which is responsible for verifying the data and authorizing payment. The PBF Unit verifies the facility reports by sending auditors to facilities on a quarterly basis on an unannounced day. Each visit is carried out by a team composed of representatives from the district, independent controllers (technical assistants who are part of the external controlling agency set up by the PBF project) and representatives of the community. To approve the activities U jit submitted for payment, they go through the facility records (independent controllers) and conduct random interview with patients (representatives of community) who have been attending the facilities in the past 3 months. During the verification process, auditors not only check that the reported patients are reported in the medical records and actually existed, but for some services (indicated by an asterisk in Table 1) also that specific process measures of quality of care were completed. In other words, activities U jit are approved only if they were delivered in the way it was supposed to (e.g. deliveries assisted by a qualified staff; prenatal care visits during which they also look through randomly chosen medical records for some of the activities (e.g. prenatal care). To determine the quality index Q it, district representatives and independent controllers rate the facility (the quality measure is done by a team of other hospitals peers for district hospitals). 1 All facilities in the intervention arm were split up in six different relatively homogenous groups based on the characteristics that were thought to drive their output (number of staff, size of catchment area, rural or urban location). 2 P j is the payment per service unit j (e.g. institutional delivery or child preventive care visit), U jit is the number of patients who have used service j in facility i in period t 7

15 2. ANALYTICAL FRAMEWORK AND STUDY METHODS 2.1. ANALYTICAL FRAMEWORK FOR EXPLORING HEALTH WORKERS PERFORMANCE 15. The proposed analytical framework to analyze the impact of PBF on health workers performance is the following: 16. The first box (extreme left) describes the three dimensions of health workers performance: - quality of care; - responsiveness (i.e. politeness toward patients, respect, reduced waiting time ); - productivity and presence at work. 17. These three dimensions of performance are influenced by two main factors (box 2). - Some factors are related to the can-do and obviously reflect the resources given to health workers to provide adequate performance. The can-do factor includes the skills and knowledge (acquired through training) as well as the working environment (adequacy of equipment and supplies). - The other factor is related to the will-do and refers to the degree of motivation/effort by health workers. 18. In turn, effort or motivation is influenced by several factors (box 3), including: - revenues earned by health workers (including PBF bonuses); - degree of monitoring by peers and supervisors; 8

16 - self-efficacy; - intrinsic motivation 19. It is assumed that some of these factors are directly influenced by PBF. Most of the variables are also influenced by contextual factors (box 5). Two of them will be explored: - the degree of management autonomy enjoyed by the facilities; - the degree of market competition between facilities (which depends on the density of facilities in each area) 2.2. STUDY DESIGN 20. This study relies on a cross-sectional survey of qualified health care providers (nurses, midwives, doctors) working in primary care facilities in the three groups. A total of 434 health workers in 135 facilities were selected following a multi-stage stratified random sampling strategy. 21. To demonstrate a minimum of a 35% difference in the outcome measures between the control and intervention facilities, with a power of 80% and a statistical significance level of 5%, we estimated a need for a sample of around 35 health centers in each of the groups. Calculations were based on results of studies looking at health workers performance using similar tools in African countries to estimate outcome means and standard deviations in the control group, and expected achievements in treatment facilities that would be significant enough to merit investment in a PBF programme and have an impact on the quality of care provided (and ultimately health outcomes). Given that health practices are clustered within districts, sample size calculations were adjusted for the variability in the number of facilities per district and the intra-cluster correlation (expected to be around 0.1), which is a consequence of randomizing at one level (districts) and analyzing at another (health workers) health centers from each group were surveyed in order to mitigate potential risks, such as high absenteeism rates and existence of ghost workers who would prevent us from observing and interviewing enough health workers. A multi-stage stratified random sampling strategy was used to select the health care facilities to be visited. First, using the baseline survey data and updated information from the PBF implementation team, we will list all facilities in the following three groups: - PBF treatment group (T1): Additional budget linked to performance - PBF control group (T2): Additional budget not linked to performance - Pure control group (C1): No additional budget 23. Then from updated information about the staffing of these facilities, only the facilities that have at least two qualified health workers were retained in the sampling frame. Facilities were then categorized as urban or rural health centers. Within these two strata, a random 9

17 sample proportional to size was chosen (31 rural centers and 14 urban centers in each group). Finally, workers from the selected facilities were sampled In each facility, we sampled several qualified health workers to take part in the survey. Where there were five or fewer workers, all the health workers in the selected facilities were invited to respond to the survey 3 and one nurse and one midwife were randomly selected to be observed. Where there were more than five HWs, we included at least one HW of each type (doctor, nurse and midwife), and the rest of the sample distributed with probability proportionate to number of workers of each type within the facility, with a maximum of five HWs in total. As in smaller facilities, only two HWs (one nurse and one midwife) were chosen to be observed QUANTITATIVE TOOLS 26. Data collection was organised between mid-january and mid-march 2015, with 15 teams of two fieldworkers 4, each visiting about nine facilities, and administering a series of survey tools to all sampled health workers. Details about the tools used are provided in Appendices. 3 Meaning that the HW questionnaire (including vignettes) will be administered to them. 4 One sociologist and one qualified medical worker (midwife, nurse or doctor). 10

18 Table 3: Summary of data collection tools used Tool Main outcomes and variables captured Administered to Facility survey - Availability of essential drugs Facility manager - Availability of essential equipment - Absenteeism - Volume and type of consultation - Financial autonomy - Support from district - Bonus PBF Health worker individual questionnaire Time and Motion survey (TMS) - Socio-demographic characteristics - Household assets - Earnings - Intrinsic motivation - Self-efficacy - Altruism towards patients - Technical knowledge and competence (vignettes) - Time spent in the facility - Productive time spent in the facility - Contact time with patients - Average duration of consultation 5 or fewer health workers. Where total number of staff exceeds sample size, workers randomly sampled as described above. 2 qualified health workers in the facility (one working on MCH activities, the other one not working on MCH activities) Direct Clinical Observations (DCO) - Responsiveness towards patients - Quality of care The 2 qualified health workers observed during the TMS. Each individual will be observed for 20 consecutive consultations Patient exit interviews - Staff responsiveness towards patients All the patients being received by the HWs observed In-depth interview - Challenges and processes in PBF bonus redistribution - Changes in management and work organization One health care staff in one every 3 facilities 11

19 27. A short questionnaire on the facility environment was administered to the most senior manager present. Where the facility did not have a designated gestionnaire, the questionnaire was administered to the most senior member of clinical staff present, or an individual delegated by this person. 28. The health-worker survey was administered to all qualified staff sampled according to the procedure detailed in the study design section. The questionnaire included a background section gathering socio-demographic information, earnings, job satisfaction, orientation towards intrinsic and extrinsic motivation, altruism 5 and clinical knowledge (assessed through six vignettes 6 ). In this study, vignettes were administered as a knowledge exercise in which the enumerator asked the HW to list the questions or activities he or she would undertake for a patient presenting specific symptoms. There were three parts to each case study: historytaking, physical examinations and diagnostic & treatment recommendations. The survey was interactive in the sense that the enumerator provided additional information to the HW based on the hypothetical questions/examinations the latter suggested s-he would do. 29. In each facility, two HWs were followed over the course of one entire working day spent at the facility (at most 8 hours). The enumerator was tasked to record the nature and duration of all activities undertaken by the HW (the use of a tablet computer automatically recorded the time at which the change in activity took place). Activities carried out by health worker were grouped into several broad categories (productive and non-productive work, clinical work or non-clinical work). 30. During this observation day, HWs were also observed in their interaction with patients. For each ANC visit, children s curative consultations and general adult outpatient consultation they observed, enumerators evaluated against specific checklists the performance of the health care provider. For ANC visits the checklist was based on national guidelines, while for children and adult curative consultations, more generic tools were developed based on the typical primary symptoms for which the patient comes (e.g. fever, cough, diarrhea, STDs for adults). All checklists included three different components: history-taking, physical examination and attitude towards patients. In addition the ANC checklist included one on treatment and recommendations. Using DCOs, we can obtain construct two index measures of the quality of providers care: one reflecting adherence to case-specific checklists of essential and recommended care and one reflecting patient-centeredness. 31. Finally, to assess staff responsiveness towards patients, we carried out patient exit interviews with all patients received by the health worker observed for an ANC or a curative consultation. The patient questionnaire aimed to capture the patients point of view with regard to staff politeness, their general attitude towards patients, and their responsiveness. We also included questions about the type of questions asked during the consultation and physical examination performed, in order to match the DCO checklist. 5 To measure altruism or dedication towards patients, a simple version of the dictator game was played. Each HW was given two envelopes at the end of the interview: one filled with ten 500FCFA bills and an empty one. The HW was told that s-he had to decide how much to put in the empty envelope, seal it and put it in a sealed box provided by the HW to be later donated to a renown local NGO. 6 Three children cases, two adult cases and one pregnant women case study. 12

20 2.4. QUALITATIVE INTERVIEWS 32. One in-depth interview was carried out by the sociologist of the team with a randomly selected health worker from one of the three main staff categories (nurses, midwives and doctors) in three facilities visited by the team (and pre-determined according to the planning of visits, in order to cover a diverse range of situations). 33. In total 32 interviews were carried out, and tacked the following topics: - Work organization and facility management - Processes and debates around revenue distribution between staff - Resources management within the facilities - Discourse of HWs on moral attitudes and values (towards patients, towards corrupt behaviours, etc.) 34. Ethical approval for the research was obtained from the LSHTM research ethics committee and from the Comité d Ethique et de Recherche of the Institut des Sciences Biomédicales Appliquées (ISBA) in Benin. In addition, permission letters from the Ministry of Health were obtained to introduce the research team to each facility visited ECONOMETRIC APPROACH 35. As there are three arms in the experiment, we report the results of the analyses comparing outcomes of health workers in the facilities from two different arms of the trial, between PBF and PBF control and also between PBF and PBF pure control. While the randomized design means the PBF and PBF control groups are expected to be balanced, the inclusion of control variables will increase the precision of the estimate of the coefficient on the treatment variable. 36. We look at a range of performance outcomes captured by our different survey tools these outcomes are sometimes measured at the health worker level, sometimes at the consultation (patient) level, sometimes at the facility level. Given the emphasis of the PBF on maternal care, we distinguish the results between ANC patients and curative patients, as we would expect the impact of the PBF to be stronger with ANC patients. 37. To test whether the performance (noted PERFi) of staff working in PBF facilities is different from that of staff working in non-pbf facilities, we use the following econometric specification: PERFi = β0+ β1pbf + β2facilityf + β3hwi where PBF will take the value 1 when the provider works in a PBF facility, 0 if they work in the control arm of the trial. ; HWi is a vector of individual-level characteristics of the health worker observed or interviewed: type of health worker, years of experience in the health care sector, whether they are new in the facility, whether they are the main bread-winner in their 13

21 household and their household wealth. FACILITYf is a vector of facility-level characteristics (such as facility type, degree of competition proxied by the number of public or private facilities within 5 km, whether the facility has electricity, whether it is located in a rural area). All specifications are clustered at the district (zone de santé) level. Given the randomization of facilities across the two arms (PBF and PBF control), the coefficient β1 can be interpreted as the impact of the PBF intervention. For the PBF and pure control arms, even if facilities were not randomized in these groups, they can be considered as comparable (see descriptive statistics). 38. To test whether the outcomes (noted Yp) of patients visiting PBF facilities are different from that of patients visiting non-pbf facilities, we use the following econometric specification: Yp = β0+ β1pbf + β2facilityf + β3patientp where PBF will take the value 1 when the provider works in a PBF facility, 0 if they work in the control arm of the trial. PATIENTp is a vector of individual-level patient characteristics of the patient observed or interviewed: age of the patient, whether the consultation is the first ANC in case of ANC consultation. FACILITYf is a vector of facility-level characteristics (such as facility type, degree of competition proxied by the number of public or private facilities within 5 km, whether the facility has electricity, whether it is located in a rural area). As before, the coefficient β1 can be interpreted as the impact of the PBF intervention. 14

22 3. RESULTS 3.1. DESCRIPTIVE STATISTICS Health care facilities 39. Health facilities in the three groups (PBF, PBF control and pure control) presents similar the basic characteristics, as shown in Table 4. The facilities are similar in terms of facility type, number of competing facilities (number of facilities within 5km), time to reference hospital and in terms of opening hours over the week. Facilities in the PBF and PBF control groups are more likely to have electricity, and have greater availability of basic equipment. However, there is no significant difference between the three groups in terms of availability of drugs and vaccines, running water or toilet facilities. There is some indication that facilities in the PBF group conducted more adult consultations in the three months prior to the facility survey than those in either the PBF control or pure control groups; however this difference is not statistically significant at the 95% confidence level, and there are no differences in the volume of other services provided, including first ante-natal appointments, vaccinations, family planning and child curative consultations. Table 4: Basic characteristics of health care facilities in the sample (from mixed methods survey 2015) PBF PBF control Pure Control Total P-value 1 Type of health facility Centre de santé de commune (CSC) Centre de santé d arrondissement (CSA) Formation sanitaire isolée Total Type of PBF intervention 0.010** % recognising PBF % recognising PBF control % don't know /not sure # of public facilities within 5km (2.25) (1.81) (1.35) (1.86) # of private facilities within 5km (3.51) (5.35) (4.91) (4.63) Catchment area size 16,158 17,383 22,181 18, (10386) (14803) (21053) (16115) Time to reference hospital (mn) (66.49) (123.56) (100.85) (100.24) Number of opening days (0.42) (0.82) (0.84) (0.72) Number of opening hours/day

23 (7.66) (8.08) (7.85) (7.82) Does the facility have (access to) electricity ** Water Toilets for patients Separate toilets for staff Telephone Equipment % of basic equipment available *** % of basic consumables available % of basic drugs available % of vaccines available Average monthly utilisation of services (Oct-Dec 2014) All consultations (268.35) (173.71) (133.83) (200.04) Adult consultations (171.04) (93.35) (69.31) (121.13) Children consultations (107.78) (95.95) (73.67) (92.88) Completely vaccinated children (58.5) (26.84) (33.1) (41.82) Deliveries (18.94) (22.73) (17.74) (19.78) First Antenatal consultation (45.57) (30.36) (25.79) (34.73) Family Planning consultations (14.91) (10.73) (30.42) (20.6) # of qualified medical staff *** (8.76) (5.85) (9.84) (8.27) In urban areas In rural areas *** Strike ratio /facility Pure absenteeism rate/facility (1) Based on Chi2 or one-way anova tests where Ho assumes equality across the three groups. (2) staff absent but not officially on strike. 40. The overall number of qualified staff (doctors, midwives and nurses) differs slightly between the three groups, with the PBF control appearing to have fewer staff, including nurses, midwives and healthcare assistants. Overall, the average ratio of qualified medical staff on strike 7 per facility was 9% in the facilities surveyed, but the average ratio was much higher for nurses (nearly 25%) and midwives (20%). 41. Surprisingly there is no statistical difference across the three groups for the financial revenues from various sources (reported in Table 5), despite the existence of PBF credits for 7 During the data collection period, there was an important strike of public health workers in Benin. 16

24 the PBF treatment and PBF control groups. Still resources available to PBF facilities were higher than financial resources available to pure control facilities (the difference may not be statistically different due to the large variation in the data, as reflected by large standard deviations). While money from PBF was highest in the PBF treatment group, the data suggest that facilities in this group received less money from the central budget allocated through the districts. On the other hand, facilities in the PBF treatment group generated more money from user charges. Table 5: Revenues of facilities in the sample (in FCFA) PBF PBF control Pure Control Total P- value 1 User fees from drugs 9,108,208 7,845,982 7,670,485 8,213,672 User fees from health services 3,119,751 2,204,551 2,437,927 2,593,167 Budget allocated through district 499,592 1,775, ,713 1,032,708 Investment fund (infrastructure) 1,282,051-6, ,603 NGO donations 46,053 89,524-45,578 PBF credits 3,224,530 2,507,987-2,875,215 Other sources 16, , , ,869 Total 2014 revenues in FCFA 16,722,025 13,406,766 11,183,889 13,770, (SD) (19,555,648) (13,299,370) (9,150,361) (14,709,165) (1) Based on Chi2 or one-way anova tests where Ho assumes equality across the three groups Health workers 42. Basic descriptive statistics from the health-worker survey, administered to up to 5 health-workers per facility, show that the demographics and administrative status of the workers in the three groups is broadly comparable (given in Table 6), though slightly more registered nurses were interviewed in the PBF facilities. Also there is some indication that workers in the PBF control facilities may have slightly smaller households and have spent slightly less time working at the current facility. There is no significant difference in altruism between the groups, as measured by the amount of money given to a charitable cause during the dictator game. 17

25 Table 6: Socio-professional profile of health worker sample PBF PBF control Pure Control Total P- value 1 Staff qualification Doctors Midwives Nurses. of which Registered nurses Professional nurses Health care assistants Total Work status % Civil servants % Permanent contract % Short contracts # of years working in the health care sector (SD) # of years working in current facility (SD) (6.14) 3.65 (4.07) (5.75) 3.10 (3.92) (5.60) 4.31 (5.17) (5.84) 3.66 (4.41) * % Female % for which their income is main revenue of household # of household members * (2.22) (2.22) (3.02) (2.51) Altruism Amount donated ( ) ( ) ( ) (1401.9) % of money donated (1) Based on Chi2 or one-way anova tests where Ho assumes equality across the three groups. 43. In terms of the revenues reported by the health workers, monthly base salary (around 125,000 FCFA) and total allowances were approximately similar between the groups (Table 7). Base salary was slightly lower in the PBF control, possibly reflecting slightly more HCAs in the sample. Despite apparent differences, PBF bonuses were not found to be significantly higher in the PBF control group than in the PBF group (although this may due to a power issue). Additionally, there is no significant difference in PBF bonus as a percent of base salary between the PBF and PBF control. There is a large difference in the size of bonuses in North Benin (departments Atacora and Alibori) compared with in the South: PBF bonuses are on average 67% as large as base salary, compared with 8% in the South. 18

26 44. Almost all workers report some unpaid salary and/or bonuses. Despite the PBF bonuses, overall worker remuneration does not appear to be higher in the PBF facilities than in the control facilities, and there is no difference in workers feelings of satisfaction with the remuneration they receive. Very few workers report receiving revenue directly from patients and there was no significant difference across groups. Table 7: Health workers self-reported revenues PBF PBF control Pure Control Total P- value 1 Monthly base salary 126, , , , Monthly total allowances 32, , , , Per diem 85, , , , N Direct (illegal) income from patients # 11,863 14,329 13,906 13, N PBF Bonus in , , , PBF Bonus in first half of , , , PBF Bonus (first half 2014) as % of base salary In the North In the South Satisfaction with total revenue (out of 10) % reporting private practice N Number of weekly private patients Revenue from private practice 47, , , , (1) Based on Chi2 or one-way anova tests where Ho assumes equality across the three groups Patient observations and interviews 45. Overall a total of 3,331 patients were interviewed, with 2,043 patients exiting ANC consultations and 1,005 having visited the facility for a curative consultation (564 for children and 441 for adults). These basic information about patients suggest some differences across the groups on population covered (Table 8). In particular, access to health facilities for patients in the pure control group seem a bit more difficult, as proxied by the travel time to facility, and the cost of care incurred at the facility. 19

27 Table 8: Patient characteristics PBF PBF control Pure Control Total p-value 1 Number of consultations, by type 0.007** Antenatal consultation Child curative consultation Adult curative consultation Other preventive consultation (FP, immunization, PNC) Total Patient gender 0.037** Male Female Patient age (child curative consultations) 0.029** Under 1y y y y y Patient age (adult curative consultations) y y y Over 31y Travel time to facility *** (30.51) (34.49) (69.95) (50.08) Cost of care Travel cost (FCFA) (276.3) (249.54) (311.14) (282.24) Total cost at facility (FCFA) ** (1800.7) (1583.1) (2996.3) (2270.7) Direct payment to HW (36.16) (32.58) (21.55) (30.26) (1) Based on Chi2 or one-way anova tests where H 0 assumes equality across the three groups. 46. Table 9 reports the total number of consultations observed and the average number of consultations observed for each health worker. The number of consultation per health worker suggest that further in-depth analysis will probably be possible for ANC consultations, but due to the small numbers of curative consultations observed, more sophisticated analysis will probably not be possible for child/adult consultations due to lack of statistical power. 20