Assessing the quality of care in a new nation: South Sudan s first national health facility assessment

Tropical Medicine and International Health doi:10.1111/tmi.12363 volume 19 no 10 pp 1237 1248 october 2014 Assessing the quality of care in a new nation: South Sudan s first national health facility assessment Sima Berendes 1, Richard L. Lako 2, Donald Whitson 1, Simon Gould 1 and Joseph J. Valadez 1 1 Liverpool School of Tropical Medicine, International Public Health Department, Liverpool, UK 2 Ministry of Health of the Republic of South Sudan, Juba, Sudan Abstract objectives We adapted a rapid quality of care monitoring method to a fragile state with two aims: to assess the delivery of child health services in South Sudan at the time of independence and to strengthen local capacity to perform regular rapid health facility assessments. methods Using a two-stage lot quality assurance sampling (LQAS) design, we conducted a national cross-sectional survey among 156 randomly selected health facilities in 10 states. In each of these facilities, we obtained information on a range of access, input, process and performance indicators during structured interviews and observations. results Quality of care was poor with all states failing to achieve the 80% target for 14 of 19 indicators. For example, only 12% of facilities were classified as acceptable for their adequate utilisation by the population for sick-child consultations, 16% for staffing, 3% for having infection control supplies available and 0% for having all child care guidelines. Health worker performance was categorised as acceptable in only 6% of cases related to sick-child assessments, 38% related to medical treatment for the given diagnosis and 33% related to patient counselling on how to administer the prescribed drugs. Best performance was recorded for availability of in-service training and supervision, for seven and ten states, respectively. conclusions Despite ongoing instability, the Ministry of Health developed capacity to use LQAS for measuring quality of care nationally and state-by-state, which will support efficient and equitable resource allocation. Overall, our data revealed a desperate need for improving the quality of care in all states. keywords quality of care, capacity building, monitoring and evaluation, national survey, lot quality assurance sampling, Africa, South Sudan, fragile state Introduction The Republic of South Sudan (RSS), the world s newest country, is in urgent need of an inexpensive, decentralised method to assess and recurrently monitor the quality of its healthcare services. South Sudan became independent in July 2011 after the 2005 peace agreement ended Africa s longest civil war, which claimed >2.5 million lives. The prolonged Sudan South Sudan conflict, domestic conflicts, frequent droughts, the influx of refugees from Sudan and internal displacement have deprived the population of basic needs and put pressure on the country s essential services. Currently, >70% of the population is illiterate, the maternal mortality ratio (2050 deaths/100 000 live-births) is the highest in The copyright line for this article was changed on 28 March 2016 after original online publication. the world, and under-five mortality (106 deaths/1000 live-births) is one of the highest (Wakabi 2011; MOH 2012). At the time of independence, South Sudan still suffered from chronically intermittent insecurity due to border conflicts with Sudan, intertribal cattle raiding, other ethnic conflicts and activities of rebel militia groups (WFP 2012; BBC 2014). However, it was stable enough to start moving from a fragmented humanitarian to a coordinated developmental approach to health service delivery. Among the priority strategic objectives in the nation s first Health-Sector Development Plan was establishing a strong monitoring and evaluation (M&E) system for evidence-based decision-making (MOH 2011b). Currently, no peer-reviewed scientific publications exist about the quality of health care (QoC) in South Sudan. The results of a health facility mapping (HFM) survey (2010 2011) documented the dire state of the country s 2014 The Authors Tropical Medicine & International Health Published by John Wiley & Sons Ltd. 1237 This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made.

health infrastructure (MOH 2011a). The HFM mapped the physical location of health facilities (HFs) and measured access and input indicators. Being a census it was time-consuming and expensive. At the time, the MOH recognised the need for a complementary, more rapid survey to regularly assess process and output indicators in addition to access and inputs. RSS decided to use the Lot Quality Assurance Sampling (LQAS) method for QoC assessments as it is relatively inexpensive, rapid, uses small samples of HFs and patients and produces data appropriate for statistical inferences on both national and state levels. LQAS is a classification method originally developed for industrial quality control (Dodge & Romig 1929) and subsequently adapted to health sciences (Valadez 1991). It has been applied in different health arenas (Robertson & Valadez 2006), including for quality assurance of health programmes (Valadez et al. 1996, 1997). Recently, we used LQAS for the rapid assessment of public and private HFs in Nigeria (Berendes et al. 2012; Oladele et al. 2012). Here, we adapt a similar LQAS design to a fragile state to rapidly assess the quality of health delivery services at the time of independence and to strengthen the MOH s capacity to perform regular rapid assessments. Methods Overall study design and eligibility This rapid health facility assessment (r-hfa) is a national cross-sectional survey among a sample of 14 16 HFs in each of 10 states totalling 156 randomly selected functioning facilities. We included public and private not-forprofit facilities from all levels [hospitals, primary healthcare centres (PHCC) and primary healthcare units (PHCU)]. Facilities were eligible if they were accessible, open for service, had a physical building, equipment and drugs, and 1 technical staff (Table 1). In each facility, we assessed infrastructure, equipment, material and supplies, interviewed health workers (HW) and reviewed patient records. In addition, we observed six consecutive consultations of sick children <5 years and interviewed their caretakers. Study instruments We adapted an r-hfa tool, developed by a multi-agency working group (USAID MCHIP 2008), for use in South Sudan to assess access, inputs, processes and HW performance of HFs. We used services for sick children <5 years as a proxy for all services, but also included select indicators on antenatal care (ANC) and neonatal care. We adapted indicator definitions to conform to the norms in the Basic Package of Health and Nutrition Services (MOH 2009), the Essential Medicine List (MOH 2007) and the WHO Integrated Management of Childhood Illnesses (IMCI) protocols for South Sudan (Table 2). We refined the tool during pretesting. Using LQAS for sampling and classification We used a two-stage LQAS sampling design (Oladele et al. 2012). Firstly, we sampled HFs to assess access, input and process indicators in each state. Secondly, within each facility, we sampled the most experienced HW present on the survey day to assess HW performance. We then sampled sick children to assess the selected HW s clinical skills. We collected and analysed rapid-hf-assessment-lqas (r-hfa-lqas) data using a sampling plan that specified sample sizes for HF (n HF ) and sick children (n SC ), and decision rules (d). d is the minimum number of HF in the sample that must pass in order for the state to be classified as reaching a performance standard (see the next section). Assessments of health facility access, inputs and processes. This r-hfa uses the hypergeometric rather than the binomial formula to compute HF sample sizes, given the small number of HFs present in each state (Valadez 1991; FANTA Project). The upper performance threshold (p-upper or p U ) for identifying acceptably performing states the MOH set at 80%; the lower threshold (p L ) was 30% lower (which is typical) at 50%. Therefore, the performance target for a state is that 80% or more of the HFs perform according to the national guidelines. The sample sizes, n HF, and decision rules, d, were selected to minimise the statistical misclassification errors so that neither alphanor beta-error exceeded 10%. These parameters result in a sample size of n HF = 14 HF and a decision rule of d = 10 for two states (Unity, and Northern Bahr-el-Ghazal) and n HF = 16 HF and d = 11 for the remaining eight states. The average alpha-error across the 10 states was 7.2% (min = 5.7%, max = 9.8%); the average beta-error was 8.2% (min = 6.5%, max = 9.6%). Assessments of health worker performance. To assess a clinician s performance in the sample of HF, we used the binomial model (Valadez 1991) with p U = 95% and p L = 50% (Valadez 1991; Valadez et al. 1997; Oladele et al. 2012). For both, the observations of consultations and patient exit interviews we used a sampling plan of n SC = 6 and d = 5 (alpha = 3% and beta = 10.9%), meaning that a HW passed as acceptable for a procedure if he/she performed according to the national guideline in 1238 2014 The Authors Tropical Medicine & International Health Published by John Wiley & Sons Ltd.

Table 1 Number and types of facilities sampled for rapid health facility assessment in South Sudan 2011 State Total no. of HFs (N) Target sample size (n) Substitutions of sampled HFs* Type and number (n) of final sample of HFs PHCU PHCC Hospital Total Public Private Total Upper Nile 107 16 4 11 5 0 16 9 7 16 Jonglei 213 16 14 4 12 0 16 7 9 16 Unity 66 14 5 11 3 0 14 3 11 14 Warrap 84 16 2 9 7 0 16 14 2 16 Northern 56 14 5 10 3 1 14 8 6 14 Bahr-el-Ghazal Western 81 16 3 10 6 0 16 16 0 16 Bahr-el-Ghazal Lakes 73 16 8 7 5 4 16 16 0 16 Western Equatoria 139 16 7 4 11 1 16 16 0 16 Central Equatoria 152 16 6 8 7 1 16 11 5 16 Eastern Equatoria 109 16 9 6 8 2 16 12 4 16 Total 1080 156 63 80 67 9 156 112 44 156 *HFs that did not meet eligibility criteria during sampling (e.g. due to geographical inaccessibility and/or insecurity) were substituted using Simple Random Sampling. Data collection had to be completed prior to the rainy season when flooding renders many areas inaccessible. However, even during the dry season some remote areas are inaccessible due to the lack or bad conditions of access ways. Security problems occurred mainly in chronically intermittently unsafe areas, such as the border areas between Sudan and South Sudan, but also elsewhere due to renegade rebel attacks, armed skirmishes, cattle raiding, and intercommunal conflict (e.g. between agriculturalists and pastoralists), or due to banditry on the road. For example, Pibor and Pochalla counties in Jonglei State had to be excluded due to a combination of bad roads and chronic insecurity. The timing of our data collection (February April 2011) overlapped with the period (March May 2011) having the most localised conflicts due to armed skirmishes and cattle raiding. It is the period prior to the lean season when food stocks are depleted, and competition for water and pasture is highest (WFP 2012). HFs that were accessible, but found to be not functioning during data collection were substituted by the nearest HF in the same area ( payam ). PHCU, primary healthcare unit: offers basic primary care, PHCC, primary healthcare centre: offers primary care, laboratory and maternity care. Hospitals that offer secondary care including government, private cooperating facilities or NGO facilities, but excluding specialty hospitals and clinics. Private non-profit. Total numbers of HFs in South Sudan are indicated in bold. five of six cases (one mistake allowed). This 6:5 design has been used previously (Valadez et al. 1996, 1997; Oladele et al. 2012). In some facilities, fewer than six children under 5 years were seen on a single day. In those cases, HF remained in the sample if the misclassification error was 20% for both alpha and beta. Facilities with too few observations to allow for classification were excluded from the analysis; no more than six facilities ever had this condition for any individual item; an additional five HF had no consultations due to the lack of patients (Table S1). See Figure S1 for details of pass / fail classifications of HW performance for different sampling plans. Data collection, entry and analysis We asked each state to select three state (or senior county-level) health staff to serve as a data collection team, with one member of each team designated as the supervisor. All staff were trained from 21 to 25 February 2011. During the following week, all teams participated in the collection of data from Central and Eastern Equatoria States. This allowed for close supervision and correction of potential errors during the initial phase of data collection. Data collection finished in all States by 4 April 2011, before the start of the rainy season. Sick-child clinics in South Sudan are held every day; we performed all assessments in the mornings with teams arriving as close to the beginning of the work day as possible. MOH staff double entered the data using CSPro-v.4.0 to verify the data entry (U.S. Census Bureau & ICF Macro 2010). We sampled the database comparing records with the questionnaires; the estimated remaining error in the database was 0.08%. Data were analysed in Excel-v.2007 â and IBM- SPSS Statistics-v.19.0 â. In addition to LQAS classifications, we computed for each indicator the weighted proportion of HFs providing adequate services at a national level using the total number of HFs in each state as the weight. We calculated 95% confidence intervals using a finite population correction (Berenson et al. 2007). 2014 The Authors Tropical Medicine & International Health Published by John Wiley & Sons Ltd. 1239

Table 2 Definitions of IMCI indicators used in the r-hfa, South Sudan 2011 Indicator Access IMCI service availability IMCI service utilisation Input Staffing Infrastructure IMCI equipment/ supplies Newborn care equipment/ supplies IMCI drug availability Vaccine availability Infection control supplies Guideline availability Process IMCI Information systems In-service training Supervision ACT logistics LLIN logistics Performance IMCI assessment Treatment of sick child Counselling for sick child Confirmatory malaria test Definition HF that offer three basic child health services (growth monitoring, immunisation and sick-child care) at least once per month HF with 1 sick-child visit per children <5 years in the catchment area in the last 12 months* Staff employed in surveyed HF at the time of the survey. Minimum standard defined as 1 community health worker and 1 pharmacy dispenser/assistant (for all HF levels), 1 maternity community health worker (for PHCUs only), 1 nurse, 1 midwife, 1 lab technician and 1 clinical officer/medical assistant (for PHCCs and Hospitals only) Essential pieces of infrastructure (a working latrine for all HF levels, any source of water in PHCUs, water from protected source in PHCCs and Hospitals, electricity in PHCCs and Hospitals, and (mobile) phone or radio in Hospitals) on day of the survey Essential supplies to support child health in HF on day of the survey (accessible and working scale for child, timing device for diagnosis of pneumonia, jar/jug/pitcher for ORS, cup and spoon to administer ORS) Essential supplies to support newborn health in HF on day of the survey (a working infant weighing scale, and tetracycline ointment for all HF levels plus a working resuscitation device for PHCCs and Hospitals) First-line medications for child health available on survey day, including ORS, oral antibiotic for pneumonia (Amoxicillin), first-line oral antibiotic for dysentery (Ciprofloxacin), first-line antimalarial (Artesunate-Amodiaquine) and vitamin A Nationally mandated vaccines (BCG, OPV, DTP, Measles, Tetanus) and a functioning fridge available on survey day Essential infection control elements, including functional autoclave, chlorhexidine, latex gloves, safety box, 5 10 ml syringe 19 21 g needle, soap, adequate waste disposal and protected waste disposal area on survey day Nationally mandated guidelines for <5-year-old children (ANC, PMTCT, IMCI, Delivery, Newborn, Post-natal care) available and accessible on survey day HF that maintain up-to-date records of sick <5-year-old children (age, diagnosis, treatment) and have report in last 3 months and evidence of data use (charts, graphs or meetings) HF in which interviewed HW reported receiving any training on any topic related to maternal, child or neonatal health in last 12 months (no set minimum standards for training content or duration) HF that received external supervision at least once in the last 6 months (including one or more of the following: checked records or reports, observed work, provided feedback, gave praise, provided updates, discussed problems or checked drug supply) HF with adequate logistics compliance for ACT had received a delivery for ACTs in previous 3 months, has ACT in stock with at least one valid sample, has written stock records, where a hand count of stock performed by enumerator agrees with stock records, ACT have not been out of stock in previous 3 months, and drugs are disposed of correctly HF with adequate logistics compliance for LLINs received delivery of LLINs in past 3 months, has written stock records, where a hand count of stock performed by enumerator agrees with stock records HF in which all IMCI key assessment tasks are made by HW (check presence of general danger signs, assess feeding practices, assess nutritional status, check vaccination status; benchmark 5 of 6 clinical observations, or as shown in Figure S2 in case of missing data) HF in which treatment by HW is appropriate to his diagnosis for child with malaria (Artesunate/ Amodiaquine combination), pneumonia (Amoxicillin), diarrhoea (ORS) or dysentery (Ciprofloxacin); no unnecessary medication was allowed (benchmark 5 of 6 clinical observations, or as shown in Figure S2 in case of missing data) HF in which the caretaker whose child was prescribed an antibiotic, antimalarial, or ORS, can correctly describe how to administer all drugs (benchmark 5 of 6 clinical observations, or as shown in Figure S2 in case of missing data) HF where HW used a malaria test to confirm a diagnosis of malaria in children with fever 1240 2014 The Authors Tropical Medicine & International Health Published by John Wiley & Sons Ltd.

ACT, artemisinin-based combination therapy; ANC, antenatal care; BCG vaccine, Bacillus Calmette Guerin vaccine (against tuberculosis); DTP, diphtheria, tetanus, and pertussis vaccine; HF, health facility; IMCI, integrated management of childhood illnesses; LLIN, long-lasting insecticide-treated bed nets; ORS, oral rehydration salts; OPV, oral polio vaccine; PHCC, primary healthcare centres; PHCU, primary healthcare units; PMTCT, prevention of mother-to-child transmission. *This definition is in line with standard WHO benchmarks and with the current South Sudan MOH targets for utilisation [USAID MCHIP 2008; MOH 2011c]. For the denominator, we used the estimated catchment area population of PHCUs (n = 15 000 people, and n = 3000 children <5-years), which is smaller than the catchment area population of PHCCs and Hospitals. It thus provides an interpretation of utilisation against the barest minimum coverage level possible. The r-hfa minimum standard for this indicator is lower than the minimum standard of the country s Basic Health Package, which requires at least 2 community health workers (for all HF levels), 2 pharmacy dispenser/assistant, 2 maternity community health workers, 1 statistical clerk, and 1 janitor/guard (for PHCUs), 4 pharmacy dispensers/ assistants, 3 nurses, 2 midwives, 2 lab technicians, 2 nutritionists, 2 clinical officer/medical assistant, 2 statistical clerks and 2 janitors/guards (for PHCCs and Hospitals only). Finally, to compare overall performance state-by-state, we computed a summary score by awarding one point for each of 19 IMCI indicators when the state was classified as acceptable (i.e. there was no evidence that performance was below the national 80% target) and one point if the state reached the national mean. For the classification using the national mean, we applied the same LQAS principles with the exception that p U = national mean rather than 80% for the given indicator, and p L is 30% lower. The resulting score (maximum possible = 38 points) was graphically represented in a colour-coded map. Ethical considerations This r-hfa was commissioned by the then Government of South Sudan, funded by the Multi-Donor Trust Fund managed by The World Bank, led by the MOH and implemented by 10 State Ministries of Health with the technical and logistical support of Liverpool School of Tropical Medicine through the Liverpool Associates in Tropical Health. The MOH reviewed all study instruments and activities, ensured they complied with their own standards for ethical clearance and gave their approval. The MOH also approved the use of oral rather than written informed consent because of the high illiteracy rate. The r-hfa was carried out as part of normal facility quality assurance and improvement. Nevertheless, informed consent was obtained from all participants in a manner consistent with the Helsinki Declaration. Results Table 1 shows the numbers and types of HFs included in our study. It also lists the numbers of substitutions made for HFs that were inaccessible or no longer functioning. We made the highest number of substitutions in Jonglei, especially among PHCUs, due to geographical inaccessibility and insecurity. Overall, our final sample included 156 HFs. In 42 facilities, we observed <6 consultations due to low HF utilisation; the total number of observations, therefore, was 829 rather than 936 (Table S1). Results for IMCI access, input, process and performance indicators are reported in Table 3 (see Table 2 for definitions); Table S2 shows ANC results. Figure 1 exhibits weighted national means for IMCI composite indicators. Figures S1 S5 depict results of the individual indicators summarised as composite indicators. Access indicators: Service availability and utilisation IMCI services availability was poor with only 16% of HFs offering all three IMCI services (child consultations, immunisation and growth monitoring) at least once a month (Table 3). No state passed as having IMCI services available in 80% of facilities. Analyses of the individual components of this composite indicator were revealing. Sick-child consultations were available in 99% (95% CI: 97 100%) of HF at 5 days a week; however, only 61% (95% CI: 53 69%) offered immunisation and 27% (95% CI: 20 34%) offered growth-monitoring services 1 day per month. IMCI service utilisation by the population was poor; all states failed to meet the 80% target (defined as 1 sick-child consultation per year per <5-year-old child in the facility s catchment area, see Table 2). Overall, only 12% of all facilities in the country passed as acceptable in terms of minimum utilisation. ANC services were available in only 59% of facilities (Table S2). None of the states met the 80% availability target of HFs providing ANC services at least monthly. Similarly, the 80% target could not be met by any state for utilisation ( 2 ANC visits per pregnant woman in a facility s catchment area). Input indicators: staffing, infrastructure, equipment and supply For all IMCI-related input indicators (Table 3), no state reached the 80% target of HFs fulfilling minimum perfor- 2014 The Authors Tropical Medicine & International Health Published by John Wiley & Sons Ltd. 1241

Table 3 Lot quality assurance sampling results for core indicators assessing the quality of IMCI and neonatal care services in South Sudan 2011 Indicator Number of HFs with acceptable performance with n = 14 and d = 10 for Unity and NBeG, and n = 16 and d = 11 for all other states (*= state meets 80% target) UN Jong Unit Warr NBeG WBeG Lake WES CES EES Weighted National average (95%CI) Access IMCI service availability 4 0 1 5 4 1 2 7 6 5 16% (11 21%) IMCI service utilisation 0 1 1 4 3 1 3 1 0 3 12% (6 18%) Input Minimum staffing 2 1 4 2 4 0 2 3 2 1 16% (8 24%) Essential infrastructure 11* 4 6 11* 10* 6 7 8 10 10 52% (44 60%) IMCI equipment/supplies 2 2 2 4 4 0 1 3 1 2 18% (10 26%) Newborn equipment/supplies 3 4 2 6 3 1 4 4 4 6 26% (19 33%) IMCI drug availability 5 9 3 14* 7 8 6 4 8 10 53% (46 60%) Vaccine availability 2 7 1 6 7 1 5 8 5 3 27% (20 34%) Infection control supplies 0 0 1 3 0 0 1 0 1 0 3% (0 6%) Child care guidelines 0 0 0 0 0 0 0 1 1 0 0% (0 0%) Process IMCI information system 1 5 4 3 1 2 1 3 2 4 25% (17 33%) In-service training 7 13* 12* 8 10* 12* 14* 15* 13* 7 67% (59 75%) Supervision 15* 11* 12* 13* 14* 11* 15* 13* 10* 14* 81% (74 88%) ACT logistics 1 1 3 5 4 2 2 1 1 3 20% (14 26%) LLIN logistics 1 10 2 8 2 0 3 2 2 2 18% (12 24%) Performance IMCI assessment 0 0 0 3 0 0 0 0 0 1 6% (3 9%) Child treatment 2 7 3 5 2 4 1 1 0 3 38% (29 47%) Confirmatory malaria test 4 3 5 7 4 7 8 6 10 6 33% (25 41%) Caretaker counselling 0 2 8* 4 1 1 0 3 2 6 33% (26 40%) n, total number of sampled HF in the state; d, decision rule based on which the states were classified as meeting 80% performance target (meaning that 80% of health facilities in the sample performed adequately for this indicator), CI, 95% confidence interval; UN, upper Nile; Jong, Jonglei; Unit, unity; Warr, Warrap; NBeG, northern Bahr-el-Ghazal; WBeG, Western Bahr-el-Ghazal; Lake, lakes; WES, Western Equatoria; CES, Central Equatoria; EES, Eastern Equatoria; IMCI, integrated management of childhood illnesses; ACT, artemisinin-based combination therapy, LLIN, long-lasting insecticide-treated bed nets. For indicator definitions see Table 2. For these indicators, the decision rules (d) were lower than d = 10 or d = 11 for some of the states, because of missing data due to underutilisation of the services, for example for Caretaker counselling in Unity the decision rule was d = 7 rather than d = 10, because the number n of sampled HF for which this indicator could be assessed was n = 10 rather than n = 14. mance standards with the following exceptions: Upper Nile, Warrap and Northern Bahr-el-Ghazal were classified as acceptable for satisfying essential infrastructure requirements and Warrap for having IMCI drugs available. On average, about half of all HFs, nationally, showed acceptable essential infrastructure (52%) and drug availability (53%). The minority of HFs had sufficient equipment and supplies to support child health (18%) and newborn health (26%). Staff shortages were severe: only 16% of HFs fulfilled staffing requirements. Nationally, only 27% of HFs had all essential vaccines and a working fridge, with little difference in the availability of each individual vaccine (all five vaccines were available in 32 35% of HFs); similarly, a functioning fridge was available in 33% of HFs (Figure S2). Only 3% of HFs had essential infection control supplies, and 0% had all five nationally mandated guidelines available. For ANC-related input indicators (Table S2), none of the states were classified as acceptable for their availability of equipment and supplies and only three states (Warrap, Western Bahr-el-Ghazal and Western Equatoria) for the availability of essential drugs. Process indicators: training, supervision and information system The quality of process indicators concerning IMCI information systems was similarly poor (Table 3). All states failed for record-keeping, reporting and data use (see Table 2 for definitions). However, seven states were clas- 1242 2014 The Authors Tropical Medicine & International Health Published by John Wiley & Sons Ltd.

Figure 1 Summary of child health-related indicators assessed during r-hfa in South Sudan 2011. * weighted national mean and 95% confidence intervals. ACT = artemisinin-based combination therapy, HF = health facility, IMCI = Integrated management of childhood illnesses, LLIN = Long-lasting insecticide-treated bed nets, MCH = mother and child health. % HFs that meet minimum standards* 100% 80% 60% 40% 20% 0% Availability/ Utilization 16% 12% 16% 52% 26% 18% 53% IMCI drug availability IMCl service availability IMCl service utilization Minimum staffing Essential infrastructure IMCI equipment/supplies Newborn equipm./supplies Input 27% 3% 0% Vaccine availability Infection control supplies Child care guidelines 25% Process 67% 81% 20% 18% IMCI information system In-service training Supervision ACT logistics LLIN logistics 6% Performance 38% 33% 33% IMCI assessment Confirmatory malaria test Child treatment Caretaker counseling sified as acceptable for training and all ten states for supervision. It should be noted, though, that during supervision visits, feedback was provided in only 51% of HFs (Figure S3). Only about one-fifth of HFs showed adequate logistics for delivery of first-line antimalaria drugs (20%) and insecticide-treated bednets (18%). No state passed for either indicator. The only ANC-related process indicator, information systems, exhibited weak performance across all states for HF offering ANC (Table S2). Only 13% of HF had all required elements, such as keeping a register and recording essential pieces of information. Performance indicators: Patient assessments, treatment and counselling Indicators for HW performance detected considerable deficiencies (Table 3). In only 6% of facilities was implementation of IMCI guidelines for assessing sick children classified as acceptable. Regarding additional diagnostics, only one-third of HFs in the country used confirmatory malaria tests, and no state reached the 80% target. Similarly, no state reached child-treatment targets with only 38% of facilities being classified as providing appropriate patient treatment for a given diagnosis. Caretakers of children receiving treatment were appropriately counselled and knew how to give the drugs in only one-third of the facilities. Only one state (Unity) was categorised as acceptable for this indicator. Summary score Weighted national averages for all IMCI-related indicators show that performance was best for training and supervision (Figure 1). State-by-state comparisons reveal that Western Bahr-el-Ghazal, Jonglei and Upper Nile were most in need of improvement (Figure 2). Discussion This first rapid-hfa, conducted in RSS s foundation year, reveals an urgent need to improve healthcare quality in all states for almost all QoC indicators. While child consultations were available in all HFs, the availability of vaccinations and growth-monitoring services was poor, although slightly higher than previously reported (MOH 2011a). Most concerning were the extremely low utilisation rates for available child and antenatal care services, which the HFM also observed in 2008 (MOH 2011a). Reasons people cited for their reluctance to use services included: long distances to HFs, poor roads, lack of transport, a dysfunctional referral system, inadequate equipment, lack of drugs and other supplies and the absence of qualified staff (MOH 2011a, 2012; IOM 2013). Some of these reasons can be deduced from our results. Perhaps most striking was the lack of staff with 84% of all HFs failing to fulfil our study s bare minimum staffing targets. The MOH also noted that only 10% of posts are 2014 The Authors Tropical Medicine & International Health Published by John Wiley & Sons Ltd. 1243

Upper Nile NB Ghazal Unity Warrap WB Ghazal Jonglei Lakes W Equatoria E Equatoria C Equatoria Lowest performance (score 15) Higher performance (score >15 and <20) Figure 2 Level of overall performance scoring by state, South Sudan r-hfa 2011. Note: Scoring based on 19 IMCI indicators with +1 point given if state reached average performance and +1 point given if state reached target performance for each indicator; none of the states had a score >20 on this scale with maximum of 38 possible points; the map was created with MapWindow GIS v4.8.8. filled by qualified HWs [1.5 physicians and two nurses/ midwives for every 100 000 citizens (MOH 2012). This compares to 30 physicians and 80 nurses/midwives per 100 000 in Sudan (World Bank 2014a,b)]. The lack of appropriate teaching facilities needs remedy if the MOH is to achieve its goal of doubling the number of doctors and quintupling the number of nurses/midwives by 2015 (MOH 2010, 2012). Despite the recent relocation of three Colleges of Medicine from Sudan to South Sudan, they lack sufficient teaching staff, accommodation, equipment and supplies. Other reasons for low HF-staffing include high staff turnover and absenteeism due to the lack of a clear retention policy and plan, poor human resources management and a poor working environment (MOH 2012). Our study assessed the number and level of staff employed, but not those actually present on the survey day. Thus, actual staffing might be even lower, which could be determined through further research on the extent and types of staff absenteeism [ planned/ unplanned and voluntary/involuntary (Belita et al. 2013) ]. The attendance monitoring system planned by the MOH will also shed light on this matter (MOH 2013). Our study showed similarly poor performance in terms of equipment and supplies for child and newborn services and for infection control with none of the states fulfilling minimum targets. However, some of these problems could be remedied with the purchase of basic, inexpensive equipment and supplies, such as soap, timers and oral rehydration therapy equipment (jug, cup and spoon). HFs rated higher for having essential infrastructure with three states classified as acceptable. Nevertheless, as long as the majority of PHCCs and hospitals lack electricity, services that need illumination and refrigeration, such as night-time deliveries and laboratory tests, cannot be guaranteed. Refrigeration is also associated with vaccine availability, as vaccination programs appear to be working in HFs with functioning refrigeration (33% of HFs). If dual-energy refrigerators cannot be supplied, an interim solution could be to provide five-day cold boxes to PHCCs and hospitals and stock vaccines weekly. During the previous HFM, even fewer facilities owned a refrigerator, and only one quarter had cold boxes for vaccine transportation (MOH 2011a). In our study, only one state was categorised as acceptable in terms of drug availability, but overall more than half of HFs stocked all essential drugs. This result shows that the drug distribution system performs comparatively well. Nevertheless, further improvements are urgently needed, as confirmed by recent reports on problems of stock-outs, unacceptable storage conditions, and substandard or counterfeit drugs (Mochache et al. 2011; MOH 2011a). The lack of a national logistics management information system contributed to these problems (Mochache et al. 2011). This deficiency might also partly explain the poor performance we detected for process indicators concerning 1244 2014 The Authors Tropical Medicine & International Health Published by John Wiley & Sons Ltd.

information systems and logistics; service delivery and stock-control both exhibited poor record-keeping. More research is needed to identify reasons for poor recordkeeping, such as low levels of staff literacy and numeracy, and ways to address deficiencies. Some of these problems might be addressed through in-service training and supervision, both of which were provided at most HFs according to our results. However, the adequacy of their content, length and quality needs assessment. Supervision has an impact only when it is supportive, that is, focussing on constructive feedback and joint problem-solving rather than inspection and control (Frimpong et al. 2011; Bradley et al. 2013; Rosales et al. 2014). It is also quite possible that supervisors might need as much support (including supervision) as do front-line clinicians and that both need to have the means to do their work and act on what they find (Rowe et al. 2010; Hill et al. 2014). In-service training and supervision if adequately performed could also be used as a means to disseminate child care guidelines, which were unavailable in most facilities. The lack of IMCI guidelines may also partly be responsible for the poor HW performance we observed during child consultations. In only 6% of facilities, HWs assessed sick children according to them, and in only 38% of facilities was treatment appropriate for the HW s diagnosis; however, only 33% of caregivers understood how to administer prescribed drugs. We have not found any other published data on HW performance in RSS to which our results could be compared, but poor performance has been reported for other low-income countries in both public and private settings (Berendes et al. 2011; Valadez & Schwarz 2012). Various strategies to address structural and behavioural factors for both providers and consumers have been suggested to improve HW performance (Mills et al. 2002; Rowe et al. 2005). As single interventions, such as disseminating written guidelines, are ineffective, multifaceted interventions have been proposed, especially those combining training, supervision, and audits with feedback (Rowe et al. 2005). Although we have attempted to summarise and compare performance data between different states, these results should be cautiously interpreted, given that all states exhibited low baseline performance. Overall Jonglei, Upper Nile and Western Bahr-el-Ghazal showed the greatest need for improvement. Their priority status might partly relate to interethnic fighting that occurred in Jonglei in 2011, border conflicts in the North of Upper Nile and Western Bahr-el-Ghazal, and the fact that the Greater Ghazal region had to absorb the greatest number of South Sudanese returnees since the 2005 Peace Agreement. However, some of these problems have also occurred in other states (Mochache et al. 2011; Centanni 2012; BBC 2014). It is interesting that while Upper Nile showed poor quality of care overall, it performed relatively well in terms of essential infrastructure. This might be partly due to increased allocation of funds to this state in the past. Until recently, there used to be a patchwork pattern of rather uncoordinated emergency funding and service provision to South Sudan by various donors and NGOs with some areas saturated and others completely underserved (Downie 2012). It would be interesting for future research to compare structural and other indicators among public vs. NGO facilities. However, the lines between them have blurred; since 2013, each state is supported by one of three large donor projects (financed by World Bank, USAID, and a consortium of five donor agencies led by DFID (MOH 2013); each of them subcontracts local agencies (public or NGO) to manage HFs together with the county health department, and most NGO facilities are increasingly managed by the government. Given the continued military conflict, future studies will have to deal with challenges related to insecurity that we encountered (BBC 2014). Problems of inaccessibility and insecurity, especially in Jonglei, required us to make HF-substitutions during sampling. The high rate of substitutions of PHCUs by PHCCs led to oversampling of PHCCs. In the future, RSS will randomly sample PHCCs and hospitals as a stratum and then randomly choose a PHCU in the catchment area for each selected PHCC, as was performed in Nigeria (Berendes et al. 2012). Another limitation was due to the low utilisation of many PHCUs, which rendered the clinical observation and exit interview modules difficult to carry out, as we had to exclude HFs if the misclassification error was 20% for both alpha and beta. We are also aware that the assessment of clinical practice through observation could have introduced a bias as HW could have performed better than they normally do. Therefore, HW performance might be even worse than is reported here. Our study shows that despite ongoing instability, it is possible to empower the MOH of a fragile state to conduct an r-hfa using LQAS in a reasonable timeframe and at relatively low costs compared with health facility surveys, such as Service Provision Assessments (Fort 2014). Our r-hfa cost approximately $25 000 per state, excluding costs for external support for design, training and analysis, which for future surveys can gradually be taken on by State MOH staff. Another advantage of using LQAS is that we have been able to report national as well as state-by-state results. In addition to the data, it is important to note that the r-hfa-lqas built local capacity, permitting health system managers, who are familiar with 2014 The Authors Tropical Medicine & International Health Published by John Wiley & Sons Ltd. 1245

the local context to assess the areas for which they are responsible (Valadez & Devkota 2002). This benefit is in line with calls for a focus on state and county-level health system development given that local officials are more knowledgeable of and responsive to local demands (Downie 2012). It also supports the MOH to rationally allocate resources from their health budget to the states and central functionaries (MOH 2012). This feature has increasing importance given that the MOH s target to increase health funding as a proportion of the national budget from 4% to 10% by 2015 may not be achieved; this is due to austerity measures resulting from the oil shut down in January 2012 March 2013 after a row with Sudan and due to the current conflict that erupted in December 2013 (MOH 2012; IRIN 2013; BBC 2014). The current fighting between government and opposition forces has left thousands of people dead and more than 1 million displaced. The crisis is most acute in Upper Nile, Jonglei and Unity, where armed forces also target health facilities (WHO 2014). The extent of the damage is not yet known, but in the most affected states, the QoC reported here has probably deteriorated. The preservation of the scarce human resources for health is now paramount. While emergency relief is critical during the current crisis, the government should not abandon its development agenda by turning back the clock to where it was prior to independence (Downie 2012; Kevlihan 2013). Under the leadership of a committed MOH, the development of a sector-wide M&E system has begun using LQAS to complement a routine health management information system in which HFs produce standardised monthly reports on communicable diseases and routine service indicators (Laku et al. 2012; MOH 2012). Despite the enormous challenges, RSS should continue on its path towards development and identify health system priorities through regular local monitoring of the quality of care (Valadez 1991). Acknowledgements The authors gratefully acknowledge William Vargas, who provided technical assistance during study planning, training and supervision, and Dr Kartini Gadroen who assisted with study implementation and hand-tabulation. We are also very grateful to the South Sudan State Ministries of Health, the South Sudan Centre for Census, Statistics & Evaluation, the County Health Departments, the staff of the Directorate of M&E in the Ministry of Health, the data collection teams from each state Ministry of Health and all study respondents. This study was funded by the Ministry of Health of the Government of South Sudan through the Umbrella Programme for Health Systems Development, part of the World Bank managed Multi-Donor Trust Fund. References BBC (2014) News Africa- South Sudan Profile: Timeline - a chronology of key events [Online]. London: BBC. Available: http://www.bbc.co.uk/news/world-africa-14019202 [Accessed 01/05/2014 2014]. Belita A, Mbindyo P & English M (2013) Absenteeism amongst health workers - developing a typology to support empiric work in low-income countries and characterizing reported associations. Human Resources of Health 11, 34. Berendes S, Heywood P, Oliver S & Garner P (2011) Quality of private and public ambulatory health care in low and middle income countries: systematic review of comparative studies. PLoS Medicine 8, e1000433. Berendes S, Adeyemi O, Oladele EA, Oresanya OB, Okoh F & Valadez JJ (2012) Are patent medicine vendors effective agents in malaria control? Using lot quality assurance sampling to assess quality of practice in Jigawa, Nigeria. PLoS ONE 7, e44775. Berenson M, Levine D & Krehbiel T (2007) Confidence Interval Estimation Pearson Education. Bradley S, Kamwendo F, Masanja H et al. (2013) District health managers perceptions of supervision in Malawi and Tanzania. Human Resources for Health 11, 43. Centanni E (2012) Updates on the changing world political map: Sudan & South Sudan s Top 8 Border Disputes [Online]. Political Geography Now. Available: http://www.polgeonow. com/2012/05/feature-sudan-south-sudan-border_26.html [Accessed 18/01/2014 2014]. Dodge HF & Romig HG (1929) A method of sampling inspection. Bell System Technical Journal 8, 398. Downie R (2012) A report of the CSIS Global Health Policy Center: The State of Public Health in South Sudan - Critical Condition. Center for Strategic and International Studies, Washington DC. FANTA Project. LQAS Sample Size Calculator (Hypergeometric Distribution Model) [Online].Washington DC: Food and Nutrition Technical Assistance (FANTA) project. [Accessed 2011]. Fort A (2014) Service Provision Assessment [Online]. New York: United Nations Population Fund (UNFPA). Available: http:// www.unfpa.org/webdav/site/global/shared/documents/publications/2010/srh_guide/tools_serviceprovision.html [Accessed 13/ 05/2014 2014]. Frimpong JA, Helleringer S, Awoonor-Williams JK, Yeji F & Phillips JF (2011) Does supervision improve health worker productivity? Evidence from the Upper East Region of Ghana. Tropical Medicine and International Health 16, 1225 1233. Hill Z, Dumbaugh M, Benton L et al. (2014) Supervising community health workers in low-income countries a review of impact and implementation issues. Global Health Action 7, 24085. IOM (2013) South Sudan Village Assessment Survey Report 2013. International Organization for Migration South Sudan, Juba. 1246 2014 The Authors Tropical Medicine & International Health Published by John Wiley & Sons Ltd.

IRIN (2013) Fighting in South Sudan capital leaves dozens dead, displaced [Online]. Juba: IRIN, the humanitarian news and analysis service of the UN Office for the Coordination of Humanitarian Affairs. Available: http://www.irinnews.org/ report/99353/fighting-in-south-sudan-capital-leaves-dozensdead-displaced [Accessed 18 December 2013 2013]. Kevlihan R (2013) Providing health services during a civil war: the experience of a garrison town in South Sudan. Disasters 37, 579 603. Laku R, Camino CM, Stoops N & Ali M (2012) Implementing a routine health management information system in South Sudan. South Sudan Medical Journal 5, 21 24. Mills A, Brugha R, Hanson K & McPake B (2002) What can be done about the private health sector in low-income countries? Bulletin of the World Health Organization 80, 325 330. Mochache D, Chinyanganya F & Ngidari J (2011) Pharmaceutical Logistics Assessment South Sudan. The Global Health Technical Assistance Project, Washington DC. MOH (2007) Southern Sudan Essential Medicine List. Ministry of Health - Government of Southern Sudan, Juba. MOH (2009) Basic package of health and nutrition services for Southern Sudan. Juba: Ministry of Health - Government of Southern Sudan. MOH & Beesley ME (2010) Off The Road: The Mapping of Health Training Facilities in Southern Sudan. Liverpool, UK: Ministry of Health- Government of Southern Sudan. MOH (2011a) Health Facility Mapping of South Sudan: Summary Report, May 2011. Juba: Ministry of Health - Government of South Sudan - Directorate of Monitoring and Evaluation. MOH (2011b) Health sector development plan 2011 2015 - Transforming the health system for improved services and better coverage (Revised Final Draft, 1st March 2011). Juba, South Sudan: Ministry of Health - Government of South Sudan. MOH (2011c) The Basic Package of Health and Nutrition Services in Primary Health Care. Juba: Ministry of Health, Government of the Republic South Sudan. MOH (2012) Health sector development plan 2012 2016 - One maternal death is one too many. Juba: Ministry of Health - Republic of South Sudan. MOH (2013) 2nd South Sudan Health Summit: Towards Better Healthcare Service that Meets People s Expectations, 4th 6th Dec. 2013 [Online]. Juba: Ministry of Health, Republic of South Sudan. Available: http://www.mohss.org/directorates/ international-health-coordination/conferences-and-workshops/ 2nd-south-sudan-health-summit/ [Accessed 2014]. Oladele EA, Ormond L, Adeyemi O et al. (2012) Tracking the quality of care for sick children using lot quality assurance sampling: targeting improvements of health services in Jigawa, Nigeria. PLoS ONE 7, e44319. Robertson SE & Valadez JJ (2006) Global review of health care surveys using lot quality assurance sampling (LQAS) 1984 2004. Social Science and Medicine 63, 1648 1660. Rosales A, Hedrick J, Amet KK et al. (2014) Maintaining knowledge and technical skills among illiterate frontline community health workers (HHPs) delivering integrated Community Case Management (iccm) in South Sudan [Online]. Washington DC: CCMCentral by USAID/MCHIP. Available: http://ccmcentral.com/documents/supervising-illiterate-chws-south-sudan/ [Accessed 26 May 2014 2014]. Rowe AK, de Savigny D, Lanata CF & Victora CG (2005) How can we achieve and maintain high-quality performance of health workers in low-resource settings? Lancet 366, 1026 1035. Rowe AK, Onikpo F, Lama M & Deming MS (2010) The rise and fall of supervision in a project designed to strengthen supervision of Integrated Management of Childhood Illness in Benin. Health Policy Plan 25, 125 134. U.S. Census Bureau & ICF Macro (2010) The Census and Survey Processing System (CSPro) [Online]. Washington, DC: U.S. Census Bureau. Available: http://www.census.gov/population/international/software/cspro/ [Accessed 2010]. USAID MCHIP (2008) Rapid Health Facility Assessment 2.1 (R-HFA) [Online]. Washington DC: United States Agency for International Development (USAID) Maternal and Child Health Integrated Program (MCHIP). Available: http://www. mchipngo.net/controllers/link.cfc?method=tools_rhfa [Accessed 21 Oct 2013 2013]. Valadez JJ (1991) Assessing Child Survival Programmes in Developing Countries: Testing Lot Quality Assurance Sampling. Harvard University Press, Boston. Valadez J & Devkota BR (2002) Decentralized supervision of community health programs: Using LQAS in two districts of Southern Nepal. Community based health care: Lessons from Bangladesh to Boston. Management Sciences for Health, Inc. Valadez JJ & Schwarz A (2012) Report of the Rapid Health Facility Assessment in Northern Uganda. For the Department for International Development, UK, Results-Based Financing Impact Evaluation Project 2012. Liverpool, UK: Liverpool Associates in Tropical Health (LATH). Valadez JJ, Brown LD, Vargas WV & Morley D (1996) Using lot quality assurance sampling to assess measurements for growth monitoring in a developing country s primary health care system. International Journal of Epidemiology 25, 381 387. Valadez JJ, Transgrud R, Mbugua M & Smith T (1997) Assessing family planning service-delivery skills in Kenya. Studies in Family Planning 28, 143 150. Wakabi W (2011) South Sudan faces grim health and humanitarian situation. Lancet 377, 2167 2168. WFP (2012) Annual Needs and Livelihood Analysis Report - South Sudan, February 2012. Rome: World Food Program. World Bank (2014a) Data: Nurses and midwives (per 1000 people) [Online]. Washington DC: The World Bank Group. Available: http://data.worldbank.org/indicator/sh.med. NUMW.P3 [Accessed 18/01/2014 2014]. World Bank (2014b) Data: Physicians (per 1000 people) [Online]. Washington DC: The World Bank Group. Available: http://data.worldbank.org/indicator/sh.med.phys.zs [Accessed 18/01/2014 2014]. WHO (2014) South Sudan Crisis Response Update: January- April 2014. 1st Quarterly report 2014. Geneva: World Health Organization. 2014 The Authors Tropical Medicine & International Health Published by John Wiley & Sons Ltd. 1247