Joseph J. Valadez 1, Baburam Devkota 1, Madan Mohan Pradhan 2, Pramod Meherda 3 *, G. S. Sonal 4, Akshay Dhariwal 2 and Rosemary Davis 1

Transcription

1 Tropical Medicine and International Health doi: /tmi volume 19 no 10 pp october 2014 Improving malaria treatment and prevention in India by aiding district managers to manage their programmes with local information: a trial assessing the impact of Lot Quality Assurance Sampling on programme outcomes Joseph J. Valadez 1, Baburam Devkota 1, Madan Mohan Pradhan 2, Pramod Meherda 3 *, G. S. Sonal 4, Akshay Dhariwal 2 and Rosemary Davis 1 1 METRe Group, Department of International Public Health, Liverpool School of Tropical Medicine, Liverpool, UK 2 National Vector Borne Disease Control Program, Bhubaneswar, Odisha, India 3 Rural Health Mission Directorate, Mission Directorate, Nayapalli, Odisha, India 4 National Vector Borne Disease Control Program, Directorate General of Health Services, Ministry of Health & Family Welfare, Government of India, Delhi, India Abstract objectives This paper reports the first trial of Lot Quality Assurance Sampling (LQAS) assessing associations between access to LQAS data and subsequent improvements in district programming. This trial concerns India s approach to addressing an increase in malaria-attributable deaths by training community health workers to diagnose, treat and prevent malaria, while using LQAS to monitor sub-district performance and make programme improvements. methods The Ministry of Health introduced LQAS into four matched high malaria burden districts (Annual Parasite Incidence >5) (N > 5 million). In each sub-district, we sampled four populations in three 6-monthly surveys: households, children <5 years, people with fever in the last 2 weeks and community health workers. In three districts, trained local staff collected, analysed and used data for programme management; in one control district, non-local staff collected data and did not disseminate results. For eight indicators, we calculated the change in proportion from survey one to three and used a Difference-in-Differences test to compare the relative change between intervention and control districts. results Coverage increased from survey one to three for 24 of 32 comparisons. Difference-in- Differences tests revealed that intervention districts exhibited significantly greater change in four of six vertical strategies (insecticide treated bed-nets and indoor residual spraying), one of six treatmentseeking behaviours and four of 12 health worker capacity indicators. The control district displayed greater improvement than two intervention districts for one health worker capacity indicator. One district with poor management did not improve. conclusions In this study, LQAS results appeared to support district managers to increase coverage in underperforming areas, especially for vertical strategies in the presence of diligent managers. keywords India, community-based management of malaria, Lot Quality Assurance Sampling, malaria control, malaria prevention, malaria treatment Introduction In 2010, an independent estimate of deaths in India caused by malaria exceeded previous estimates 10-fold, dwarfing WHO s estimate of (WHO *Current posting: Revenue Divisional Commissioner of the Northern Region, Odisha, India. 2008) and the number of Indian certified malaria deaths (~1000) (National Vector Borne Disease Control Programme 2011b). That report is not the first to suggest that the malaria burden is considerably higher than earlier reported (Yadav et al. 2003; Kumar et al. 2007; Dash 2009; Dhingra et al. 2010; Hay et al. 2010). Although another study provided a lower annual mortality estimate (50 000) (Murray et al John Wiley & Sons Ltd

2 2012), adding to the uncertainty (Basnyat 2011; Shah et al. 2011; Valecha et al. 2011), what is certain it that malaria is a leading cause of mortality. India s attempt to eradicate malaria during the 1950s resulted in a major reduction in the malaria burden. Although the prevalence subsequently increased in the 1970s, it never reached earlier levels. However, eastern states persistently report a high malaria burden, thereby prompting the Government of India (GoI) to introduce new control measures. The World Bank funded nine high malaria burden states (The World Bank 2008) under the leadership of the National Vector Borne Disease Control Programme (NVBDCP) to introduce these new approaches. They included (i) adopting rapid diagnostic tests (RDT) for point-of-care diagnosis of malaria including at the village level; (ii) introducing Artemisinin-based Combination Therapy (ACT) to treat falciparum malaria (P.f.); (iii) adopting long-lasting insecticidal nets (LLIN); (iv) creating the position of malaria technical supervisors (MTS) as staff dedicated to collecting and analysing malaria programme information at the sub-district level, presenting their conclusions to local managers, and resolving local logistical problems; and (v) using village health volunteers for malaria case management (National Vector Borne Disease Control Programme 2011a). These volunteers were primarily Accredited Social Health Activist (ASHA). The existing monitoring system consisted of ASHA maintaining a paper and pencil record of activities, which formed part of the recurrent health information system. NVBDCP managers recognised that this system was not appropriate for these new approaches and decided to introduce Lot Quality Assurance Sampling (LQAS), which permitted decentralised monitoring at both district and sub-district levels (called Block Community Health Centres: B-CHC), was low cost, and could be implemented by local health workers. Odisha, the focus of this paper, accounts for 25% of India s malaria cases and deaths (National Vector Borne Disease Control Programme 2011b), despite having <4% of the population. In its elemental form, LQAS is a classification system categorising areas as high or low performing, thereby allowing managers to address problems. Due to the small sample size used for these classifications, LQAS has been in high demand; a recent LQAS literature review identified more than 800 studies (Robertson & Valadez 2006). Odisha VBDCP managers designed the first state application in November 2009 as a trial to appraise LQAS s utility for programme strengthening (Figure 1). The use of LQAS has burgeoned in that last 20 years (Robertson & Valadez 2006) along with the growing concern to address problems in health service delivery (Kim et al. 2013). While other studies have documented benefits of using LQAS data to guide decision-making (Valadez et al. 1995, 2005; Valadez & Devkota 2002), this study documents the first trial of LQAS as it affects decision-making at such a large scale. Methods This trial uses a Non-Equivalent Control Group Design (Campbell & Stanley 1963) in which three districts (Mayurbhanj, Sundargarh and Nabrangpur) had access to LQAS results and one district (Kandhamal) did not (Figure 2). Each district has a population exceeding people (Table 1). A stakeholder group consisting of VBDCP managers, World Bank advisors, NGO managers and DFID programme consultants matched districts according to four criteria: a large proportion of low-income and tribal populations, >5 Annual Parasite Incidence (API) (>5 confirmed malaria cases reported per 1000 persons per year), received World Bank funding to support malaria prevention and treatment, and the presence of a functioning health system (inclusion criterion: each district s vaccination coverage approximated Odisha s average vaccination coverage) (Table 1). To reduce inter-district contamination, the districts were distributed throughout Odisha and did not share borders. The four selected districts exhibited the least inter-district variation among the selection criteria. Odisha VBDCP managers conducted three LQAS surveys in the intervention districts: November December 2009, July August 2010 and November December 2010; the first and third surveys also were conducted in the control district in intervention districts. MTS were responsible for collecting and analysing data and presenting results to B-CHC and district managers. In the control, an outside survey team collected the data and did not share data or results with anyone until after the third survey in MTS collected the data for the third survey. We used standard LQAS procedures (Valadez 1991; Robertson et al. 1997). LQAS is an established analysis technique originally developed as a classification method for industrial quality control during the 1920s and adapted to health sciences in the mid-1980s (Reinke 1988; Valadez 1991; Robertson et al. 1997) to classify management units, referred to as supervision areas (SA) according to a performance target. In each SA, a sample of n individuals is assessed. A d is selected that determines the cut-off number below which the area is classified as low performance for a specified indicator. The decision rule d depends on the sample size, thresholds for classifying high and low performance, and selection 2014 John Wiley & Sons Ltd 1227

3 Facilitating factors LQAS survey: baseline data Accurate data collection Strong district leadership LQAS survey: follow-up data (increased coverage proportions) Sufficient motivated programme staff MoH support Timely corrective action Identification of underperforming districts Sufficient and uninterrupted supplies Inter-district experience sharing Planning strategic and tactical changes Problem and solution analysis Adequate funds Figure 1 Lot Quality Assurance Sampling Process Cycle. of two misclassification errors: the risk of misclassifying an SA with very low coverage as high (b error), and the risk of misclassifying an SA with high coverage as low (a error). National stakeholders set the upper threshold, pu, for identifying acceptably performing SA while low performers, pl, are often 30% lower (Pagano & Valadez 2010). In our surveys, 19 villages were selected in each B-CHC, which is the largest administrative unit within a district, and which served as the SA for this study (Table 2). The SA sample size, n = 19, ensures that a and b errors do not exceed 10% for all pu, and all corresponding values of pl that are 30 percentage points less than pu, as is the case for this trial. SA with pl < p <pu are also classified. A high classification is more likely the nearer p is to pu (and vice versa for pl); p located in the middle of this so-called grey area has an approximately equal chance of being classified in the high or low category. Other monitoring Odisha, like other Indian States, does not have sufficient public health staff to monitor the malaria programme in all villages. MTS were intended to remedy this deficiency. However, they are too few. Nevertheless, MTS and district supervisors carried out regular monitoring and supervision, in addition to LQAS, in all four districts. For example, they visited ASHAs and health facilities to replenish supplies, collect recurrent information and solve problems John Wiley & Sons Ltd

4 Nuapada Nabarangapur Kalahandi Jharsuguda Balangir Debagarh Sambalpur Bargarh Rayagada Sonapur Baudh Kandhamal Sundargarh Kendujhar Baleshwar Bhadrak Anugul DhenkanalJajapur Kendrapara Cuttack Nayagarh Khordha Jagatsinghapur Puri Ganjam Mayurbhanj koraput Gajapat Figure 2 Map of Odisha State, India highlighting four districts included in this study. Malkangiri Table 1 Comparison of four districts in Odisha State used in this study High burden districts Geographical location Population 2009 Number of Block CHC No. block CHC with >5 API* No. MTS working in block CHC with >5 API Full routine immunization coverage, % LQAS districts 1 Sundargarh North-west Mayurbhanj North Nabrangpur South-west Non-LQAS districts 1 Kandhamal Central Block CHC, Block Community Health Centre; LQAS, Lot Quality Assurance Sampling; API, Annual Parasite Index; MTS, Malaria Technical Supervisors are government technical staff responsible for 1 2 Block CHC. *API = Total positive slides for parasites in a year /Total population. Data collection Surveys used a three-stage sampling design. At Stage 1, probability proportional to size sampling identified 19 villages in each of 51 B-CHC. At Stage 2, MTS used segmentation sampling (Turner et al. 1996; Milligan et al. 2004; Valadez et al. 2007) to randomly select a household. At Stage 3, interviewers randomly selected an eligible respondent in the sampled household. All stages were repeated for each sampling element in each survey. The survey questionnaire has four modules, each requiring different respondents: a household head, a caretaker of a child aged between 0 and 59 months, a person with fever in the last 2 weeks and ASHA. We designed the questionnaire to measure key NVBDCP s programme indicators; LLIN distribution and use, in-door residual spraying (IRS) and community knowledge and practices related to early detection and treatment of malaria. The questionnaire was designed collaboratively with Odisha VBDCP managers and approved by the State Secretariat for Health s Technical Working Group John Wiley & Sons Ltd 1229

5 Table 2 Surveys completed, sample sizes and use of results in four districts: November 2009 August 2011 Survey 1 Nov Dec 2009 Survey 2 July Aug 2010 Survey 3 Nov Dec 2010 District Total sample size (no. SA) Used results for management Total sample size (no. SA) Used results for management Total sample size (no. SA) Used results for management Sundargarh (intervention) 285 (15) Yes 285 (15) Yes 285 (15) Yes Mayurbhanj (intervention) 266 (14) Yes 266 (14) Yes 266 (14) Yes Nabrangpur (intervention) 171 (9) Yes 190 (10)* Yes 190 (10) Yes Kandhamal (control) 228 (12) No Not conducted NA 228 (12) Yes SA, Supervision Area; API, Annual Parasite Index: Total positive slides for parasite in a year /Total population. *One additional Block CHC was added to Nabrangpur during 2010 as NVBDCP determined API >5. In October 2009, prior to the first data collection, the authors trained 27 MTS in the three intervention districts, 12 staff from an external data collection agency for the control district and 13 supervisors on the LQAS methodology and sampling procedures. Refresher training was carried out prior to each survey along with new MTS receiving a comprehensive training. Data management, dissemination and analysis Once data collection finished, we trained intervention district MTS to hand tabulate data and classify B-CHC as having reached or not reached state performance targets for key indicators. (Valadez et al. 2007). Following the hand tabulation, MTS shared the results with district public health officials to discuss: (i) reasons why targets were not met and (ii) strategies that address problems. We then aggregated data across all B-CHCs (weighted by B-CHC population sizes) to calculate district wide coverage estimates. Identical procedures were implemented in all intervention districts and facilitated by one or more of the authors. After each survey round, questionnaire data were double entered into an ACCESS database to eliminate data entry errors and analysed using SPSS-19. We measured change in coverage estimates for key indicators within each district using the first and third survey time points. We calculated 95% confidence intervals (95% CI) for the change estimates with Stata-12.1 using logistic regression. Thereafter, we used difference-in-differences (DiD) tests. To use this procedure, we first calculated within each district the change between the baseline and final time point; then we compared the difference between each intervention district and the control district. We tested for significance using the 95% CI, which when crossing zero, indicates no evidence of difference. This trial design has three independent comparisons of intervention districts with the control district, with each district in the comparison having comparable sample sizes. We opted for this approach rather than collapsing the intervention districts into a single comparison group because districts in India have a large population size with a corresponding health system bureaucracy that are similar to regions in other countries. Collapsing the districts would have obfuscated the inter-district variation. District senior managers have styles that can vary considerably from other districts that may be relevant to this trial. Hence, a district level comparison is a fairer analysis than aggregating the intervention data as a single pool. Ethics This project was reviewed by the NVBDCP of India, the Technical Working Group of the World Bank, the Office of the Odisha Secretary of Health, and by the Odisha VBDCP. They reviewed the study design, questionnaires and informed consent procedures, and data security. The monitoring and evaluation methods included in this study are incorporated into GoI s contract with the World Bank. Results This study reports on eight indicators, NVBDCP used to monitor the malaria programme; each intervention district is compared separately with the control district (Table 3). Indicators 1 and 2 report on the proportion of people protected with either IRS or LLINs. Indicators 3 and 4 assess the treatment-seeking behaviour of people with fever and whether they received appropriate malaria treatment. Indicators 5 8 assess an ASHA s capacity to diagnose and treat malaria. Table 3 shows aggregated coverage figures for each indicator by district at three time points and the proportional change between surveys one and three with 95% CI John Wiley & Sons Ltd

6 Table 3 Coverage proportions for eight indicators, three time points, four districts with proportional change and Difference-in-Differences tests Indicators District Survey* 1 Nov 2009 Survey 2 Aug 2010 Survey 3 Dec 2010 % Change 95% CI of the Change: P-value Diff in diff 95% CI of the change: P-value Vertical Intervention Section Target 30% 40% 50% 1 Proportion of people protected by either Sundargarh ITN/LLIN or IRS the night preceding Mayurbhanj the survey Nabrangpur NS 2 Proportion of children protected by either ITN/LLIN or IRS the night preceding the survey Kandhamal (c) Sundargarh Mayurbhanj Nabrangpur NS Kandhamal (c) Treatment-Seeking Behaviour Section target 30% 40% 50% 3 Proportion of people with fever in the last 2 weeks whose blood was tested with an RDT within 1 day of the fever and told the result on same day as the test 4 Proportion of people with fever in the last 2 weeks who were RDT+ve and received ACT Sundargarh NS Mayurbhanj Nabrangpur NS NS Kandhamal (c) NS Sundargarh NS NS Mayurbhanj NS NS Nabrangpur NS NS Kandhamal (c) NS Community Health Workers/ASHA capacity Target 50% 60% 60% 5 Proportion of ASHA who have at least a Sundargarh NS 2-months RDT stock Mayurbhanj NS Nabrangpur Kandhamal (c) Proportion of ASHA who have sufficient P.f. drugs on hand (per the current guidelines) 7 Proportion of ASHA who know the waiting time prior to reading the RDT 8 Proportion of ASHA who know the correct ACT dose to treat P.f. Target 80% 80% 80% Sundargarh Mayurbhanj Nabrangpur Kandhamal (c) NS Target 80% 80% 80% Sundargarh NS Mayurbhanj NS Nabrangpur NS NS Kandhamal (c) Target 80% 80% 80% Sundargarh Mayurbhanj Nabrangpur NS Kandhamal (c) ITN/LLIN, Insecticide Treated Net/Long-Lasting Insecticidal Net; IRS, Indoor Residual Spraying; RDT, Rapid Diagnostic Test; ACT, Artemisinin-based Combination Therapy; ASHA, Accredited Social Health Activist; P.f., Plasmodium falciparum; C, Control; NS, Not significant. *Targets (pu) are listed for each time point and indicator. All values of pl are 30 percentage points lower. Decision rules (DR) used: DR = 3 (30%), DR = 5 (40%), DR = 7 (50%), DR = 9 (60%), DR = 13 (80%) John Wiley & Sons Ltd 1231

7 The change in coverage proportion between surveys one and three ranged from 35.9% (95% CI 26.2%) (Nabrangpur, Indicator 5) to 49.6% (95% CI 6.7%) (Mayurbhanj, Indicator 8). Of the 32 within district changes (8 indicators 9 4 districts), 23 (71.9%) showed a significant positive change in coverage from survey one to three. These changes included eight of eight prevention indicators, two of eight treatment-seeking behaviour indicators and 13 of 16 community health worker indicators. Significant change was detected in both intervention and control districts. In DiD tests, intervention districts had significantly greater change in four of six (66.7%) prevention indicator comparisons with the control district, one of six (16.7%) treatment-seeking behaviour indicator comparisons, and four of 12 (33%) ASHA capacity indicator comparisons. Sundargarh and Mayurbhanj account for eight of these nine significant results; Nabrangpur accounts for one only. The control displayed greater improvement vis-a-vis Nabrangpur and Sundargarh for one indicator each among the ASHA capacity building indicators. At least one intervention district had a significantly greater change in coverage proportion compared to the control for five of the eight indicators. For both prevention indicators (Indicator 1, Indicator 2), Sundargarh and Mayurbhanj exhibited improvements exceeding the control district. They displayed significantly more improvement for only one of the two treatmentseeking indicators (Indicator 3): people were tested with a RDT within 1 day of the onset of a fever and received the result. However, the data reveal no significant improvement for treating RDT+ve individuals with an appropriate antimalarial (ACT) (Indicator 4). Sundargarh and Mayurbhanj displayed significantly larger increases than the control for one of four ASHA capacity indicators, namely, ASHA had sufficient ACT on hand to treat RDT+ve cases (Indicator-6); Nabrangpur also increased the value more than the control for this indicator, but this was the only occasion any of its indicators exceeded the control. Mayurbhanj also displayed one additional improvement as compared with the control district, namely, the proportion of ASHA who know the correct ACT dose to treat patients with a positive RDT result (Indicator-8). Interestingly, the control district, exhibited significantly more improvement vis-a-vis Sundargarh for Indicator-8; and Nabrangpur for Indicator-5 (the proportion of ASHA who have at least a 2-month RDT stock). Indicator-7 (the proportion of ASHA knowing the waiting time prior to reading the RDT) was already a high value for all districts at time 1 and displayed no significant improvement by time 3. Discussion The analyses produced four main findings. Firstly, intervention districts, Sundargarh and Mayurbhanj, exhibited greater increases between survey periods time 1 and time 3, compared with the control district for 100% of the four prevention indicator comparisons, and 25% of the 12 health-seeking behaviour and ASHA capacity indicator comparisons. Secondly, the third intervention district, Nabrangpur, demonstrated significantly fewer improvements than other intervention districts. In only one of eight comparisons, that is measuring ASHA capacity, did Nabrangpur exceed the control. Thirdly, the control district exhibited greater increases relative to the intervention districts in only two instances: relative to Nabrangpur and Sundargarh. Fourthly, indicators concerning the supply chain (indicators 4 6) for RTDs and ACTs exhibited the poorest performance relative to the control with only 33% (three of nine comparisons) displaying a significant positive effect. This last finding is particularly important and may signal basic health system weaknesses; strengthening supply chain management in the periphery is a main aim of supervision. Odisha managers ought to examine why MTS were not able to improve logistics. Although the trial was not designed to systematically study the role of good management, the findings concerning Nabrangpur suggest that the coverage figures for key indicators improved more in the presence of access to both LQAS information and committed managers in Sundargarh and Mayurbhanj; both had worked in those position at least 3 years prior to the study. Results suggest that in this study, locally produced LQAS data were effective when managers able to use the findings were present in the districts. Our interpretation is also due to the Nabrangpur s consistently lower coverage for indicators despite being an intervention site as compared to Sundargarh and Mayurbhanji. During the trial, the Nabrangpur District Malaria Officer (who had been in his position <1 year) was unable to formulate an acceptable management response to his district s low indicator results. This deficiency may explain the higher dropout rate of MTS during year 1 (60%). With their departure, no personnel were in place to use LQAS results. By contrast, Odisha s VBDCP praised the control district s manager. Although he had been in his post <1 year, he had worked in other health sector management positions for a long time. Also he regularly visited the field to support MTS in problem solving. However, without data he could not perform as effectively as the two managers in Sundargarh and Mayurbhanj who had access to LQAS results John Wiley & Sons Ltd

8 The variation in intervention district performance across the three intervention categories is potentially important. The data suggest that the two districts with effective managers performed better than Nabrangpur for the prevention interventions and treatment-seeking interventions. They performed marginally better for ASHA capacity indicators. We know that during early 2010, shortly after dissemination of the baseline survey results that Odisha VBDCP managers initiated several system changes. For example, they prioritized low performing B-CHC for focused LLIN distribution, IRS spraying, and supply chain management reforms. They also, institutionalised LQAS surveys at the district level for evidencebased decision-making. However, these and other changes were effective in some but not all districts. This variation may suggest that in the study area strong central leadership ought to be complemented by equally strong district and B-CHC managers. However, the variation among indicator types may also be due to managers not being able to address all problematic areas simultaneously. After the baseline survey, NVBDCP prioritised the prevention indicators. As IRS and LLIN coverage took place within a vertical programme, problems were addressed by a combination of strong state leadership and district responsiveness. Sundargarh and Mayurbhanj managers responded more effectively than Kandhamal, possibly because they had the LQAS data to target problematic B-CHCs. However, the Nabrangpur manager remained non-responsive as he had poor relations with MTS and with Medical Officers In-Charge in B-CHC. Kandhamal s managers reported a larger proportional change compared to Nabrangpur although this difference was not significant. Between the two treatment-seeking behaviours, Indicator 3 (the proportion of people with fever in the last 2 weeks whose blood was tested with an RDT within 1 day of the fever and told the result on the same day as the test) displayed a significant increase over three time points whereas, Indicator 4 (the proportion of people with fever in the last 2 weeks who were RDT+ve and received ACT) did not. The difference between the two indicators is that the former requires district and B-CHC managers to train ASHA to correctly use RDTs, and maintain adequate supply chain management for RDT provision, as well as good supervision to monitor their appropriate use. Indicator 3 increased in Sundargarh and Mayurbhanj, but not in Nabrangpur and the control district. This deficiency may be because district managers were not engaged in using the LQAS data or in mobilising B-CHC staff. Indicator 4 is controversial as the use of ACTs had two systemic problems: Firstly, ASHA were newly introduced into the malaria programme for diagnosis and treatment of uncomplicated malaria cases. ACT was not adequately supplied from higher levels at the time, as a supply chain had not linked Federal state and district health systems. Secondly, not all B-CHC clinicians supported ASHA using ACT; during 2010, for example, some clinicians restricted ASHAs from using ACT. Once informed, NVBDCP began rectifying the problem. Earlier research in Peru (Lanata et al. 1990), albeit not organised as a trial, also concluded that LQAS usage increased coverage by alerting programme implementers to underperforming areas. In that research, immunisation coverage increased from 72% (59 85%) to 88% (84 92%) over 3 months when LQAS was used promptly after each immunisation campaign. However, the presence of LQAS results alone does not produce improvement. Only results combined with an understanding of performance problems and timely corrective action foster change. When LQAS was integrated into communitybased health care in Nepal to measure maternal and child health programmes, the programme displayed a significant improvement across all indicators over four time points (Valadez & Devkota 2002). A key part of the LQAS process highlighted in that project was a semiannual planning meeting to share results across SAs and jointly solve problems. Other longitudinal examples exist in Nicaragua (Valadez et al. 2005), Costa Rica (Valadez et al. 1995), and Uganda (Sprague et al. 2012). While the results of this trial cannot be generalised to other settings, when considered with other longitudinal applications of LQAS, they suggest the following testable hypothesis: managers use of LQAS data to prioritize problems and formulate programme solutions are associated with improvements in health indicators. Other non-lqas related research conducted among mid-level health managers (Loevinsohn 1994) in an unnamed developing country revealed a serious lack of confidence and skill in the analysis and use of data; only 24% of the respondents in that study could identify their best and worst performing districts for a specific indicator. The conclusions suggested that in addition to appropriate training, data must be presented simply and managers need an enabling environment promoting the routine use of data. Other research in India (Moreland et al. 2010) exploring barriers to effective data use found the majority of respondents not having timely access to survey results and limited computer skills or access to this technology. Lot Quality Assurance Sampling is a process and a classification method (Pagano & Valadez 2010); health workers are trained to analyse and interpret data to identify substantially underperforming SA. Using these 2014 John Wiley & Sons Ltd 1233

9 classifications they can develop plans to address SA problems while maintaining the programme intervention in areas classified as reaching targets. This process removes the need for computer data entry as hand tabulation is often preferred by local health workers, and allows prompt local use of results. In this trial the total per district cost for this use of LQAS per round, assuming a district with 10 B-CHC, was approximately $4000 (2011 US$) (including: training, data collection and hand tabulation); this cost does not include international technical assistance. Although LQAS has been used for more than 20 years in health sector M&E, this study is the first trial of LQAS that indicates benefits obtained by competent managers in Odisha using LQAS data to make strategic and tactical changes to their programmes (also see (Myatt et al. 2003)). Following the first-round LQAS surveys, intervention district results were shared with the state VBDCP who, acting upon the low coverage, improved the IRS and LLIN activities which they rolled-out to all districts, thereby contaminating the control. Further, in 2009 Kandhamal reported the highest number of deaths, thereby instigating NVBDCP to take precautions during 2010 by supplying higher quantities of LLINs, intensifying IRS, and intensifying early diagnosis and treatment by ASHA. An important limitation is the inclusion of only four districts in the trial. But in the case of India, using few districts may be inevitable given the nature of the research question, which involves assessing district level use of LQAS results. The four districts comprise a population exceeding five million people. Although the population is enormous, the few districts used in the design lead us to treat this study as a proof of principle; we do not generalise the results to other settings. Yet, while we are cautious about our statistical findings, LQAS did unmask a range of system weaknesses the VBDCP found useful such as: programme improvements were dependent on the district manager, the MTS workforce was unstable, supervision of district supply chain management was ineffective, some clinicians working at the B-CHC level undermined the strengthening of curative services by restricting ASHA from treating P.f. malaria with ACT. Another concern was the weak district leadership and attrition of MTS in Nabrangpur, which the State attributed to its distant location away from the State headquarters, and fewer health facilities. While these characteristics are a limitation, these deficiencies also indicate the importance of good management for successful use of M&E information. The study duration of 12 months was a limitation. A longer intervention may have resulted in larger intervention impacts and produced more stable trend data than this short study. Conclusion The study suggests that when functioning programme management was present in an intervention district in this trial, the use of LQAS for M&E was associated with programme strengthening and improved outputs and outcomes; in eight of 16 comparisons with the control district Sundargarh and Mayurbhanj had significant improvements. This result needs replication in a larger study and in diverse settings, due to the small number of districts used in the current study. By design, LQAS data inform local health. Using classifications, programme staff can quickly identify underperforming areas and, through structured problem analysis with committed district and state leaders, achieve improvements. These improvements can be made while maintaining areas that are classified as not currently a priority. However, state leadership was also essential to address systemic problems impeding the programme, especially for the prevention interventions. By 2013, Odisha s LQAS application had become a national showcase despite the need for many improvements (Hussain et al. 2013). Representatives from low resource countries visit Odisha to learn about its integration of LQAS into the routine work of field workers. LQAS activities expanded to all 21 high malaria-burden districts (population = ). In India, Odisha VBDCP managers have a unique capacity to compare results within and across districts, and assess state performance relative to other states, making it a standard for excellence. This may serve other states or countries, which can learn from the Odisha experience. While this is a small study (covering a large population), it revealed systemic problems within Odisha problems that can potentially be resolved by a confluence of efforts by state, district and B-CHC managers. Although this trial produced results that were relevant to Odisha, we do not generalise them to other settings. Rather we recommend replication of this trial in other settings with a larger number of districts. Acknowledgements We are grateful for the guidance and support of Anu Garg without whom this project would not have been possible. We also are grateful to our colleagues at DFID including Billy Stewart, Jyoti Tiwari and Manjula Singh for their continuous support. We also thank Michelle John Wiley & Sons Ltd

10 Stanton for her highly useful comments on earlier version of the manuscript. We are also deeply grateful to Marcello Pagano and Caroline Jeffery for their essential statistical advice and support. This work was supported by UK aid from the Department for International Development. However, the content does not necessarily reflect that of DFID or official UK government policy. References Basnyat B (2011) Malaria-attributed death rates in India. Lancet, 377, 993; author reply Campbell DT & Stanley J (1963) Quasi Experimental Design. Rand McNally, Chicago, IL. Dash AP (2009) Estimation of True Malaria Burden in India. A Profile of National Institute of Malaria Research. National Institute of Malaria Research, New Delhi, India. Dhingra N, Jha P, Sharma VP et al. (2010) Adult and child malaria mortality in India: a nationally representative mortality survey. Lancet 376, Hay SI, Gething PW & Snow RW (2010) India s invisible malaria burden. Lancet 376, Hussain MA, Dandona L & Schellenberg D (2013) Public health system readiness to treat malaria in Odisha State of India. Malaria Journal 12, 351. Kim JY, Farmer P & Porter ME (2013) Redefining global health-care delivery. Lancet 382, Kumar A, Valecha N, Jain T & Dash AP (2007) Burden of malaria in India: retrospective and prospective view. The American Journal of Tropical Medicine and Hygiene 77, Lanata CF, Stroh GJ, Black RE & Gonzales H (1990) An evaluation of lot quality assurance sampling to monitor and improve immunization coverage. International Journal of Epidemiology 19, Loevinsohn BP (1994) Data utilization and analytical skills among mid-level health programme managers in a developing country. International journal of epidemiology 23, Milligan P, Njie A & Bennett S (2004) Comparison of two cluster sampling methods for health surveys in developing countries. International Journal of Epidemiology 33, Moreland S, Misra SN, Agrawal S, Gupta RB & Harrison T (2010) Data Use in the Indian Health Sector Research report, MEASURE Evaluation, Chapel Hill, NC, pp. 63. Murray CJL, Rosenfeld LC, Lim SS et al. (2012) Global malaria mortality between 1980 and 2010: a systematic analysis. Lancet 379, Myatt M, Limburg H, Minassian D & Katyola D (2003) Field trial of applicability of lot quality assurance sampling survey method for rapid assessment of prevalence of active trachoma. Bulletin of the World Health Organization 81, National Vector Borne Disease Control Programme (2011a) Malaria control strategies. Delhi: Government of India. National Vector Borne Disease Control Programme (2011b) Malaria situation Delhi: Government of India. Pagano M & Valadez JJ (2010) Commentary: Understanding practical lot quality assurance sampling. International Journal of Epidemiology 39, Reinke WA (1988) Industrial Sampling Plans: Prospects for Public Health Applications. Institute for International Programs, The Johns Hopkins University School of Hygiene and Public Health, Baltimore, MD. Robertson SE & Valadez JJ (2006) Global review of health care surveys using lot quality assurance sampling (LQAS) Social Science & Medicine 63, Robertson SE, Anker M, Roisin AJ, Macklai N, Engstrom K & Laforce FM (1997) The lot quality technique: a global review of applications in the assessment of health services and disease surveillance. World Health Statistics Quarterly Rapport Trimestriel De Statistiques Sanitaires Mondiales 50, Shah NK, Dhariwal AC, Sonal GS, Gunasekar A, Dye C & Cibulskis R (2011) Malaria-attributed death rates in India. Lancet, 377, 991; author reply Sprague DA, Jeffery TAHC, Ouma J, Lwanga SK, Valadez JJ & Roberts GO (2012) Delivery practices among women across Uganda Royal Society for Tropical Medicine and Health. Univerisity of Warwick. The World Bank (2008) National vector borne disease control and polio eradication support project. Project Appraisal Document. African Region, Washington, DC. Turner AG, Magnani RJ & Shuaib M (1996) A not quite as quick but much cleaner alternative to the Expanded Programme on Immunization (EPI) Cluster Survey design. International Journal of Epidemiology 25, Valadez JJ (1991) Assessing Child Survival Programs in Developing Countries: Testing Lot Quality Assurance Sampling. Harvard University Press, Boston, MA. Valadez JJ & Devkota BR (2002) Decentralized supervision of community health program using LQAS in two districts of Southern Nepal. In: Community-Based Health Care: Lessons from Bangladesh to Boston (eds J Rhode & J Wyon) Management Sciences for Health, Boston, MA, pp Valadez JJ, Weld L & Vargas WV (1995) Monitoring community health workers performance through lot quality-assurance sampling. American Journal of Public Health 85, Valadez JJ, Hage J & Vargas W (2005) Understanding the relationship of maternal health behavior change and intervention strategies in a Nicaraguan NGO network. Social Science and Medicine 61, Valadez JJ, Weiss W, Leburg C & Davis R (2007) Assessing Community Health Programs: A Participant s Manual and Workbook: Using LQAS for Baseline Surveys and Regular Monitoring. TALC, St Albans. Valecha N, Staedke S, Filler S, Mpimbaza A, Greenwood B & Chandramohan D (2011) Malaria-attributed death rates in India. Lancet 377, World Health Organization (2008) World Malaria Report World Health Organisation, Geneva John Wiley & Sons Ltd 1235

11 Yadav RS, Bhatt RM, Kohli VK & Sharma VP (2003) The burden of malaria in Ahmedabad city, India: a retrospective analysis of reported cases and deaths. Annals of Tropical Medicine and Parasitology 97, Corresponding Author Joseph J. Valadez, Liverpool School of Tropical Medicine, Pembroke Place, Liverpool, L3 5QA, UK. Tel.: +44 (0) ; Joseph.Valadez@lstmed.ac.uk John Wiley & Sons Ltd