HEALTH AND NUTRITION TRACKING SERVICE (HNTS) Review of Publicly Available Surveys, North Kivu, DRC,

The Health and Nutrition Tracking Service (HNTS) is an interagency initiative hosted by WHO HEALTH AND NUTRITION TRACKING SERVICE (HNTS) Review of Publicly Available Surveys, North Kivu, DRC, 2006-2008 The Health and Nutrition Tracking Service (HNTS), an interagency initiative hosted by WHO, was created in response to a request made by the United Nations Emergency Relief Coordinator as part of the Humanitarian Reform process. The HNTS was established in late 2007 by the Inter-Agency Standing Committee (IASC) Health and Nutrition Clusters. The Health and Nutrition Tracking Service (HNTS) aims to provide impartial, credible and timely information on mortality and nutrition rates in populations affected by crises and emergencies, especially the least funded and publicized ones, using standardized data collection and analysis methods wherever possible. The information gathered will help improve humanitarian operations by (1) rapidly detecting excess mortality and malnutrition in crises using key indicators, (2) promoting mutual accountability between the humanitarian community and beneficiaries, and (3) ensuring evidence-based information on health and nutrition needs in crises is available to high-level decision-makers. The HNTS has two main functions. It offers operational support to humanitarian staff in the field by peer-reviewing guidelines and other documents, participating in assessment missions, advising on the design of surveys, and providing technical advice to various agencies. Its normative functions include developing standards for data collection and measurement through its Expert Reference Group, collecting, analysing and disseminating data, and providing independent technical advice on various issues related to method development and validation studies. Contact email: hnts@who.int 1

The Health and Nutrition Tracking Service (HNTS) is an interagency initiative hosted by WHO Contact email: hnts@who.int 2

The Health and Nutrition Tracking Service (HNTS) is an interagency initiative hosted by WHO Geneva, Tuesday, 19 May 2009 Dear all, Please find enclosed a review of publicly available surveys, in North Kivu, 2006-2008. It was carried out by Epicentre (www.epicentre.msf.org) upon HNTS' request. A short presentation (see attached PowerPoint presentation) was also done during the first Expert Reference Group meeting held 17 th and 18 th of February 2009 in Geneva. As you will see, conclusions and recommendations show the need for the HNTS to develop its technical support to relief organizations and other partners conducting surveys in the region (but not exclusively) and to strengthen its peer review function. During a recent HNTS field visit to DRC, the recommendations of this review were useful to discuss current needs about quality of collection, analysis and interpretation of data, especially with national authorities and Health and Nutrition Clusters' partners. With the coming deployment in Goma of a HNTS epidemiologist to support national authorities and relief organizations working in the region, we hope to be able to provide adequate technical support in the field of data collection and analysis, and quality and validation of mortality and nutrition surveys. Main conclusions of the review The surveys reviewed here vary in methodological quality and utility for planning interventions. Despite these differences, the results obtained for the CMR, U5MR and nutritional status are within the same range. Although this provides some degree of assurance, without a gold standard it is difficult to know whether these consistent results are also accurate. All surveys, except one, employed cluster-based sampling. Two of the surveys had as primary objective nutritional status and mortality and health indicators as secondary objective. The primary objective of the IRC survey concerned documentation of mortality and the long-lasting effects of populations in conflict, rather than guiding specific operational responses. The limitations of performing this review highlight the need for ongoing training and discussion about the utility and conduct of retrospective mortality surveys and nutritional assessments. First, the very fact that performing this review necessitates an understanding of both field epidemiological methods and constraints inherent in the conduct of such surveys limits the utility of this review for a wider audience. Second, we only included reports of surveys where an actual document could be assessed and available to the public. We clearly did not include all surveys conducted in North Kivu during that time. Furthermore, as we based this review on the written document without having seen the data collected or other relevant documentation, some issues are difficult to assess. Among the key weaknesses present in the surveys reviewed here, several are worth further discussion. The first weakness, which could be considered a detail, is that none of the survey reports, except the IRC, states within the text of the report that they obtained oral informed consent from participants. This may have been the case in practice, or may have been contained within other documents (training guides for interviewers for example), but there is no explicit description of how the survey was explained to participants. This is important for both the conduct of an appropriate survey, but also in terms of communicating to participants why their participation is important and providing them with a means to obtain the results of the survey if they wish. This plays a major role in the credibility of surveys if they are to be used further for publication to a wider audience and in discussions with donors, who may require this information to use survey results in policy decision-making. None of the surveys sought ethical committee approval prior to their conduct. The subject of the need and feasibility of obtaining ethical committee approval for such surveys is debated and not the focus of this paper. Contact email: hnts@who.int 3

The Health and Nutrition Tracking Service (HNTS) is an interagency initiative hosted by WHO The second key weakness is also methodological. Three surveys used cluster based sampling, which although used most frequently, is also often misinterpreted. As in the conduct of any survey, the sample size should correspond to the stated survey objectives. The sampling scheme should be described with potential sources of bias listed and their potential impact on the validity of the survey discussed. In the specific case of cluster-based sampling, authors should report the design effect and state that they accounted for it in the calculation of 95% confidence intervals around estimates. Three surveys reviewed here used short recall periods, which lead to wider confidence intervals around estimates. The choice of a recall period should be justified by quantitative or anecdotal evidence about past mortality in the population, and by the stated survey objectives. For further information on best practices in the conduct of retrospective mortality surveys, the monograph by F. Checchi and L. Roberts discusses these issues in detail. The third set of weaknesses concerns the interpretation of results. Although the objectives of all surveys are stated, what is unclear is whether the results of the surveys were actually interpreted clearly. In all reports except that of the IRC, an experienced reader is left with the impression that these surveys were mechanical and conducted as a reflex rather than providing either internal guidance on policy making or for a wider audience. The recommendations are similar and in some cases miss the forest for the trees. Meaning that in all surveys measles vaccination coverage was dangerously low, but does not appear to have sounded the alarm. Although the organizations conducting these surveys may not be capable of supporting or conducting a mass vaccination intervention in response, this indicates the vertical nature of the recommendations. There was little evidence of a holistic assessment of overall health status and possible interventions that may follow by the respective operational organizations. Complicating matters further, retrospective mortality surveys and nutritional assessments have important strengths, but their conduct may come too late to guide a meaningful intervention. As their very nature and objective is different than that of a prospective surveillance system, evaluating the immediate impact of specific interventions is virtually impossible using this methodology. In some cases, prospective surveillance may not be feasible, and retrospective or cross-sectional surveys the only option, but their use as a program evaluation tool necessitates their repetition at intervals relevant to the program objectives. Recommendations Although this review looked at only 4 surveys and is related to only one area of humanitarian action, the recommendations that follow apply to other contexts. First, many surveys are conducted by experienced field epidemiologists, but there is not a formal mechanism for NGOs to present their survey protocols for methodological review. Ethical committee approval may ensure that participants are respected, but methodological questions and the specificity of the conduct of mortality and nutritional surveys still lacks sufficient forum for discussion. The recently formed Expert Reference Group of the HNTS, or another similar body such as the Technical Advisory Committee of SMART, could provide a mechanism for peer-review of these protocols before implementation. This would ideally help to prevent serious methodological pit-falls, and improve the overall quality of the information collected. Having experienced field epidemiologists review survey protocols serves several purposes. First, NGOs may have internal assurance that if their protocols are adhered to, their results will be informative for their stated objectives. Second, official review of surveys meant principally for advocacy purposes, whether this is stated clearly or not, can help ensure that they will be met with less criticism if publicized or submitted to peer-reviewed publications. Third, the act of presenting a protocol would ensure that it is actually written, discussed before implementation and the objectives and expected results clarified. Second, there is a clear and perpetual need to look beyond retrospective mortality surveys and punctual nutritional assessments. In some cases, organizations are present in an area for an extended period and the implementation of a prospective surveillance system, however rudimentary, is possible. Contact email: hnts@who.int 4

The Health and Nutrition Tracking Service (HNTS) is an interagency initiative hosted by WHO Surveillance systems may not always be possible in a context of ongoing violence, but should be investigated. Third, initiatives such as SMART have emerged to ensure standardization of planning, training, analysis and minimum reporting requirements. The initiative aims to revise guidance on a continual basis taking into account feedback from field epidemiologists. The SMART initiative also provides user-friendly software for the conduct of mortality and nutritional assessments. However, the use of the software alone is insufficient to ensure the appropriate conduct of surveys. The guidelines that accompany the software are just as essential. Nevertheless, through initiatives like SMART, some degree of standardization could be assured allowing results to be compared and shared. It would ensure that new interventions can be planned and existing ones adapted. The HNTS Technical Secretariat is pleased to share this review of surveys. It remains at your disposal to discuss its conclusions and recommendations, and to assess the best ways to strengthen the future quality of collection, analysis and interpretation of data in DRC. Sincerely Pierre Salignon Project director Health and Nutrition Tracking Service (HNTS) Mobile: 0041-79-50-90-631 Phone: 0041-22-791-14-48 E-mail: salignonp@who.int Contact email: hnts@who.int 5

The Health and Nutrition Tracking Service (HNTS) is an interagency initiative hosted by WHO Contact email: hnts@who.int 6

1 2 Review of Publicly Available Surveys, North Kivu, DRC, 2006-2008 Benjamin Coghlan Rebecca Freeman Grais Emmanuel Greletty Francisco J.Luquero Heloise Pham March 2009 Centre Collaborateur de l OMS pour la Recherche en Epidémiologie et la Réponse aux Maladies Emergentes TELEPHONE : 00 33 (0)1 40 21 28 48 FAX : 00 33 (0)1 40 21 28 03 E-MAIL : EPIMAIL@EPICENTRE.MSF.ORG WEB : HTTP://WWW.EPICENTRE.MSF.ORG ASSOCIATION LOI 1901. 1

SUMMARY BACKGROUND: Field epidemiologists often conduct mortality and nutritional assessments via surveys to guide humanitarian decision-making and to document a crisis. Here, we review publicly available mortality and nutritional surveys conducted in North Kivu, Democratic Republic of Congo (DRC) published from January 2006 to January 2009. Our aim was to provide an overview of the surveys as a means to explore how these surveys can be used to inform humanitarian decision-making. METHODS: We performed a PubMed/Medline search for articles published from January 1, 2006 to January 1, 2009. To identify non-peer-reviewed reports, we performed the same search in publicly available databases and clearinghouses for humanitarian relief. We excluded meta-analysis, commentaries and reports on DRC, but in which no information about North Kivu was available. To evaluate the surveys, we based our criteria on that described by Mills et al. for mortality surveys with additional criteria specific to nutritional assessments. RESULTS: We identified 38 reports through our search strategy, of which 4 surveys met our inclusion criteria. The surveys varied in methodological quality and utility for planning interventions. The results obtained for the CMR, U5MR and nutritional status are within the same range. Although this provides some degree of assurance, without a gold standard it is difficult to know whether these consistent results are also accurate. All of the surveys, except one, employed cluster-based sampling. Two of the surveys had as primary objectives nutritional status and mortality and health indicators as secondary objectives. The primary objective of the IRC survey concerned documentation of mortality and the long-lasting effects of populations in conflict, rather than to guide specific operational responses. CONCLUSIONS: The limitations of performing this review highlight the need for ongoing training and discussion about the utility and conduct of retrospective mortality surveys and nutritional assessments. Retrospective mortality surveys and nutritional assessments have important strengths, but their conduct may come to late to guide a meaningful intervention. There is a clear and perpetual need to look beyond the specific results of punctual retrospective mortality surveys and nutritional assessments by placing results in a larger context. 2

1 INTRODUCTION... 5 2 METHODS... 6 2.1 SETTING...6 2.2 LITERATURE REVIEW...7 2.3 EVALUATION CRITERIA...7 3 RESULTS... 8 3.1 SURVEYS...9 3.1.1 Action Contre la Faim (ACF)... 9 3.1.2 Cooperazione Internazionale (COOPI)... 9 3.1.3 International Rescue Committee (IRC)... 10 3.1.4 Epicentre... 10 4 CONCLUSIONS... 11 5 RECOMMENDATIONS... 12 6 REFERENCES... 14 3

This review was performed at the request of the Health Nutrition Tracking Service and the results were presented at the Expert Reference Group on February 18, 2009. 4

1 Introduction In media and agency reports on crises stemming from conflict, such as the on-going crisis in North Kivu, Democratic Republic of Congo (DRC), an estimate of the number of people who have died since the start of the conflict, or during the current episode of violence, is often quoted. Although a discriminating reader may understand that these are just estimates, we rarely question how, or from where, these numbers were obtained. Even if we assume the estimates are reasonable, it may be difficult to conceptualize what they actually mean: what makes this particular situation an emergency, rather than another and how the humanitarian response required differs depending on the mortality estimate. The same is true for nutritional crises. Frequently, mortality and nutritional assessments are conducted together with both estimates provided. Whether a crisis is considered an emergency has implications for funding, how the event is perceived in the eyes of the world and the scope of the humanitarian response (1). There are different definitions, but the term complex humanitarian emergency usually refers to a situation where a civilian population is displaced from their homes by war, conflict or natural disaster. Often, living conditions have deteriorated, resulting in significant excess mortality either in the short or long term (2). As complex emergencies mostly occur in contexts where births and deaths are not reported routinely and there is no functioning health or demographic surveillance system, a retrospective survey is used to obtain a mortality estimate and determine the nutrition status of the population. Cluster based sampling often provides the quickest and most logistically feasible option. It does not require a list of sampling units, and can be conducted in any setting. In the first stage, the area under study is divided into clusters based on geographical location (e.g. villages, towns, urban zones). An estimation of the population size of each cluster is made and clusters can then be randomly selected for inclusion in the survey. This is accomplished through probability proportional to size sampling (PPS), which ensures that in densely populated areas more clusters are sampled than in less dense areas. In the second stage, once sample clusters are chosen, households are selected. A representative is interviewed concerning the number of household inhabitants, new arrivals, births, deaths and people who have left the household. Interviews are designed to provide information for a specific time-frame on the number of people currently living in the household, the number who have died (crude mortality rate (CMR) and under five mortality (U5MR) and their presumed cause of death (3). To assess nutritional status, weight, height and the mid-upper arm circumference (MUAC) are measured in children. Also, children are assessed clinically for the presence of bilateral oedema. Often data on other indicators including vaccine coverage, access to food and non-food items, sanitation and other nutritional information (for example, vitamin A) are collected at the same time. Different methods of conducting such surveys are the subject of debate among epidemiologists and their strengths and weakness have been widely described in the literature (4-7). However, even if the sampling strategy were ideal, interpreting mortality and nutritional surveys presents another problem. Field epidemiologists face difficulties in interpreting survey results and how to use them to guide humanitarian response. Here, we perform a review of publicly available mortality and nutritional surveys conducted in North Kivu, DRC published from January 2006 to January 2009. The aim of this review was to provide an overview of the surveys as a means to explore how these surveys can be used to inform humanitarian decision-making. 5

2 Methods 2.1 Setting The current crisis in the eastern DRC Province of North Kivu is the most recent episode of violence in a country where, since 1998, an estimated 5.4 million people have lost their lives (Figure 1). The Province of North Kivu hosts more than half the internally displaced population in DRC. Renewed violence has forced some 250,000 people to flee their homes since August 2008, despite the official end of the war following the signature of peace agreements in December 2002. Reports indicate that another 20,000 persons were displaced in October 2008 and population displacement continues (8). Figure 1: Map of North Kivu, Democratic Republic of Congo 6

2.2 Literature review We performed a PubMed/Medline search for articles published from January 1, 2006 to January 1, 2009, in English, French, German, or Spanish. We searched using the key words mortality (major topic) OR nutrition (major topic) AND Congo (text word) OR Democratic Republic of Congo OR North Kivu. To identify non-peer-reviewed reports on mortality and nutrition surveys, we performed the same search in the following locations: the Human Impact of Complex Emergencies Complex (CE-DAT) database Relief-Web (a media and NGO repository maintained by the Office for the Coordination of Humanitarian Affairs); RDC-humanitaire.net; and the websites and databases: International Rescue Committee, Merlin, Action Contre la Faim, UNICEF, UNHCR, Medecins Sans Frontieres and Epicentre. We also contacted individual organizations and requested additional information from the Health and Nutrition Tracking Service (HNTS). To be included in the review, the survey needed to be documented in writing by means of either a peer-reviewed publication or internal report. We excluded meta-analysis, commentaries on mortality and nutrition studies and reports or articles reporting on DRC, but in which no information about North Kivu was available. Multi-sectorial agency evaluations, humanitarian plans of action and rapid assessments of small and non-randomized populations were excluded. The aim of the literature was not to conduct a systematic review, but rather perform a search of information easily available to field epidemiologists. 2.3 Evaluation criteria To evaluate the surveys, we based our criteria on that described by Mills et al. (9) for mortality surveys. We added additional criteria specific to nutritional assessments. The criteria include three key areas: validity, results and utility. We assessed the overall strengths and weaknesses of the methodology and addressed the following specific elements: Were the objectives of the survey clearly stated? Was authorization for the survey obtained? Was the studied sample representative of the underlying population? Was random sampling performed? Was a large proportion of the chosen sample been interviewed? Were there specific strategies to ensure data accuracy? Were HH revisited to confirm findings? How large is the mortality rate? How precise is the estimate of the mortality rates? What is the absolute death toll over the period of analysis? Can the results be applied to my setting? What were the specific causes of death? Was seasonality of malnutrition taken into account? (survey conducted pre/post harvest) Were food security, health, water and sanitation taken into consideration? Could findings be corroborated with other sources? What were the recommendations provided as a result of the survey? Where possible, we also assessed if either a new or modified humanitarian response followed as a result of the survey. For the nutritional surveys, we examined the indicators used (National Center for Health Statistics references and WHO growth standards, MUAC and 7

presence of bilateral oedema). We also noted whether the surveyors used age, height or both in the assessment of individual children to determine their nutritional status. 3 RESULTS We identified 38 agency reports through our search strategy (Figure 2). Of these 7 were found through PubMed/MEDLINE, 4 through CE-DAT, 1 through Reliefweb, 23 through RDC-humanitaire.net, and 3 via individual agency/organizations web sites. No additional reports were identified through citations in articles or reports. We were able to obtain 36 of the 38 documents. The two that we were unable to obtain as written documents were a rapid field assessment of critical needs for nutrition, hygiene, water and sanitation conducted by Action Contre la Faim in November 2008, and a nutritional survey conducted by World Vision in Rwanguba health zone in March 2007. Of these 38 reports, only 4 surveys met our inclusion criteria. Those excluded included 22 multi-sectorial evaluations, 2 humanitarian action plans and 1 survey covering the entire country, but without specific mention of North Kivu. None of the articles referenced in PubMed/Medline met our inclusion criteria. The retained surveys are discussed below. 7 abstracts identified in Pubmed database 31 studies identified in NGO s databases 7 excluded 2 surveys not found * 29 full-text articles reviewed 2 Humanitarian action plans (OCHA) excluded 1 survey on all DRC excluded (EDS/RDC) 22 multisectorial evaluations (Unicef and Norwegian Refugee council) excluded 4 surveys included in the analysis 1 mortality survey 3 mortality and nutrition surveys Figure 2: Flow diagram of studies included in review 8

3.1 Surveys Four surveys meeting the inclusion criteria were conducted in North Kivu between January 1 2006 and January 1, 2009. Three surveys were conducted in 2008. The nongovernmental organizations (NGOs) conducting the studies were Action Contre la Faim, Cooperazione Internazionale, the International Rescue Committee and Epicentre (Table 1) (10-13). All surveys stated that authorization from the Ministry of Health was obtained. No survey sought ethical committee approval. The objectives of all surveys included the estimation of CMR and U5MR. Three surveys also included an assessment of the nutritional status and measles vaccination coverage. A description of the methodology and mortality results are presented in Table 2, the nutritional assessment results in Table 3, the evaluation criteria for mortality surveys in Table 4 and for nutritional surveys in Table 5. 3.1.1 Action Contre la Faim (ACF) The ACF survey was conducted in Kibua (North Kivu), where the estimated population was 81,175 inhabitants, with 13,799 children under the age of five. They carried out a cross-sectional study using a two-stage household based cluster sampling strategy. Clusters were selected proportional to population size based on population data provided by the Bureau of the Census. Second-stage (household level) sampling was conducted using the WHO-EPI spin the pen method. The stated sample size was 900 children 6 to 59 months to obtain a representative estimate of the prevalence of acute malnutrition with 95% confidence intervals (translated from French). Household members of all ages were included for the estimation of CMR and U5MR. The recall period for CMR and U5MR was 90 days. A total of 30 clusters, each with 30 families was desired, but 32 clusters were selected to replace clusters that were inaccessible due to security or logistic reasons. The authors obtained a representative sample, using random sampling and sampled a large proportion of the target population. No information was provided concerning strategies to ensure data accuracy or information concerning whether households were revisited to confirm the findings. The definition of a household was stated as persons eating in the same cooking pot but children residing less than 3 months in the family were excluded from the analysis. The total number of deaths estimated is not reported. Other missing information includes the main causes of death and no comparisons with other sources of information were provided. For the nutritional assessment, the results presented are those for the percentage of the reference median using both the NCHS reference and WHO standards. The MUAC and oedema were also assessed. The survey was conducted in the first month of the harvest (June) and this is stated, although information concerning seasonality in previous years or quality of harvest is not mentioned. The surveyors also report an estimated measles vaccination coverage of 11.3% by card verification. The recommendations derived from the survey include: the provision of nutritional education and treatment, an assessment of food security, improving routine immunization and continued monitoring of the situation. 3.1.2 Cooperazione Internazionale (COOPI) The COOPI study was conducted in Binza within Rutshuru province (Nord Kivu), where the estimated population was 102,284 inhabitants, with 24,355 children under five. They carried out a cross-sectional study using a two-stage household based cluster sampling 9

strategy. Clusters were selected proportional to population size based on population data provided by the Bureau of the Census. Second-stage (household level) sampling was conducted ostensibly using the WHO-EPI spin-the-pen method, but this was not implemented. The surveyors included households only within the first 30 households to the center of each cluster, thereby conducting a convenience sample. The stated sample size was 815 children 6 to 59 months, with precision of 3% and an estimated prevalence of global acute malnutrition of 11% (based on a previous survey conducted in 2005). The authors state that they employed a design effect of 2 and provide the formula employed for their sample size calculation. The authors also report that they increased their desired sample size to 915 due to the security situation. Household members of all ages included for the estimation of CMR and U5MR. The recall period for CMR and U5MR was 90 days. A total of 30 clusters, each with 30 families was sampled. No information regarding the proportion of the chosen sample that was interviewed, strategies to ensure data accuracy and whether households were revisited to confirm findings was reported. The absolute number of deaths is not provided, nor the main reported causes, nor comparisons with other information to corroborate findings. The definition of a household is not provided in the report. Indicators using the NCHS reference, WHO growth standard, MUAC and presence of bilateral oedema are reported. The surveyors report estimated measles vaccination coverage of 11.2% by card. The recommendations derived from the survey center on the need to strengthen health services. 3.1.3 International Rescue Committee (IRC) The IRC study was conducted to estimate mortality in the whole country. Estimates are provided for different regions including North Kivu. Within North Kivu, Itebero, Lubero and Rwanguba were included in the survey. Investigators used three-stage cluster sampling. The recall period covered 16 months and both the CMR and the U5MR were estimated. They did not perform a nutritional survey. Unlike the three other surveys, the IRC survey is the fifth in a series of surveys with the aim of documenting the humanitarian situation in DRC resulting from the conflict by providing mortality estimates. This survey was conducted for advocacy purposes rather than to inform directly the immediate humanitarian operations. The fourth survey in the series was published in a peer-reviewed journal and the investigators have used the same methodology over time (12). The surveys conducted by the IRC have been widely publicised and are the subject of debate. The debate focuses on the calculation of excess deaths attributed to the conflict and the validity of retrospective mortality surveys (14). Calculation of excess deaths requires a solid baseline relevant for a country as large and diverse as DRC. In the absence of solid demographic data, counting deaths by visiting households is an alternative. Concerning the evaluation criteria, this study fulfilled all the criteria but the revisiting of households. The investigators state that they were unable to revisit empty households due to logistic and security constraints, and that as a result, the final sample lost more than 20% of the expected number of households. 3.1.4 Epicentre The Epicentre study was conducted in Bambu (Nyanzale, Nord Kivu), where the estimated number of households was 1701. They carried out a cross-sectional survey using systematic sampling. The recall period covered 60 days and both the CMR and the U5MR were reported. Regarding the quality criteria, this study fulfilled most of the criteria but they did not report the total estimated death toll and they did not corroborate their result with other 10

information available in the discussion. Further, although this survey is greatly detailed, the conduct was that of a rapid assessment, although it is not stated as such. The survey teams were unable to survey the populations they intended and report that they selected their survey site by default. The Epicentre survey was the only survey not to use cluster-based sampling and the only to note that a prospective surveillance system was put in place immediately after the conduct of the survey. 4 Conclusions The surveys reviewed here vary in methodological quality and utility for planning interventions. Despite these differences, the results obtained for the CMR, U5MR and nutritional status are within the same range. Although this provides some degree of assurance, without a gold standard it is difficult to know whether these consistent results are also accurate. All of the surveys, except one, employed cluster-based sampling. Two of the surveys had as primary objectives nutritional status and mortality and health indicators as secondary objectives. The primary objective of the IRC survey concerned documentation of mortality and the long-lasting effects of populations in conflict, rather than to guide specific operational responses. The limitations of performing this review highlight the need for ongoing training and discussion about the utility and conduct of retrospective mortality surveys and nutritional assessments. First, the very fact that performing this review necessitates an understanding of both field epidemiological methods and constraints inherent in the conduct of such surveys limits the utility of this review for a wider audience. Second, we only included reports of surveys where an actual document could be assessed and available to the public. We clearly did not include all of the surveys conducted in North Kivu during this time. Futher, as we based this review on the written document, without having seen the data collected or other relevant documentation, some issues are difficult to assess. Of the key weaknesses present in the surveys reviewed here, several are worth further discussion. The first weakness, which could be considered a detail, is that none of the survey reports, except the IRC, state within the text of the report that they obtained oral informed consent from participants. This may have been the case in practice, or contained within other documents (training guides for interviewers for example), but there is no explicit description of how the survey was explained to participants. This is important for both the conduct of an appropriate survey, but also in terms of communicating to participants why their participation is important and providing them with a means to obtain the results of the survey if they wish. This plays a larger role in the credibility of surveys if they are to be used further for publication to a wider audience and in discussions with donors, who may require this information for use of survey results in policy decision-making. None of the surveys sought ethical committee approval prior to their conduct. The subject of the need and feasibility of obtaining ethical committee approval for such surveys is debated and not the focus of this paper. The second key weaknesses are also methodological. Three surveys used cluster based sampling, which although used most frequently, is also frequently misinterpreted. As in the conduct of any survey, the sample size should correspond to the stated survey objectives. The sampling scheme should be described with potential sources of bias listed and their potential impact on the validity of the survey discussed. In the specific case of cluster-based sampling, authors should report the design effect and state that they accounted for it in the calculation of 95% confidence intervals around estimates. Three surveys reviewed here used short recall periods, which lead to wider confidence intervals around estimates. The choice of a recall period should be justified by quantitative or anecdotal evidence about past mortality in the 11

population, and by the stated survey objectives. For further information on best practices in the conduct of retrospective mortality surveys, the monograph by Checchi and Roberts discusses these issues in detail (15). The third set of weaknesses concerns the interpretation of results. Although the objectives of all surveys are stated, what is unclear is whether the results of the surveys were actually interpreted clearly. In all reports except that of the IRC, an experienced reader is left with the impression that these surveys were mechanical and were conducted as a reflex rather than to provide either internal guidance on policy making or for a wider audience. The recommendations are similar and in some cases miss the forest for the trees. Meaning, that in all surveys measles vaccination coverage was dangerously low, but does not appear to have sounded the alarm. Although the organizations conducting these surveys may not be capable of supporting or conducting a mass vaccination intervention in response, this indicates the vertical nature of the recommendations. There was little evidence of a holistic assessment of overall health status and possible interventions that may follow by the respective operational organizations. Complicating matters further, retrospective mortality surveys and nutritional assessments have important strengths, but their conduct may come to late to guide a meaningful intervention. As their very nature and objective is different than that of a prospective surveillance system, evaluating the immediate impact of specific interventions is virtually impossible using this methodology. In some cases, prospective surveillance may not be feasible, and retrospective or cross-sectional surveys the only option, but their use as a program evaluation tool necessitates their repetition at intervals relevant to the program objectives. 5 Recommendations Although this survey review looked at only 4 surveys and in only one area of humanitarian action, the recommendations that follow apply to other contexts. First, many surveys are conducted by experienced field epidemiologists, but there is not a formal mechanism for NGOs to present their survey protocols for methodological review. Ethical committee approval may ensure that participants are respected, but methodological questions and the specificity of the conduct of mortality and nutritional surveys still lacks sufficient forum for discussion. The Expert Review Group of the HNTS could provide a mechanism for peer-review of these protocols before implementation. This would ideally help prevent serious methodological pit-falls, some of which were discussed here, and improve the overall quality of the information collected. Having experienced field epidemiologists review survey protocols serves several purposes. First, NGOs may have internal assurance that if their protocols are adhered to, their results will be informative for their stated objectives. Second, official review of surveys meant principally for advocacy purposes, whether this is stated clearly or not, can help ensure that they will be met with less criticism if publicized or submitted for peer-reviewed publications. Third, the act of presenting a protocol would ensure that a protocol is actually written, discussed before implementation and the objectives and expected results clarified. Second, there is a clear and perpetual need to look beyond retrospective mortality surveys and punctual nutritional assessments. In some cases, organizations are present in an area for an extended period and the implementation of a prospective surveillance system, however rudimentary, is possible. Surveillance systems may not always be possible in a context of ongoing violence, but should be investigated. Third, initiatives such as SMART have emerged to ensure standardization of planning, training, analysis and minimum reporting requirements. The initiative aims to revise guidance on a continual basis taking into account 12

feedback from field epidemiologists. The SMART initiative also provides user-friendly software for the conduct of mortality and nutritional assessments. However, use of the software alone is insufficient to ensure the appropriate conduct of surveys. The guidelines that accompany the software are just as essential. Nevertheless, through initiatives like SMART, some degree of standardization could be assured allowing for results to be compared, shared to ensure that new interventions can be planned and existing interventions adapted. 13

6 References 1. Spiegel P. Differences in World Responses to Natural Disasters and Complex Emergencies. JAMA. 2005; 293: 1915-1918. 2. National Research Council (2002). Demographic Assessment Techniques in Complex Humanitarian Emergencies: Summary of a Workshop. Holly Reed, Rapporteur, Roundtable on the Demography of Forced Migration, Committee on Population. Washington, DC: National Academy Press. 3. World Food Program. A manual: Measuring and interpreting malnutrition and mortality. CDC and WFP: Rome; 2005. 4. Brown V, Checchi F, Depoortere E, Grais RF, Greenough PG, Hardy C, Moren A, Richardson L, Rose AM, Soleman N, Spiegel PB, Sullivan KM, Tatay M, Woodruff BA. Wanted: studies on mortality estimation methods for humanitarian emergencies, suggestions for future research. Emerg Themes Epidemiol. 2007 Jun 1;4(1):9 5. Soleman N, Chandramohan D, Shibuya K. Verbal autopsy: current practices and challenges. Bull World Health Organ. 2006 Mar;84(3):239-45. Epub 2006 Mar 22 6. Turner AG, Magnani RJ, Shuaib M: A not quite as quick but much cleaner alternative to the Expanded Programme on Immunization (EPI) cluster survey design. Int J Epidemiol 1996, 25(1):198-203. 7. Grais RF, Rose AMC, Guthmann J-P: Don t spin the pen: Alternative methods for second-stage sampling in retrospective cluster surveys. Emerg Themes Epidemiol. 2007 Jun 1;4(1):8 8. See for example: http://ec.europa.eu/echo/aid/sub_saharian/rdc_en.htm. 9. Mills E, et al. User s guide to the medical literature: how to use an article about mortality in an humanitarian emergency. Conflict and Health 2008, 2:9. 10. Action Contra La Faim. Rapport d Enquête Nutritionnelle Anthropométrique, Zone de Santé de Kibua, Province Du Nord Kivu, Juin 2008, http://www.cedat.be/tools/database/index.php?id=3 (Accessed February 2009) 11. Rizzi D. Rapport d Enquête Nutritionnelle et de Mortalité dans la Zone de Santé de Binza Province du Nord Kivu, République Démocratique du Congo, Juillet 2008, Cooperazione Internazionale, http://www.cedat.be/tools/database/index.php?id=3 (Accessed February 2009) 12. Coghlan B, Ngoy P, Mulumba F, Nkamgang Bemo V, Brennan R, Mortality in the Democratic Republic of Congo: An ongoing crisis, April 2007, http://www.theirc.org/resources/2007/2006-7_congomortalitysurvey.pdf (Accessed February 2009) 14

13. Grellety E, Ronsse A. Retrospective Mortality Survey, North Kivu. Epicentre, Medecins Sans Frontieres. 14. See for example, http://www.congoresources.com/2009/01/new-mortality-studyslashes-estimates.html for an overview and http://www.obsac.com/e20090105172451/index.html for a critique of the IRC study. 15. Checchi F, Roberts L. Interpreting and Using Mortality Data in Humanitarian Emergencies: A Primer for Non-epidemiologists, Network Paper 52 (London: HPN, 2005). 16. Standardised Monitoring and Assessment of Relief and Transitions (SMART): Measuring mortality, nutritional status, and food security in crisis situations: SMART methodology. Protocol, Version 1, January 2005. (Unpublished document; available at: http://www.smartindicators.org/smart_protocol_01-27-05.pdf) 15

ORGANIZATION Time Place Place in Kivu MoH agreement Objectives Methods Population data Recall period CMR (per 10,000 per day) U5MR (per 10,000 per day) ACF international June 2008 Nord Kivu Kibua (Kibuwa) Yes o o o o o To estimate the prevalence of acute and chronic malnutrition among children 6-59 mo To determine the crude and the under five mortality rate To estimate the measles vaccine coverage To estimate the vitamin A supplementation coverage To assess the deparasitation among children with Mebendazol Two-stage household based cluster sampling Total 81,174 Children 0-59 mo: 13,799 90 days 0.38 [0.18 0.58] 1.10 [ 0.45 1.76] COOPI July 2008 Nord Kivu Binza Yes o o o o o To estimate the prevalence of acute and chronic malnutrition among children 6-59 mo To determine the crude and the under five mortality rate To estimate the measles vaccine coverage To estimate the vitamin A supplementation coverage To assess the feeding practices among young children Cross-sectional survey using cluster sampling Total = 102.284 Children 6-59 mo = 24,355 90 days 0.53 [0.30-0.76] 0.88 [0.25-1.51] IRC January 2006- April 2007 RDC Itebero Lubero Rwanguba Yes o To evaluate the current humanitarian situation in DR Congo by providing an update on mortality Three-stage household based cluster sampling Total = 69.9 million East stratum 32.6 million West stratum 37.3 million Inaccessible 44,000 16 months DRC: 0.73 [0.70-0.77] - Lubero: 0.27 [0.17-0.4] - Itebero: 0.6 [0.47-0.73] - Rwanguba: 0.73 [0.47-0.97] DRC: 0.67 [1.53-1.77] - Lubero: 0.6 [0.17-1.03] - Itebero: 1.27 [0.83-1.7] - Rwanguba: 1.5 [0.87-2.13] 16

Epicentre July- August 2008 Nord Kivu Nyanzale, Birambizo Yes o o o o To assess the mortality rate To assess the nutritional status of the children To evaluate the measles vaccination coverage To implement a mortality surveillance system Cross-sectional study using systematic sampling 1701 households 60 days 0.48 [0.22-1.05] 1.08 [0.37-3.14] 17

Will the results help you care for the population you are serving? ORGANIZATION Time Place Can the results be applied to my setting? Seasonality is taken into consideration? Food security, health, water and sanitation context are taken in consideration? Corroborate findings from local independant sources? GAM & SAM (NCHS) GAM & SAM (WHO) ACF international June 2008 N Kivu??? COOPI July 2008 N Kivu?? X? Epicentre July 2008 N Kivu?? X? 3.8%[2.8-4.8] 0.7%[0.3-1.2] 4.9%[3.2-6.7] 0.2% [0.0-0.5] 3.0%[1.8-5.1] 0.7% [0.2-2.0] 4.8% [3.2-6.3] 0.5% [0.1-1.0] 5.1% [3.3-7.0] 1.0% [0.3-1.7] 2.6% [1.4-4.5] 0.9% [0.4-2.4] 18

References Are the results of the study valid? What are the results? Will the results help you care for the population you are serving? ORGANIZATION Time Place Studied sample representative of the underlying population Random sampling large proportion of the chosen sample interviewed Specific strategies to ensure data accuracy HH revisited to confirm findings How large is the mortality rate? How precise is the estimate of the mortality rates? What is the absolute death toll over the period of analysis? Can the results be applied to my setting? What are the specific causes of death? corroborate findings from local independant sources? ACF international June 2008 N Kivu?? X X X COOPI July 2008 N Kivu? X??? X X X IRC January 2006- April 2007 DRC X Epicentre July- August 2008 N Kivu? X X 19

Sampling validity and precision for Nutritional Assessments Estimated Prevalence (%) Design Effect selected Precision (%) No Clusters Selection of households/children Planned sample size (U5) Sample size By age or by height or both Definition of a HH ACF international??? 30x32 EPI 960 959 Age Yes COOPI 11 2 3 30x30 NA 900 921 Age No Epicentre 17.5 2 3.5 NA sampling interval of 3 HH 453 461 Height Yes Measurements - Definition - Calculation Measurements Definition Calculations ORGANIZATION Correct measurements of Height/Length Weight / Oedema Inclusion of oedema as severe acute malnutrition Prevalence expressed as Z- scores CI given* % of oedema given NCHS reference and WHO standards Exclusion of z-scores given Absents or non respondants stated Reference of formulae used ACF international No Yes Yes Yes Yes Yes No Yes No COOPI Yes Yes Yes Yes Yes Yes Yes Yes Yes Epicentre Yes Yes Yes Yes Yes Yes Yes Yes Yes ORGANIZATION MUAC Correct measurements Standard definition CI given* ACF international Yes Yes No COOPI Yes Yes Yes Epicentre U U U 20

Correct measurements of Height/Length Weight / Oedema: Age = events calendar Height/Length = U5 85 cm measured upright, U5 < 85 cm length with precision of 1 mm / weight = precision of 100g) 21

22

Review of mortality and nutritional surveys in North Kivu, DRC Francisco Luquero Heloise Pham Emmanuel Grellety Ben Coghlan Rebecca Freeman Grais February 18, 2009

Preface: Independent Review? Members of ERG and HNTS formerly employed by MSF, Epicentre, WHO, etc. Epicentre received funding for this review sits on steering committee of HNTS performed one of the surveys in the review has received funding from various ERG members institutions collaborates with members of ERG One of the authors of this review served as a reviewer of two of these surveys for publication Two of the authors conducted surveys reviewed here IRC Epicentre MSF HNTS WHO

Source: UNOCHA, population movements through July 2008

Source: UNOCHA 830,000 IDPs (Aug. 2008)

Background Media and agency reports on conflict vary Magnitude of mortality and nutritional prevalence has implications for funding, how the events are perceived and humanitarian response Strengths/weaknesses in retrospective mortality surveys or nutritional surveys using cluster based sampling well known Sometimes we miss the forest for the trees

Objective Review available mortality and nutritional surveys performed in North Kivu, DRC 2007-2008 Methodology Opportunities or missed opportunities Ideas for HNTS in North Kivu

Methods: Literature Review 7 abstracts in Pubmed 31 surveys in NGO databases 7 excluded 2 surveys not found * 29 full-text articles/reports 2 Humanitarian action plans (OCHA) excluded 1 report covering DRC excluded 22 multisectorial evaluations (Unicef and Norwegian Refugee Council) excluded 4 surveys included in the analysis 1 mortality survey 3 mortality and nutrition surveys