The objective of the RSPA is to provide reliable information on the following:

Chapter 1 Survey Methodology 1.1 Overview The Rwanda Service Provision Assessment (RSPA) is the first national survey of health facilities for Rwanda. It was undertaken to provide a picture of how the health facilities function and of the quality of the reproductive and child health services available. Specific service areas assessed were child health, family planning, maternal health, and services for sexually transmitted infections (STIs), including HIV/AIDS. The goal of the RSPA is to describe facility-based health services and to recommend improvements to service delivery. The survey provides provincial- and national-level representative information on both public health facilities and government-assisted health facilities (GAHFs). Findings supplement the household-based health information collected in the 2000 Demographic and Health Survey in Rwanda Enquête Démographique et de Santé du Rwanda, 2000 (EDSR-II). The EDSR-II provides information on the health status of the population of Rwanda and the utilization of health services (ONAPO and ORC Macro, 2001). 1.2 Institutional Framework and Objectives of the Study The RSPA was conducted by the National Population Office (ONAPO) at the request of the Ministry of Health (MoH). Technical assistance was provided by ORC Macro through the MEASURE DHS+ project. The U.S. Agency for International Development (USAID) financed the survey. The objective of the RSPA is to provide reliable information on the following: 1. The availability of specific maternal, child, and reproductive health services; 2. The availability of infrastructure, equipment and supplies, staff, and health system components that contribute to quality of services; 3. The existence of management practices supportive of quality services; 4. The extent to which service providers adhere to quality standards when providing antenatal care (ANC) or consultation services for sick children; and 5. The health service experience from the client perspective. An additional objective is to strengthen the capacity of the MoH, and ONAPO in particular, to conduct similar studies and to analyze and utilize health system data and health services data for program development. 1.3 Data Collection Instruments Data were collected using structured printed instruments. These instruments were based on generic questionnaires developed in the MEASURE DHS+ project and were adapted after consulting with technical specialists from the MoH, nongovernmental organizations (NGOs), and other organizations knowledgeable about the health services and service program priorities covered by the RSPA. Operational definitions were developed for the health system components that were measured. These were revised for the RSPA after discussions with MoH officials in Rwanda and after the pretest. A training manual was developed and distributed to all data collectors to support standardized data collection. SURVEY METHODOLOGY 1

Data were gathered through interviews with key informants at facilities, observation, and interviews with health care providers and clients. Specific data collection instruments were as follows: Health facility inventory. This form collected information on the type of facility and the operating authority. It also collected information regarding furnishings, equipment, personnel, and other items for each service assessed by the RSPA that was provided by the facility. One questionnaire was completed for each facility. Health service provider interview. Providers of relevant services were interviewed regarding their technical qualification, supervision received, continuing education received, and experience providing the services that were assessed. Observation checklists. Checklists specific to quality curative child care and ANC were used to collect information on procedures conducted and information shared between the provider and the client. Exit interviews. Exit interviews were conducted with clients whose ANC consultation had been observed and with the caretaker of observed sick children. The interview covered their perception of what had occurred during the consultation and their opinion on issues related to client satisfaction. The inventory questionnaire was administered in French, with terminology that was identified as difficult during training translated into Kinyarwanda so that all data collectors would use similar terms. Observation and exit interviews were in French but were also translated into Kinyarwandan for use when appropriate. 1.4 Sample A representative sample of facilities, a sample of health service providers at each facility, and a sample of ANC and child health clients were selected. 1.4.1 Sample of Facilities The sample was selected to provide national- and provincial-level representation of health facilities offering maternal, child, and reproductive health services. These included hospitals, health centers, and dispensaries managed by the government (public) or by NGOs operating under agreement with the government (GAHFs). Private pharmacies, doctor s offices, and private clinics were not included in the sample. All hospitals were surveyed. Using a list of facilities supplied by the MoH, all government and government-assisted health centers and dispensaries were listed by facility type, province, and operating authority and then systematically selected. The assigned numbers of facilities to be selected for each province were determined to ensure adequate provincial representation of facilities. The sampling universe thus established contained 361 health facilities. Table 1.1 gives the distribution of these health facilities by type, by operating authority (public or GAHF), and by province. The final sample included 57 percent of the government-operated health centers and dispensaries (excluding special facilities for prisons or schools), 58 percent of the government-assisted health centers and dispensaries, and 100 percent of the hospitals (excluding psychiatric facilities) (Table 1.1). To ensure the sample included an appropriate number of facilities to permit analysis according to the type of facility and province, the facilities in some provinces were over-sampled. Because the sample distribution for the selected health facilities was not directly proportional to the distribution of the facilities in the universe, there was a potential for the findings to be biased. Therefore, data were weighted during analysis to account for the differentials caused by over-sampling. 2 SURVEY METHODOLOGY

Table 1.1 Rwanda SPA facility sample, actual and weighted numbers facilities in the sample, percentage of eligible facilities in the final sample, and weighted sample numbers, by type of facility, operating authority, and province, Rwanda SPA 2001 Province Public GAHF Health centers and dispensaries hospitals 1 Public GAHF 2 Number in sample Percent eligible Weighted number Number in sample Percent eligible Weighted number Butare 3 1 13 68 11 13 68 11 Byumba 2 0 14 61 13 3 100 2 Cyangugu 2 2 8 80 7 6 46 4 Gikongoro 1 1 6 46 5 6 60 5 Gisenyi 3 1 10 53 15 2 40 2 Gitarama 0 3 14 67 15 8 67 9 Kibungo 2 0 19 70 13 3 100 4 Kibuye 1 3 2 50 3 10 53 10 Kigali City 2 0 8 38 9 4 50 6 Kigali Ngali 1 1 10 56 11 3 50 3 Ruhengeri 1 1 10 48 12 5 63 5 Umutara 1 2 9 50 10 3 50 3 Total 19 15 123 57 124 66 58 64 1 All hospitals were surveyed. 2 Government-assisted health facilities All selected facilities were visited. Three selected facilities were no longer functioning and were replaced by three randomly chosen facilities with the same characteristics as those initially chosen. In addition, two health units were found to be of a different operating authority than indicated in the sample. These were surveyed, and the facilities were reclassified from government-assisted health facilities to public facilities. Descriptive information on facilities included in the RSPA is presented in Appendix Tables A-1.1-A-1.4. The data include the size of catchment populations, utilization statistics for outpatient adults (Appendix Table A-1.1), monthly average number of overnight patients and number of overnight beds (Appendix Table A-1.2), numbers and qualifications of staff assigned to facilities (Appendix Table A-1.3), and the years of basic and technical training reported by providers interviewed in the RSPA (Appendix Table A- 1.4). 1.4.2 Sample of Health Service Providers The sample of health service providers was selected from providers who were present in the facility on the day of the survey and who provided services that were assessed by the RSPA. In facilities with fewer than 10 health providers, all of the providers present on the day of the visit to the unit were interviewed. In facilities where there were more than 10 providers, all providers whose work was observed were interviewed, and a random selection of the providers not selected for observation was interviewed to compile a minimum of 10 provider interviews. The selection was carried out to ensure that, if available, at least one provider from each service was interviewed even if no observations were conducted for that service. The results of the RSPA are potentially biased because the staff who were present the day of the survey may not be representative of the staff who normally provide the services of interest in the facility. Table 1.2 furnishes information on the eligible and interviewed providers. Provider data were weighted for analysis to ensure that analysis provided data representative of the eligible providers. There were no refusals for the interviews. SURVEY METHODOLOGY 3

Table 1.2 Sample of interviewed health care providers and weighted values for providers providers assigned to facilities, number present the day of the survey (eligible), percentage of total staff eligible for interview, number of interviewed (sample) staff, percentage of eligible providers interviewed, and weighted value for provider interviews, by type of provider, type of facility, and operating authority, Rwanda SPA 2001 Type of facility/ operating authority staff assigned to facility 1 staff present on the day of the survey (eligible for interview) Percentage of all staff who were present the day of the survey staff interviewed Percentage of eligible staff who were interviewed Weighted number of providers PHYSICIANS Public 98 43 44 25 58 40 GAHF 39 26 67 15 58 19 Health center Public 0 na na 0 na 0 GAHF 1 0 0 0 na 2 Dispensary Public 1 0 0 0 na 0 GAHF 0 na na 0 na 0 Total 139 69 50 40 58 61 NURSE A1 AND A2 Public 538 181 34 112 62 173 GAHF 218 144 66 83 58 75 Health center Public 309 230 74 216 94 108 GAHF 261 184 70 169 92 87 Dispensary Public 27 21 78 20 95 11 GAHF 23 23 100 22 96 12 Total 1,376 783 57 622 79 466 AUXILIARY AND NURSE A3 Public 247 98 40 29 30 77 GAHF 173 94 54 23 24 42 Health center Public 355 288 81 211 73 215 GAHF 226 187 83 93 50 100 Dispensary Public 21 14 67 13 93 16 GAHF 15 13 87 7 54 15 Total 1,037 694 67 376 54 465 na = Not applicable 1 From administrator list 4 SURVEY METHODOLOGY

1.4.3 Sample for Observations Outpatient consultation services for sick children under age 59 months and ANC client consultations were observed. The sample of observations was opportunistic, meaning that clients were selected for observation as they arrived because there was no way to know how many eligible clients would attend the facility the day of the survey. When there were several eligible clients waiting for service, an effort was made to ensure that children with sickness (rather than injury or skin or eye infections) were selected for observation and that there was a mixture of new and follow-up ANC clients observed. The ratio observers aimed for was 2 new for every 1 follow-up case for ANC. Cases were not always available to allow this objective to be met. Where numerous clients were eligible for observation, the rule was to observe a maximum of 5 clients for each provider of the service, with a maximum number of observations in any given facility for each service to be 15. In practice, more clients were observed in some facilities while fewer clients than were eligible were observed in others. The latter occurred because logistic arrangements sometimes resulted in missed observations. Table 1.3 provides information on the eligible and observed clients. An attempt was made to interview the caretaker for all observed sick children before leaving the facility and to interview all ANC clients before leaving the facility. There were no refusals for observation of the sick children; however, there were nine refusals for exit interviews. There were no refusals for observation of ANC clients; however, there were four refusals for exit interviews. Refusals for exit interviews by caretakers of sick children were because Table 1.3 Sample of observed and interviewed clients sick children/antenatal care (ANC) clients attending facility on the day of the survey (eligible), number of clients observed, and percentage of eligible clients who were observed, by type of client, type of facility, and operating authority, Rwanda SPA 2001 Type of facility/ operating authority clients present on the day of the survey (eligible for observation) SICK CHILDREN clients observed Percentage of eligible clients who were observed Public 66 61 92 GAHF 39 34 87 Health center and dispensary Public 695 680 98 GAHF 519 464 89 Total 1,319 1,239 94 ANC CLIENTS Public 49 42 86 GAHF 128 97 76 Health center and dispensary Public 2,143 1,812 85 GAHF 1,248 991 79 Total 3,568 2,942 82 SURVEY METHODOLOGY 5

the child was seriously ill and being admitted or referred elsewhere. Reasons for refusals for exit interviews by ANC clients were not provided in the data; however, anecdotal reports were that refusals were because the woman felt she needed to leave or was lost to follow up when she went elsewhere in the facility (laboratory or pharmacy) for additional services. 1.5 Study Implementation 1.5.1 Training and Supervision of Data Collectors Researchers from ONAPO were trained on the RSPA methodology and data collection instruments July 17-27, 2001. Data collectors were primarily recruited from applicants who were trained in nursing sciences. The data collectors were trained over a three-week period, August 6-24, 2001. Nine teams of three people each collected the data. Each team was made up of a team leader and two investigators. The team leader was responsible for the organizing the work of the team and ensuring quality control of the data collected. The team leader completed the inventory questionnaire and the provider interviews. One investigator conducted the observations and the other conducted the exit interviews. Each group of three teams was under the direction of a supervisor, who was also a team leader. The Technical Coordinating Team, made up of members from ONAPO and the resident advisor from ORC Macro, made weekly visits to each group to ensure the work was being conducted according to correct survey methodology and to provide quality control of the data collected. RSPA data were collected from September 10 to November 17, 2001. 1.5.2 Methods for Data Collection Each team received a list of facilities to be visited. Data collection took one day in most facilities, with two days being allotted to hospitals, if required. In addition, if one of the observed services (consultation for sick children or ANC) was not being offered the day of the survey, the teams returned on a day when the service was offered. If the service was offered, the clients for that day were observed. If the service was offered but no clients came (as occurred occasionally for consultations for sick children), teams did not revisit the facility. The team leader was instructed to ensure that the informant for each component of the facility survey was the most knowledgeable person for the particular health service or system component being addressed. Where relevant, the data collector indicated if a specific item being assessed was observed, reported available but not observed, not available, or it was uncertain if the item was available. Equipment, supplies, and resources for specific services were required to be in the relevant service delivery area or in an immediately adjacent room to be accepted as available. Informed consent was obtained from observed and interviewed providers and from clients for observations and exit interviews. 1.5.3 Data Analysis Items were accepted as available if they were observed either in the service delivery areas or in an area immediately adjacent. If the service was not being offered on the day of the survey, an attempt was still made to observe each item. In some instances, however, the item may have been locked away or the knowledgeable staff was not present. In these cases, only if the service was not being provided on the day of the survey, availability was expanded to accept reported available if a facility staff member could verify that the item was present and in working order. This was applicable for curative child care, family planning, ANC, and STI services. Only observed items were accepted when assessing resources for delivery services because delivery services must be available whenever the facility is open. If an item was 6 SURVEY METHODOLOGY

locked away and could not be seen, it was evident it was not available for service. In none of the analyses did reported responses exceed 1 percent. In looking at the observation data, it should be noted that many facilities provide routine services for clients separately from the actual consultation (e.g., taking blood pressures and temperatures). Often, there is a period of time between these events and the point at which the primary provider assesses the client. Although RSPA observers were instructed to follow a client through the entire system, this was not always possible logistically. Thus, when services were being provided outside the observed consultation on the day of the survey, the observed client was assumed to have received these services. Where this type of (functional) system applies, multiple providers participate in the services received by each client. The provider who ultimately diagnosed and prescribed was defined as the primary provider. Aggregating the data into subsets makes it possible to analyze many pieces of information and to see how they relate to the overall capacity to provide quality services. It also enables monitoring changes in capacity to provide services and changes in adherence to standards, since there may be improvement in some items but not in others. There are not yet generally accepted aggregates of the health information collected in the RSPA. Initial decisions regarding what should comprise a particular aggregate can be difficult, with inclusion or exclusion of items equally valid depending on the objective of the user. The aggregate variables presented in this report are an initial phase in the process of developing health information aggregates. They will be refined as users provide feedback on the aggregate variables found useful (or not useful) to policymakers and program implementers. 1.6 Process for Data Management and Report Writing Data management and analysis were carried out according to the following steps: Management of questionnaires. Completed and verified questionnaires were collected by supervisors and sent to ONAPO, where they were edited and classified to ensure all questionnaires were accounted for. Data entry. Data entry was conducted by five Rwandan data entry personnel supervised by an ORC Macro technical advisor and ONAPO staff. CSPro software developed by ORC Macro and the U.S. Census Bureau was used for data entry. Double entry of all the questionnaires was carried out to catch errors. This operation took place from November 12, 2001 to January 22, 2002. Quality control and data editing. Quality control and data editing took place at the same time as data entry. Where there were inconsistencies, the questionnaires were reviewed and questions were recoded when the correct response could be determined. Data analysis. The design of the tabulation plan and the preparation of the programs for the production of statistical tables were carried out from February to June 2002. Data analysis and clarification of questionable results were carried out from February to September 2002. Development of final report. The final report was written with input from ORC Macro technical staff and ONAPO and MoH technical personnel. After the draft report was finalized, MoH technical staff and other partners were further consulted to share findings and make corrections and changes before publication of the report. This took place during February 2003. SURVEY METHODOLOGY 7