EVALUATION PERFORMANCE EVALUATION OF THE CLINICAL HIV/AIDS SYSTEM STRENGTHENING PROJECT IN NIASSA PROVINCE

Transcription

1 EVALUATION PERFORMANCE EVALUATION OF THE CLINICAL HIV/AIDS SYSTEM STRENGTHENING PROJECT IN NIASSA PROVINCE December 30, 2015 This report was produced by AGEMA Consultoria Lda, for the United States Agency for International Development/Mozambique. Authors: Peter S. Wandiembe, PhD, Rosemary Barber-Madden, PhD, Esther Kazilimani-Pale, MPH, and Verona Parkinson, PhD

2

3 PERFORMANCE EVALUATION OF CLINICAL HIV/AIDS SYSTEM STRENGTHENING PROJECT IN NIASSA PROVINCE December 30, 2015 Agema Consultoria Lda. submits this report to USAID/Mozambique as a deliverable under Contract No. 656-O for the Performance Evaluation of the Clinical HIV/AIDS System Strengthening Project in Niassa, FHI 360 with sub-award to Abt Associates Inc. DISCLAIMER The authors views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

4

5 CONTENTS ACRONYMS AND ABBREVIATIONS... iii EXECUTIVE SUMMARY... v I. EVALUATION PURPOSE AND EVALUATION QUESTIONS... 1 I.1 EVALUATION PURPOSE... 1 I.2 EVALUATION QUESTIONS... 1 II. PROJECT BACKGROUND... 3 II.1 GENERAL CONTEXT... 3 II.2 OVERVIEW OF CHASS NIASSA IMPLEMENTATION... 4 III. EVALUATION METHODS AND LIMITATIONS... 5 III.1 EVALUATION METHODS... 5 III.2 SITE SELECTION AND DATA COLLECTION... 5 III.3 DATA ANALYSIS... 6 III.4 STUDY LIMITATIONS... 6 IV. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS... 7 IV.1 FINDINGS... 7 IV.2 OVERALL CONCLUSIONS IV.3 FUTURE DIRECTIONS IV.4 SUMMARY OF RECOMMENDATIONS REFERENCES ANNEX I. EVALUATION SCOPE OF WORK ANNEX II. DATA COLLECTION INSTRUMENTS: KII, FGD ANNEX III. LIST OF INTERVIEWEES ANNEX IV. SOURCES OF INFORMATION ANNEX V. SUMMARY OF QUANTITATIVE DATA ANNEX VI. EVALUATION TEAM MEMBERS Performance Evaluation of the CHASS Niassa Project i

6 LIST OF TABLES, FIGURES AND CHARTS Table 1: Key indicators for HIV care, ART and PMTCT programs... vii Table 2: Categories of key informant interview respondents... 6 Table 3: Recommendations for future programs Figure 1: Patients already on ART at the beginning of the year and those newly enrolled... 7 Figure 2: Indicators in the PMTCT Cascade... 8 Figure 3: Retention and community linkage indicators Chart 1: Weaknesses in Activities in Clinical Services and Recommendations... 9 Chart 2: Weaknesses in Activities for HSS and Recommendations Chart 3: Weaknesses in Activities for HSS and Recommendations (2) Chart 4: Challenges to Community linkages Chart 5: Constraints to activities aimed at improving retention and recommendations Chart 6: Challenges associated with improvement of health facility management Chart 7: Challenges to HF data management capacities Chart 8: Challenges and weaknesses in scaling-up GBV services ii Performance Evaluation of the CHASS Niassa Project

7 ACRONYMS AND ABBREVIATIONS Acronym or Abbreviation Definition (English) Definition (Portuguese) AIDS Acquired Immune Deficiency Syndrome Síndrome de Imunodeficiência Adquirida ANC Antenatal care Cuidado pré natal APEs Agentes Polivantes Elementer ART Antiretroviral treatment Tratamento Anti-retroviral ARV Antiretroviral Anti-retroviral CBO Community-based organization Organização Baseada na Comunidade CCM Community case managers CD4 Cluster of Differentiation 4 Cluster de Diferenciação 4 CHASS Clinical HIV/AIDS System Strengthening Project DPS Provincial Health Directorate Direcção Provincial de Saúde EPTS Electronic patient tracking system Sistema Electrónico de Seguimento de Pacientes FILA ARV drug pick-up form Folha de Informação de Levantamentos de ARV FOGELA Strengthening Lab Management for Fortalecimento da Gestão de Laboratório Accreditation para Acreditação FY Fiscal year GAACs Community Support and Adherence Groups Grupo de Apoio e Adesão da Comunidade GBV Gender-based violence Violência Baseada no Género HCT HIV counseling and testing Aconselhamento e Testagem em Saúde HF Health facility Unidade de saúde HIV Human Immunodeficiency Virus Vírus da Imunodeficiência Humana HR Human resources Recursos Humanos HSS Health system strengthening M2M Mother-to-Mother Groups Mãe para Mãe MCH Maternal and child health Saúde Materno-Infantil MOH Ministry of Health Ministério da Saúde PCC Community Care Project Projecto de Cuidados Comunitários PCR Polymerase Chain Reaction Reacção em Cadeia da Polimerase PEPFAR U.S. President s Emergency Plan for AIDS Relief PIMA CD4-analyzing machine Máquina para Analisar as Amostras de CD4 PLHIV People living with HIV Pessoas que Vivem com o HIV PMTCT Prevention of mother-to-child transmission Prevenção da Transmissão Vertical QA Quality assurance Controlo da qualidade QI Quality improvement Avaliação da Qualidade de Dados RH Reproductive health Performance Evaluation of the CHASS Niassa Project iii

8 Acronym or Abbreviation Definition (English) Definition (Portuguese) SA Sub-agreement SDSMAS District Health Directorate Serviços Distritais de Saúde, Mulher e Acção Social SIFO Continuing Training Information System Sistema de Informação de Formação Contínua TB Tuberculosis Tuberculose USAID United States Agency for International Development iv Performance Evaluation of the CHASS Niassa Project

9 EXECUTIVE SUMMARY EVALUATION PURPOSE The Clinical HIV/AIDS System Strengthening Project in Niassa province (CHASS Niassa) was a five-year project funded by the United States Agency for International Development (USAID). It was implemented by FHI 360 with an overall goal of strengthening the Niassa provincial health system by maximizing access, quality and sustainability in the delivery of comprehensive HIV/AIDS and related primary health services. The purpose of this performance evaluation was to determine how the project activities were performing relative to their objectives. The evaluation was to provide an objective view of progress toward the expected results. The main objectives were to (a) assess CHASS Niassa achievements, emphasizing project-level results; (b) identify implementation successes, as well as any internal and external constraints that hindered the implementation of planned activities; and (c) propose recommendations for future directions of the CHASS project. KEY EVALUATION QUESTIONS The evaluation was guided by 10 questions, 1 with the following four key questions: 1) What are the strengths and weaknesses of the activities as seen by the stakeholders, and how can these weaknesses be improved? 2) What constraints have the activities faced in improving retention of patients in pre-art and ART programs? 3) To what extent have knowledge (community and health worker) and utilization of gender-based violence (GBV) services increased over the life of the project? 4) What are the benefits and challenges of the activities model of working with government (at provincial and district levels) through sub-agreements? PROJECT BACKGROUND The CHASS Program was designed to address the HIV situation in Mozambique and focused on three components: (a) Improving service quality in HIV prevention, care and treatment services; (b) Enhancing program linkages and integration to provide a continuum of accessible services; and (c) Creating stronger and more sustainable systems and institutions. In Niassa, CHASS was implemented by FHI360 from August 2010 to July EVALUATION DESIGN AND METHODS The performance evaluation of CHASS Niassa was conducted September 22-30, 2015, with the aim of achieving the aforementioned objectives. The evaluation was based on a non-experimental design using a mixed-methods approach. In addition to reviewing the existing quantitative project data and documents, the evaluation team conducted field key informant interviews with eight project staff, staff from five health facilities, 24 staff from health and social welfare departments at the province and the three districts, three staff from the Ministry of Health (MOH), five staff from USAID/Mozambique, and five staff from community-based organizations (CBOs) supporting CHASS Niassa. The evaluation used multichannel data collection tools, using mobile technology when possible and paper-based methods when handheld computers were not appropriate. All of the interviews were recorded on tablets. Some of the 1The 10 questions are listed in Section 1 of this report: Evaluation Purpose and Evaluation Questions. Performance Evaluation of the CHASS Niassa Project v

10 members handwrote their field notes directly onto the tablets using a stylus beta enhanced Open Data Kit (ODK) program. The qualitative data were analyzed using a qualitative data analysis matrix. Information from this analysis was triangulated with the available quantitative data from CHASS project documents. EVALUATION LIMITATIONS The limitations of the evaluation are as follows: Firstly, in many cases, the evaluation team was unable to ascertain consistent factors responsible for the project s success and weaknesses as the project evolved over time. Therefore, some of the findings reported are from a historical perspective. Secondly, the findings reported here are responses provided by the interviewees to the evaluation team. The likelihood of interviewee bias and recall bias cannot be ruled out. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS The findings for each evaluation question are summarized below: 1) Project achievements and challenges: Activities in HSS: The project focused directly on delivering HIV/AIDS and related clinical services at the health facility (HF) level. The project made efforts to train the provincial directorate of health (DPS) and district directorate of health (SDSMAS) staff in program management, supply and logistics, and financial management. In addition, laboratory services were improved through renovations, provision of logistics for transportation of medications, supplies, and CD4 and PCR samples and results. The project contributed significantly to the human resource (HR) system and capacities through training and providing logistics for implementation of HR systems such as the Continuing Training Information System (SIFO) at DPS and SDSMAS, providing on-the-job training, training nurses to prescribe antiretrovirals (ARVs), and funding pre-service training of about 60 health workers. Six health workers received Master s degree-level training in management. In addition, procurement staff were recruited for all SDSMAS. It was clear that the project lacked a well-planned and appropriate exit strategy. Most of the project s operational costs were managed by the project team, and hence limited experiential learning by DPS and SDSMAS. The quality assurance/improvement (QA/QI) approach that helped DPS to identify service-delivery weaknesses and develop, test and implement improvement strategies was not regularly done, and some key informants from the DPS were not well versed with the concepts of the approach. Overall, the project staff focused more on helping the DPS and SDSMAS staff to implement activities and did little to build their capacities. Activities in clinical services delivery: The HIV/AIDS clinical services were improved and decentralized or expanded to new HFs through staff trainings and mentoring, minor facility renovations, and demand creation and active referrals from the communities through activists, in coordination with CBOs. The project expanded the number of antiretroviral treatment (ART) sites and scaled up Option B+ in prevention of mother-to-child transmission (PMTCT). A summary of the indicators, showing trends in clinical services, coverage and quality, is presented in Table 1 below. Most of the project s set targets were achieved by fiscal year (FY) The improvements made by the project are threatened by scarce human resources and high rates of staff transfers. Stakeholders at provincial, district and HF levels reported an inadequate number of staff to manage improved activities, meet increased demands and maintain highquality services. The logistics and supply chain was not working well, and many health facilities reported periodic stock-outs of drugs and HIV test kits. Children reached and quality of services were below the project targets. vi Performance Evaluation of the CHASS Niassa Project

11 Table 1: Key indicators for HIV care, ART and PMTCT programs Indicator FY 2011 FY 2012 FY 2013 FY 2014 % of 2014 Target achieved Number currently enrolled on ART 3,316 7,383 10,768 13,334 (2,699) (7,622) (7,189) (11,012) 121% Number newly enrolled on ART 861 2,812 3,967 5,261 (1,615) (5,103) (2,518) (2,919) 180% Percent of currently enrolled on ART patients that are men 27% 36% 30% 26% Percent of currently enrolled on ART patients that are children 7% 6% 11% 9% Retention rate in 12-month ART cohort (adult) 71% 75% 63% 70% 82% Percent of pregnant women with known HIV status 92% 83% 90% (newly tested and known positive at ANC entry) * (90%) (90%) (90%) 100% Percent of HIV-positive pregnant women in ANC who have initiated Cotrimoxizole * 52% 33% 59% Percent of HIV-positive pregnant women who received ARVs to reduce risk of MTCT * 72% 56% 82% Percent of infants born to HIV-positive women who received an HIV test within 12 months of birth * 26% 24% 53% Percent of HIV test results for infants born to HIVpositive women who received an HIV test within 12 months of birth that are positive * 15% 7% 4% Indicators without targets have this column left blank. *No data were available for most of the PMTCT-related indicators in FY 2011 and are left blank in the table. 2) Most-improved HSS component: The project supported the strengthening of systems and capacities of the various components of the health system, with the HR component identified by key informants from the DPS and SDSMAS as the most significantly improved. The DPS and SDSMAS staff were trained on HR management and use of electronic HR information systems, and health workers were trained through mentoring and tutoring, on-the-job training and pre-service training. 3) Ready-to-transition project activity or component: The project worked jointly with SDSMAS and DPS in executing most of its activities. Whereas most of the routine project activities were led by the project team, they were part of MOH activities that were supposed to be implemented by provincial and district technical teams. The key project-specific activity that should be transitioned to the DPS and SDSMAS team leadership is the QI/QA methodology that was effective in improving clinical practices and processes. 4) Strengthening community linkages: The project had exemplary community linkage strategies for clinical and GBV services, including developing the graduation path for HFs. Through three CBOs, the project provided logistics to activists or community case managers (CCMs) for casefinding of pre-art care and ART defaulters, community sensitization, mobilization and communitybased HIV testing. The facility-based case managers hired by CBOs were instrumental in generating lists of defaulters, receiving referrals from the communities and communicating with health workers. Through provision of training and logistics to the adherence support groups (GAACs), people living with HIV (PLHIV) and mother-to-mother (M2M) activities, the project improved knowledge of services and managed to maintain retention in ART programs by at least 70 percent. 5) Constraints to improving retention rates: Retention rates persisted at below 85 percent of target across the province. One of the key constraints was the insufficient number of staff providing Performance Evaluation of the CHASS Niassa Project vii

12 adequate counseling at some of the HFs and communities. Retention is a cycle that begins with counseling, and if counselors do not take time to counsel patients and answer their questions so that they understand that HIV/AIDS needs regular clinical reviews or adherence to ARVs, retention becomes a problem. Other constraints that limited the achievements of retention-improvement strategies (see point 6 below) were the long distances traveled by CCMs, stigma in the community, wrong addresses given by patients, community movements during the farming season, and time constraints, especially for working men. 6) Most effective method for improving retention: The key strategies aimed at improving retention included (1) active case-finding, where CCMs use a defaulters list generated by the HF case managers to search for defaulters in communities; (2) GAAC members picking up drugs for each other and also providing psychosocial support; (3) community mobilization by CCMs, encouraging retention and adherence to ART; (4) M2M following HIV-positive pregnant women; and (5) psychosocial support from PLHIV groups. Although it was reported to be costly and did not cover all communities, active case-finding was the most effective strategy. The activities of the M2M groups were moderately effective, followed by the activities of the PLHIV and GAACs. Community mobilization by activists and lay counselors also played a supportive role. 7) Improvement in HF management: Use of one-stop PMTCT and TB models, training of HF staff in process organization, and reinstallation of the fluxogram (flowchart) card system led to improved patient flow and reduced waiting times. The HFs restricted physical spaces limited the improvement of patient flow. 8) Improvement in HF data management capacity: The error rates of filling in the HIV/AIDS registry decreased substantially, and correct filing and summarizing of the registry records for data entry improved moderately. However, these improvements are still below the desired levels, especially the pediatric care registries. 9) Knowledge of GBV (among health workers and community) and service utilization: CHASS provided training for focal points at provincial and district levels and trained activists who could refer cases to HFs. Community sensitization was done by activists and supported by other organizations, such as MULEIDE. The CCM did an active search for the GBV victims in the community. Overall, there was an increase in cases reported at the HF from the community, police and the judiciary. However, the GBV services are not well integrated with other clinical services, except maternal and child health (MCH). In addition, sociocultural reasons were cited to have limited victims reporting to HFs or police. 10) Benefits and challenges of working with government through sub-agreements (SAs): Project activities were implemented through SAs, which enabled the development of systems and capacities of the SDSMAS and DPS through forced learning, and to some extent fostered mutual understanding of the activities and wider ownership. However, the SAs were based on the project activities, with no allowance for engaging in any other activities outside that scope. Further, there were always delays in reimbursement throughout the five years of the project. The capacity of the DPS and SDSMAS to manage direct funding needs further strengthening. CONCLUSION In general, the project achieved most of its targets, but more efforts are needed to improve these further and sustain them. In particular, more efforts and resources are required in health system strengthening (HSS). There was limited direct support to HSS at the DPS and SDSMAS, and limited operations research was done to understand reasons for low retention rates, poor patient flow especially in rural HFs, and low recruitment of children on treatment. viii Performance Evaluation of the CHASS Niassa Project

13 RECOMMENDATIONS Health System Strengthening 1) Involve the DPS and SDSMAS in the project design, and give them an active role at all stages of the project. The transition plan should be agreed on with the SDSMAS, DPS and MOH and should be explicitly incorporated in the project documents through the first year. Face-to-face interactions should be a preferred method of communication with these entities. The MOH should actively play its role as agreed with the project to ensure its deliverables as well as success and sustainability. 2) Introduce a component of operations research in project activities and also in local capacity building. 3) Develop tools and methodologies for regular assessment of the project s support for HSS. The graduation path assessment that uses a 23-item tool to assess systems and capacities of various components against standards should be used to measure progress in HSS. 2 The action plans following the assessment will benefit from the use of the QI approach. 4) Put a training mechanism in place. This could include a peer-to-peer model, in which trained health workers can do tutoring at the nearby HFs; work with training institutions to establish courses in HSS for all health workers; and training a group of national and district-level trainers. 5) Expand mhealth in project areas to more sites. Most aspects of data management and patient follow-up can be performed with mobile technology. In the future, the project should expand on its mhealth activities. This will include the use of tablets, PDAs or mobile phones by CCMs or activists to input client information; text messaging reminders for patient clinic visits, including mothers returning babies for HIV testing and adherence to ARVs; and working with a mobile phone company to provide online tutorials for health workers on the tablets. 6) The supply chain and logistics system is still weak. A private company should be hired to manage it until the DPS and SDSMAS have developed the necessary capacity. 7) Advocate for and support the MOH in rolling out the electronic patient tracking system (EPTS) to moderate- to large-volume HFs. In the meantime, encourage and train health workers to correctly fill in ARV drug pick-up forms (FILAs) and other forms on retention and file them appropriately, and initiate conversations with DPS about the roll-out plan for EPTS once the MOH approves it, including but not limited to data validation processes and tools, plan and terms of reference for data managers, use of data from EPTS for reporting, and the QI program. Clinical Services (enrollment and retention) 8) The GRM/MOH should design a better staff development and retention package or plan to promote staff retention in post for at least three years to allow for consolidation of experience. It should also increase staffing levels to cope with increased client load, especially for HIV/AIDS services. 9) The GRM/MOH, with support from partners, should invest in improving HFs physical structure and space. 10) Implement an augmented MOH strategy of a lay counselors workforce by recruiting and training lay counselors or retraining case managers and Agentes Polivantes Elementer (APEs) as lay counselors. 2The graduation path assessment tool was developed by Abt Associates Inc., one of the implementing partners of CHASS- Niassa. Performance Evaluation of the CHASS Niassa Project ix

14 Some counselors should be stationed at the HF and one in each community to provide services such as adequate counseling for ART initiation, community sensitization against HIV stigma, psychosocial support to HIV-positive pregnant women to disclose HIV status, psychosocial support to GAACs and individual PLHIV, and adherence counseling. 11) Reinvigorate linkages between HF and community-based services, and promote regular information exchange for cross-referrals and health information messaging to communities: Conduct operations research to map and improve the functioning of community linkages. Explore adding new activities to CBOs, including creating links with faith-based organizations, traditional healers, traditional birth attendants, APEs and others. Expand the mobile ART outreach into a mobile clinic with mhealth to provide HIV counseling and testing (HCT), HIV care, ARVs, antenatal care (ANC) and PMTCT, CD4 testing, TB screening services and health education. Maintain the family approach model and community drama to reach couples and children. 12) Expand the HIV prevention, care and support services for adolescents at HFs, with emphasis on adolescent girls. Develop a school-based integrated health program that includes adolescent health and HCT services. 13) Improve HIV testing and ART initiation among infants through training all HF health workers in: PCR sample-taking to provide HIV testing in all pediatric entry points, including the children of patients on ART; linkage of records in labor/delivery units and at-risk child consultations; and adequate counseling of pregnant women. In addition, distribute SMS printers to rural health facilities. Community Linkages 14) Expand the current activities of the project (see Recommendation 8). 15) Partner with employer-based health programs to provide training and technical support to institute worksite HCT, referral, and dispensing ARVs, particularly for male workers. GBV Services 16) Expand the current GBV community linkages and collaborations to other communities. 17) Develop the capacity of all health workers so that GBV screening is done at all HF points of care. Sub-agreements 18) Establish a budget item for non-hiv-related services: This can be used to fund emergencies or disease outbreaks that are not necessarily within the scope of CHASS. 19) Improve on the reimbursement process: Review internal financial controls to make them realistic for GRM collaborators x Performance Evaluation of the CHASS Niassa Project

15 I. EVALUATION PURPOSE AND EVALUATION QUESTIONS I.1 EVALUATION PURPOSE The five-year Clinical HIV/AIDS System Strengthening Project in Niassa province (CHASS Niassa) was funded by the United States Agency for International Development (USAID). It was implemented by FHI 360 from August 2010 to July The project s goal was to strengthen the Niassa provincial health system by maximizing access, quality and sustainability in the delivery of comprehensive HIV/AIDS and related primary health services. This performance evaluation of CHASS Niassa was commissioned by USAID, and its purpose was to determine how the project s activities were performing relative to their objectives. The evaluation was to provide an objective view of progress toward the expected results. The outcomes of the evaluation will inform the transition to the future activity that will support USAID system strengthening and clinical service delivery activities. The evaluation s main audience is USAID. In addition, results will be shared with FHI 360, the Ministry of Health (MOH), U.S. Government agencies, and other stakeholders. The results will also be made available on the Development Experience Clearinghouse. The objectives of the evaluation are to: Assess CHASS Niassa achievements, emphasizing objectives and activity- and project-level results Identify implementation successes, as well as any internal and external constraints that hindered the implementation of planned activities Propose recommendations for future directions of CHASS Niassa and for future activities in system strengthening and service delivery to support improved performance in addressing the HIV epidemic, in line with the GRM HIV/AIDS acceleration plan An additional goal was to use mobile technology, when possible and appropriate, to increase the efficiency, transparency and accuracy of performance data, and to take advantage of multiple data sources (pictures, videos, GPS data). I.2 EVALUATION QUESTIONS The three main evaluation question areas include a total of 10 evaluation questions. All areas were answered fully and completely, underscoring both positive and negative outcomes. Where sufficient data were available, gender analysis was done with quantitative data. In order to accomplish the aboveidentified evaluation objectives, the evaluation sought to answer the following 10 questions: Question Area 1: Project Achievements and Challenges 1) What are the strengths and weaknesses of the activities, as seen by the implementing partner staff, Provincial Directorate of Health (DPS), District Directorate of Health and Social Welfare (SDSMAS) and chief medical officer, HFs, USAID, and the USAID-funded Community Care Project (PCC), and how can weaknesses be improved according to the stakeholders listed here? 2) Where has the most progress been seen in strengthening systems (e.g., planning, financial management, supply and logistics, information systems)? 3) Which activities or project components will be most feasible to transition from the project to the GRM? Performance Evaluation of the CHASS Niassa Project 1

16 Question Area 2: Linkages 4) To what extent has the project been able to create and strengthen linkages between health facilities and communities to allow for increased service uptake, specifically in the areas of: a. Community-based counseling and testing to treatment (for both men and women); b. Retention of pre-art and ART patients (through the use of adherence groups, active case-finding and other community groups); and c. Knowledge, demand and access of services by men 5) What constraints have the activities faced in improving retention of patients in pre-art and ART? 6) What has been the most effective method found by the project to improve retention? Question Area 3: Health System Strengthening 7) To what extent has HF management (improved patient flow, etc.) improved over the life of the project? 8) To what extent is data management capacity built at the HFs with regard to HIV/AIDS registry data? 9) To what extent has knowledge (community and health worker) and utilization of GBV services increased over the life of the project? 10) What are the benefits and challenges of the activities model of working with government (at provincial and district levels) through sub-agreements? Specific questions that guided the performance evaluation are stated in the evaluation matrix included in Appendix II. 2 Performance Evaluation of the CHASS Niassa Project

17 II. PROJECT BACKGROUND II.1 GENERAL CONTEXT In July 2009, USAID/Mozambique issued a request for applications for a results-oriented five-year project to improve HIV clinical services in Manica, Niassa, Sofala and Tete provinces within a strengthened, comprehensive primary health care system. This project was designed to address the HIV situation in Mozambique and focused on three components: 1) Improving service quality in six important areas: HIV counseling and testing (HCT), laboratory services, prevention of mother-to-child transmission (PMTCT), adult care and treatment, pediatric care and treatment, and the prevention, diagnosis and treatment of HIV/tuberculosis (TB) co-infection 2) Enhancing program linkages and integration to provide a continuum of accessible services, including MCH and reproductive health (RH) services, within facilities and between facility and community-based services 3) Creating stronger and more sustainable Mozambican systems and institutions At the time of project design in 2010, the average HIV prevalence in Mozambique was estimated at 16 percent nationwide (ANC survey in pregnant women aged years). Nearly 1.6 million people were living with HIV, and nearly half of all HIV-infected (48.4 percent) were identified at the time as having active TB. Within the Central Region, Sofala and Manica provinces have the highest prevalence, 23 percent and 16 percent respectively. Tete was identified as having a very mature epidemic and existing infrastructure that was unable to accommodate the numbers of patients requiring care and treatment. Niassa was considered a particularly underserved province, with vastly inadequate infrastructure. To address these issues, two separate agreements were awarded for the Clinical HIV/AIDS System Strengthening (CHASS) Project (Sofala, Manica, Tete, Niassa) to two implementing partners: FHI 360 for CHASS Niassa, and Abt Associates Inc. for CHASS-SMT (Sofala, Manica and Tete). CHASS Niassa was funded at $36,538,233 for the period of August 1, 2010 to July 31, The implementation of CHASS Niassa started in late 2010 and was to support USAID/Mozambique s Country Assistance Strategy s (CDCS DO4) priority goal number three, Improved health of Mozambicans, and the following focal areas in USAID s Health Results Framework: (a) Improved access to and delivery of quality integrated services; (b) Increased adoption of healthy behaviors and informed use of services; and (c) Strengthened health systems. After these two activities were awarded, the MOH developed the HIV Acceleration Plan in 2011; signed the Political Declaration on HIV/AIDS: Intensifying the Efforts to Eliminate HIV/AIDS; and created the Mozambique HIV and AIDS Response Strategic Acceleration Plan to respond to the commitment to an end of AIDS. The latter three-year strategy reflects the united vision of all stakeholders to achieve a Generation Free of AIDS in Mozambique, focusing on three major goals, and one added objective on gender-based violence: Increase the percentage of eligible HIV-infected adults and children receiving ART to 80 percent by Reduce the rate of transmission of HIV from mother to child to less than 5 percent by Reduce the number of new infections by 50 percent by In addition, both CHASS activities (Niassa and SMT) included programming to address GBV within the HIV platform, with a total life-of-project funding of approximately $1.5 million. GBV fosters the spread of HIV by limiting a person s ability to negotiate safe sexual practices, disclose Performance Evaluation of the CHASS Niassa Project 3

18 HIV status, access services (due to fear of reprisal), adhere to treatment, and access care. Activities were implemented in all four provinces to: (a) Expand and improve coordination and effectiveness of GBV prevention efforts; (b) Improve policy implementation in response to GBV; and (c) Improve the availability and quality of GBV services. The project s results framework is included in Appendix VII. II.2 OVERVIEW OF CHASS NIASSA IMPLEMENTATION CHASS Niassa s goal was to strengthen the Niassa provincial health system by maximizing access, quality and sustainability in the delivery of comprehensive HIV/AIDS and related primary health services. The activity s objectives were to: 1. Increase access, quality and utilization of HCT, laboratory services, PMTCT, adult care and treatment, pediatric care and treatment, and HIV/TB co-infection services. 2. Provide a continuum of accessible HIV and related primary health care services, including MCH and RH services, and to improve linkages and referrals within and between facilities and communities. 3. Support stronger and more sustainable Mozambican systems and institutions through emphasis on strengthening government and community capacity to deliver and manage services. The project has supported implementation and scaling up of HIV services by the DPS/MOH through training, mentorship, structural refurbishments, strengthening of monitoring and evaluation systems, and provision of equipment and medical supplies. Through sub-agreements, the project worked jointly with the DPS and the SDSMAS on activities that led to improved health system and expansion of high-quality HIV/AIDS services. The bulk of the funds identified in the SAs were expended directly by CHASS team on behalf of the DPS and SDSMAS. The project introduced a QI collaborative methodology that was reported by the DPS and SDSMAS to be very effective in systematically identifying weaknesses in clinical and auxiliary service practices, processes and environment at the HFs, and in developing elaborate action plans. In addition, through the project s peer education program, which included community-based peer case management and mobilization, community knowledge and utilization of clinical services and GBV services increased over time. The project introduced a graduation path for large facilities. Following a QI/QA process, a HF with sustained standard clinical and auxiliary service practices and processes and that delivers high-quality care according to MOH standards was supposed to be graduated from the project s support to run independently. The project worked with three CBOs (Associação Renascer a Vida, Conselho Cristão de Moçambique and Concelho Islamico de Moçambique) through SAs to strengthen linkages between the HFs and the communities, but also in coordination with other projects such as PCC and TB Care. 4 Performance Evaluation of the CHASS Niassa Project

19 III. EVALUATION METHODS AND LIMITATIONS III.1 EVALUATION METHODS This performance evaluation was based on a non-experimental design and was executed by a team independent and external to the USAID/Mozambique CHASS Project. The evaluation used a mixedmethods approach, utilizing mostly qualitative data collection and evaluation methods. The evaluation involved extensive desk review and analysis of existing quantitative project data and documentation, and primary collection and analysis of qualitative data. Quantitative data were extracted from the quarterly progress reports and project datasets held by the USAID/Mozambique team. The evaluation was conducted September 22-30, Primary data collection methods included key informant interviews and observational analysis at HFs. The key informant interviews served to validate and, where possible, verify project approaches or activities, interventions, achievements, extent of gains and changes over time. They also identified gaps and weaknesses in project activities or performance. The key informants included staff from the MOH, DPS, SDSMAS, HFs, the CHASS implementing partner and CBOs that participated in the project. Some members of the USAID/Mozambique team were interviewed. The structured key informant interview guides are included in Appendix III. They were pretested in Beira city before fieldwork began. Interviews were not held with clients or patients, as this required ethical approval that would have substantially shifted the timeframe for the evaluation. The evaluation used multi-channel data collection: mobile technology when possible and paper-based methods when handheld computers/tablets were not appropriate. The team noted that the use of mobile technology is feasible, and some of the members handwrote their field notes using the ODK program, which was enhanced with a stylus beta program. However, the team notes that for the mobile technology to be most useful for collection of qualitative data, the key informant and focus group guides should be semi-structured. Further, time is required for extensive practice of handwriting on the tablets. This was not possible in this evaluation due to time constraints. III.2 SITE SELECTION AND DATA COLLECTION Three districts were selected at random to represent dominantly rural, semi-urban and urban districts. These included Lichinga, Chimbunila and Lago districts. Within each district, the team selected HFs purposively to represent low and high client volumes, extent of CHASS activities at the site, and rural vs. urban locations. Five HFs were visited and various categories of health workers interviewed (Table 2). The DPS/SDSMAS senior staff included the directors and chief medical officers while the HF management team included the health facility in-charges. A total of eight CHASS Niassa staff, eight PCC staff, and five staff from three CBOs that worked closely with CHASS Niassa were interviewed. In addition, two staff from the MOH and five from USAID were interviewed. Performance Evaluation of the CHASS Niassa Project 5

20 Table 2: Categories of key informant interview respondents Group Number of male key informants Number of female key informants Total key informants DPS/SDSMAS senior staff Health facility management team HIV focal points/technicians (at DPS, SDSMAS and HF) SMI focal points/technicians (at DPS, SDSMAS and HF) HSS/Monitoring and evaluation focal points (at DPS and 2 0 SDSMAS) 2 Logistics/finance (at DPS and SDSMAS) GBV focal points (at DPS, SDSMAS and HF) Other (lab, pharmacy) Total Percent target 100 III.3 DATA ANALYSIS The data analyzed included those in the performance monitoring system and program reports. Trends in results and progress made on planned results were analyzed. The qualitative data were analyzed using a qualitative data analysis matrix. Information from this analysis was integrated or triangulated with the available quantitative data from CHASS documents and reports. The analysis was guided by the 10 evaluation questions listed in the scope of work (Appendix I) and in Section 1 of this report. For the evaluation questions where gender-related data or information are relevant, gender-related differences are presented. The end summary of the analysis was focused on the priority issues for CHASS to address and the main lessons learned, based on the answers provided in examining the 10 evaluation questions. III.4 STUDY LIMITATIONS The limitations of this evaluation are as follows: Firstly, the performance evaluation was delayed; it was initially planned to be in the fourth year of the project but was conducted a year later. Secondly, the findings reported here are based on what the respondents reported to the evaluation team. The likelihood of interviewee bias and recall bias cannot be ruled out. Thirdly, the project was closing at the time of the evaluation, and to some extent, this constrained the sample size for the key informant interviews. Lastly, the evaluation was conducted when the project staff was transitioning, which delayed some of the activities of the evaluation team. 6 Performance Evaluation of the CHASS Niassa Project

21 IV. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS IV.1 FINDINGS Question 1: What are the strengths and weaknesses of the activities, as seen by the interviewed stakeholders, and how can the weaknesses be improved? Findings on the strengths and weaknesses of the project activities are grouped according to project results I-III. The findings are as reported by the key informants and project reports. The achievements reported here should not be attributed solely to the CHASS strategies, considering the interventions of other actors in the province. Strengths and Weaknesses of Activities to Improve Clinical Service Delivery The project made considerable progress toward achievement of its set targets for the results. The HIV/AIDS clinical services were improved and decentralized or expanded to new HFs through staff training and mentoring, improvement of processes and practices through the QI/QA collaborative model, minor facility renovations, and creation of demand and active referrals from the communities to HFs through activists (peer case managers) and coordination with CBOs. The project trained nurses to prescribe ART and provided clinical, laboratory and pharmaceutical supplies, equipment, job aids, care and treatment protocols, and logistical support for the transportation of drugs, lab supplies and samples and results for CD4 and PCR. HIV care and treatment support: In addition to decentralization of HIV/AIDS care and treatment from 25 sites in FY 2011 to 46 sites in FY 2014, the project established mobile HIV care ART outreach teams to take ARVs to the outlying posts for pickup by the clients. The peer or community case managers actively reached out to their community members to test for HIV and enroll into pre-art care and the ART program. Figure 1: Patients already on ART at the beginning of the year and those newly enrolled 16,000 48% 14,000 38% Numbers 12,000 10,000 8,000 6,000 4,000 2,000-27% 861 2,455 7% 36% 30% 26% 2,812 4,571 6% 3,967 11% 6,801 5,261 8, New on ART Already on ART 9% 28% 18% 8% -2% -12% -22% Performance Evaluation of the CHASS Niassa Project 7

22 The number of individuals currently on ART increased from 3,318 in FY 2011 to 13,334 in FY 2014, achieving at least 120 percent of the set target in The percentage of clients currently on ART who are men declined from 36 percent in 2012 to 26 percent in This is partly due to scale-up of Option B+ among pregnant women. This is expected, as HIV-positive men are mainly reached through HCT programs, but HIV-positive women are reached through both HCT and PMTCT programs. The number of clients newly enrolled on ART increased five-fold since 2011 (Figure 1). In FY 2014, the project achieved more than 100 percent of its set target for clients newly enrolled on ART. PMTCT services and integration with MCH services: In addition to expanding PMTCT services to 65 sites by the end of FY 2014 and training nurses to prescribe ART, the project developed innovative strategies to get more pregnant women tested and put on Option B+. Through active mobilization by the CCMs, the number of women seen in ANC increased from 34,961 in 2012 to 53,693 in 2014 (Figure 2), above the project s initial targets. Of these women, at least 82 percent were tested for HIV for the first time in ANC. The percentages initiated on ART in 2012 and 2013 were, however, below the set target of 80 percent. The involvement of M2M groups in following up with HIV-positive women also played a role in getting pregnant women onto ART. The one-stop shop model for PMTCT services was important in ensuring better patient flow and quality of services at the health facility. Figure 2: Indicators in the PMTCT Cascade No. of women who attended ANC Numbers 70,000 60,000 50,000 40,000 30,000 20,000 10,000 - % pregnant women with known HIV status (newly tested + known positive at ANC entry) 92% 72% 56% 16% 18% 40,295 83% 90% 54,726 82% 34% 60, % 98% 78% 58% 38% 18% -2% -22% Source: CHASS USAID progress report data Pediatric care and treatment: Effective strategies to improve enrolment into pediatric care and treatment included training of MCH nurses, the one-stop shop model, ensuring that HIV testing is part of the services at all points of care at the HF, reactivation of adolescent health services at some HFs, and community mobilization through the CCM and M2M. Nurses reported an increased number of youth accessing services. Further, the percentage of patients currently on ART who are children increased from 7 percent in 2011 to 9 percent in Despite this, there is still more work needed to improve enrolment into pediatric care and treatment services. Although the percentage of infants tested for HIV within 12 months increased from 26 percent in 2011 to 53 percent in 2014, this is still below target. TB/HIV co-infection: In most HFs, TB patients were referred to an ART clinic to initiate ART after completion of TB treatment. Similarly, the project made HCT a pre-requisite to access TB treatment. In large-volume facilities, a one-stop shop model for TB and HIV services was introduced, in collaboration with the TB Care project. 8 Performance Evaluation of the CHASS Niassa Project

23 HCT established at health facilities: The project has significantly improved access to HCT services, achieving more than 100 percent of its set targets. A total of 370,281 clients were tested for HIV since 2011, of whom about 50 percent were from provider-initiated counseling and testing, and 30 percent from community HCT. Community-based HCT has become an increasingly large source of HCT clients. Laboratory services: Through minor infrastructural renovations, training of staff at 18 micro and functioning labs, and provision of logistical support for the transportation of test kits, reagents, and test samples and results, the coverage of CD4, PCR and other relevant services have improved. For example, the number of samples subjected to CD4 counts using PIMAs increased from below 200 per month in 2011 to more than 600 in The project also supported registration of the Provincial Hospital Clinical Laboratory into the Management of Laboratory Accreditation Building Programme (FOGELA). The summary of other findings and recommendations are listed in Chart 1. Chart 1: Weaknesses in Activities in Clinical Services and Recommendations Weaknesses Recommendations Adequate space for auditory and visual privacy is very limited in many HFs in Niassa, and structural modifications are critical for effective PMTCT interventions. Storage facility for medication is lacking in some HFs. The quality and quantity of laboratory supplies in place were not sufficient to meet the needs of the HIV/AIDS response. In some HFs, standard laboratory equipment, especially the biochemistry and hematology analyzer, were not available or were not functioning at the time of visit. There are frequent stock-outs, with a poor logistical system between district warehouses and the HFs. There is an insufficient number of staff, and improved activities aimed at high quality of care have placed more demands on health workers. Service coverage of children is limited. The GRM, with support from partners, should invest in structural modifications of the physical space at HFs. Expansions should include storage space. Improve on the logistics and supply chain by hiring a private company to run the system until the DPS has the capacity to run it independently. Put a system in place for reporting consumption rates for the various drugs, kits, reagents, e.g., through electronic tracking system with SMS facility to link the HF to staff at the warehouses. Develop a kit básico (basic package of clinical and lab consumables and equipment) and store at the district warehouses and develop interdistrict route maps to ease regular supply of the kits. The GRM/MOH and partners should urgently invest in buying necessary equipment for the different levels of HFs to ensure better quality of service. Reinforce community-hf linkages and communication, and expand on the number and type of CBOs (e.g., actively involve faith-based organizations, traditional birth attendants, APEs, traditional healers, etc.). Advocate with the MOH to increase the number of health workers and develop better transfer plans. Expand the coverage of a mobile clinic model to include CD4 and PCR sample taking, HCT, PMTCT, and TB screening services. Promote and extend school-based integrated health program, including adolescent health and HCT services. Implement an augmented MOH strategy of a lay counselors workforce by recruiting and training lay counselors or retraining case managers and APEs as lay counselors to provide services. Improve HIV testing and ART initiation among infants through training all HF health workers in: taking PCR samples to provide HIV testing in all pediatric entry points, including children of patients on ART; linkage of records in labor/delivery units and at-risk child consultations (improve pediatric care registries); adequate counseling of pregnant women about early ART initiation among infants and HIV status disclosure to partners, and counseling against HIV stigma; and DBS processing and storage. Also, distribute SMS printers to rural HFs. Performance Evaluation of the CHASS Niassa Project 9