Monitoring & Evalua/on Ari Probandari
Learning Objec/ves Students are able to explain the importance of monitoring and evalua/on a program management Students are able to apply concepts of monitoring and evalua/on in a program management
What is Monitoring and Evalua/on (M&E)? Group Work 1
Dimension Monitoring Evalua2on Frequency Periodic, occurs regularly Episodic Func2on Tracking/oversight Assessment Purpose Focus Methods Informa2on source Cost Improve efficiency, provide informa/on for reprogramming to improve outcome Inputs, outputs, processes, workplans (opera/onal implementa/on) Rou/ne review of reports, registers, administra/ve databases, field observa/ons Rou/ne surveillance system, field observa/on reports, progress reports, rapid assessment, program review mee/ng Consistent, recurrent costs spread across implementa/on period Improve effec/veness, impact, value for money, future programming, strategy and policy making Effec/veness, relevance, impact, cost-effec/veness (popula/on effect) Scien/fic, rigorous research design, complex and intensive Same Popula/on based surveys, vital registra/on, special studies Episodic, osen at the midpoint and end of implementa/on period The Global Fund. (2011). Monitoring and Evalua/on Toolkits.
Monitoring gives informa/on on where a policy, program, or project is at any given /me (and over /me) rela/ve to respec/ve targets and outcomes. It is descrip/ve in intent. Evalua/on gives evidence of why targets and outcomes are or are not being achieved. It seeks to address issues of causality. Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
M&E Monitoring Clarifies program objec/ves Links ac/vi/es and their resources to objec/ves Translates objec/ves into performance indicators and sets targets Rou/nely collects data on these indicators, compares actual results with targets Reports progress to managers and alerts them to problems Evalua2on Analyzes why intended results were or were not achieved Assesses specific causal contribu/ons of ac/vi/es to results Examines implementa/on process Explores unintended results Provides lessons, high- lights significant accomplishment or program poten/al, and offers recommenda/ons for improvement Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Why M&E are needed? To ensure that resources are used effec/ve and efficient (input) To ensure that the program is implemented as planned (process) To inform about the level of achieved output, therefore correc/ons may be conducted if needed (output/outcome) To assess impact of program (impact)
How do we design M&E systems in a program management? What aspects of M&E systems should be planned?
Planning of M&E Systems Indicators, baseline, target Data source + Methods of data collec/on Analysis + Synthesis Communica/on + Use of informa/on
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank. Results based M&E Results Goal Outcome Long term, widespread improvement in society Intermediate effect of outputs on clients Implementa/on Outputs Ac/vi/es Input Products and services produced Task perssonel undertake to transform inputs into outputs Financial, human, and material resources
Example: Results based M&E Results Goal Outcome To end TB epidemic in Indonesia Increase the case no/fica/on rate annualy to achieve a minimun case detec/on rate of 70%, by 2019, as compared to 33% in 2014 Implementa/on Outputs Ac/vi/es Input 34 provincial level training 1 na/onal workshop on the guideline Intensified Case Finding (ICF) is implemented in all 34 provinces Guidelines, training, implementa/on of ICF Trainers, Trainee, DraS of guideline, etc
Global Context Na/onal Context Goal of Program Other Relevant Context
Situa/on analysis 2 Situa/on analysis 1 Situa/on analysis 3 Outcome
Indicators, Baseline, Target Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Indikator yang Baik Clear Relevant Economic Adequate Monitorable Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
M&E Data Source Agregate Annual Report Monthly Report Daily Register Individual Medical Records Detail Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Design an evalua/on Ovretveit J. 2003. Evalua/ng Health Interven/on. Open University Press Purpose Evaluator role: internal/external evaluator? How osen When the evalua/on is undertaken: before/ during/aser? Methods Focus of evalua/on Scope: limited or comprehensive?
Focus of evalua/on Needs Demands Inputs Processes Outputs Outcome Feasibility Process/Forma/ve Look at the effect
Methods Quan/ta/ve Qualita/ve
Let s learn from a program evalua/on research
Assessment of DOTS strategy implementa/on in hospitals in Indonesia: 2005-2007 Adi Utarini, Ari Probandari, Trisasi Lestari, Hary Sanjoto, Ktut As/, Agus/na, Mohammad Arifin
Provid er Pa/- ent Servic e The Study The Perspec/ves Burden of TB TB services in hospital Prescrip/on DOTS strategy implementa/on Pa/ent Provider DOTS Centre Lab District wasor ISTC 32
Focus of evalua/on Needs Demands Inputs Processes Outputs Outcome Process/Forma/ve
Methods Quan/ta/ve Qualita/ve
Process in external linkage 35
Free TB drugs and choices of facili/es for treatment 36
Sputum collec/on and prac/ces in the ward HIV pa/ents were put in the same room with TB, TB- HIV or other pa/ents with infec/ous diseases (Field notes) 37
Use of secondline drugs in TB treatment Common pattern : -HRE-Cipro (Cat 1) -HRZE-Cipro (Cat 1) -Cipro (Cat II) 38
Most Common Source of TB drugs used by TB pa/ents Category I, II and Extra-pulmonary TB Cases: first line generic drugs (41-64%) FDC provided by NTP (21-40%) Children: first line generic (82%) first-line branded (18%) 39
Deficiencies in TB drugs prescrib/on and dosage in TB Adult Cases 100% 90% 80% 70% 60% 50% 40% Type Dosage Secondline drug 30% 20% 10% 0% 1 2 3 4 5 6 7 8
ISTC Standard Criteria Hosp (%) Chest Clinic (%) Standard 1 Cough for 2-3 weeks as TB suspects 88.9 50.0 Standard 2 Sputum microscopy for TB diagnosis: Sputum test carried out 94.4 100.0 At least 2 specimens for diagnosis 94.4 100.0 Standard 3 Microscopy examina/on for EP-TB 66.7 50.0 Standard 4 Sputum examina/on for sugges/ve of TB 94.4 100.0 Standard 5 Diagnosis of SS (-) TB: Repeat sputum examina/on 22.2 50.0 Repeat sputum examina/on&x Ray 22.2 0.0 Clinical judgement&x Ray 33.3 0.0 Standard 6 Scoring system for pediatrict TB 11.1 100.0 41
ISTC Standard Criteria Hosp (%) C.Clinic (%) Standard 7 Treatment monitoring 77.8 100.0 Standard 8 Standard 9 TB treatment: First line drugs 94.4 100.0 Dossage conform to interna/onal recommenda/on Treatment adherence: 94.4 100.0 Treatment supporter 72.2 50.0 Referral for poor adherence 83.3 100.0 Standard 10 SS test for monitoring treatment 88.9 100.0 Standard 11 Recording all medica/ons 88.9 100.0 Standard 12 TB-HIV: HIV risk assessed 50.0 50.0 Referral to VCT 77.8 50.0 42
ISTC Standard Criteria Hosp (%) Standard 13 ARV treatment ini/a/on: C.Clinic (%) ARV and TB drugs given concurrently 55.6 50.0 Cotrimoxazole profilaxis 50.0 50.0 Standard 14 Monitoring of drug resistance 44.4 50.0 Standard 15 Drug resistance TB: Consult to MDR expert 72.2 100.0 Effec/ve four drug regimen 44.4 50.0 Standard 16 Contact tracing 50.0 0.0 Standard 17 Repor/ng all TB cases to health authori/es 94.4 100.0 43
ISTC Public Health Responsibili/es ISTC Standard Criteria Hosp (%) Standard 16 Contact tracing 50.0 0.0 Standard 17 Repor/ng all TB cases to health authori/es Chest Clinic (%) 94.4 100.0 44
STRUCTURE 1. Ownership of hospitals 2. Quality assurance system in hospital 3. TB case load 4. Trained staffs 5. DOTS team 6. DOTS unit 7. NTP Guidelines 8. Financial incen/ves from DHO to hospitals Findings PROCESS 1. Commitment from hospital 2. Adherence to standard of diagnosis and treatment 3. Conversion rate 4. Case holding process OUTCOME 1. Treatment success rate 2. Treatment comple/on rate 3. Default rate 4. Coverage of DOTS within a hospital Probandari A, Utarini A, Hur/g AK. Achieving quality in the DOTS strategy implementa/on process: a challenge for hospital PPM in Indonesia. Global Health Ac/on 2008. DOI: 10.3402/gha.v1i0.1831 45
Probandari A, Lindholm L, Stenlund H, Utarini A, Hur/g AK. Missed opportunity for standardized treatment among adult TB pa/ents in hospitals involved in PPM-DOTS in Indonesia. BMC Health Services Research 2010; 10: 113 TB suspects Outpa/ent Unit TB suspects perform sputum test TB suspects do not perform sputum test Lab register Hospital Laboratory Medical Record Report ICD X A. 15-A.19 Loss of follow up among SS (+) TB cases: 8-18% Outpa/ent Unit TB cases Not TB cases DOTS pabent register DOTS unit Treated within the hospital not under DOTS unit Refer to other health service facili/es 20-53% Referral rate from Irawa/ et. al. (2007) 46
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Repor/ng M&E Results Full wriren report Execu/ve summary (1-4 pages) Oral presenta/on Policy brief
M&E Result U/liza/on Mass-Media Internet E-government Annual Report Public Hearing with House of Representa/ve Public Discussion
Con/nuing M&E Demand Clear roles and responsibility Trustworthy and credible informa/on Accountability Capacity Incen/ves
What you have learned from this session?