EVALUATION of MCH PROGRAMS: ISSUES & METHODS (Part II)

Transcription

1 EVALUATION of MCH PROGRAMS: ISSUES & METHODS (Part II) Patricia O Campo Ph.D. May 2013 pat.ocampo@utoronto.ca

2 Topics Key Concepts in Evaluation Review of Logic Models, Theory & Evaluation cycle Design of evaluations Using evaluations

3 Key concepts in Evaluation (quick review)

4

5 CYNEFIN framework (Snowden 2002)

6 **Evidence-based and best practices Simple BAKING A CAKE Complicated SENDING A ROCKET TO THE MOON Complex RAISING A CHILD Recipe is essential No particular expertise required A good recipe produces nearly the same cake every time The best recipes give good results everytime A good recipe notes quanitity and nature of the parts needed and notes order in which to combine them but there is room for experimentation Rigid protocols or formulas needed Sending one rocket increases the likelihood that the next will be a success High levels of expertise and training in a variety of fields are necessary for success Key elements of each rocket MUST be identical to succeed There is a high degree of certainty of outcome Success depends upon a blueprint that directs both the development of parts and the exact way to assemble them Rigid protocols have limited application /counter productive Raising one child provides experience but is no guarantee of success with next Expertise helps but only when balanced with responsiveness to the particular child Every child is unique and must be understood as an individual Uncertainty of outcome remains Cannot separate the parts form the whole; essence exists in the relationship between different people, different experiences, different moments in time Getting to Maybe

7 Simple BAKING A CAKE Complicated SENDING A ROCKET TO THE MOON Complex RAISING A CHILD Get good recipes Closely adhere to them Minimize modifications Success every time Formulate good protocols, need many of them Stick with protocols that lead to success Adhere closely Success increases with experience Protocols cannot be developed Guidelines can be followed and tailored to child Successes should be noted and monitored and refined iterative process Success at one stage won t guarantee success at subsequent stages same for failure (don t give up after a failure) *** Excellent for formulating Best Practices Getting to Maybe

8

9 Developmental evaluation design possibility for Healthy Start -Serve women in the Community. -Reach 70% or more of women to ensure we re getting high risk participants. -Get women into Prenatal Care Early. -Serve women in the Community. -Reach 70% or more of women to ensure we re getting high risk participants. -Engage them on their own issues especially urgent needs first. -Once urgent needs resolved, engage clients in prenatal care. -Serve women in the Community. -Reach 70% or more of women to ensure we re getting high risk participants. -Engage them on their own issues especially urgent needs first. -Once urgent needs resolved, engage medically high risk clients in prenatal care. -Provide women with knowledge and skills to become more economically independent (education, job experience) YEARS

10

11 Systematic Reviews Evidence based practice Adapted from: navigatingeffectivetreatments.org.au

12

13 Realist Systematic Review of Partner Violence Screening in Health Care Settings High institutional support Thorough initial and ongoing Staff training Effective screening protocols & valid tools High screening confidence Routine or Universal Screening Strong community Linkages Easy referral to onsite or offsite services Supported by existing socialcognitive theories O Campo et al, 2011 Community Institutional Provider/clinical/service Level of influence

14

15 Program evaluation is a systematic method for collecting, analyzing, and using information to answer questions about projects, policies and programs*, particularly about their effectiveness and efficiency. Wikipedia Was the program effective? Why and how and for whom did the program work? *Program is defined as a plan of action aimed at accomplishing a clear objective, with details on what work is to be done, by whom, when, and what means or resources will be used. adapted from Businessdictionary.com

16 Planning & Evaluation Cycle Evaluation Plan Assessment Monitor Capacity & Strategies Implement Do Plan

17 Recommended Framework For Program Evaluation

18 Standards for Effective Evaluation CDC 1999

19 Engage Stakeholders throughout the Evaluation Process

20 A logic model will assist you in communicating the underlying theory (logic) that you have about why your activities are a good solution to the problem identified. Link short/long term outcomes to each other and to program activities using the identified for your program (illustrate cause and effect); Link Evaluation Questions to the Program Logic Select indicators to match Program Logic and Evaluation Questions; Become a common reference point for staff, stakeholders, constituents and funding agency CDC Logic Model Planning

21 TOC and Program Logic Models Assumptions & Theories of Change DO GET Strategies Results

22 Program Logic Model Parenting Program

23 Evaluation Logic Model

24 Evaluation Logic Model

25

26 Logic models can help you answer the following about evaluation: Are the results specified with shared meaning among all stakeholders? Did we uncover our assumptions and carefully examine research, practice, and theory as the grounding for our choices in strategies including other similar programs so that we can design an evaluation around these issues if needed? Does the model clearly show the relationship of strategies to results so that we can design an evaluation? Do our evaluation questions reflect key aspects of the program as well as priorities of the stakeholders? Do our evaluation indicators match the processes and outcomes of the program?

27 (your turn) Homework: Is there a existing program or intervention for which you, solely or as part of a team, will be involved in the evaluation? What do you know about the program? Do you know: TOC, goals/objectives, target population, how long it has been in operation, whether it is still evolving, size/scope of the program, program stakeholders? If not, how can you find out? Does a Logic Model for the program exist and/or can you create one? Can you ensure that the LM is vetted by relevant stakeholders?

28 Recommended Framework For Program Evaluation CDC 1999

29 Focussing the Evaluation

30 Examples of evaluation questions which should be prioritized by stakeholders with considerations of feasiblity????? Was the program successful in achieving its goals? What is the need for the program? Is the program relevant? Was the structure/logic of the program appropriate? Was the program implemented as intended? Was the program technically efficient? Was the program responsible for the outcomes that actually occurred? Did the program achieve its intended objectives? Was the program cost-effective? Ws the program cost-beneficial? Was the program adequate?

31 Focussing an Evaluation

32 Formative Process or Implementation Outcome Can we effectively reach the target population? What proportion of clients attend all sessions? What proportion of clients achieved each of the main outcome? Can we implement the intervention to the desired intensity? Did/do we have to make adaptations to our program and if so how? Were there subgroups for whom the program does not work? Is the staff to client ratio sufficient? How can we explain unintended outcomes? Was there a difference in effect by dose or intensity? Is the outcome being measured accurately? Are there conditions under which this program is not ideal or not working? Are there unexpected outcomes and how did it impact clients?

33 Focussing an Evaluation Formative, Process or Outcome?

34 Focussing your Evaluation Resources (funds, data, time, capacity, etc) Stakeholder Priorities: outcome, implementation, replication, etc Program Logic Models (TOC, Program) Evaluation Design

35 Consider issues of feasibility

36 Should the evaluation be undertaken? Assessing the feasibility of an evaluation (ask throughout the design phase) Can the program and its assumptions be fully articulated? Or what is the logic of the program? What resources are available to do the evaluation most consistent with stakeholder priorities? What kind of environment does the program operate in and how does that affect the comparisons available to an evaluator? Which research design alternatives are desirable and feasible? What data sources are available and appropriate, given stakeholder priorities, evaluation issues, program structure, and the operating environment? Given all issues raised above, which evaluation strategy is the most feasible and defensible?

37 (your turn) Data sources State-wide Domestic Violence Screening/Identification /Referral Program Initiative starts Jan 1, 2014 in all hospitals in State Resources for Evaluation: 2 FT persons for 3 years & access to public data (e.g., vital records, PRAMS, etc) Stakeholders interested in: Process: In Year 1, are at least 30% of hospitals complying? Process: Is screening process acceptable to 80% of victims who are women and men? Outcomes: Are 50% or more of victims receiving referrals? What are candidate data sources?

38 (your turn) Data sources State wide WIC program, evaluating selected objectives: Short Term Outcome: Every pregnant client will receive BF support from a CPA during the third trimester, and this will be documented in the chart. The support will either be by class, individual education, online interactive education, or phone call. Long Term Outcome Increase breastfeeding duration at 6 months by 3 %. What are candidate data sources?

39 Choosing the Design (for outcome/impact evaluations)

40 There cannot be any plausible rival explanatory factors that could account for the correlation between the program and the outcome Causality ( Evidence from Randomized Clinical Trial) The program has to precede the observed outcomes The program theory has to predict the evaluation outcomes The presence or absence of the program has to be correlated with the presence or absence of the observed outcome(s)

41 Evaluations Only about 15% of published evaluations are RCTs Quasi-experimental, another 32% Non-experimental designs, 48% Christie and Fisher 2010

42 Design Quality & Four types of Validity for Evaluations McDavid et al., 2013

43 Threats to validity Internal validity assesses the extent to which we can confidently attribute the cause of the observed changes to the program. Want to eliminate competing explanations of what we observe about program success External validity assesses the degree of confidence we have in generalizing our findings to populations and settings beyond what we observed for our particular program.

44 Major threats to internal validity History Maturation Testing Instrumentation Statistical regression Selection Mortality

45 Major threats to internal validity History Events over time affect the generalizability or the ability to evaluate the program (e.g., suicide prevention programs and austerity practices/recession) Maturation Testing Instrumentation Statistical regression Selection Mortality

46 Major threats to internal validity History Maturation The aging process of participants affects evaluation (e.g., child development programs) Testing Instrumentation Statistical regression Selection Mortality

47 Major threats to internal validity History Maturation Testing Administration of tests may serve as an intervention and result in program getting more credit that it deserves (e.g., child assessments, they can get better over repeated administrations) Instrumentation Statistical regression Selection Mortality

48 Major threats to internal validity History Maturation Testing Instrumentation Questionnaires/surveys serve to call attention to the issue and may result in behavior change (e.g., lots of questions about breastfeeding prepregnancy may influence BF initiation/duration) Statistical regression Selection Mortality

49 Major threats to internal validity History Maturation Testing Instrumentation Statistical regression extreme values tend toward the population mean (e.g., high and low scorers, tend to move toward average over time) Selection Mortality

50 Major threats to internal validity History Maturation Testing Instrumentation Statistical regression Selection Those who are willing to participate may be different than those who are not willing to receive services/treatment Mortality

51 Major threats to internal validity History Maturation Testing Instrumentation Statistical regression Selection Mortality--Follow-up/drop out may be influenced by program characteristics and/or vary by demographic, behavioral, other specific traits (resulting in a selection type of effect)

52 (your turn) Impact Evaluation: Example 1 Home visiting program for first time teen mothers to promote continued education 300 moms enrolled in program and 120 stayed long enough to get (up to) 3 home visits over a 3 month period to promote the importance of education & obtaining a HS diploma. (rest were lost) Reported that 60% of 120 were in school at end of study. Concluded that this was a high proportion and that the program was a success

53 Impact evaluation: Example 2 Smoking cessation intervention (3 weeks) Enrolled all those wanting to quit from a primary care clinic (N=150) Brief counseling, telephone booster sessions, and 1 follow-up visit at 3 weeks. Assessed quit rates at 5 weeks Comparison group 1 was the smoking rate among those who said they didn t want to quit, and were contacted at 5 weeks post-recuitment (N=50). Comparison group 2 was the smoking rate among those who wanted to quit but did not ever return to the clinic for the intervention (N=80) Treatment and comparison quit rates: 18% versus 2% (1) and 7% (2) Even if adjustments were made for demographic differences between T & C, differences were significant (14% difference T vs. 1, 7% T vs 2) Intervention a success!!!

54 Example 3 Program preventing repeat teen births among girls having undergone 3 or more foster care placements in the past year (very high risk). Girls obtained a professionally trained big sister who became a case manager & a stabilizing force in their lives as well as assisted with things like parenting, schooling, etc. Randomized design (N=100 per group) and repeat births to teens was substantially reduced among the treatment group (T vs. C=5% vs. 18%) over a 2 year period Concluded that the big sister program can prevent all teen births and should be widely implemented.

55 Study design Want optimal designs to Promote causal inference Minimize threats to internal, construct and external validity For achieving high internal validity, the best design is the an experimental design: randomized trial Identify target population Randomize assignment to treatment Intention to treat analysis is most conservative Quasi-experimental designs, has comparison group but not selected randomly Selection of the comparison group is one of the most important decisions

56 Quasi-experimental & Non-experimental Study designs and Threats to Internal Validity

57 Evaluation Design Resources (funds, data, time, capacity, etc) Stakeholder Priorities Choices between various comparisons & feasibility / resources Evaluation Design

58 Combining designs within a single evaluation (patch quilt)

59 Consider Sample size during your design stage There is a standard alpha and beta often used Are those the best values for our study? Should we challenge the status quo?

60 Type I and Type II error Study Sample Reality Association/Difference is Found No Association/ Difference is Found Group A ¹ Group B Association/Difference Exists Correct conclusion Power Type II error, Beta (b) Group A = Group B Association/Difference does not Exist Type I error Alpha (a) Correct conclusion Confidence Type I: the error of rejecting a true null hypothesis Type II: the error of accepting the null hypothesis when in fact the null is incorrect

61 New Behavioral Intervention to Prevent HIV Infection Among Youth Which is more important, alpha, beta or both? Are the typical values appropriate (α=0.05 & β=0.20)? For this intervention which costs millions of dollars, what are some ideal values of alpha? Beta?

62 Sample size: summary If evaluation lacks power, then the effort is a waste of resources Power considerations must be undertaken BEFORE data collection There are no hard cutoffs with power analyses just as there are no hard cutoffs when conducting statistical analyses: In analyses we use (approx) p<0.05 but are flexible In power analyses, we like to see power=0.8 but we remain flexible We may want to use a smaller level of beta to minimize the chance of committing TYPE II ERROR

63 Measurement error & power Devine 2003

64 Choosing comparison group; Example Community based project: Baltimore City Healthy Start Program Geographically defined target areas for service /intervention delivery Targeted highest risk areas of the City for services Identified high risk areas using quantitative assessments of economic, social and reproductive risk Challenge was to find an appropriate comparison group for the local evaluation

65 Baltimore City Census Tracts, 1997 High Economic and Reproductive Risk Targeting specific neighborhoods

66 Healthy Start LM PROGRAM (e.g.,) --Serve women in the Community. -Reach 70% or more of women to ensure we re getting high risk participants. -Engage clients on their own issues especially urgent needs first. -Once urgent needs resolved, engage clients in health/social services (e.g., PNC). -Provide women with knowledge and skills to become more economically independent (education, job experience) _Hire community residents to work in HS -Provide women with knowledge and skills about parenting, child development, and infant health -Special programs for dads -SHORT TERM & INTERMEDIATE OUTCOMES -less economic deprivation -more knowledge about infant well-being (child development and health) -more contact with health care providers (prenatal and pediatric) -child care support -improved maternal skills (reading, job readiness) -economic opportunities -lower LBW, PTB, infant mortality for index child LONG TERM OUTCOMES -better family health -better outcomes for subsequent pregnancies -better economic selfsufficiency -better child development

67 PROGRAM Healthy Start LM -SHORT TERM & INTERMEDIATE OUTCOMES -less economic deprivation -more knowledge about infant well-being (child development and health) -more contact with health care providers (prenatal and pediatric) -child care support -improved maternal skills (reading, job readiness) -economic opportunities -lower LBW, PTB, infant mortality for index child LONG TERM OUTCOMES -better family health -better outcomes for subsequent pregnancies -better economic selfsufficiency -better child development -engaged community with improved economic well-being

68 National evaluation (FYI)

69 Baltimore City Census Tracts, 1997 High Economic and Reproductive Risk Targeting specific neighborhoods

70 (your turn) What are some candidates for a comparison group? Group Data source Outcomes

71 Baltimore City Healthy Start Comparison Group 1 Chose the target areas during needs assessment Only thinking about service provision, not the impact evaluation Political considerations & consequences of depriving one side of the city of services were enormous Precluded servicing one side of the City with the other side serving as the comparison Had impact evaluation been considered earlier, may have configured the target areas differently to accommodate the comparisons May have had smaller target areas on each side reserving some comparison census tracts on each side May have chosen a mix of high/moderately high target areas with corresponding comparison groups May have spent much of the early part of the planning period collecting baseline/comparison data Challenge is that our (evaluation) measures evolved as the program unfolded and as the program matured

72 Census tracts with a # were highest quintile of reproductive and social risk What are potential comparison areas? *Green circles Target Area

73 Healthy Start Baltimore comparison group 2 ( ) Each target area (TA) had a Neighborhood Healthy Start Center (NHSC) Center offered intensive recruitment (>75% in TA), home visiting, emergency need resources, on site services (e.g., WIC, parenting, laundry facilities, drop in child care, GED, etc) New mothers enrolled up to 6 months postpartum

74 Strengths of Postpartum NHSC ENROLLEES as Comparison Group for Prenatal Outcomes exposed to same neighborhood context comparable levels of eligibility for program participation comparable levels of medical and social risk (and can adjust for several such factors) motivation for program participation similar

75 The Effect of Healthy Start Participation on Pregnancy Outcome Treatment group - prenatal enrollee 1714 prenatal enrollees from May 1993 to January st pregnancy in HS, singleton, & had case management activity Comparison group - postpartum enrollees 1039 clients given birth as of January (92%) linked to birth certificate data 1314 postpartum enrollees from May 1993 to January st HS infant, singleton birth & case management contact 847 (91%) linked to birth certificate data

76 (your turn) What are the threats to validity for Comparison group 2?

77 Percent The Effect of Healthy Start Participation on Pregnancy Outcome LBW** VLBW** Preterm Delivery** Treatment Group Comparison Group *p=.05 **p<.01 ***p<.001

78 Demographic Differences Treatment vs. Comparison 2 Women Treatment Comparison N=956 N=847 % % Age 17 or less < 12 years education Substance users Ever employed Below 100% poverty < 18 mos since last birth*** Parity 3 or more* Initiated PNC 1st trimester*** *p<0.05 ***p<0.001

79 Odds Control vs. Treatment The Effect of Healthy Start Participation on Pregnancy Outcome LBW+ VLBW* Preterm Delivery+ Adjusted Odds for Control vs. Treatment

80 Women who enroll late in pregnancy may not have been exposed to the intervention long enough to impact the outcome. Limit the prenatal enrollees/treatment group to those who enrolled before the 3 rd trimester Interest in the impact on high risk groups too substance users

81 Percent The Effect of Healthy Start Participation on Pregnancy Outcome: Early Prenatal Enrollees LBW** VLBW** Preterm Delivery** <32 weeks Comparison Group *p<.05

82 Percent The Effect of Healthy Start Participation on Pregnancy Outcome: Substance Users LBW* VLBW Preterm Delivery Substance Users in Treatment Group Substance Users in Comparison Group p<.10

83 (your turn) Comparison Group Choice State-wide Domestic Violence Screening/Identification /Referral Program Initiative starts Jan 1, 2014 in all hospitals in State Resources for Evaluation: 2 FT persons for 3 years & access to public data (e.g., vital records, PRAMS, etc) Evaluation priorities: Reduction in DV rates, Provider confidence in Screening (key ingredient) Logic Model? Candidate Designs including data sources? Threats to validity?

84 Justify your conclusions

85 While quantitative associations matter, (expert) judgement is a large driver of Evaluation conclusions

86 Engage Expert Stakeholders in Assessing the Conclusions Reached using the Data

87 At the end of your analyses Review your data collection strengths/challenges. How did the actual process of implementing your evaluation activities impact evaluation confidence & threats to validity? How do you describe those in your report? There is no perfect evaluation! Expect to describe the limitations and challenges of your evaluation activities.

88 Example: Expansion of HIV Testing and Counselling in Ontario Evaluation of improvements

89 Example Many stakeholders Program Funder, Providers all over the province Large steering committee with key players on board *tho not all key members willingly assisted us Patchwork evaluation design (18 months): WHO ARE THE RISK GROUPS BEING TESTED? WHAT ARE THE TESTING AND COUNSELLING EXPERIENCES OF CLIENTS? WHAT FACILITATORS AND CHALLENGES DO PROVIDERS FACE IN ADHERING TO THE GUIDELINES FOR HIV TESTING AND COUNSELLING? HOW WELL ARE REFERRALS MADE AND ACCESSED?

90 Populations (province wide): MSM, IDUs, immigrants from endemic countries, (not) pregnant women, worried well (resource drain), youth/street youth. HIV Testing and Counselling in Ontario Stakeholders: Ontario Ministry of Health; HIV Testing Providers; Clients;

91 Surveys with providers and clients Providers: all providers sent packets of anonymized surveys to fill out and return in prepaid envelopes to ask about (1) guidelines and (2) counselling experiences Clients: providers asked to hand out surveys to each testing client for a set duration (e.g., March 1-March 12) Response rates low, we had hoped to have 1500 surveys returned but we got 800 Had hoped to have a greater proportion of high risk clients, tho had very few respond. **we relied on provider reports and case studies to talk about high risk clients We sought to do 18 case studies and we completed 17

92 Lessons learned about the evaluation Flexibility on design issues were key throughout the 18 month evaluation as changes were necessary as barriers were encountered Stakeholder involvement was energy consuming and impacted key decisions (e.g., anonymity of the survey, whom to survey, what to ask about and HOW) but worth all the effort as the findings were accepted and used Staff qualifications and communication deserves a lot of attention to ensure high quality data collection & analysis.

93 Ensure use and share evaluation findings

94 Report: Exec Summary (2 pages), 2 pages of methods, 13 pages of results, 2 appendices with the survey & detailed tables

95 Short summary circulated widely in newsletters

96 Engaging Stakeholders: Ensuring Utility

97 Theory of change & short and long term outcomes

98

99

100 Think out of the box when presenting findings in reports and in short communications Planning a good report/communication strategy takes time Include someone on your team with expertise

101 Questions???