Do you or don t you? Measuring Fidelity to Evidence- Based Supervision Dr. W. Carsten Andresen Dr. Geraldine Nagy Travis County Adult Probation 2011 APPA Summer Conference - Chicago, Illinois 1
Let s go beyond the EBP sound bite
Please use EBP today
That s EBP!
EBP has plenty of dash
These are pretty pictures, but Please use EBP today EBP has plenty of dash Where is the evidence? That s EBP!
Overview: Supervision Fidelity Training and Learning Objectives Evidence-Based Practices model Department Studies Officer Supervision (aka fidelity study) Residential Treatment study
Training and Learning Objectives 1. Define and identify examples of supervision fidelity and client outcomes 2. Explain the relationship between supervision fidelity and client outcomes
Training and Learning Objectives 3. Identify the dimensions of community supervision that are important to measure (fidelity study) 4. Describe methods of evaluating supervision fidelity
Training and Learning Objectives 5. Discuss approaches to present measures of supervision fidelity to the general public
Moving beyond the sound What do prior studies tell us about EBP: bite
Objective One: Define and identify examples of supervision fidelity and client outcomes. Outcomes: eventual effects of a program on some condition (Maxfield, 2001). - Probation Violations - Revocations - New Arrests
Supervision fidelity is less studied Fidelity: presence of EBP elements and quality of implementation - Assess actuarial risk-needs - Motivational Interviewing - Target Interventions o o o o o Risk principle Need principle Responsivity Dosage Treatment - Positive Reinforcement - Engage ongoing community support
Most evaluations involve a leap of faith Program Evaluations focus solely on outcomes Evaluations sidestep fidelity measures 14
Most studies only measure outcomes What about supervision quality? How can you measure if EBP is happening? Can we measure supervision quality and outcomes?
Travis County Adult Probation: We primarily had outcome evaluations External Evaluators: strong outcomes - Dr. Tony Fabelo, Justice Center, Council of State Government Our own data: strong outcomes
Travis Steepest Decline in Felony Revocations - 19.6% (Fabelo, 2009)
Travis Lowest Revocation Rate Out of Population 9% (Fabelo, 2009)
Travis Steepest Decline in Technical Revocations - 47.7% (Fabelo, 2009)
Travis Lowest Rate of Technical Revocations 3.4% (Fabelo, 2009)
Reduced Felony Revocations - 270-294 Revocations Technical Revocations
Reduced Felony Absconders - 1,659
Reduction in New Felony Absconders - 422
Do we successfully use EBP? Outcomes High Outcomes due to other factors Effective Program Low 24
Objective Two: Explain the relationship between supervision fidelity and client outcomes
EBP Model: Outcomes Outcomes High Outcomes due to other factors Effective Program Low What about our Supervision Fidelity? 26
EBP Model: Fidelity and Outcomes Outcomes High Low Outcomes due to other factors Poorly Implemented Effective Program Ineffective Program Low High Fidelity 27
Spring 2009: Study Supervision Fidelity TCIS measured at beginning of implementation Initial baseline identifies areas to improve But where should we begin? Unannounced pop quiz of TCIS fidelity Future fidelity studies can track progress
Travis Community Impact Supervision Measuring Fidelity TCIS TCIS measured at beginning of implementation Initial baseline identifies areas to improve Unannounced pop quiz of TCIS fidelity Future fidelity studies can track progress
EBP Process and Context Measuring Fidelity to EBP Measured at beginning of implementation Initial baseline identifies areas to improve Unannounced pop quiz of EBP fidelity Future fidelity studies can track progress
Two Travis County Supervision Studies Felony Cohort (Jan 2008 - Placements) Follow supervision practices for six months Focus on cases direct for at least three months Follow up: Cohort (Jan 2009 - Placements) Sample of Felony-reduced and Misdemeanants
Study Descriptives Study 1 Study 2 1. Cohort Sample - Felons 134 102 - Felony-Reduced 17 - Misdemeanants 46 2. Coders - Administrators X X - Operations-Research X X - Managers X
Objective Three: Identity the dimensions of community supervision that are important to measure (Fidelity study)
Purpose: To measure our fidelity to EBP
Objective Four: Describe methods of evaluating supervision fidelity
Case File Review Form Fidelity to EBP 36
Methodology Probationer Case File CSS Case File Review Form 37
Methodology Case File Review Form Database 38
Objective Five: Discuss approaches to present measures of supervision fidelity to the general public
Fidelity Study: Fidelity to EBP Results (Jan 2009)
Supervision Fidelity: Do we create accurate assessments? We have validated our risk assessment instrument three times We have conducted Inter-rater reliability tests on our Diagnostic Report Process
Do we create accurate assessments? 95% 95% 89% 89% We checked scoring on each diagnostic instrument for each person
Supervision Fidelity: What about the Judiciary / Courts?
Court-Ordered Conditions of Probation 90% 44
Supervision Fidelity: What about Supervision Process?
Do we focus on risk-needs and use motivational interviewing? We examined each point carefully. 46
Correct Caseload 97% Caseload match Probationer? - Maintenance Caseload - Regular Caseload - Correct Specialized 47
Supervision Agreement 85% Note: Some felony probationers were excluded from this analysis because they were unable to create a Supervision Agreement for various reasons (i.e. absconding, reoffending) 48
Supervision Agreement 85% 49
Responsivity / Motivation 50
Responsivity / Motivation We used a 5-point scale to measure Source: High and Medium Risk Felons Motivational Interviewing - 83% some use - 53% frequent use 51
Sanctions / Incentives 67% 52
Sanctions / Incentives 2009: 10 rush warrants - all 10 addressed properly - all 10 timely response 67% 53
Sanctions / Incentives 75% 54
Overview of Internal Studies 55
EBP Model: Fidelity and Outcomes Outcomes High Low Outcomes due to other factors Poorly Implemented Effective Program Ineffective Program Low High Fidelity 56
EBP Model: Fidelity and Outcomes Outcomes High Low Effective Program Low High Fidelity 57
What about Fidelity Outcomes? What happened to the probationers in this supervision study? Felony probationers: two year follow-up Revocations Arrests EBP Supervision appears correlated with better outcomes for felony probationers, with caveats Small felony sample Difficult to account for complicated interactions between variables
Overall Supervision Measure - Was the Supervision Appropriate? Data Source: Coder provided summary ranking of supervision quality.
The Supervision Agreement is more than just a piece of paper We looked for evidence that this document played a role in the supervision process. We examined if the document had the following: Probationer s signature Indications that the creation of the Supervision Agreement was a collaborative process Chronological Notes suggesting that the Supervision Agreement was the foundation for office visits
Did the Supervision Agreement show signs of EBP Yes: Signature, evidence of cooperation, evidence it was used as the foundation for office visits
Now we will move away from supervision to discuss SMART our Residential Drug Treatment Program.
Fidelity Evaluation: Drug Treatment What does our SMART Program do? Does it engage in Evidence-Based practices? Quality of work?
EBP Model: Fidelity and Outcomes Outcomes High Low Outcomes due to other factors Poorly Implemented Effective Program Ineffective Program Low High Fidelity 64
Prior SMART Residential Analysis Corrections Institute-University of Cincinnati Correctional Program Checklist (CPC) CPC 5 Domains - Leadership - Staff - Assessment -Treatment -Quality Assurance Highly Effective Effective Needs Improvement Ineffective 65
Corrections Institute University of Cincinnati SMART Program Rank (400+ Programs) 7% Highly Effective SMART Program 18% Effective 33% Needs Improvement 42% Ineffective Shaffer, DK and C Thompson. (2008). Evidence-Based Correctional Program Checklist SMART Program. University of Cincinnati. 66
SMART Residential Program 2004 Sept 2005 Sept 2006 Sept 2007 Sept 2008 Sept 2009 Sept Nov 2008: CPC Study: 3 yrs successful completions - 18 month Feb 2007 follow-up SMART Sep 2005 - Aug 2008 SOMP start 18 Comparison Group Sep 2005 - Aug 2008 67 18
Creating Comparison Group Individual cases matched on seven factors Smart - Risk Score - Fiscal Year - Age Range - Race-Ethnicity - Gender - Offense Degree - Offense Type Comparison N = 559 N = 489 Many studies avoid risk / lack comparison groups 68
It is critical to match for Risk Score 9% 25% 67% - Felony Placements 0% 14% 86% - Smart 69
All Arrest Categories (by group) Note: this captures # of arrests, not # of people arrested. SMART: % decrease in total arrests 27 % fewer total arrests 12% fewer total felony arrests 15 % fewer total misd arrests 70
EBP Model: Fidelity and Outcomes Outcomes High Low Outcomes due to other factors Poorly Implemented Effective Program Ineffective Program Low High Fidelity 71