Do you or don t you? Measuring Fidelity to Evidence- Based Supervision

Transcription

1 Do you or don t you? Measuring Fidelity to Evidence- Based Supervision Dr. W. Carsten Andresen Dr. Geraldine Nagy Travis County Adult Probation 2011 APPA Summer Conference - Chicago, Illinois 1

2 Let s go beyond the EBP sound bite

3 Please use EBP today

4 That s EBP!

5 EBP has plenty of dash

6 These are pretty pictures, but Please use EBP today EBP has plenty of dash Where is the evidence? That s EBP!

7 Overview: Supervision Fidelity Training and Learning Objectives Evidence-Based Practices model Department Studies Officer Supervision (aka fidelity study) Residential Treatment study

8 Training and Learning Objectives 1. Define and identify examples of supervision fidelity and client outcomes 2. Explain the relationship between supervision fidelity and client outcomes

9 Training and Learning Objectives 3. Identify the dimensions of community supervision that are important to measure (fidelity study) 4. Describe methods of evaluating supervision fidelity

10 Training and Learning Objectives 5. Discuss approaches to present measures of supervision fidelity to the general public

11 Moving beyond the sound What do prior studies tell us about EBP: bite

12 Objective One: Define and identify examples of supervision fidelity and client outcomes. Outcomes: eventual effects of a program on some condition (Maxfield, 2001). - Probation Violations - Revocations - New Arrests

13 Supervision fidelity is less studied Fidelity: presence of EBP elements and quality of implementation - Assess actuarial risk-needs - Motivational Interviewing - Target Interventions o o o o o Risk principle Need principle Responsivity Dosage Treatment - Positive Reinforcement - Engage ongoing community support

14 Most evaluations involve a leap of faith Program Evaluations focus solely on outcomes Evaluations sidestep fidelity measures 14

15 Most studies only measure outcomes What about supervision quality? How can you measure if EBP is happening? Can we measure supervision quality and outcomes?

16 Travis County Adult Probation: We primarily had outcome evaluations External Evaluators: strong outcomes - Dr. Tony Fabelo, Justice Center, Council of State Government Our own data: strong outcomes

17 Travis Steepest Decline in Felony Revocations % (Fabelo, 2009)

18 Travis Lowest Revocation Rate Out of Population 9% (Fabelo, 2009)

19 Travis Steepest Decline in Technical Revocations % (Fabelo, 2009)

20 Travis Lowest Rate of Technical Revocations 3.4% (Fabelo, 2009)

21 Reduced Felony Revocations Revocations Technical Revocations

22 Reduced Felony Absconders - 1,659

23 Reduction in New Felony Absconders - 422

24 Do we successfully use EBP? Outcomes High Outcomes due to other factors Effective Program Low 24

25 Objective Two: Explain the relationship between supervision fidelity and client outcomes

26 EBP Model: Outcomes Outcomes High Outcomes due to other factors Effective Program Low What about our Supervision Fidelity? 26

27 EBP Model: Fidelity and Outcomes Outcomes High Low Outcomes due to other factors Poorly Implemented Effective Program Ineffective Program Low High Fidelity 27

28 Spring 2009: Study Supervision Fidelity TCIS measured at beginning of implementation Initial baseline identifies areas to improve But where should we begin? Unannounced pop quiz of TCIS fidelity Future fidelity studies can track progress

29 Travis Community Impact Supervision Measuring Fidelity TCIS TCIS measured at beginning of implementation Initial baseline identifies areas to improve Unannounced pop quiz of TCIS fidelity Future fidelity studies can track progress

30 EBP Process and Context Measuring Fidelity to EBP Measured at beginning of implementation Initial baseline identifies areas to improve Unannounced pop quiz of EBP fidelity Future fidelity studies can track progress

31 Two Travis County Supervision Studies Felony Cohort (Jan Placements) Follow supervision practices for six months Focus on cases direct for at least three months Follow up: Cohort (Jan Placements) Sample of Felony-reduced and Misdemeanants

32 Study Descriptives Study 1 Study 2 1. Cohort Sample - Felons Felony-Reduced 17 - Misdemeanants Coders - Administrators X X - Operations-Research X X - Managers X

33 Objective Three: Identity the dimensions of community supervision that are important to measure (Fidelity study)

34 Purpose: To measure our fidelity to EBP

35 Objective Four: Describe methods of evaluating supervision fidelity

36 Case File Review Form Fidelity to EBP 36

37 Methodology Probationer Case File CSS Case File Review Form 37

38 Methodology Case File Review Form Database 38

39 Objective Five: Discuss approaches to present measures of supervision fidelity to the general public

40 Fidelity Study: Fidelity to EBP Results (Jan 2009)

41 Supervision Fidelity: Do we create accurate assessments? We have validated our risk assessment instrument three times We have conducted Inter-rater reliability tests on our Diagnostic Report Process

42 Do we create accurate assessments? 95% 95% 89% 89% We checked scoring on each diagnostic instrument for each person

43 Supervision Fidelity: What about the Judiciary / Courts?

44 Court-Ordered Conditions of Probation 90% 44

45 Supervision Fidelity: What about Supervision Process?

46 Do we focus on risk-needs and use motivational interviewing? We examined each point carefully. 46

47 Correct Caseload 97% Caseload match Probationer? - Maintenance Caseload - Regular Caseload - Correct Specialized 47

48 Supervision Agreement 85% Note: Some felony probationers were excluded from this analysis because they were unable to create a Supervision Agreement for various reasons (i.e. absconding, reoffending) 48

49 Supervision Agreement 85% 49

50 Responsivity / Motivation 50

51 Responsivity / Motivation We used a 5-point scale to measure Source: High and Medium Risk Felons Motivational Interviewing - 83% some use - 53% frequent use 51

52 Sanctions / Incentives 67% 52

53 Sanctions / Incentives 2009: 10 rush warrants - all 10 addressed properly - all 10 timely response 67% 53

54 Sanctions / Incentives 75% 54

55 Overview of Internal Studies 55

56 EBP Model: Fidelity and Outcomes Outcomes High Low Outcomes due to other factors Poorly Implemented Effective Program Ineffective Program Low High Fidelity 56

57 EBP Model: Fidelity and Outcomes Outcomes High Low Effective Program Low High Fidelity 57

58 What about Fidelity Outcomes? What happened to the probationers in this supervision study? Felony probationers: two year follow-up Revocations Arrests EBP Supervision appears correlated with better outcomes for felony probationers, with caveats Small felony sample Difficult to account for complicated interactions between variables

59 Overall Supervision Measure - Was the Supervision Appropriate? Data Source: Coder provided summary ranking of supervision quality.

60 The Supervision Agreement is more than just a piece of paper We looked for evidence that this document played a role in the supervision process. We examined if the document had the following: Probationer s signature Indications that the creation of the Supervision Agreement was a collaborative process Chronological Notes suggesting that the Supervision Agreement was the foundation for office visits

61 Did the Supervision Agreement show signs of EBP Yes: Signature, evidence of cooperation, evidence it was used as the foundation for office visits

62 Now we will move away from supervision to discuss SMART our Residential Drug Treatment Program.

63 Fidelity Evaluation: Drug Treatment What does our SMART Program do? Does it engage in Evidence-Based practices? Quality of work?

64 EBP Model: Fidelity and Outcomes Outcomes High Low Outcomes due to other factors Poorly Implemented Effective Program Ineffective Program Low High Fidelity 64

65 Prior SMART Residential Analysis Corrections Institute-University of Cincinnati Correctional Program Checklist (CPC) CPC 5 Domains - Leadership - Staff - Assessment -Treatment -Quality Assurance Highly Effective Effective Needs Improvement Ineffective 65

66 Corrections Institute University of Cincinnati SMART Program Rank (400+ Programs) 7% Highly Effective SMART Program 18% Effective 33% Needs Improvement 42% Ineffective Shaffer, DK and C Thompson. (2008). Evidence-Based Correctional Program Checklist SMART Program. University of Cincinnati. 66

67 SMART Residential Program 2004 Sept 2005 Sept 2006 Sept 2007 Sept 2008 Sept 2009 Sept Nov 2008: CPC Study: 3 yrs successful completions - 18 month Feb 2007 follow-up SMART Sep Aug 2008 SOMP start 18 Comparison Group Sep Aug

68 Creating Comparison Group Individual cases matched on seven factors Smart - Risk Score - Fiscal Year - Age Range - Race-Ethnicity - Gender - Offense Degree - Offense Type Comparison N = 559 N = 489 Many studies avoid risk / lack comparison groups 68

69 It is critical to match for Risk Score 9% 25% 67% - Felony Placements 0% 14% 86% - Smart 69

70 All Arrest Categories (by group) Note: this captures # of arrests, not # of people arrested. SMART: % decrease in total arrests 27 % fewer total arrests 12% fewer total felony arrests 15 % fewer total misd arrests 70

71 EBP Model: Fidelity and Outcomes Outcomes High Low Outcomes due to other factors Poorly Implemented Effective Program Ineffective Program Low High Fidelity 71