Standardized Program Evaluation Protocol [SPEP] Treatment Quality Rating Guide

Transcription

1 Standardized Program Evaluation Protocol [SPEP] Treatment Quality Rating Guide Florida Department of Juvenile Justice Bureau of Quality Improvement (Revised May 2014)

2 Table of Contents SPEP Quality of Service Delivery Ratings...1 Service Classification...1 Service Evaluation and Rating...1 SPEP Quality of Service Delivery Indicators...2 Indicator 1: Facilitator Training...3 Indicator 2: Primary Service Manual/Protocol...5 Indicator 3: Observed Adherence to Primary Service Manual/Protocol...7 Indicator 4: Facilitator Turnover...9 Indicator 5: Internal Fidelity Monitoring Indicator 6: Corrective Action Based on Fidelity Monitoring Indicator 7: Evaluation of Facilitator Skill Delivering the Primary Service SPEP Quality of Service Delivery Rating... 15

3 [SPEP] Treatment Quality Rating Guide 1 SPEP Quality of Service Delivery Ratings Service Classification A program offers many delinquency intervention and mental health/substance abuse treatment services. However, some of these services are not classified as a Primary services. Technical Assistance staff members determine which delinquency intervention and mental health/substance abuse treatment services meet the requirements to be classified as a Primary Service. Service Evaluation and Rating Each delinquency intervention and mental health/substance abuse treatment service that is classified as a Primary service will be reviewed, evaluated, and rated annually by the Bureau of Quality Improvement. Therefore, each Primary Service gets its own Quality of Service Delivery Rating. For example, the program has three intervention services that were rated as Primary Services: Thinking for a Change (T4C), Young Men s Work, and Individual Counseling. The programs three Primary services would each be reviewed, evaluated, and rated on Quality of Service Delivery and therefore, would receive three ratings one rating for each service. NOTE: The review period for the Quality of Service Delivery Rating shall be one year from the date of the current Quality Improvement review.

4 [SPEP] Treatment Quality Rating Guide 2 SPEP Quality of Service Delivery Indicators There are seven Quality of Service Delivery indicators that will be reviewed, evaluated, and rated annually. 1. Facilitator Training 2. Service Manual/Protocol 3. Observed Adherence to Primary Service Manual or Protocol 4. Facilitator Turnover 5. Internal Fidelity Monitoring 6. Corrective Action Based on Fidelity Monitoring 7. Facilitator Evaluations of Skill in Delivering the Service The guidelines and ratings are discussed further in the next section.

5 [SPEP] Treatment Quality Rating Guide 3 SPEP Quality of Service Delivery Evaluation Guidelines Indicator 1: Facilitator Training *This indicator shall be rated based on services as currently provided (at the time of the review). Guidelines 1. The facilitator has completed formal training, from a qualified trainer, in the Primary Service. 2. If more than one staff facilitates or co-facilitates a Primary Service, all staff delivering the Primary Service must have completed formal training in the Primary Service. 3. Simply being licensed or supervised by a licensed clinician does not constitute training in a Primary Service. 4. Some Primary Services do not have formal training required; to be rated as 1, there must be evidence of a written training protocol and training in the Primary Service and document as in-service/on the job training hours in each staff training files or the Department s Learning Management System. 5. Individual counseling does not have a standardized formal training. The expectation is that a written training protocol must capture how the program intends for individual sessions to be structured. Individual counseling training should be documented as inservice/on the job training hours in each staff training files or the Department s Learning Management System. NOTE: The following services have a formal training required by DJJ: Aggression Replacement Training (ART) Boys Council (The Council) Cannabis Youth Treatment (CYT) (also called MET/CBT) Dialectical Behavioral Therapy (DBT) Girls Moving On (GMO) Impact of Crime; Addressing the Harm to Victims and the Community (IOC) LifeSkills Training (LST) Moral Reconation Therapy (MRT) Seven Challenges Thinking for a Change (T4C) T4C 2.0 has 22 lessons and T4C 3.1 has 25 lessons The following names are interchangeable:

6 [SPEP] Treatment Quality Rating Guide 4 MET/CBT 5 or MET/CBT 12 = Cannabis Youth Treatment (CYT) Boys Council=The Council Rating = 0 or 1 1. The rating is 0: a. The facilitator has not completed formal training in the Primary Service; and, b. Co-facilitators (if any) have not completed formal training in the Primary Service. 2. The rating is 1: a. The facilitator completed formal training in the Primary Service; and, b. Co-facilitators (if any) have completed formal training in the Primary Service. Examples: Example 1: Staff Jones completed the 4-day Thinking for a Change (T4C) facilitator training and is now running T4C groups two times per week. Rating = 1. Example 2: Staff Jones completed the 4-day Thinking for a Change (T4C) facilitator training and is now running T4C groups two times per week. Staff Doe co-facilitates the T4C groups with Staff Jones but has not received the 4-day Thinking for a Change (T4C) facilitator training. Rating = 0.

7 [SPEP] Treatment Quality Rating Guide 5 Indicator 2: Primary Service Manual/Protocol *This indicator shall be rated based on services as currently provided (at the time of the review). Guidelines 1. Delinquency Intervention and Mental Health/Substance Abuse Group Sessions: a. Brand-name interventions will have a manual for the facilitator to adhere to when running the intervention that contain lesson plans and/or instructions and addresses how the curriculum is to be implemented (e.g., number of facilitators per session, group member selection, group size, frequency, materials needed, lesson plans, session length, facilitator selection criteria, etc.). b. Some non-brand-name interventions may have a manual or protocol to adhere to when running the intervention. c. For those interventions that are not manualized, programs will have developed protocols and/or manuals for each intervention. The manual/protocol explicitly states the topics covered and the order in which they are covered, with instructions on how to facilitate each lesson or session. d. A manual for non-curriculum-based interventions states the premises of the intervention and the general progression of therapy. The manual states the parameters of the intervention and guides the therapy (e.g., FFT has several phases in which the youth progress consecutively). 2. Individual Counseling: a. Individual counseling (if a Primary Service) has a protocol documenting how therapists are to conduct individual sessions. b. The topics may not be defined but the structure of the session and requirements for the therapists are defined. Rating = 0, 1, or 2 1. The rating is 0: a. There is no manual or protocol for the delivery of the service; and, b. There is a non-specific protocol stating that the service will be delivered and there is a general (not specific) description of the service. 2. The rating is 1: a. There is a manual/protocol specifying the frequency and duration of the service, a general outline or lesson plan that applies to each topic delivered and the order in which it the topics are to be delivered. 3. The rating is 2:

8 [SPEP] Treatment Quality Rating Guide 6 a. The manual/protocol is detailed to the extent that it explains how each specific session is delivered and there is a script or outline for each session (refer to the examples in item 1 of the above guidelines). Brand Name Example Example 1: The program provides the evidence-based curriculum LifeSkills Training to all youth admitted to the program. The program has on site the LifeSkills Training Middle School Teachers Manual and 12 LifeSkills Training Student Guides. The manual contains the order of the lessons to be delivered, lesson plans for each session, and implementation (delivery) guidelines. RATING = 2. Generic Program Example: Example 1: The program provides the generic curriculum Male Responsibility is My Responsibility to all youth admitted the program. The program has on site the Male Responsibility is My Responsibility Facilitator and Student Guides. The manual contains lesson plans with basic instructions for each lesson. RATING = 2. Example 2: Counseling group will be delivered two times per week. Each session will have a check in, a worksheet, and a processing component. The topics covered are in order, as follows: introduction; knowing your feelings; goal setting; problem-solving; triggers; relapse prevention; conclusion/certificates. There is no individual outline for each session. Rating= 1. Example 3: Counseling group will be delivered two times per week, but there is no written protocol for the therapist to follow. Counseling group is a service where a personal relationship is developed with a responsible therapist who attempts to exercise influence on the juvenile s feelings, cognitions, and behavior. Rating = 0.

9 [SPEP] Treatment Quality Rating Guide 7 Indicator 3: Observed Adherence to Primary Service Manual/Protocol *This indicator shall be rated based on services as currently provided (at the time of the review). Guidelines 1. The QI Reviewer is to interview (informal) the facilitator(s) of the Primary Service to be observed as to whether or not there is a manual/protocol for the lesson/session/topic to be observed. Interview Questions that may be used: a. Please show me the manual or protocol used during your delivery of. b. How do you know which lesson/session of the manual to deliver on any given day? c. Explain to me how you choose which lesson to deliver. d. Can you bring in additional materials from other sources? If so, to what extent? 2. The QI reviewer should obtain a copy of the manual/protocol before the session begins to review implementation guidelines (delivery requirements) and the lesson plan or outline, including trainer notes. Be sure to ask the facilitator what lesson/session/topic they will be delivering during your observation. 3. The QI Reviewer is to observe a group session of each intervention rated as a Primary Service during the QI Review period, except individual mental health counseling. This observation is to enable the QI Reviewer to determine if the facilitator adheres and follows to the manual/protocol. NOTE: If the Primary Service is not delivered during the time of the Quality Improvement review, the indicator shall be rated as N/A. The standard language when rating as N/A shall be, This curriculum was not able to be observed during the review week; therefore, this indicator rates as non-applicable. Additionally, please document information from the facilitator(s) interview in the narrative. Rating = 0 or 1 1. The rating is 0 if the facilitator did not adhere to the manual/protocol or there was no manual/protocol for the facilitator to use. 2. The rating is 1 if the facilitator adhered to the manual/protocol. Examples Example 1: A QI Reviewer, with a copy of the lesson plan in hand, observes a Thinking for a Change group session and follows along as a facilitator teaches the material. The QI

10 [SPEP] Treatment Quality Rating Guide 8 Reviewer notices that the facilitator does not read the script verbatim during each activity, but uses his/her own words to convey the meaning of the topic under discussion, and does so correctly. However, the QI Reviewer notices that the facilitator skips several activities including setting up the role-plays and requiring group member feedback. RATING = 0. Example 2: A QI Reviewer, with a copy of the lesson plan in hand, observes a Thinking for a Change group session and follows along as a facilitator teaches the material. The QI Reviewer notices that the facilitator does not read the script verbatim during each activity, but uses his/her own words to convey the meaning of the topic under discussion, and does so correctly. The facilitator adheres to the lesson plan in every other respect, but does not complete the lesson by the end of the session. RATING = 1.

11 [SPEP] Treatment Quality Rating Guide 9 Indicator 4: Facilitator Turnover *This indicator shall be rated based on services that have been applied and demonstrated throughout the 12 month SPEP period (in keeping with the existing QI applied and demonstrated criteria). Guidelines 1. Determine the extent to which facilitators of the Primary Service have changed within the SPEP evaluation period (1 year back from time of review). 2. Interview Clinical/Treatment Director and Facilitators of the Primary Service using Motivational Interviewing skills. 3. Examine attendance/sign-in sheets or the JJIS Evidence-Based Services Module for gaps in delivery of the Primary Service. 4. Some gaps in service may be due to the closed group nature of many interventions; 5. Gaps in the middle of a group are indicative of turnover issues. Rating = 0, 1 or 2 1. The rating is 0: a. There was a gap in delivery of the Primary Service for longer than one month due to facilitator turnover. b. The therapist leaves a youth or family without transitioning the youth to a new therapist. 2. The rating is 1: a. There has been a gap in the Primary Service for more than two weeks, but less than one month, due to facilitator turnover. b. The facilitator of the Primary Service changed during service provision to a youth/group of youth, but the service continued uninterrupted. c. The therapist leaves a youth or family after transitioning the youth to a new therapist, and the transition is documented in the case notes. 3. The rating is 2: a. There has been no gap in service delivery and no turnover of facilitator/therapist since the last SPEP evaluation (Note: some gaps in service, even over 1 month, are due to the service not being needed during the gap. Examples include waiting for enough youth to begin a new closed group, no youth entering the program with a need for the service). Examples

12 [SPEP] Treatment Quality Rating Guide 10 Example 1: Trained facilitators leave or are reassigned and the Primary Service is stopped or discontinued. RATING = 0. Example 2: The facilitator of anger management groups left the program in March. A trained therapist began conducting the groups in June. RATING= 0 (gap longer than 1 month). Example 3: The facilitator of a T4C group changed after lesson 5; a trained facilitator took over and conducted lessons 6-25 with no break in service. RATING= 1 (facilitator changed during service provision). Example 4: A trained facilitator completed ART with a group of 10 youth and began a new ART group with 10 youth two months later. All youth completed ART prior to release. RATING = 2 (gap in service was due to closed group requirement).

13 [SPEP] Treatment Quality Rating Guide 11 Indicator 5: Internal Fidelity Monitoring *This indicator shall be rated based on services that have been applied and demonstrated throughout the 12 month SPEP period (in keeping with the existing QI applied and demonstrated criteria). Guidelines 1. Staff members responsible for performing fidelity monitoring have completed the formal training, delivered by a qualified trainer, in the Primary Service, or were the developers of the Primary Service. 2. The program has a documented fidelity monitoring observation and reporting process. This may include detailing how trained staff (identified in #1 above) will document observations, findings, and corrective action required, if any, as well as, how closely actual implementation matches the manual/protocol. 3. If using a brand-name or other manualized Primary Service, staff members responsible for performing fidelity monitoring follow the manual/protocol (e.g., lesson plan, outline) to ensure the facilitator delivers each component (lessons and elements of each lesson) of the curriculum in the order and manner prescribed. 4. For individual counseling, detailed supervision case notes related to what was discussed in the sessions are used for fidelity monitoring. NOTE: Fidelity monitoring provided by Bureau of Quality Improvement staff does not qualify as institutionalized fidelity monitoring at the program, nor does monitoring one s self of one s own delivery of an intervention. The link to a sample of fidelity monitoring checklists can be found: Rating = 0, 1 or 2 1. The rating is 0: a. Staff members performing fidelity monitoring have not been trained, by a qualified trainer, in the delivery of the Primary Service; b. Fidelity monitoring has not be performed or completed; c. Fidelity monitoring may have been completed, but is not documented; d. Supervision notes do not specifically discuss events that actually happened during individual counseling sessions; or, e. The rating is 0 if generic supervision notes do not specifically discuss what occurred within the mental health group or individual counseling session. 2. The rating is 1:

14 [SPEP] Treatment Quality Rating Guide 12 a. Staff members performing fidelity monitoring have been trained, by a qualified trainer, in the Primary Service AND b. Fidelity monitoring of the Primary Service is occurring as evidenced by fidelity monitoring reports or checklists specific to the intervention; or c. Supervision notes specifically discuss what occurred during the individual counseling being provided. 3. The rating is 2 (all three of these must occur): a. Staff members performing fidelity monitoring have been trained, by a qualified trainer, in the Primary Service AND b. Internal fidelity monitoring reports, checklists, supervision notes are specific to the Primary Service AND c. Internal fidelity monitoring of the Primary Service is scheduled and occurs at regular intervals (at a minimum once a month per facilitator for those delivering group).

15 [SPEP] Treatment Quality Rating Guide 13 Indicator 6: Corrective Action Based on Fidelity Monitoring *This indicator shall be rated based on services that have been applied and demonstrated throughout the 12 month SPEP period (in keeping with the existing QI applied and demonstrated criteria). Guidelines 1. The program has a process by which corrective action is applied and demonstrated based on the fidelity monitoring of the Primary Service. 2. Corrective action may include: training/re-training of staff, increased frequency of fidelity monitoring and coaching and co-facilitation. Rating = 0 or 1 1. The rating is 0: a. Indicator 5, Internal Fidelity Monitoring, is rated as 0; b. If corrective action is not applied and demonstrated, based on the documented findings in the fidelity monitoring report; and, c. There is no documentation of deficiencies being corrected. 2. The rating is 1: a. Indicator 5, Internal Fidelity Monitoring, is rated as 1; b. Corrective action is applied and demonstrated, based on the documented findings in the fidelity monitoring report; c. There is evidence that the program responded to the results of the fidelity monitoring report and made changes to the implementation or delivery of the Primary Service, for example, facilitator was re-trained, groups are now cofacilitated; d. Corrective action is in direct response to the findings of fidelity monitoring report; e. There was discussion and documentation of findings during fidelity monitoring that led to the corrective action; f. There is documentation that identified deficiencies have been corrected, such as more recent fidelity monitoring reports indicate progress in delivery of service; and g. There is documentation that corrective action was not required (ex. fidelity monitoring reported perfect adherence). Examples Example 1: Fidelity monitoring discusses the need for youth to bring homework to group, corrective action included scheduling time during the day for youth to complete homework and have it checked by staff. Rating = 1.

16 [SPEP] Treatment Quality Rating Guide 14 Indicator 7: Evaluation of Facilitator Skill Delivering the Primary Service *This indicator shall be rated based on services that have been applied and demonstrated throughout the 12 month SPEP period (in keeping with the existing QI applied and demonstrated criteria). Guidelines 1. Performance evaluations of all facilitators of the Primary Service include evaluation of skill in delivering the Primary Service. 2. Performance evaluations should be reviewed for all facilitators of the Primary Service during the review period. Rating = 0 or 1 1. The rating is 0: a. There is no evidence of annual evaluations; or, b. There is evidence of annual evaluations, but they do not include assessment of the facilitator specifically on facilitation of the Primary Service. 2. The rating is 1: a. There is evidence of annual performance evaluations of facilitators including assessment of the facilitators proficiency in delivering the Primary Service. NOTE: 1. If facilitator does not yet have an annual evaluation, a probationary evaluation will suffice. 2. If facilitator is not due for a probationary evaluation, the rating would be N/A only for that particular facilitator. 3. If all facilitators of the Primary Service are not due for a probation/annual evaluation, the rating would be N/A only for that particular Primary Service.

17 [SPEP] Treatment Quality Rating Guide 15 SPEP Quality of Service Delivery Rating Each Primary Service will be assigned ratings on all seven Quality of Service Delivery Indicators. 1. Facilitator Training (0, 1) 2. Primary Service Manual/Protocol (0, 1, 2) 3. Observed Adherence to the Primary Service Manual/Protocol (0, 1) 4. Facilitator Turnover (0, 1, 2) 5. Internal Fidelity Monitoring (0, 1, 2) 6. Corrective Action based on Fidelity Monitoring (0, 1) 7. Evaluation of Facilitator Skill Delivering the Primary Service (0, 1) Each Primary Service will be scored overall as Low, Medium, or High by summing the ratings of the seven indicators based on the table displayed below. Rating Total Rating Points Low 0 3 Medium 4 7 High 8 10