Standardized Program Evaluation Protocol [SPEP] Treatment Quality Rating Guide for Monitoring and Quality Improvement

Transcription

1 Standardized Program Evaluation Protocol [SPEP] Treatment Quality Rating Guide for Monitoring and Quality Improvement Florida Department of Juvenile Justice Bureau of Quality Improvement (Revised October 2015)

2 Table of Contents SPEP Quality of Service Delivery Ratings... 1 Service Classification... 1 Service Evaluation and Rating... 1 SPEP Quality of Service Delivery Indicators... 2 Indicator 1: Facilitator Training... 3 Indicator 2: Primary Service Manual/Protocol... 5 Indicator 3: Observed Adherence to Primary Service Manual/Protocol... 7 Indicator 4: Facilitator Turnover/Gaps in Service... 9 Indicator 5: Internal Fidelity Monitoring Indicator 6: Corrective Action Process Based on Fidelity Monitoring Indicator 7: Evaluation of Facilitator Skill Delivering the Primary Service SPEP Quality of Service Delivery Rating... 16

3 [SPEP] Treatment Quality Rating Guide 1 SPEP Quality of Service Delivery Ratings Service Classification A program offers many delinquency intervention and mental health/substance abuse treatment services. However, some of these services are not classified as a Primary services. The Department, in conjunction with the service provider, determine which delinquency intervention and mental health/substance abuse treatment services meet the requirements to be classified as a Primary Service. Service Evaluation and Rating Each delinquency intervention and mental health/substance abuse treatment service that is classified as a Primary service will be reviewed, evaluated, and rated annually by the Bureau of Monitoring and Quality Improvement. Therefore, each Primary Service gets its own Quality of Service Delivery Rating. Members of the Office of Program and Technical Assistance may also assist in the Quality of Service Delivery Rating. Observations of the primary service by Technical Assistance staff may occur outside of the annual review visit. NOTE: The review period for the Quality of Service Delivery Rating shall be one year from the date of the current annual review.

4 [SPEP] Treatment Quality Rating Guide 2 SPEP Quality of Service Delivery Indicators There are seven Quality of Service Delivery indicators that will be reviewed, evaluated, and rated annually. 1. Facilitator Training 2. Service Manual/Protocol 3. Observed Adherence to Primary Service Manual or Protocol 4. Facilitator Turnover/Gaps in Service 5. Internal Fidelity Monitoring 6. Corrective Action Process Based on Fidelity Monitoring 7. Facilitator Evaluations of Skill in Delivering the Service The guidelines and ratings are discussed further in the next section.

5 [SPEP] Treatment Quality Rating Guide 3 SPEP Quality of Service Delivery Evaluation Guidelines Indicator 1: Facilitator Training *This indicator shall be rated based on services as currently provided (at the time of the review). Guidelines 1. The facilitator has completed formal training, from a qualified trainer, in the Primary Service. 2. If more than one staff facilitates or co facilitates a Primary Service, all staff delivering the Primary Service must have completed formal training in the Primary Service. 3. Simply being licensed or supervised by a licensed clinician does not constitute training in a Primary Service. 4. Some Primary Services do not have formal training required; to be rated as 1, there must be evidence of a written training protocol and training in the Primary Service and should be documented as in service/on the job training hours in each staff training files or the Department s Learning Management System (SkillPro). 5. Individual counseling does not have a standardized formal training. The expectation is a written training protocol must capture how the program intends for individual sessions to be structured. Individual counseling training should be documented as in service/on the job training hours in each staff training files or the Department s Learning Management System (SkillPro). NOTE: The following services require verification by DJJ: Impact of Crime; Addressing the Harm to Victims and the Community (IOC) LifeSkills Training (LST) Thinking for a Change (T4C) The following names are interchangeable: MET/CBT 5 or MET/CBT 12 = Cannabis Youth Treatment (CYT) Boys Council = The Council for Boys and Young Men Rating = 0 or 1 1. The rating is 0: a. The facilitator has not completed formal training in the Primary Service; and,

6 [SPEP] Treatment Quality Rating Guide 4 b. Co facilitators (if any) have not completed formal training in the Primary Service. 2. The rating is 1: a. The facilitator completed formal training in the Primary Service; and, b. Co facilitators (if any) have completed formal training in the Primary Service. Examples: Example 1: Staff Jones completed the 4 day Thinking for a Change (T4C) facilitator training and is now running T4C groups two times per week. Rating = 1. Example 2: Staff Jones completed the 4 day Thinking for a Change (T4C) facilitator training and is now running T4C groups two times per week. Staff Doe co facilitates the T4C groups with Staff Jones but has not received the 4 day Thinking for a Change (T4C) facilitator training. Rating = 0.

7 [SPEP] Treatment Quality Rating Guide 5 Indicator 2: Primary Service Manual/Protocol *This indicator shall be rated based on services as currently provided (at the time of the review). Guidelines 1. Delinquency Intervention and Mental Health/Substance Abuse Group Sessions: a. Brand name interventions will have a manual for the facilitator to adhere to when running the intervention that contain lesson plans and/or instructions and addresses how the curriculum is to be implemented (e.g., number of facilitators per session, group member selection, group size, frequency, materials needed, lesson plans, session length, facilitator selection criteria, etc.). b. Some non brand name interventions may have a manual or protocol to adhere to when running the intervention. c. For those interventions that are not manualized, programs will have developed protocols and/or manuals for each intervention. The manual/protocol explicitly states the topics covered and the order in which they are covered, with instructions on how to facilitate each lesson or session. d. A manual for non curriculum based interventions states the premises of the intervention and the general progression of therapy. The manual states the parameters of the intervention and guides the therapy (e.g., FFT has several phases in which the youth progress consecutively). 2. Individual Counseling: a. Individual counseling (if a Primary Service) has a protocol documenting how therapists are to conduct individual sessions. b. The topics may not be defined but the structure of the session and requirements for the therapists are defined. Rating = 0, 1, or 2 1. The rating is 0: a. There is no manual or protocol for the delivery of the service; and, b. There is a non specific protocol stating that the service will be delivered and there is a general (not specific) description of the service. 2. The rating is 1: a. There is a manual/protocol specifying the frequency and duration of the service, a general outline or lesson plan that applies to each topic delivered and the order in which it the topics are to be delivered. 3. The rating is 2:

8 [SPEP] Treatment Quality Rating Guide 6 a. The manual/protocol is detailed to the extent that it explains how each specific session is delivered and there is a script or outline for each session (refer to the examples in item 1 of the above guidelines). Brand Name Example Example 1: The program provides the evidence based curriculum LifeSkills Training to all youth admitted to the program. The program has on site the LifeSkills Training Middle School Teachers Manual and 12 LifeSkills Training Student Guides. The manual contains the order of the lessons to be delivered, lesson plans for each session, and implementation (delivery) guidelines. RATING = 2. Generic Program Example: Example 1: The program provides the generic curriculum Male Responsibility is My Responsibility to all youth admitted the program. The program has on site the Male Responsibility is My Responsibility Facilitator and Student Guides. The manual contains lesson plans with basic instructions for each lesson. RATING = 2. Example 2: Counseling group will be delivered two times per week. Each session will have a check in, a worksheet, and a processing component. The topics covered are in order, as follows: introduction; knowing your feelings; goal setting; problem solving; triggers; relapse prevention; conclusion/certificates. There is no individual outline for each session. Rating= 1. Example 3: Counseling group will be delivered two times per week, but there is no written protocol for the therapist to follow. Counseling group is a service where a personal relationship is developed with a responsible therapist who attempts to exercise influence on the juvenile s feelings, cognitions, and behavior. Rating = 0.

9 [SPEP] Treatment Quality Rating Guide 7 Indicator 3: Observed Adherence to Primary Service Manual/Protocol *This indicator shall be rated based on services as currently provided (at the time of the review). Guidelines 1. The MQI Reviewer is to interview (informal) the facilitator(s) of the Primary Service to be observed as to whether or not there is a manual/protocol for the lesson/session/topic to be observed. Interview Questions that may be used: a. Please show me the manual or protocol used during your delivery of. b. How do you know which lesson/session of the manual to deliver on any given day? c. Explain to me how you choose which lesson to deliver. d. Can you bring in additional materials from other sources? If so, to what extent? 2. The MQI reviewer should obtain a copy of the manual/protocol before the session begins to review implementation guidelines (delivery requirements) and the lesson plan or outline, including trainer notes. Be sure to ask the facilitator what lesson/session/topic they will be delivering during your observation. 3. The MQI Reviewer is to observe a group session of each intervention rated as a Primary Service during the QI Review period, except individual mental health counseling. This observation is to enable the MQI Reviewer to determine if the facilitator adheres and follows to the manual/protocol. NOTE: If the MQI Reviewer is not able to observe the Primary Service during the annual review, it may be observed by a Technical Assistance Specialist up to 14 days after the annual review. Fidelity Monitoring completed by a Technical Assistance Specialist within 60 days prior to the annual review may be substituted for the observation during the review, for interventions supported by the Office of Programming and Technical Assistance (Thinking For a Change, LifeSkills, and Impact of Crime). If the Primary Service is not able to be observed, the indicator shall be rated as N/A. The standard language when rating as N/A shall be, This curriculum was not able to be observed; therefore, this indicator rates as non applicable. Additionally, please document information from the facilitator(s) interview and reason why the observation could not take place in the narrative. Rating = 0 or 1 1. The rating is 0 if the facilitator did not adhere to the manual/protocol or there was no manual/protocol for the facilitator to use.

10 [SPEP] Treatment Quality Rating Guide 8 2. The rating is 1 if the facilitator adhered to the manual/protocol. Examples Example 1: An MQI Reviewer, with a copy of the lesson plan in hand, observes a Thinking for a Change group session and follows along as a facilitator teaches the material. The MQI Reviewer notices that the facilitator does not read the script verbatim during each activity, but uses his/her own words to convey the meaning of the topic under discussion, and does so correctly. However, the MQI Reviewer notices that the facilitator skips several activities including setting up the role plays and requiring group member feedback. RATING = 0. Example 2: An MQI Reviewer, with a copy of the lesson plan in hand, observes a Thinking for a Change group session and follows along as a facilitator teaches the material. The MQI Reviewer notices that the facilitator does not read the script verbatim during each activity, but uses his/her own words to convey the meaning of the topic under discussion, and does so correctly. The facilitator adheres to the lesson plan in every other respect, but does not complete the lesson by the end of the session. RATING = 1.

11 [SPEP] Treatment Quality Rating Guide 9 Indicator 4: Facilitator Turnover/Gaps in Service *This indicator shall be rated based on services that have been applied and demonstrated throughout the 12 month SPEP period (in keeping with the existing QI applied and demonstrated criteria). Guidelines 1. Determine the extent to which facilitators of the Primary Service have changed within the SPEP evaluation period (1 year back from time of review). Focus should be placed on the exit or departure of the facilitator from the program during a group cycle. 2. Interview Clinical/Treatment Director and Facilitators of the Primary Service using Motivational Interviewing skills. 3. Examine attendance/sign in sheets or the JJIS Evidence Based Services Module for gaps in delivery of the Primary Service. 4. Some gaps in service may be due to the closed group nature of many interventions; 5. Gaps in the middle of a group are indicative of turnover issues. Rating = 0, 1 or 2 1. The rating is 0: a. There was a gap in delivery of the Primary Service for longer than one month due to facilitator turnover. b. The therapist leaves a youth or family during group/treatment without transitioning the youth to a new therapist. 2. The rating is 1: a. There has been a gap in the Primary Service for more than two weeks, but less than one month. b. The facilitator of the Primary Service changed during service provision to a youth/group of youth, but the service continued uninterrupted. c. The therapist leaves a youth or family after transitioning the youth to a new therapist, and the transition is documented in the case notes. 3. The rating is 2: a. There has been no gap in service delivery and no turnover of facilitator/therapist since the last SPEP evaluation. Turnover that occurs in between group cycles is most appropriate and does not impact the rating. For groups being co facilitated, both facilitators should be documenting their participation in group consistently, for a majority of group sessions.

12 [SPEP] Treatment Quality Rating Guide 10 Note: some gaps in service, even over 1 month, are due to the service not being needed during the gap. Examples include waiting for enough youth to begin a new closed group, no youth entering the program with a need for the service. For any gaps in service greater than 1 month, the MQI reviewer should inquire further as to the reasons for the gap, assess those reasons in formulation of the rating, and document the reasons in the narrative. Examples Example 1: Trained facilitators leave or are reassigned and the Primary Service is stopped or discontinued. RATING = 0. Example 2: The facilitator of anger management groups left the program in March. A trained therapist began conducting the groups in June. RATING= 0 (gap longer than 1 month). Example 3: The facilitator of a T4C group changed after lesson 5; a trained facilitator took over and conducted lessons 6 25 with no break in service. RATING= 1 (facilitator changed during service provision). Example 4: A trained facilitator completed ART with a group of 10 youth and began a new ART group with 10 youth two months later. All youth completed ART prior to release. RATING = 2 (gap in service was due to closed group requirement and not enough youth in program to start a new group).

13 [SPEP] Treatment Quality Rating Guide 11 Indicator 5: Internal Fidelity Monitoring *This indicator shall be rated based on services that have been applied and demonstrated throughout the 12 month SPEP period (in keeping with the existing QI applied and demonstrated criteria). Guidelines 1. Staff members responsible for performing fidelity monitoring have completed the formal training, delivered by a qualified trainer, in the Primary Service, or were the developers of the Primary Service. 2. The program has a documented fidelity monitoring observation and reporting process. This may include detailing how trained staff (identified in #1 above) will document observations, findings, and corrective action required, if any, as well as, how closely actual implementation matches the manual/protocol. 3. If using a brand name or other manualized Primary Service, staff members responsible for performing fidelity monitoring follow the manual/protocol (e.g., lesson plan, outline) to ensure the facilitator delivers each component (lessons and elements of each lesson) of the curriculum in the order and manner prescribed. 4. For individual counseling, detailed supervision case notes related to what was discussed in the sessions are used for fidelity monitoring. NOTE: Fidelity monitoring provided by Bureau of Monitoring and Quality Improvement staff does not qualify as internal fidelity monitoring at the program, nor does monitoring one s self of one s own delivery of an intervention. The program should establish and practice their own process for fidelity monitoring beyond the Department s assistance. The link to a sample of fidelity monitoring checklists can be found: Rating = 0, 1 or 2 1. The rating is 0: a. Staff members performing fidelity monitoring have not been trained, by a qualified trainer, in the delivery of the Primary Service; b. Fidelity monitoring has not been performed or completed, or has not been performed or completed more than one time in the last 12 months; c. Fidelity monitoring may have been completed, but is not documented; d. Supervision notes do not specifically discuss events that actually happened during individual counseling sessions; or,

14 [SPEP] Treatment Quality Rating Guide 12 e. The rating is 0 if generic supervision notes do not specifically discuss what occurred within the mental health group or individual counseling session. 2. The rating is 1: a. Staff members performing fidelity monitoring have been trained, by a qualified trainer, in the Primary Service AND b. Fidelity monitoring of the Primary Service has occurred twice or more in the last 12 months as evidenced by fidelity monitoring reports or checklists specific to the intervention; or c. Supervision notes specifically discuss what occurred during the individual counseling being provided. 3. The rating is 2 (all three of these must occur): a. Staff members performing fidelity monitoring have been trained, by a qualified trainer, in the Primary Service AND b. Internal fidelity monitoring reports, checklists, supervision notes are specific to the Primary Service AND c. Internal fidelity monitoring of the Primary Service is scheduled and occurs at regular intervals (at a minimum once a month per facilitator for those delivering group).

15 [SPEP] Treatment Quality Rating Guide 13 Indicator 6: Corrective Action Process Based on Fidelity Monitoring *This indicator shall be rated based on services that have been applied and demonstrated throughout the 12 month SPEP period (in keeping with the existing QI applied and demonstrated criteria). Guidelines 1. The program has a process by which corrective action is applied and demonstrated based on the fidelity monitoring of the Primary Service. 2. Corrective action may include: training/re training of staff, increased frequency of fidelity monitoring and coaching and co facilitation. Rating = 0 or 1 1. The rating is 0: a. Indicator 5, Internal Fidelity Monitoring, is rated as 0 or 1; b. Fidelity monitoring has been completed twice or more in the last 12 months, but has not taken place during the last six months (if corrective action has been applied and demonstrated, the rating should be 1); c. Corrective action is not applied and demonstrated, based on the documented findings in the fidelity monitoring report; and, d. There is no documentation of deficiencies being corrected. 2. The rating is 1: a. Indicator 5, Internal Fidelity Monitoring, is rated as 1 or 2; b. Corrective action is applied and demonstrated, based on the documented findings in the fidelity monitoring report; c. There is evidence that the program responded to the results of the fidelity monitoring report and made changes to the implementation or delivery of the Primary Service, for example, facilitator was re trained, groups are now cofacilitated; d. Corrective action is in direct response to the findings of fidelity monitoring report; e. There was discussion and documentation of findings during fidelity monitoring that led to the corrective action; f. There is documentation that identified deficiencies have been corrected, such as more recent fidelity monitoring reports indicate progress in delivery of service; and g. There is documentation that corrective action was not required (ex. fidelity monitoring reported perfect adherence).

16 [SPEP] Treatment Quality Rating Guide 14 Examples Example 1: Fidelity monitoring discusses the need for youth to bring homework to group, corrective action included scheduling time during the day for youth to complete homework and have it checked by staff. Rating = 1. Example 2: Fidelity monitoring was only completed three times for one facilitator during the past year. No fidelity monitoring has taken place during the last seven months. Rating = 0.

17 [SPEP] Treatment Quality Rating Guide 15 Indicator 7: Evaluation of Facilitator Skill Delivering the Primary Service *This indicator shall be rated based on services that have been applied and demonstrated throughout the 12 month SPEP period (in keeping with the existing QI applied and demonstrated criteria). Guidelines 1. Performance evaluations of all facilitators of the Primary Service include evaluation of skill in delivering the Primary Service. 2. Performance evaluations should be reviewed for all facilitators of the Primary Service during the review period. Rating = 0 or 1 1. The rating is 0: a. There is no evidence of annual evaluations; or, b. There is evidence of annual evaluations, but they do not include assessment of the facilitator specifically on facilitation of the Primary Service. 2. The rating is 1: a. There is evidence of annual performance evaluations of facilitators including assessment of the facilitators proficiency in delivering the Primary Service. NOTE: 1. If facilitator does not yet have an annual evaluation, a probationary evaluation will suffice. 2. If facilitator is not due for a probationary evaluation, the rating would be N/A only for that particular facilitator. 3. If all facilitators of the Primary Service are not due for a probation/annual evaluation, the rating would be N/A only for that particular Primary Service.

18 [SPEP] Treatment Quality Rating Guide 16 SPEP Quality of Service Delivery Rating Each Primary Service will be assigned ratings on all seven Quality of Service Delivery Indicators. 1. Facilitator Training (0, 1) 2. Primary Service Manual/Protocol (0, 1, 2) 3. Observed Adherence to the Primary Service Manual/Protocol (0, 1) 4. Facilitator Turnover/Gaps in Service (0, 1, 2) 5. Internal Fidelity Monitoring (0, 1, 2) 6. Corrective Action Process Based on Fidelity Monitoring (0, 1) 7. Evaluation of Facilitator Skill Delivering the Primary Service (0, 1) Each Primary Service will be scored overall as Low, Medium, or High by summing the ratings of the seven indicators based on the table displayed below. Rating Total Rating Points Low 0 3 Medium 4 7 High 8 10