1 Program Description: PROGRAM EVALUATION GRANT AWARD 2006 GOAL SETTING AND CAFAS EVALUATION Program Evaluation Grant # 162606-051 As a non-profit Children s Mental Health Center with a long history of serving youth and their families, we provide a continuum of services from Crisis/Intake, Day treatment, Intensive In-home services to Residential Treatment and Rehabilitative Custody Programs. The Brief Child and Family Phone Interview (BCFPI) is used as an intake triaging tool and the Child and Adolescent Functional Assessment Scale (CAFAS) as an admission assessment and an outcome measurement tool. CAFAS is administered for all youth in our Ministry of Children and Youth Services funded programs, as well as for the youth receiving residential treatment in our outside paid residences and youth receiving rehabilitative programming within our Youth Justice residences. We have a multidisciplinary implementation steering committee to assist our practitioners integrate these tools into their everyday work with youth and to inform decision makers. The Program Evaluation grant has been beneficial, as it has allowed us to advance its data analysis, information sharing and knowledge exchange efforts. This grant allowed us to intensely focus on the analysis of our goal setting areas compared to the Child and Adolescent Functional Assessment Scale areas. Expected outcomes and deliverables: 1) An increase in specific knowledge related to CAFAS evaluation and outcome measurement. The Child and Adolescent Functional Assessment Scale (CAFAS) is currently used as an admission assessment tool and an outcome measurement tool across the agency. Familiarity and comfort in using the tool, although increasing, continues to vary with staff who are identified as primary workers for youth. Staff have differing degrees of knowledge related to CAFAS and integrated goal setting. This variability of knowledge is related to a couple of factors, including newness in the role and staff assignment changes moving staff into areas where it is used, whereas in their previous assignment with dually diagnosed youth it had not been used. Some staff acknowledged reviewing information available at the time of the initial Plan of Care meeting, including the CAFAS, Brief Child and Family Phone Interview (BCFPI) and Risk Needs Assessment, in addition to other reports, for example probation reports, psychiatric evaluation, education reports, or discharge reports from the facility where the youth previously resided. Other staff were non committal in their knowledge and use of the CAFAS. In order to increase internal capacity and knowledge in CAFAS evaluation and outcome measurement, a training package was developed to educate individuals and small groups
2 of front line residential staff, who are involved in a primary care role with youth, and thus responsible for developing and documenting goals within the case plan. The CAFAS training and goal setting packages included: background information on CAFAS; how it is used at the agency; goal setting framework; demonstration and discussion of real examples in individual goal planning, program outcomes and outcomes for the agency. Team meetings were initially targeted as the best option to provide training to staff. Presentations occurred at three program meetings. In the training it became evident that individualized one-on-one tutoring and mentoring would further advance the knowledge of some staff and provide a better venue for training others. Nineteen staff and supervisors received the CAFAS training through individual or small groups. This training material and model has been well established and will continue further into the future through our CAFAS implementation team members. The training package will be placed on our Intranet as an additional resource. A hard copy of the training package, with attached samples will be given to managers for each of the residential units. 2) Analysis of the congruence between contracted goals and CAFAS problem areas Evaluation occurred through an analysis of our goal setting methods in comparison to our ministry mandated assessment and outcome measurement tool. As identified in the program evaluation award, our practitioners struggle with reconciling the different assessment perspectives, youth and family wishes in goal setting with the knowledge of outcome data being collected through CAFAS (page 3). Much work has and is being done in this area within the agency. As part of this project, an examination of problem areas identified in CAFAS, BCFPI and the Risk/Needs Assessment was analyzed. A chart was developed which clearly establishes the links between assessment areas identified by CAFAS, BCFPI and RNA. Twelve assessment areas were identified. Each assessment area has a set of possible goals that can be utilized which would be specific to the individual needs of the youth and their family and their wishes. Methodology to achieve the goals would be further individualized. Additionally, specific research occurred on our goal setting methods in comparison to our Ministry mandated assessment and outcome measurement tool - CAFAS. Specific analysis examined the correlation between goal setting and CAFAS. This was accomplished by reviewing our contracted client goal areas to determine congruence with CAFAS problem areas. A convenient sample of thirty individual cases was reviewed. These cases included those reported to CAFAS in Ontario, as well as those the agency has chosen to collect data on that are not mandated to report. Thus, a broader scope of cases was included, beyond the Ministry mandated cases. Youth admitted to treatment after January 1, 2005 were included in the review. In all of the cases reviewed, the youth were over 12 years of age at the time of admission and included both males and females, although a significantly higher number of males were
3 represented. The goals reviewed were from their initial Plan of Care. The CAFAS scores that were used were their admission CAFAS scores, that is, rated on their behaviour thirty days prior to admission. One clinician completed an extensive audit of cases manually so that the subjective rating of the goals to CAFAS scales was consistent, with only one assessor making the determination. Consultation occurred with a second clinician regarding the approach that would be taken to review the data. Goals were taken at face value as written, rather than reviewed to determine if they were truly a goal or an objective. As noted, further work is being done in the area of goals setting. There was wide variance in the goals as written. There appears to be a variance in understanding regarding goals, methodology and outcomes as written in some plans of care. This is an area being specifically addressed through staff training as previously indicated. Some treatment areas were captured in other sections of the Plan of Care. Treatment issues relating to substance use or inappropriate sexual behaviour may have only be addressed within Specialized Services section of the Plan of Care, i.e. Drug and Alcohol Counseling or Sex Offender Group. Also, for some of the older youth in the care of the Children s Aid Society and headed towards independence, there were goals identified relating to transition planning for independent living, such as obtaining a part time job or needing to look for affordable and suitable living accommodations. Although these goals may be very appropriate goals for the youth, there is limited correlation to the CAFAS needs areas for the transition to independent living. In summary, the only goals that were examined for this project were the goals identified in the Plans of Care under the heading Youth or Family Long Term Goals and Expected Outcomes or from the Goals section of the Case Management/Reintegration Plan. Of the thirty cases reviewed, the least number of goals per Plan of Care was two and the most number of goals was five, with all falling within that range. Each goal was reviewed and then it was determined if there was a match to a CAFAS scale, and if that scale had an impairment score of 20 or 30. That is, at the time of admission, the youth was rated at the moderate or severe impairment level. For each youth, the total number of goals was identified, and the total number of goals that correlated to a CAFAS scale, with a score of 20 or 30 was also tabulated. A percentage of correlation was obtained per youth.
4 Data Collection Sheet for CAFAS Problem Areas and Corresponding POC Goals CaseNumbers CAFAS 1st Evaluation Scores AVERAGES School/work 23 Home 28 Community 20.33333333 Behaviour 20.66666667 Mood 16 Harm 5.333333333 Subst. 9.333333333 Thinking 2 TOTAL 1ST EVALUATION CAFAS SCORE 124.6666667 # Goals Suggested by Identified (>/=20)Problem Area in Initial Plan of Care 2.666666667 # Goals in Initial Plan of Care 23.23636364 Percent of CAFAS-Related-Goals Found in all POC Goals 75.72% With the sample of thirty cases, the average percent of congruence to CAFAS related goals found in the Plans of Care was 76%. The range of congruence was from 20-100%. As identified in the Relevance statement section of the grant application, we expected that there would be a high level of congruence between what CAFAS identified as main problem areas and other assessment findings and ultimate contracted goals. There is, however, room for improvement. As noted earlier, some of the goals established at time of admission, although very appropriate for the youth, especially in the area of transition planning to independent living, may not have been captured as an assessment area rated within the CAFAS scales. A secondary outcome was an increase in knowledge of how CAFAS scores were being examined against goals set at initial Plans of Care with front line staff. Informal conversations about the nature of the project and what was being examined in the files, with projected outcomes, seemed to be listened to and questions were asked spontaneously as this process was occurring unit to unit, with staff who were working that day. These conversations may have increased understanding while reducing some of the mystique and possible fear/resistance that staff may feel when the files are being audited for compliance purposes. Although this was one of the outcomes identified, to increase capacity and knowledge and comfort level, the method for doing this in this manner was not specifically identified at the beginning of the project.
5 3) A Summary Report and Power Point presentation for agency staff and management on the aggregate analysis of congruence between goals and CAFAS areas and outcomes. A detailed report has been presented to the agency Management team on the outcomes and deliverables of the Goal Setting and CAFAS Evaluation project. Elements of the training package generated from this evaluation have been used as part of a Power Point Presentation. Further components of the evaluation, training package and associated goal evaluation information are available for further presentation creation. This presentation could be used as a stand-alone presentation. Plans are in place for further training on assessment areas in CAFAS, BCFPI and Risk/Needs Assessment and goals, as referred to within the Analysis section of this report. Presentations will be made to the Management team and to staff teams regarding the information developed. Future opportunities for knowledge exchange regarding the information studied and shared within this evaluation project will continue to enhance our capacity and further develop the knowledge exchange along a learning continuum. Each of these progressive steps are building blocks to further increase our understanding and utilization of the assessment and outcome measurement tools and connect them to our goal setting practices.
Eligible Budget Items Accounting Summary of Expenses GOAL ANALYSIS PROGRAM Peg. 162606-051 Cost per Item ($) Personnel Costs Data Input and Analysis, Presentation and Evaluation 5,061 5,061 6 Total Cost ($) Consultation Costs Consultation of Program Evaluation, Statistics and Evaluation Techniques 611 611 Computer Costs (hardware and software; Max. $3,000) Hardware to Facilitate Program Evaluation Activities 2,906 2,906 Training Tools/Questionnaires Administrative Costs (details required; Max. $1,000) Administrative Support and Accounting 1,016 1,016 Office Supplies (Max. $500) Paper, pens, photocopying, postage, telephone, etc. 302 302 Web-Design (Max. $500) Travel (for data collection only) For Data Collection - Staff travelling between sites 162 162 Total Cost of All Expenses (Max. $10,000) $10,058 $10,058