Monitoring Client Outcomes

Transcription

1 Spported Employment Implementation Resorce Kit Monitoring Client Otcomes DRAFT VERSION 2003 What are client otcomes? Client otcomes are those aspects of clients lives that we seek to improve or to manage sccessflly throgh the delivery of mental health services. Medications help clients manage their symptoms. Spported employment programs help clients find work in the commnity. Dal disorders grops help clients redce their dependence on alcohol and illicit drgs. Relapse prevention programs help clients stay ot of the hospital. Some otcomes are the direct reslt of an intervention, sch as getting a job throgh participation in a vocational program, whereas others are indirect, sch as improvements in qality of life de to having a job. Some otcomes are concrete and observable, sch as the nmber of days worked in a month, whereas others are sbjective and private, sch as satisfaction with vocational services. Every mental health service intervention, whether considered treatment or rehabilitation, has both immediate and long-term client goals. In addition, clients have goals for themselves, which they hope to attain throgh the receipt of mental health services. These goals translate into otcomes, and the otcomes translate into specific measres. For example, the goal of a spported employment program is commnity integration throgh employment. The otcome for clients is obtaining and holding reglar jobs in the commnity. The otcome measre for a spported employment program may be the nmber of weeks that a client has worked at competitive jobs dring the past qarter. DRAFT 2003 MONITORING CLIENT OUTCOMES 1

2 Why monitor client otcomes? Client otcomes are the bottom-line for mental health services, like profit is in bsiness. No sccessfl bsinessperson wold assme that the bsiness was profitable jst becase the enterprise was prodcing a lot of widgets (e.g. cars, clothes) or employees were working hard. This does not mean that the owner does not need to pay attention to prodctivity, bt rather one wold not make the assmption that prodctivity necessarily leads to profit. In mental health, prodctivity measres, sch as the nmber of conseling sessions or the nmber of clients served, tell s very little, if anything, abot the effects of services on clients and their welfare. This fact has led to a broad-based call for otcome monitoring. At the policy and systems level, the Government Performance and Reslts Act of 1993 reqires that all federal agencies measre the reslts of their programs and restrctre their management practice to improve these reslts. In a parallel fashion, there is a significant movement in hman service management toward client otcome-based methods (Rapp & Poertner, 1992). Stdies have shown that an otcome orientation of managers leads to increased service effectiveness in mental health (Gowdy & Rapp, 1989). This has led Patti (1985) to arge that effectiveness, meaning client otcomes, shold be the philosophical linchpin of hman services organizations. Recovery and client otcomes Recovery means more than controlling symptoms. It s abot getting on with life beyond the mental health system. As Pat Deegan (1988) wrote: The need is to meet the challenge of the disability and to reestablish a new and valed sense of integrity and prpose within and beyond the limits of the disability; the aspiration is to live, work, and love in a commnity in which one makes a significant contribtion (p.15). While the goals of each individal are niqe in detail, people with severe mental illness generally desire the same core otcomes that we all want: 1. To live independently in a place called home 2. To gain an edcation, whether for career enhancement or personal growth 3. To have a job that enhances or income, provides a means to make a contribtion, and en ables s to receive recognition 4. To have meaningfl relationships 5. To avoid the spirit-breaking experiences of hospitalization, incarceration, or sbstance abse DRAFT 2003 MONITORING CLIENT OUTCOMES 2

3 If this is tre, then mental health services shold be focsed on the most powerfl methods available to help consmers achieve these otcomes. The evidence-based practice that is described in this resorce kit was chosen for its ability to achieve one or more of these otcomes. A powerfl resorce for program leaders If fnds are the lifeblood of an organization, then information is its intelligence. Collecting and sing client otcome data can improve organizational performance. Consider the following vignette. Participants in a partial hospitalization program sponsored by a commnity mental health center were consistently showing very little vocational interest or activity. Program staff began gathering data monthly on clients vocational stats and reporting this to their program consltant. He retrned these data to program staff sing a simple bar graph every three months. The reslt of gathering and sing information on clients vocational activity was evident almost immediately. Three months after institting this monitoring system, the percentage of the program s clients showing no interest or activity in vocational areas declined from an original 64 percent to 34 percent. Three months later this percentage decreased an additional 6 percent, so that 72 percent of program participants were now involved in some form of vocational activity. This example shows that when information is made available, people respond to it. Peters and Waterman (1982) in their stdy of sccessfl companies observed: We are strck by the importance of available information as the basis for peer comparison. Srprisingly, this is the basic control mechanism in the excellent companies. It is not the military model at all. It is not a chain of command wherein nothing happens ntil the boss tells somebody to do something. General objectives and vales are set forward and information is shared so widely that people know qickly whether or not the job is getting done-and who s doing it well or poorly (p. 266). They observed that the data were never sed to browbeat people with nmbers (p.267). The information alone seemed to motivate people. What is clear from these examples is this: The collection and feedback of information inflences behavior. Crrent research sggests several principles to improve organizational effectiveness: The role of information in an organization is to initiate action and inflence organizational behavior. The act of collecting information (measrement) generates hman energy arond the activity being measred. To ensre that information directs hman energy toward enhanced performance, data collection and feedback mst be sed: to foster and reinforce desired behaviors; to identify barriers to performance and ways to overcome them; and to set goals for ftre performance. DRAFT 2003 MONITORING CLIENT OUTCOMES 3

4 Feedback directs behavior toward performance when it provides ces to workers to identify clear methods for correction and when it helps workers learn from their performance. Feedback motivates behavior toward performance when it is sed to create expectations for external and internal rewards, is linked to realistic standards for performance, and is directed toward the ftre verss sed pnitively to evalate past performance. Managers who are committed to enhancing client otcomes have a powerfl tool. By proactively and systematically collecting and sing client otcome information, managers can enhance the goaldirected performance of program staff, as well as increase their motivation, professional learning, and sense of reward. Minimally, spervisors and managers shold distribte (or post) the otcome data reports and discss them with staff. Team meetings are sally the best time. Nmbers reflective of above average or exceptional performance shold trigger recognition, compliments, or other rewards. Data reflecting below average performance shold provoke a search for nderlying reasons and the generation of strategies that offer the promise of improving the otcome. By doing this on a reglar basis the manager has begn to create a learning organization characterized by consistently improving client otcomes. Otcomes and evidence-based practices The fondation of evidence-based practices is client otcomes. The decision to implement an evidence-based practice is based on its ability to help clients achieve the highest rates of positive otcomes. Therefore, one key component of the implementation of an evidence-based practice is the carefl monitoring and se of client otcome data. The problem for many mental health providers is that crrent data systems do not captre relevant client otcomes or are nable to prodce meaningfl and timely reports. Providers mst find ways to develop evidence-based practices information systems that are easy to implement and to maintain. The following material is designed to gide programs that are implementing an evidence-based practice in developing a practical and sefl information system. Some programs may go their own way and develop a system anew. Other programs may adapt existing information systems to sit their needs for monitoring client otcomes. These gidelines will help programs to make sch beginnings and adaptations. In addition, programs may wish to expand the evidence-based practices information systems that we describe, to bild on the sccess they have had sing a basic system or to cstomize a system to their needs and context. We encorage sch expansion once a basic system has been implemented sccessflly, and we make recommendations for sch enhancements at the end of this section. DRAFT 2003 MONITORING CLIENT OUTCOMES 4

5 We begin with advice on getting started, and then we describe a simple, yet comprehensive, system for monitoring evidence-based practice otcomes. We follow this with ideas on sing tables and graphs of otcome data to improve practice and on expanding basic systems. Gidelines for an evidence-based practices information system Many practitioners feel overwhelmed by the demands of their jobs and cannot imagine adding the brden of collecting client otcomes. Reporting systems already exist in many mental health settings, bt they are time-consming, and they do not provide sefl feedback to improve practice. Ths, resistance is likely when implementing a new system to monitor client otcomes. To overcome this resistance we recommend starting with a very simple system and making the system practical and immediately sefl. Start simply At the otset, the system mst be simple to implement, se, and maintain. Complexity has doomed nmeros well-intended attempts to collect and se client otcome data. One way to keep it simple is to limit the amont and sorces of information that it contains. Begin with a few key client otcomes and bild the system arond them. Collect data from practitioners, withot the initial need for data collection from clients and families. Start with simple reports that tablate reslts for the past qarter and show time trends, and then let experience with the system determine what additional reports are needed. Fit the needs of practitioners The system mst not create nde brden for practitioners, and it mst provide information to them that is sefl in their jobs. If possible, the system shold collect already known information abot clients, and it shold reqire little time to record the data. The system shold fit into the workflow of the organization, whether that means, for example, making ratings on paper or directly into a compter. It shold collect information on participation in evidence-based services and on client otcomes. Program leaders and practitioners can then keep track of what services clients are sing and how they are doing on key otcomes. It shold prodce easy-to-read and timely reports that contribte to planning and lead to action, for individal clients, for treatment teams, and for the program as a whole. These two gidelines may lead to a system that consists of a single otcome measre that is collected reglarly and sed by the program leader and practitioners to monitor their progress toward stated goals for an evidence-based practice. For example, a spported employment program may decide to monitor the rate of competitive employment among those clients who have indicated a DRAFT 2003 MONITORING CLIENT OUTCOMES 5

6 desire to work. Practitioners may be asked to indicate whether each client has worked in a competitive job dring the past qarter. These data can then be tallied for the entire program to indicate the employment rate dring the past qarter, which can be compared to prior qarters and can be sed to develop performance goals based on client choices for the pcoming qarter. The system sggested by these two gidelines can be implemented in a variety of ways, from paper and pencil to mlti-ser compter systems. Begin with whatever means yo have available and expand the system from there. In the beginning, data may be collected with a simple report form, and hand-tallied smmaries can be reported to practitioners. A compter with a spreadsheet program (e.g., EXCEL) makes data tablation and graphing easier than if it is done by hand. A compterized system for data entry and report generation presents a clear advantage, and it may be the goal, bt do not wait for it. Feedback does not have to come from a sophisticated compter system to be sefl. It is more important that it is meaningfl and freqent. As a client otcome monitoring system develops, program leaders and practitioners will weave it into the fabric of their day-to-day rotines. Its reports provide tangible evidence of the se and vale of services, and they will become a basis for decision-making and spervision. At some point, the practitioners may wonder how they did their job withot an information system, as they come to view it as an essential ingredient of well-implemented evidence-based practices. Once a basic system has been implemented for a single evidence-based practice, we encorage programs to consider expanding to a comprehensive system for monitoring mltiple evidence-based practices. We provide two additional gidelines for developing sch a system. Inclde all evidence-based practices in one system The system shold monitor the participation of clients in all evidence-based practices. This can be as simple as recording whether clients are eligible for each practice, and in which practices they have participated dring the past qarter. For those practices that are implemented, participation rates can be monitored over time, as a means of monitoring the penetration of the practices in the poplation of eligible clients. For those practices that are not yet implemented, the system will create incentive to do so. Likewise, the system shold monitor a core set of otcomes that apply across evidence-based practices and that are valed by clients and families, as well as by providers and policymakers. For example, keeping people with mental illness in stable commnity hosing, rather than in instittions or homeless settings, is an agreed-pon otcome for several evidence-based practices. Conseqently, keeping track of qarterly rates of hospitalization, incarceration, and homelessness will enable evalation of the effectiveness of a range of services. DRAFT 2003 MONITORING CLIENT OUTCOMES 6

7 Make the data reliable and valid For an information system to be sefl, the data mst reliable and valid. That is, the data mst be collected in a standardized way (reliability), and the data mst measre what it is spposed to measre (validity). Ths, the otcomes mst be few in nmber and concrete, in order for practitioners to stay focsed on key otcomes, to nderstand them in a similar way, and to make their ratings in a consistent and error-free fashion. To enhance reliability and validity, we recommend simple ratings (e.g., Did the client hold a competitive job in this qarter?), rather than more detailed ones (e.g., How many hors dring this qarter did the client work competitively?). In addition, reliability will be enhanced if the events to be reported are easy to remember, and ths we recommend collecting data at reglar and short intervals, sch as qarterly at the otset, and we recommend collecting data for salient events. We recommend the following otcomes: psychiatric or sbstance abse hospitalization incarceration homelessness independent living competitive employment edcational involvement stage of sbstance abse treatment These few otcomes reflect the primary goals of the evidence-based practices. Assertive commnity treatment, family psychoedcation, and illness management and recovery share the goal of helping clients to live independently in the commnity. Ths, their goal is to redce hospitalization, incarceration, and homelessness, and to increase independent living. Spported employment and integrated dal disorders treatment have more direct otcomes, and ths it is important to assess work/school involvement and progress toward sbstance abse recovery, respectively. A Qarterly Report Form is presented at the end of this section as an example of a simple, paper-based way to collect participation and otcome data on a reglar basis. A stand-alone compterized client otcome monitoring system has been developed for the Evidence-Based Practices Project. It follows the above gidelines closely and is available to those programs who wish to start with sch a system. DRAFT 2003 MONITORING CLIENT OUTCOMES 7

8 Using tables and graphs in reports The single factor that will most likely determine the sccess of an information system is its ability to provide sefl and timely feedback to practitioners. It is all well and good to worry abot what to enter into a system, bt ltimately its worth is in converting data into information. For example, the data may show that twenty consmers worked in a competitive job dring the past qarter, bt it is more informative to know that this represents only 10 percent of the consmers in the spported employment program and only three of these were new jobs. For information to inflence practice, it mst be nderstandable and meaningfl, and it mst be delivered in a timely way. In addition, the monitoring system mst tailor the information to sit the needs of varios sers and to answer the qeries of each of them. The otcome monitoring system shold format data for a single client into a smmary report that tracks participation in practices and otcomes over time. This report cold be entered in the client s chart, and it cold be the basis for a discssion with the client of treatment and rehabilitation progress and options. Frther vale of a monitoring system comes in prodcing tables and graphs that smmarize the participation and otcomes of grops of clients. Below are some examples of tables and graphs that are sefl when implementing and sstaining an evidence-based practice. Qarterly smmary tables Whether for an entire program, for a specific team, or for a single practitioner s caseload, rates of participation in practices and client otcomes shold be displayed for the past qarter. Sch a table can address the following kinds of qestions. How many of my clients participated in or spported employment program last qarter? How many of my clients worked competitively dring the last qarter? What proportion of clients in or program for persons with severe mental illness were hospitalized last qarter? How did the hospitalization rate for those on assertive commnity treatment teams compare to the rate for clients in standard case management? How many clients with a sbstance se disorder have yet to participate in or integrated dal diagnosis treatment program? Simple percentages or proportions, based on qarterly tallies, provide important feedback for both program management and clinical service provision. DRAFT 2003 MONITORING CLIENT OUTCOMES 8

9 Movement tables Movement tables smmarize changes from the previos qarter. They are created by cross-tablating the same variable from two sccessive qarters. For example, participation in the family psychoedcation program can be cross-tablated as shown below. Participation dring Q2 no yes Participation dring Q1 no yes This table indicates that, ot of 120 clients overall, 50 clients did not participate in the program dring either qarter (no/no), 40 participated dring both qarters (yes/yes), 20 began participation dring Qarter 2 (no/yes), and 10 stopped participation after Qarter 1 (yes/no). Ths, there was a net gain of 10 clients in the family psychoedcation program from Qarter 1 to Qarter 2. The same kind of table can show changes in otcomes between qarters as well. This wold answer a qestion sch as, Were more clients working in competitive jobs dring the most recent qarter, as compared to the previos qarter? Movement tables can be prepared for varios gropings of clients. For example, the net gain in competitive employment cold be compared across caseloads from mltiple case managers or across mltiple vocational specialists. Longitdinal plots A longitdinal plot is an efficient and informative way to display participation or otcome data for more than two sccessive periods. The idea is to plot a participation or otcome variable over time, to view performance in the long term. A longitdinal plot can be for an individal, a caseload, a specific evidence-based practice, or an entire program. A single plot can also contain longitdinal data for mltiple clients, caseloads, or programs, for comparison. Below is an example comparing one case manager s caseload to all other clients in a spported employment program over a twoyear period. DRAFT 2003 MONITORING CLIENT OUTCOMES 9

10 This plot reveals that JP s clients were slower to find employment in the first year (Qarters 1-4), when compared to other clients in the program, bt they made contined progress throghot year two (Qarters 5-8), whereas the rate of employment for the other clients has leveled off. Longitdinal plots are powerfl feedback tools, as they permit a longer-range perspective on participation and otcome, whether for a single client or a grop of clients. They enable a meaningfl evalation of the sccess of a program, and they provide a basis for setting goals for ftre performance. Recommendations for additions to the basic evidence-based practices information system Mental health service programs that are sophisticated in sing information systems or that have been sccessfl in implementing a start-p system may want to collect and se more information than we recommend for a basic system. For example, programs may want more detailed participation data, sch as the nmber of grop sessions attended or the nmber of contacts with a case manager. They may want to inclde additional client otcomes or to collect them in a more detailed way. Programs may also want to collect feedback directly from consmers and family members. Recipients of services are important informants for programs seeking to improve otcomes. Programs may want to know if clients are satisfied with their services and the otcomes they have achieved. They may seek inpt from consmers abot how to improve the services, practically and clinically. Programs may want to know if the services are helping consmers and families to achieve their goals. These are worthy ambitions, and sch data have become part of many monitoring and qality improvement systems. DRAFT 2003 MONITORING CLIENT OUTCOMES 10

11 We did not recommend collecting data from consmers and family members as part of a basic system for monitoring client otcomes for a nmber of reasons. First, we recommend starting with a set of otcomes that practitioners can reported qickly and accrately. The task of collecting data from clients and families cold impede progress and distract focs. Second, there are no well-validated qestionnaires to assess many of the constrcts that are freqently inclded in consmer and family srveys. Otcomes sch as satisfaction, qality of life, and recovery are mltifaceted and difficlt to measre objectively. Third, it is hard to obtain a representative sample of respondents. Mailed srveys are often not retrned. Interviews may be done with those individals who are easy to reach and cooperative. Qestions may be asked only of those who show p for rotine appointments. Unless the data are collected from a representative sample, it is difficlt to interpret the findings, becase it is not clear to whom they generalize. Forth, there may be better ways to get feedback from consmers than by trying to collect qantitative data from them. A program may be better off holding focs grops for consmers or families to discss a specific evidence-based practice with the practitioners or with qality improvement personnel. Likewise, a program may learn more abot consmer perceptions of services and their feelings abot recovery from qalitative interviews with a small grop of consmers. Fifth, qality improvement personnel may be better able and qalified to collect, analyze, and interpret data from consmers and families. A treatment team may collect informal feedback from consmers throgh their day-to-day contacts, bt it may be better left to others to collect systematic data. In many agencies, formal reporting systems already inclde client-based assessments, and it may be possible to bild on these efforts rather than to dplicate them. Yet, programs may want to collect data from the recipients of their services. If a basic otcome monitoring system has been implemented, then expanding data collection to inclde consmers and family members may be appropriate and feasible. Programs are encoraged to explore their options, althogh it is important to remain mindfl of the isses discssed above. We inclde the Kansas Consmer Satisfaction Srvey, and a Qality of Life Self-Assessment developed in New York, as examples for programs to consider. When thinking abot expanding data collection beyond the basic set of otcomes, it is important to realize that more is not necessarily better. Unless the data can be reported reliably and validly, the vale of adding more data to the monitoring system is illsory. The old adage, garbage in, garbage ot, mst be kept in mind when the temptation is present to expand a working system. Feedback that is based on nreliable, invalid, or nrepresentative data may be no better for a system than no feedback at all. Nevertheless, the thoghtfl and gradal expansion of a working system for collecting and sing client otcome can increase the vale of the feedback. The litms test is not what and how mch data a program collects, bt rather whether the program ses the data to inform and improve the practice. DRAFT 2003 MONITORING CLIENT OUTCOMES 11

12 References Deegan, P. E. (1988). Recovery: The lived experience of rehabilitation. Psychosocial Rehabilitation, 11(4), Gowdy, E., & Rapp, C. A. (1989). Managerial behavior: The common denominators of effective commnity based programs. Psychosocial Rehabilitation Jornal, 13, Patti, R. (1985, Fall). In search of prpose for social welfare administration. Administration in Social Work, 9(3), Peters, T.J., & Waterman, R.H. (1982). In search of excellence. New York: Harper & Row. Rapp, C. A., & Poertner, J. (1992). Social Administration: A Client-Centered Approach. New York: Longman. DRAFT 2003 MONITORING CLIENT OUTCOMES 12

13 Client Otcomes Qarterly Report Form Client ID: Date: Reported by: Qarter: Indicate the client s stats dring the past 3 months. Check all that apply: Evidence-Based Practice Eligible Enrolled Integrated Dal Disorders Treatment q q Spported Employment q q Assertive Commnity Treatment q q Illness Management & Recovery q q Family Psychoedcation q q In the past 3 months, how many weeks has the client: Held a competitive job? Been homeless? Been incarcerated? Been hospitalized for psychiatric reasons? Been hospitalized for sbstance se reasons? What has been the client s stage of sbstance abse treatment dring the past 3 months? Circle one. N/A Engagement Persasion Active treatment Relapse prevention What is the client s crrent living arrangement? Circle one. 1. Psychiatric hospital 2. Sbstance se hospitalization 3. General hospital psychiatric ward 4. Nrsing home or IC-MH 5.Family care home 6. Lives with relatives (heavily dependent for personal care) 7.Grop home 8. Boarding hose 9. Lives with relatives (bt is largely independent) 10. Spervised apartment program 11. Independent living 12. Other (specify) 13. Emergency shelter 14. Homeless What is the client s crrent edcational stats? Circle one. 1. No edcational participation 2. Avocational/edcational involvement 3. Pre-edcational explorations 4. Working on GED 5. Working on English as second langage 6. Basic edcational skills 7. Attending vocational school or apprenticeship, vocational program (CNA training) or attending high school 8. Attending college 1-6 hors 9. Attending college 7 or more hors 10. Other (specify) DRAFT 2003 MONITORING CLIENT OUTCOMES 13

14 Definitions for Qarterly Report Form Each person completing the form shold become familiar with the definitions of the data elements in order to provide consistency among reporters. Heading information Client ID The client ID that is sed at yor agency. This is sally a name or an identifying nmber. This information will only be accessible to the agency providing the service. Reported by The name of the person who completed the form the case manager or other staff member from the mental health agency who have access to the desired information. Date The date the report was completed. Qarter The time frame for the reporting period. For example, Janary March, April Jne, Jly September, October December. Evidence-based practice Eligible Does the client meet the participation criteria for a specific EBP? For example, all persons who have a severe mental illness and a drg/alcohol diagnosis are eligible for participation in integrated dal disorders treatment. Each EBP has criteria for program participation that shold be sed to determine eligibility. DRAFT 2003 MONITORING CLIENT OUTCOMES 14

15 Enrolled Is the client participating in a particlar EBP service? Note: aggregate data abot eligibility and enrollment can be sed to determine the penetration of services to eligible persons served by a mental health agency. For the following incidents, the qarterly report shold record the nmber of weeks the client spent in the specific incident category dring the 3 months of the reporting period. Employment In the past 3 months, how many weeks has the client held a competitive job? Competitive employment is viewed as working in a paid position (almost always otside the mental health center) that wold be open to all commnity members to apply. This wold exclde persons working in sheltered workshops, transitional employment positions, or volnteering. It may inclde persons who are self-employed bt the person mst work reglarly and be paid for the work. Incidents reporting Been homeless? Record the nmber of weeks the client spent homeless dring the reporting period. This refers to individals who lack a fixed, reglar, and adeqate nighttime residence. Been incarcerated? Record the nmber of weeks the client spent incarcerated in jails or other criminal jstice lock-ps dring the reporting period. Been hospitalized for psychiatric reasons? Record the nmber of weeks the client spent hospitalized primarily for treatment of psychiatric disorder(s) dring the reporting period. This incldes both pblic and private hospitals whose primary fnction is the treatment of mental disorders. Been hospitalized for sbstance se reasons? Record the nmber of weeks the client spent hospitalized primarily for treatment of sbstance se disorder(s) dring the reporting period. This incldes those both pblic and private hospitals whose primary fnction is the treatment of sbstance se disorders. DRAFT 2003 MONITORING CLIENT OUTCOMES 15

16 Stage of sbstance abse treatment What has been the client s stage of sbstance abse treatment dring the past 3 months? For those persons participating in integrated dal disorders treatment, please indicate the appropriate stage of sbstance abse treatment. N/A is sed for persons who do not have a sbstance se problem or diagnosis. Engagement. This category incldes Pre-engagement and Engagement. The person does not have any reglar contacts with an assigned case manager, mental health conselor, or sbstance abse conselor. The lack of reglar contact implies lack of a working alliance. Persasion. This category incldes Early Persasion and Late Persasion. The client has reglar contacts with a conselor bt has not yet redced sbstance se for more than a month (early persasion), or has redced sbstance se for at least one month while discssing sbstance se isses or attending grops (late persasion). Reglar contacts imply a working alliance and a relationship in which sbstance abse can be discssed. Active Treatment. This category incldes Early Active Treatment and Late Active Treatment. The client is engaged in treatment, is discssing sbstance se or attending a grop, has redced se for at least one month and is working toward abstinence as a goal, even thogh he or she may still be absing (early active treatment). This category also incldes persons engaged in treatment, who have acknowledged that sbstance abse is a problem, and have achieved abstinence bt for less than 6 months (late active treatment) Relapse Prevention. This category incldes Relapse Prevention, and In Remission or Recovery. The client is engaged in treatment, has acknowledged that sbstance abse is a problem, and has achieved abstinence for at least 6 months. Occasional lapses, not days of problematic se, are allowed (relapse prevention). This category also incldes clients who have had no problems related to sbstance se for over one year and are no longer in any type of sbstance abse treatment (in remission or recovery). DRAFT 2003 MONITORING CLIENT OUTCOMES 16

17 Residential and edcational stats These data provide yor agency with an ongoing record of the client s residential and edcational stats. Record the stats that applies to the client on the last day of the reporting period. What is the client s crrent living arrangement? 1. Psychiatric hospital. This incldes those hospitals, both pblic and private, whose primary fnction is the treatment of mental disorders. This incldes state hospitals and other freestanding psychiatric hospitals. 2. Sbstance se hospitalization. This incldes those hospitals, both pblic and private, whose primary fnction is the treatment of sbstance se disorders. 3. General hospital psychiatric ward. This category incldes psychiatric wards located in general medical centers that provide short-term, acte crisis care. 4. Nrsing home or IC-MH. This category incldes facilities that are responsible for the medical and physical care of a client and have been licensed as sch by the state. 5. Family care home. This category is for sitations in which a client is living in a single family dwelling with a non-relative who provides sbstantial care. Here (as with #8), sbstantial care is determined by the degree that the nonrelative(s) is responsible for the daily care of the individal. Sch things as medication management, transportation, cooking, cleaning, restrictions on leaving the home, and money management are considered. The non-relative may have gardianship responsibilities. If the client is not able to do a majority of the daily living tasks withot the aid of the caretaker, the caretaker(s) is providing sbstantial care. 6. Lives with relatives (heavily dependent for personal care). Here the individal client and relatives shold be conslted to the degree that family members are responsible for the daily care of the individal client. An important distinction between this stats and #9 is to ask, If the family was not involved, wold the person be living in a more restrictive setting? In assessing the extent to which the members provide sbstantial care, sch things as taking medication, transportation, cooking, cleaning, control of leaving the home, and money management can be considered. If the client is nable to independently perform a majority of the daily living fnctions, the family member(s) is providing sbstantial care. 7. Grop home. A grop home is defined here as a residence that is rn by staff who provide many fnctions (shopping, meal preparation, landry, etc.) that are essential to independent living. 8. Boarding hose. A boarding home is a facility that provides for a place to sleep and meals, bt it is not seen as an extension of a mental health agency, nor is it staffed with mental health personnel. These facilities are largely privately rn, and clients have a high degree of atonomy. DRAFT 2003 MONITORING CLIENT OUTCOMES 17

18 9. Lives with relatives (bt is largely independent). As with stats #8, an assignment to this category reqires information provided by the client and family. The key consideration relates to the degree that the individal is able to perform the majority of tasks essential to daily living withot the spervision of a family member. 10. Spervised apartment program. Here, the client is living (fairly independently) in an apartment sponsored by a mental health agency. In determining whether someone fits in this category, look at the extent to which mental health staff have control over key aspects of the living arrangements. Example characteristics of control inclde: the mental health agency signs the lease, the mental health agency has keys to the hose or apartment,. the mental health agency provides onsite day or evening staff coverage, or the mental health agency mandates client participation in certain mental health services medication clinic, day program, etc. in order to reside in the hose or apartment. Clients only receiving case management spport or financial aid are NOT inclded in this category; they are considered to be living independently (#11). 11. Independent living. This category describes clients who are living independently and are capable of self-care. It incldes clients who live independently with case management spport. This category also incldes clients who are largely independent and choose to live with others for reasons not related to mental illness. They may live with friends, a spose, or other family members. The reasons for shared hosing cold inclde personal choice related to cltre and/or financial considerations. 12. Other. This stats shold be clearly defined in the space provided by those completing the form. 13. Emergency shelter. This category incldes temporary arrangements de to a crisis or misfortne that are not specifically related to a recrrence of the client s illness. While many emergency shelters provide emotional spport, the need for emergency shelter is de to an immediate crisis not related to the client s mental illness. 14. Homeless. This category incldes individals who lack a fixed, reglar, and adeqate nighttime residence. DRAFT 2003 MONITORING CLIENT OUTCOMES 18

19 What is the client s crrent edcational stats? 1. No edcational participation. 2. Avocational/edcational involvement. These are organized classes in which the client enrolls consistently and expects to take part for the prpose of life enrichment, hobbies, recreation, etc. These classes mst be commnity based, not rn by the mental health center. Classes are those that any citizen cold participate in, not jst persons with severe mental illness. If any of these activities involve college enrollment, se stats #8 or #9. 3. Pre-edcational explorations. Individals in this stats are engaged in edcational activities with the specific prpose of working towards an edcational goal. This incldes individals who attend a college orientation class with the goal of enrollment, meet with the financial aid office to apply for scholarships, or apply for admission for enrollment. This stats also incldes those persons who attend a mental health center sponsored activity focsing pon an edcational goal, e.g., camps visits with a case manager to srvey the location of classrooms; meetings with the case manager and college staff to secre entitlements. 4. Working on GED. This stats incldes people who are taking classes to obtain their GED. 5. Working on English as second langage. This incldes those who are taking classes in English as a second langage in a commnity setting. 6. Basic edcational skills. This incldes those who are taking adlt edcational classes focsed on basic skills sch as math and reading. 7. Attending vocational school or apprenticeship, vocational program (CAN training), or attending high school. This stats incldes those participating in commnity based vocational schools; learning skills throgh an apprenticeship, internship, or in a practicm setting; involved in on the-job training to acqire more advanced skills; participating in correspondence corses which lead to job certification; and yong adlts attending high school. 8. Attending college: 1 6 hors. This individal attends college for 6 hors or less per term. This stats contines over breaks, etc., if the individal plans to contine his/her enrollment. This stats sggests reglar attendance by the individal. Incldes correspondence, TV, or video corses for college credit. 9. Attending college: 7 or more hors. This individal attends college for more than 7 hors per term. This stats contines over breaks, etc., if the individal plans to contine his/her enrollment. Reglar attendance with expectations of completion of corse work is essential for assignment to this stats. 10. Other. This stats shold be clearly defined in the space provided by those completing the form. DRAFT 2003 MONITORING CLIENT OUTCOMES 19

20 Kansas Consmer Satisfaction Srvey KANSAS CONSUMER SATISFACTION SURVEY Mental Health Agency: Conty Where Yo Live: This srvey asks for yor opinions abot the mental health services yo receive. Yor feedback will be sed to help improve the services that are available to yo and others. No names are attached to the srvey forms, so the information yo provide is strictly confi dential. Yor answers will not be shown to staff at the agency where yo receive yor services. Below are listed age, gender, and race/ethnic grop categories. Please place a check mark by the categories that fi t yo. (Note: Yo may leave this section blank if yo prefer not to give this information.) Age: Gender: Race or Ethnic grop: Female American Indian or Alaskan Native Male Asian/Pacifi c Islander Black/African American Hispanic White Over 65 Mltiple Race/Ethnicity Some services offered by the Mental Health Center are listed below. Please make a check mark by the services that yo have sed: Case Management Medication Services Psychosocial Services Partial Hospital Vocational Services Attendant Care Edcational Services Compeer Other DRAFT 2003 MONITORING CLIENT OUTCOMES 20

21 INSTRUCTIONS: There are no right or wrong answers. Please answer each qestion by CIRCLING the nmber of the choice which matches yor opinion at the present time. (Note: The response, Does Not Apply, means that yo have not sed this service or the service is not available where yo live.) Please circle the one choice that best describes yor opinion for each statement. 5 Strongly Agree 4 Agree 3 In Between 2 Disagree 1 Strongly Disagree 0 Does Not Apply 1. I have good access to the program (distance, pblic transportation, parking, etc.) 2. As a reslt of the services I have received here, I deal more effectively with daily problems. 3. I believe that the staff have my best interest in mind. 4. If I am having a problem with my case manager, the program will make staff changes. 5. I am rarely lonely or bored. 6. The doctor here listens to my concerns and vales my opinion. 7. The program s services and staff help me to stay ot of the hospital. 8. As a reslt of the services I have received here, I am better able to deal with crisis. 9. I am free to make choices abot my life withot fear of losing the help I get from the program. 10. If I have an emergency at night or on the weekend, I am able to get help from the program 11. Staff follow throgh on promises they make. 12. I can choose where I live. 13. Staff do a good job of telling me abot my rights as a consmer. DRAFT 2003 MONITORING CLIENT OUTCOMES 21

22 Please circle the one choice that best describes yor opinion for each statement. 5 Strongly Agree 4 Agree 3 In Between 2 Disagree 1 Strongly Disagree 0 Does Not Apply 14. My opinions and ideas are inclded in my treatment plan. 15. The staff here treat me like an adlt, not a child. 16. The staff help to overcome the problems that go along with getting and keeping a job. 17. To the best of my knowledge, staff have kept my personal information confi dential. 18. As a reslt of the services I have received here, I do better with my leisre time. 19. Overall, I am satisfi ed with the services I receive. 20. If I don t want the services the staff recommend, they will give me other choices. 21. The staff I work with are competent and knowledgeable. 22. Staff have helped me to maintain a home or apartment in the commnity. 23. I know who the consmer representative is on the Mental Health Center s Governing Board. 24. As a reslt of the services I have received here, I do better in social sitations. 25. Staff are willing to see me as often as I feel it is necessary. 26. My doctor tries to fi nd the medications that work best for me. In the space below, please give s any comments yo wold like to make abot what yo like and dislike abot the services yo receive, and sggestions for how to make things better. (Yo may attach additional pages if more space is needed for comments.) DRAFT 2003 MONITORING CLIENT OUTCOMES 22

23 Qality of Life Self-Assessment This srvey asks yo to tell s how things are going for yo these days. It shold take yo abot 5 mintes to complete. When finished, please give the srvey to yor care Coordinator so that yo can review the reslts together. Please print yor name, yor Care Coordinator s name and today s date below. Yor name (please print): Yor Care Coordinator s name: Today s date: In this section, we ask yo to rate how things are going in different areas of yor life. For each statement below, circle the answer that best matches yor experience. Overall, how wold yo rate (Circle one choice for each statement) Shold this be on yor service plan? The place where yo live (yor hosing). Poor Fair Good Excellent Yes or No The amont of money yo have to by what yo need. Poor Fair Good Excellent Yes or No Yor involvement in work, employment. Poor Fair Good Excellent Yes or No Yor level of edcation. Poor Fair Good Excellent Yes or No Yor access to transportation to get arond. Poor Fair Good Excellent Yes or No Yor social life. Poor Fair Good Excellent Yes or No Yor participation in commnity activities (leisre, sports, Poor Fair Good Excellent Yes or No spirital, volnteer work). Yor ability to have fn and relax. Poor Fair Good Excellent Yes or No Yor physical health. Poor Fair Good Excellent Yes or No Yor level of independence. Poor Fair Good Excellent Yes or No Yor ability to take care of yorself (staying healthy, Poor Fair Good Excellent Yes or No eating right, avoiding danger). Yor self-esteem (how yo feel abot yorself). Poor Fair Good Excellent Yes or No The effect of Alcohol & other drgs on yor life. Severe Moderate Minimal None Yes or No Yor mental health symptoms. Severe Moderate Minimal None Yes or No Overall, how things are going in yor life? Poor Fair Good Excellent Yes or No Is there anything else that yo want on yor service plan? DRAFT 2003 MONITORING CLIENT OUTCOMES 23

24 Spported Employment Implementation Resorce Kit Simple Employment Otcome Measres DRAFT VERSION 2003 The following gives an otline for measring employment otcomes at a program level. Staff and/ or administrators in a spported employment program or in an agency seeking to monitor its own progress over time cold be assigned to collect the information. The methods are also sitable for a state agency seeking to monitor a grop of programs across the state. Monitoring otcomes is important for any evidence-based practice. For spported employment, the main otcome is competitive employment. Althogh there are many aspects of competitive employment that wold be desirable to know, the primary otcome of interest is whether or not a consmer is working or not in competitive employment. The definition of competitive employment incldes the following: pays at least minimm wage the employment setting incldes co-workers who are not disabled the position can be held by anyone, that is, one does not need to be a member of a poplation with a disability to hold the job Some employment programs may choose to measre other types of employment in addition to competitive employment. The system below can be adapted to do so (e.g., se different codes for agency-rn bsiness), bt every addition to a reporting database componds the complexity of one s method. DRAFT 2003 MONITORING CLIENT OUTCOMES 24

25 The employment reporting database We strongly recommend the prospective collection of data. Althogh in theory one can retrospectively collect program activity over a prior period of time, or experience is that retrospective data collection, especially when the time period is long and the nmber of consmers to track is large, is ssceptible to clerical and data entry errors. We recommend that the reporting grid be pdated at a reglarly schedled meeting, ideally at least weekly. To minimize data entry errors one individal shold be assigned the responsibility to pdate the database. This person obtains the information directly from the employment specialists. The basic format for recording employment data is an EXCEL spreadsheet, (see Employment Reporting Grid), with the rows consisting of consmer names (or ID #s, depending on isses of confidentiality) and the colmns consisting of weeks. The second colmn gives the admission date into the program and the third colmn gives the termination date if the consmer is closed. A start date mst be chosen for data collection. The names consist of all active consmers in the program as of the start date. In each cell is recorded a W for working or a N for not working in a particlar week. Working means that the person actally worked in that week. New names are entered at the bottom of the list as they are added to the roster. An end date for data reporting e.g., one year after the start date mst also be chosen and then comparisons can be made. A prototype for the database is attached. Employment otcomes The Employment Reporting Grid permits the calclation of the following: Percentage of consmers who were employed at any time dring follow-p. The nmerator consists of the nmber of people who worked at least one week dring follow-p. Denominator consists of the nmber of people who were active at least one week dring the follow-p. Percentage of consmers employed at follow-p. Nmerator = all employed at follow-p. Denominator = all active at follow-p. Average weeks worked among clients enrolled in program. Nmerator = total nmber of weeks of employment across all consmers. Denominator = total nmber of consmers enrolled at any time. DRAFT 2003 MONITORING CLIENT OUTCOMES 25