Jocelyn Fontaine, Ph.D., and Helen Ho with Kaitlin Greer

Transcription

1 Lessons Learned through The Health Foundation of Greater Cincinnati s Substance Use Disorder and Severe Mental Illness in the Criminal Justice System Initiative Jocelyn Fontaine, Ph.D., and Helen Ho with Kaitlin Greer March 2013

2 Copyright 2013 by The Health Foundation of Greater Cincinnati. All rights reserved. To cite this work, please follow this format: Health Foundation of Greater Cincinnati, The (2013). Lessons Learned through The Health Foundation of Greater Cincinnati s Substance Use Disorder and Severe Mental Illness in the Criminal Justice System Initiative. Cincinnati, OH: Author. Permission is granted to reproduce this publication provided that these reproductions are not used for a commercial purpose, that you do not collect any fees for the reproductions, that our materials are faithfully reproduced (without addition, alteration or abbreviation), and that they include any copyright notice, attribution or disclaimer appearing on the original. Free copies of our publications are available; see About the Health Foundation on page 4 for details. 2 LESSONS LEARNED THROUGH THE CRIMINAL JUSTICE SYSTEM INITIATIVE

3 Contents About the authors, Foundation staff, about the Health Foundation Introduction Importance of the Initiative Substance Use Disorder and Severe Mental Illness in the Criminal Justice System Initiative The Urban Institute Study Grantee Interviews Methods Findings Summary Grantee Outcomes Methods Activities Funded Sustainability Findings Grantee Outcomes Summary Lessons Lessons for Criminal Justice and Behavioral Health Practitioners Lessons for Criminal Justice Practitioners Lessons for Behavioral Health Practitioners Lessons for Policymakers, Public Administrators and Other Decisionmakers Lessons for Researchers Lessons for Funders Conclusions 22 Endnotes 22 Appendix: Data Limitations, Methods, and Results 23 A.1. Data Limitations 23 A.2. Methods 25 A.3. Findings Recidivism, Mental Health, and Substance Use Outcomes 26 A.4. Findings Cost Benefit Outcomes 32 THE HEALTH FOUNDATION OF GREATER CINCINNATI 3

4 About the authors Jocelyn Fontaine, Ph.D., is a senior research associate in the Justice Policy Center of the Urban Institute. Dr. Fontaine s research portfolio is focused mostly on evaluating innovative community-based crime reduction and reentry initiatives targeted to vulnerable populations. She is experienced in using both quantitative and qualitative research methodologies to explore the impact of community-based initiatives on a range of individual and community outcomes. Helen Ho is a research assistant in the Justice Policy Center of the Urban Institute. Her research focuses on criminal justice reforms, reentry programs, and racial disparities in the criminal justice system. Kaitlin Greer is an analyst in the Executive Office of the President (EOP). Before joining EOP, Ms. Greer was a research assistant in the Justice Policy Center of the Urban Institute where she contributed to several criminal justice-related projects. Before joining the Urban Institute, Ms. Greer was a policy intern at the Pennsylvania Prison Society and the Vera Institute of Justice. Foundation staff Janice Bogner, Senior Program Officer, Severe Mental Illness Jeff Williams, Director, Publications Ann Barnum, Senior Program Officer, Substance Use Disorders Patricia O Connor, Vice President, Chief Operating Officer About The Health Foundation of Greater Cincinnati Since 1997, The Health Foundation of Greater Cincinnati has invested more than $128 million to address health needs in 20 counties in Ohio, Kentucky and Indiana that surround Cincinnati. Our mission is to improve the health of the people of the Cincinnati region. Our vision is to be one of the healthiest regions in the country. For more information about the Health Foundation and our grantmaking interests and capacity building programs for nonprofits, please call us at or toll-free at , or visit our website at Copies of this and other Health Foundation publications can be downloaded at org/foundation-publications-and-reports. 4 LESSONS LEARNED THROUGH THE CRIMINAL JUSTICE SYSTEM INITIATIVE

5 1.0. Introduction In the late 1990s, The Health Foundation of Greater Cincinnati began exploring how to improve outcomes for people with severe mental illnesses and substance use disorders in the criminal justice system. It commissioned a paper to better understand the scope of the problem and where funding could be most useful. With this information, the Foundation began the Substance Use Disorder and Severe Mental Illness in the Criminal Justice System Initiative in Through the Initiative, the Foundation has invested more than $12 million in 99 grants in the Greater Cincinnati area. The grants cover a wide range of activities focused on diversion and reentry for people with behavioral health issues in the criminal justice system. 1 Although the Initiative formally ended in 2008, several grantees continue to work on multi-year projects. Beginning in November 2011, the Urban Institute (UI) has been working with the Foundation to disseminate the findings and lessons learned from the Initiative, lessons that are relevant for interested stakeholders within the Greater Cincinnati area and beyond. Specifically, UI has gathered information from interviews and focus groups with Health Foundation staff, grantees, and criminal justice and behavioral health stakeholders in the Greater Cincinnati area, as well as grantee reports. This report summarizes the findings and lessons learned from the perspective of the UI researchers and is intended for five audiences: behavioral health practitioners, criminal justice practitioners, policymakers and public administrators, researchers, and funders. This report aims to contribute to each audience s understanding of the Initiative and the outcomes for the population served by the Initiative s grantees. The following sections within the introduction outline the importance of focusing on people with behavioral health issues in the criminal justice system, a brief overview of the Initiative, and the UI study. Next, a section describes the findings from interviews with stakeholders. Program outcomes are discussed in a subsequent section and the report ends with specific findings and lessons learned for the five audiences outlined Importance of the Initiative As documented extensively in existing research, people with behavioral health issues are overrepresented in the juvenile and adult criminal justice systems. 2 And in many cases, individuals behavioral health conditions directly influence their participation in crime. 3 Unfortunately, the criminal justice system is ill-equipped to address the needs of these people effectively. 4 Behavioral health services provided in prisons and jails are limited 5 and many people would be better and more effectively served by behavioral health diversion and reentry programs in the community. 6 As such, it is critical that information detailing the opportunities for alternative programs for individuals with behavioral health disorders be available to a wide array of stakeholders. Further, it is equally important that information about the successes and challenges of implementing and sustaining alternative programs are disseminated widely to contribute to the field Substance Use Disorder and Severe Mental Illness in the Criminal Justice System Initiative Beginning in 1999, 99 grants were funded by the Health Foundation under the Initiative. The 99 grants funded activities across the Greater Cincinnati area, including 20 counties in the states of Ohio, Kentucky, and Indiana. Grantees applied THE HEALTH FOUNDATION OF GREATER CINCINNATI 5

6 for funding by responding to a request for proposals (RFP). The application process was iterative; the Foundation would give feedback to applicants throughout the RFP process to strengthen applications. As a condition of funding, the Foundation required grantees to develop evaluation plans that described how they would evaluate their processes and their participants outcomes. The Foundation also encouraged grantees to describe how they would obtain funding to sustain their programs when the grant period ended. Grantees funded by the Health Foundation through the Initiative varied widely. Programs ranged from mental health courts to crisis intervention teams to multisystemic therapy. Programs included services for both juveniles and adults and focused on diverting people with behavioral health disorders from incarceration (diversion) or serving them upon their release from incarceration (reentry). In addition to paying for the implementation of programs, the Initiative included funding for program planning and development activities, such as paying for practitioners to attend research conferences and funding multi-organization collaborative bodies to evaluate behavioral health needs in the local criminal justice system. Some of these planning and development activities led to program implementation funding by the Health Foundation under the Initiative. The grantees also varied widely, including a range of stakeholders from departments of corrections and prosecutors, mental health and substance use disorder treatment providers, to nonprofit or faith-based agencies, such as churches The Urban Institute Study To understand the overall findings and distill the lessons learned from the Initiative, the Health Foundation tasked UI with collecting and analyzing qualitative and quantitative data from two primary sources. First, we conducted interviews and focus groups with former and current grantees. In these conversations, we discussed grantees perspectives on the Initiative, the Foundation, and program implementation successes and challenges, and their experiences with sustaining the program after the Initiative funding ended. Second, we received administrative data collected about and by the grantees, including the information that the Health Foundation routinely collected from the grantees during the course of their program performance and the Foundation staff s observations. Using these data, we assessed the results of the Initiative to understand what overall lessons could be learned. 6 LESSONS LEARNED THROUGH THE CRIMINAL JUSTICE SYSTEM INITIATIVE

7 2.0. Grantee Interviews To improve outcomes for the people in the criminal justice system with behavioral health disorders and reduce system cycling and the associated costs, it is the perspective of the Health Foundation that the criminal justice and behavioral health systems must modify their practices, policies, and procedures to better serve this population. To assess whether the Initiative achieved its goals, UI conducted semi-structured interviews and focus groups with grantees and other key stakeholders throughout the Greater Cincinnati area. The grantee interviews were used to document changes in local and state treatment of individuals with behavioral health disorders. These discussions revealed difficult, but rewarding experiences with developing cross-system collaborations, effective and sustained interventions, and meeting the Health Foundation s grant requirements Methods UI conducted 20 semi-structured telephone interviews and facilitated one focus group with stakeholders involved in executing grants funded by the Health Foundation. The interviews ranged from 30 to 60 minutes in length. The focus group lasted 120 minutes. In total, UI spoke with 38 stakeholders, representing 30 different grantees in addition to four staff members at the Foundation who designed and managed the Initiative. The discussion themes that emerged from the discussions fell into six categories: The Foundation s planning-to-implementation grant structure Stakeholder support Interagency collaboration Data collection Sustainability Working with the Foundation These main themes were those most consistently cited by grantees and stakeholders as factors that contributed to the adoption (or lack thereof) of effective strategies and interventions for the population under focus Findings The role of the planning-to-implementation grant structure There was consensus among stakeholders who worked on planning and implementation grants that the planning grants were very helpful in laying the groundwork for the implementation grants. Specifically, the planning grants provided the grantees with time and resources to determine which program or design to pursue and then to build support from relevant stakeholders. One grantee credited the planning process for establishing their project because everyone took ownership of the project, because they got invested in the planning of it. Planning grants also helped grantees identify and address potential issues that may arise during implementation. One negative comment that One grantee credited the planning process for establishing the program as everyone took ownership of the project, because they got invested in the planning of it. did emerge regarding the grant structure was that the planning grant perhaps inflated the sense of stakeholder support and collaboration. Stakeholders had supported the abstract ideas generated during the planning grant; however, support waned as implementation challenges arose. Regardless, the grantee who encountered these issues noted that the planning-to-implementation grant structure was an effective one. The role of stakeholder support All of the grantees discussed stakeholder support, either as a positive indicator of program adoption, a negative indicator, or both. Stakeholder support was gained at different stages of the grant process. In some cases, stakeholders had existing support and a history of collaboration before they applied for THE HEALTH FOUNDATION OF GREATER CINCINNATI 7

8 a planning or implementation grant. In other cases, the collaborative group was brought together specifically to apply for a grant. In most cases, though, a small group of stakeholders applied for and were awarded a grant and they had to gain additional stakeholder support as the grants were implemented. Stakeholder support took many forms. For example, support was gathered in two grants through prominent One grantee stated that stakeholders with stakeholder support was enough political clout challenging because the to recruit supporters. In another grant, consensus to address the need was there, but how there was a clear and to do it wasn t, mainly agreed-upon need because of resources. for services, such that stakeholder support was easy to gather. In addition to stakeholder support, political support was important to the success of grants, and in one grant, the planning stakeholders anticipated a negative political reaction to the program and made changes that would gather more support. Stakeholder support was also a challenge for some grantees because of a lack of understanding of the program being implemented, lack of patience with or interest in working with the client population, and logistical difficulties in service delivery to clients in rural areas. One grantee stated that stakeholder support was challenging because the consensus to address the need was there, but how to do it wasn t, mainly because of resources. Some grantees were able to work through their stakeholder support challenges by engaging leaders with in-person meetings and by framing collaboration as a way to avoid duplicating services. Two grantees that worked on grants not sustained after Health Foundation funding ended cited lack of stakeholder support as one of the reasons they were not sustained. The role of interagency collaboration Interagency collaboration was also named as either a challenge or a contributor to success by almost all of the grantees interviewed. As with stakeholder support, interagency collaboration was formed at different times during the grant. The mental health, drug, and substance abuse/mental illness (SAMI) courts, especially, emphasized the importance of having a long history of interagency collaboration. For those without a history of collaboration, the Health Foundation grant provided an opportunity where people met each other who had never met each other the sheriff met providers, adult education providers were able to network, and parole talked with (clients support networks). This collaboration was not only useful during the planning phases of grants, but in one case, two service providers collaborated to deliver services within a drug court program. Four grantees mentioned that they used boundary spanners or liaisons to help connect clients with services and that these boundary spanners helped agencies work together to deliver services. Another four grantees mentioned that interagency collaboration was a minor challenge during For those without a history of collaboration, the Health Foundation grant provided an opportunity where people met each other who had never met each other the sheriff met providers, adult education providers were able to network, and parole talked with (clients support networks). implementation of their grants, but those mentions were brief and did not come up as many times as did stakeholder support challenges. Lack of referrals, mistrust between agencies, and confidentiality challenges when sharing client information were mentioned by interviewees as barriers to collaboration, as was an insular service-provider community wary of ideas from outsiders. The role of data collection While all of the grantees were required by the Health Foundation to collect program and outcome data during their implementation grant, not all of the grantees maintained data collection after the grants ended. The data required by the Foundation and collected after the grants 8 LESSONS LEARNED THROUGH THE CRIMINAL JUSTICE SYSTEM INITIATIVE

9 have been used to leverage sustainability funding and gather political support as well as target services to the greatest number of people. One grantee mentioned that mental health court data are being collected and reported to county commissioners as a contingency plan in case the need for additional funding arises. The grantee hopes that should the need arise, the commissioners will see a history of reported success and agree to provide funding. Efforts to sustain programs As mentioned, the Health Foundation encouraged grantees to develop sustainability plans as a condition of funding. One grantee appreciated the sustainability plan requirement because if we hadn t committed to that at the beginning, we wouldn t have done the fundraising. Many, but not all, of the programs funded by the Foundation were sustained after the financial support ended. Sustainability funding came from many different sources, which depended on the type of program being funded (for example, some mental health courts received funding from local mental health boards in Ohio). Funding types included local levies, a builtin payer source (such as Medicaid), private foundation grants, fees collected from program participants, state departments of corrections funding and federal grants. While all of the grantees dealt with sustainability funding, only two mentioned non-monetary sustainability, such as continued regular meetings of stakeholders involved in the implementation of the grant. Even though most grantees were One grantee appreciated the sustainability plan sustainability funding successful in finding requirement because if we and support, a handful of the grantees hadn t committed to that at the beginning, we wouldn t indicated that their have done the fundraising. programs had not been sustained after the funding ended. Barriers to sustainability funding included the lack of federal grants, lack of time to develop stakeholder support, lack of stakeholder support for finding continuation funding, and state budget cuts. Working with the Health Foundation Overall, the grantees found the Health Foundation to be flexible and supportive during the planning process and during implementation, specifically when grantees needed help to meet the Foundation s data collection requirements or to work through unforeseen implementation challenges. One grantee One grantee praised the praised the Health Health Foundation: They Foundation: They thought this was their thought this was their project, too. There was so project, too. There was much support. so much support. In addition, some grantees reported that the Foundation helped gather stakeholder support and identify effective partners for grantees during the planning stages. One grantee used the phrase night and day to describe the positive experience of working with the Foundation compared with other foundations and funders Summary In summary, funding from the Health Foundation led to changes in the behavioral health and criminal justice systems involved in the funded programs. The success of the Initiative in leading to sustainability funding is notable as are the grantees highlighting of the critical role of collaboration and stakeholder support. It appears that much of the stakeholder support and collaboration was generated specifically through the Foundation s planning grants, which led to more successful implementation. While grantees did not often cite specific examples of changed policies and practices, the funding appears to have led to new and/or strengthened modes of contact between behavioral health and criminal justice system stakeholders. THE HEALTH FOUNDATION OF GREATER CINCINNATI 9

10 3.0. Grantee Outcomes The logic of the Health Foundation s Initiative is that grantees outcomes were intended to be broad, such that the programs were intended to have an impact on criminal justice outcomes as well as behavioral health outcomes (mental health and/or substance use disorders). The synergies between different systems created by grantees were also intended to be costbeneficial across these systems. As such, the Foundation required grantees to report their clients criminal justice involvement and behavioral health assessments as well as cost savings. Overall, grantees reported that the majority of their clients had reduced involvement with the criminal justice system, reduced mental health symptoms, and reduced substance use. Those that reported cost savings reported significant savings from reduced criminal justice involvement Methods Of the 99 grants funded, UI received reports and materials for 79 grants. After reviewing the data at length, we encountered multiple challenges to conducting quantitative analyses for comparison purposes. Grantees defined their outcomes in various ways, used different outcome periods, used a breadth of interventions with minimal overlaps, did not report outcomes for all of their participants, reported aggregate data without individual-level data, and did not use comparison groups. Most of these issues emerged because grantees collected data to generate and report on program benchmarks. Therefore, more systematic analyses of outcomes across grantees could not be conducted. The Appendix provides more information on the data analysis methods and limitations. The following section describes grantee characteristics and provides an overview of some of the outcomes that grantees reported. Then, we present a scan of practice of the grantee s most frequently used program models to contextualize the outcomes Activities Funded Grantees were funded for a variety of purposes: planning, research, technical assistance, operations support, and implementation. The distribution of funding purposes is displayed in Table 1. The majority (52 percent) of grants were given to implement a program. A large portion of grantees received funding for planning and development activities, which included creating a task force, hiring a project coordinator, and developing a business plan. The remaining grants were used for technical assistance, operations support, and research. The activities funded under these grants included sending organization representatives to conferences, studying reentry outcomes for those released from an Ohio prison, and preparing for litigation to improve reentry services. As echoed in the stakeholder interviews previously described, the Initiative s planning and technical assistance grants facilitated the establishment or improvement of programs. Fifty-three percent of the planning and technical assistance grants evolved into an implemented program funded by the Initiative. Table 1: Number of Grants by Grant Type Grant Type Count Percentage Planning Research 1 1 Technical Assistance 5 6 Operations Support 5 6 Implementation Total 79 99* * Numbers do not add to 100% because of rounding. 10 LESSONS LEARNED THROUGH THE CRIMINAL JUSTICE SYSTEM INITIATIVE

11 The grantees that received implementation grants employed a variety of treatment models and approaches to interacting with people with behavioral health issues in the criminal justice system. Table 2 in Appendix A lists the treatment models and the frequency of their use among the grantees. The most common approach was creating a diversion court, although each of the eight diversion courts funded had different focuses or models. The next two most common models implemented were the Assertive Community Treatment (ACT) 7 and Crisis Intervention Teams (CIT) 8, with five and four grantees, respectively. Two grantees each used the Assessment, Planning, Identifying, Coordinating (APIC) model 9 ; a boundary spanner, or someone who knows both the criminal justice and behavioral health systems and facilitates collaboration between the systems; or drug treatment. The rest of the models were used by only one grantee each. Notably, some grantees combined more than one approach or model. For example, one SAMI court used Integrated Dual Disorders Treatment (IDDT) in addition to its diversion strategy while another diversion court used ACT as its treatment model. Other grantees did not employ a specific model and provided standard services, such as screening, treatment, and case management Sustainability Findings Program sustainability was a major focus of the Initiative. As part of the Initiative s requirements, grantees developed plans to sustain their programs after the Health Foundation s funding ended. Throughout the grant period, grantees reported on their progress with fundraising, writing grant proposals, and securing funding commitments from stakeholders. Grantees were also asked to report whether they were able to sustain their programs when they submitted their final reports or when Foundation program officers wrote their grant close-out reports. The vast majority of grantees sustained their programs Likely because of the Health Foundation s emphasis on sustainability from the outset of program implementation, the vast majority of the Initiative s implementation projects were sustained at the end of their grant periods. Figure 1 displays the percentage of programs that were sustained. Of the 41 implementation projects, 37 had closed their grant periods at the time of our analyses. Of these 37, 33 (or 89 percent) sustained their programs. Figure 1: Percentage of Projects Sustained after Health Foundation Funding Ended 89% 11% Sustained (33 projects) Not sustained (4 projects) Multiple factors contributed to the discontinuation of some programs Because only four grantees were not sustained, it was difficult to determine the factors that promoted and hindered sustainability. One of the programs that was not sustained had not developed a detailed funding plan. In its application, the grantee had not identified specific funding avenues to pursue while all of the other grantees did. At the end of the grant period, the grantee still had not identified potential funding sources and could not continue its operations. The other program that was not sustained experienced severe staff turnover. This led to a costly process to maintain the certification THE HEALTH FOUNDATION OF GREATER CINCINNATI 11

12 for its evidence-based treatment model, which the program could not afford. One grantee was initially sustained at the close of its grant period, but it had to discontinue its program because of state government budget cuts. Grantees relied on a variety of funding sources for their programs As for the grantees that were sustained, they initially planned to use a variety of funding sources and ultimately were able to acquire funds from many different sources. Grantees mainly planned to receive funds from fees for services; reallocations of the grantee s own budget; grants from foundations and federal, state, or local governments; budget reallocations from state or local government agencies; and levy taxes. Built-in payer sources Billing revenue from Medicaid, private insurance companies, or clients was a fairly reliable source of continuation funding for grantees. Eight of the 11 grantees that planned to use billing revenue and specified their final funding sources were able to rely on billing revenue as part of their funding package. In addition, two grantees that had not planned to use billing revenue were able to support their programs with this method. Grantee funds Nine of the grantees used their own funds to sustain the programs after their grant periods ended. These grantees were mostly county-level mental health and substance use disorder boards or another type of government agency. State and federal government agencies Many grantees were sustained using public funds that were not from their own budgets, suggesting that activities funded by the Initiative filled important service gaps. Some of the grantees attributed this to their ability to demonstrate success to external public agencies. Five grantees were sustained by state department funds, usually from a behavioral health department or a department of corrections. A state department of jobs and family services also provided funding, notable since this department is outside of the criminal justice, mental health, and substance use disorder systems. Thirteen grantees were sustained by local funding sources, including probation departments, county government mental health and substance use disorder boards, county commissioners, and local courts. The availability of federal grant funds was much more limited. Of the seven grantees that planned to apply for federal grants, only one received federal grant money. Public levies Public agencies planned to use funds from a levy, which were thought to be unreliable. Seven grantees, most of which were mental health and/or substance use disorder boards, planned to use levies. Some levies were slated to be approved or renewed by voters and the grantees were not sure whether the levies would pass. Additionally, one grantee whose grant period had not ended was able to use funds from a levy that passed, but reported that there was less tax revenue than expected because of the economic recession Grantee Outcomes More than three-quarters of implementation grantees reported outcomes for mental health, substance use disorders, and criminal justice, where applicable (see Table 3 in Appendix A for reporting rates). The majority of grantees reported very positive outcomes for each of these three outcome categories. Mental health outcomes Grantees used a variety of methods to measure mental health outcomes. Most grantees used scored instruments such as the BASIS-24, the Youth Outcome Questionnaire, and the Ohio Mental Health Consumer Outcomes Instrument. Two CIT programs measured whether mental health cases resulted in referrals to mental health services. These outcomes were reported for participants 12 LESSONS LEARNED THROUGH THE CRIMINAL JUSTICE SYSTEM INITIATIVE

13 who were active in the program at the time of measurement or were discharged successfully. Grantees reported that between 33 and 100 percent of their clients had improved mental health at some point after admission to their programs. Most grantees reported that more than 50 percent of the clients showed improvement, though it is not appropriate to compare results across grantees because of differences in their client populations. The two CIT grantees reported that 72 percent and 93 percent of cases were linked to treatment services. Graph 1 displays the distribution of client improvements across programs. Few grantees reported on the magnitude of their clients improvements. The grantees that did report changes indicated improvements in mental health outcomes between 3 and 48 percent. Graph 1: Improvements in Mental Health or Referrals to Mental Health Services Number of programs % %-50% 51%-75% 76%-100% Percentage of Clients Reporting Improvements or Receiving Referrals Note: Five grantees reported only percentage improvements in mental health assessment scores and were not included in this chart. One grantee reported a range for its percentage of clients who improved without noting the number of clients it assessed. The median of the range (45.5%) was used as a proxy for this grantee s outcome. 9 Substance use disorders outcomes Most grantees measured substance use using clients self-reports. Some grantees used parole office and case manager reports and urine testing. All but six grantees reported that more than 50 percent of their clients had reduced substance use. The outcomes were measured for participants active in the program at the time or measurement or for those who were successfully discharged. Graph 2 displays the distribution of reduced substance use across programs. Similar to the mental health outcomes, it is not appropriate to compare outcomes across grantees because of differences in client populations and their outcome definitions. For example, some grantees used being substancefree as their definition of having reduced substance use, a higher bar than those reporting on substance use reductions. The grantees that measured whether their clients were substance-free reported that between 27 and 88 percent of their clients had no substance use by the end of their programs. Graph 2: Reductions in Substance Use Number of programs % 26%-50% 51%-75% 76%-100% Percentage of Clients Reporting Reductions Note: One grantee reported only the percentage of substance-free days clients experienced and was not included in this chart. 9 9 THE HEALTH FOUNDATION OF GREATER CINCINNATI 13

14 Criminal justice outcomes Grantees used a variety of measures to assess clients involvement in the criminal justice system, including charges, convictions, and incarcerations. They reported that between 50 and 100 percent of their clients had no new involvements or reduced involvement with the criminal justice system either during or after their programs. All but two grantees reported that 60 percent or more of their clients had reduced involvement with the criminal justice system. For the CIT and Mobile Crisis Teams, between 93 and 99 Graph 3: Reductions in Criminal Justice Involvement Number of programs % %-50% 51%-75% 76%-100% Percentage of Clients Reporting Reductions 21 percent of calls involving people with mental health issues were resolved at the scene or resulted in a referral to a mental health service provider instead of resulting in an arrest. Graph 3 shows the distribution of reduced criminal involvement across programs. Cost savings Of the 41 implementation grants, 17 reported cost analyses in their evaluation documents. The reported per month costs saved ranged from a loss of $1,517 per client to a savings of $45,316 per client, with only one grantee reporting an overall loss. Although the self-reported cost analyses are encouraging, they should be taken with caution. First, none of the grantees compared its clients outcomes to a comparison group; therefore, the cost effectiveness of the grantee programs relative to no intervention, another intervention, or business-as-usual services is unknown. Second, some grantees did not subtract treatment costs from their calculations, inflating the program s net benefits. Table 5 in Appendix A displays reported costs and additional data limitations Summary In summary, it appears that the Initiative has led to notable benefits among program participants, who were served by an array of treatment models, some of which were based in evidence. As mentioned throughout this section, we cannot make definitive claims as to the benefit of the specific programs or the overall Initiative in leading to the outcomes observed for a variety of reasons. Yet, from the self-reported outcomes provided by the grantees, the Initiative succeeded in helping individuals with behavioral health disorders in the criminal justice system achieve generally positive health and criminal justice outcomes. 14 LESSONS LEARNED THROUGH THE CRIMINAL JUSTICE SYSTEM INITIATIVE

15 4.0. Lessons Using the information gathered from the stakeholders during the interviews and focus groups, coupled with the grantee data on outcomes, successes, and challenges discussed in the data provided to UI, we have generated some lessons learned from the Initiative Lessons for Criminal Justice and Behavioral Health Practitioners Anticipate the difficulty involved in collecting and using data Data from other systems, such as criminal data and medical data, can be difficult to obtain, especially cost data. One grantee wanted to measure the program s effect on child services, but could not access that department s data. Some of the data collection issues encountered by the grantees were resolved with memoranda of understanding or information release protocols that allowed grantees to access data from other systems. Grantees also found it difficult and time-consuming to follow up with clients to measure longterm outcomes, particularly those who were no longer attached to the program. Grantees should develop realistic plans for working with data early in their program s implementation, especially with regard to clients who do not complete programs or cannot be reached for data collection. While there are likely to be issues collecting data, data can be a powerful tool in obtaining new funding and support Grantees reported that the outcomes they tracked were helpful in persuading funders and other stakeholders to support their programs. This helped programs sustain operations after funding from the Health Foundation ended. A slow economy presents challenges and opportunities for new initiatives Not only was funding scarcer in the slower economy, but collaborative support also diminished. Nonprofit organizations and businesses may be less willing to consider new collaborations as they focus on their own organizations issues. Yet, budget cuts can provide the impetus for organizations to find ways to be more efficient through collaboration. Practitioners need to anticipate and adapt to the changing environment caused by the current economic realities. Collaborative efforts and learning opportunities build capacity for future initiatives Even though some projects were not sustained or implemented after funding from the Health Foundation expired, grantees were able to make connections to professionals in different systems and levels of government that may lead to future collaborations. The grantees also gained the experience of developing plans, navigating federal funding rience, if you lay the In one grantee s expe- proposals, working groundwork, you might through implementation issues, and beginning, but you might not get something at the learning behavioral get something later. health practices. This new capacity will likely be useful for future efforts focused on people with behavioral health issues in the criminal justice system. In one grantee s experience, if you lay the groundwork, you might not get something at the beginning, but you might get something later. Adaptations in implementation to account for local context may be necessary Health Foundation grantees included both urban and rural communities, communities that face different challenges and realities. In particular, programs in rural communities reported issues with recruiting and retaining qualified staff and providing transportation for clients. These issues were not as pronounced for grantees in more THE HEALTH FOUNDATION OF GREATER CINCINNATI 15

16 urban settings. Meanwhile, rural communities found that communication and collaboration was facilitated by the smaller number of relevant stakeholders in their jurisdiction as compared with urban communities. Additionally, since some program models may have been developed in urban areas, some program adaptations may be necessary before being implemented in rural areas. Legal leverage can be used to motivate clients One benefit of collaborating with the criminal justice system is the use of legal leverage to motivate clients to continue with the program and to comply with treatment. However, behavioral health and criminal justice practitioners must discuss and agree on appropriate sanctions for violating the terms of a program. Ongoing training is necessary to sustain knowledge and practice Ongoing professional development is critical to maintaining model fidelity and best practices, especially when there is staff turnover. Even without significant staff turnover, staff may revert to past techniques over time, particularly when dealing with a difficult client or an intractable system issue. Programs should think about finding new funding sources early Grantees learned that verbal support from collaborators did not always translate into support, financial or otherwise. Further, the slow economy meant that grants and public funding were cut, often unexpectedly. Programs should anticipate and prepare for staff turnover The loss of key personnel among the grantees hampered their ability to implement their programs and increase services. Grantees should plan to mitigate the problems stemming from staff turnover. Helpful ways to deal with staff turnover include ongoing documentation of institutional knowledge, creating plans for smaller staffs, and planning training opportunities for new staff. In choosing models, service providers should consider the costs of model fidelity, training, and certification Although practitioners may believe in the success of certain models, they should consider very practical barriers to model fidelity. For example, a model that requires a great deal of staff training, such as functional family therapy or multi-systemic therapy, may be very costly for programs that experience staff turnover Lessons for Criminal Justice Practitioners A focus on behavioral health can inform criminal justice work Learning about substance use disorder or mental health issues may help criminal justice practitioners more effectively supervise their clients and address the causes of offending or recidivism. For example, staff in a women s unit of a jail began providing trauma-informed services because the training and technical assistance that was funded by the Initiative helped them better understand substance use disorders and mental health issues. Collaboration with behavioral health practitioners can provide an opportunity to teach them about the criminal justice system Although criminal justice and behavioral health practitioners work with some of the same clients, there is often a lack of understanding between the two systems. Those in the behavioral health system may create programs without fully understanding the nuances of the criminal justice system. For example, one of the Health Foundation s grantees trained only dayshift police officers when night-shift officers had the most contact with the program s potential clients. Thus, exposing behavioral health practitioners to the criminal justice system can better prepare them for future collaborations. Behavioral health practitioners can provide knowledge of other community services It may be difficult for criminal 16 LESSONS LEARNED THROUGH THE CRIMINAL JUSTICE SYSTEM INITIATIVE

17 justice practitioners to know about all of the services in the community to which they can refer clients. Collaboration with behavioral health practitioners can provide a link to the array of service providers in the community Lessons for Behavioral Health Practitioners Criminal justice practitioners have varying levels of understanding of behavioral health issues Some criminal justice practitioners may be fully aware of addiction and treatment practices and the value of diverting offenders with substance use disorder and mental health problems, while others may not. One grantee reported that some defendants would rather plead to serve jail time than participate in programs, Another grantee described success with and some attorneys changing criminal justice advised defendants practitioners approach against participating in the program. to the client population: We had to change the Discussions with language people used attorneys and judges but that s been satisfying. could improve program participation by Now police officers want to help instead of just explaining the benefits of participating in arresting them. behavioral health programs. Another grantee described success with changing criminal justice practitioners approach to the client population: We had to change the language people used but that s been satisfying. Now police officers want to help instead of just arresting them. Emphasizing the importance of treating behavioral health issues in one program may spur action in other programs or parts of the system Exposure to the intersection of behavioral health and criminal justice can spill over to other jurisdictions and parts of the criminal justice system. In Dearborn and Ohio counties, for example, because of the new juvenile drug court program, the traditional court counterpart began requiring more drug screenings and more intensive probationary supervision because they could see the value in treating behavioral health issues. In Hamilton County, after implementing Mobile Crisis Teams (MCT) to assist police officers in two districts, three other districts requested MCT presence. While engaging families of defendants may be difficult, it can be rewarding Some grantees were able to engage clients families with great responses. For example, parents in one program believed that the parenting classes they received from the grantee helped them regain control of their children. Other programs could not involve families in their services and sometimes received resistance to family involvement. One grantee learned that families with open children services cases could be skeptical of service providers, who they perceived as outsiders. Another grantee reported that some clients refused to allow the grantee to contact their families. Social Security and other benefits take time to be approved Although clients may qualify for benefits, the approval process may be long and arduous. One program was able to apply for benefits two weeks before a client s release from prison, but it still took two to 12 months for clients to receive benefits. Thus, maintaining stable lifestyles will be difficult for clients who do not yet have the money to pay for housing and other necessities. Be aware of the extra burdens that behavioral health and criminal justice collaborations place on criminal justice practitioners Incorporating behavioral health services into criminal justice processes may present a burden on criminal justice professionals. For example, judges may not get credit for cases on the mental health docket to reduce their caseload on other dockets, THE HEALTH FOUNDATION OF GREATER CINCINNATI 17

18 increasing their workload. Finding ways to rework incentives or performance indicators to reward collaboration may reduce reluctance on the part of criminal justice practitioners to incorporate behavioral health programs. Anticipate clients with co-occurring disorders Some grantees that had planned to treat either substance use or mental health disorders found that their clients often had both. Service providers should develop treatment and funding plans to address overlapping issues. Program staff should create a presence in criminal justice settings to improve referrals and linkages Criminal justice practitioners and people in the criminal justice system may not be aware of or understand behavioral health programs. Making visits to criminal justice settings, speaking with front-line staff, and having program staff on site will help engage criminal justice practitioners and inmates. Programs should anticipate using funds for wrap-around services Grantees observed that wrap-around services were needed to stabilize clients. One grantee explained, We re trying to fix cracks so the clients don t fall through them there can t be gaps in the housing situation. Leaving a need, such as housing, unaddressed can One grantee explained, We re trying to fix cracks interfere with behavioral health service so the clients don t fall through them. efficacy. Thus, programs should plan to use funds for wrap-around services even if those services are not explicitly part of their models Lessons for Policymakers, Public Administrators, and Other Decisionmakers For the criminal justice population with behavioral health issues, collaboration between systems is critical Awareness of and collaboration among the systems fosters support for program operations and referrals. Health Foundation grantees reported that in the beginning of the implementation process, the criminal justice and behavioral health systems had different goals and languages for describing the problem and solutions. There was also a lack of trust across agencies. Thus, communication is critical for continued collaboration and understanding. Use and support boundary spanners A boundary spanner is a person who knows and operates in both the criminal justice and behavioral health systems and serves as a bridge between them. These boundary spanners helped Health Foundation grantees foster support and collaboration in order to connect clients with services more effectively. The resulting cross-system learning and development of relationships spurred other criminal justice and behavioral health cooperative projects. Change can be slow Getting systems, such as the substance use disorder, mental health and criminal justice systems, to work in new ways and collaboratively takes time. These nuanced systems often speak in different languages, and operate with different missions and goals. Changing the way these systems work together can be a slow process. As one of the stakeholders aptly said, It s not all As one of the stakeholders aptly said, It s or nothing. Change is not all or nothing. slow, but it s about heading in the right direction. Change is slow, but Decisionmakers who can it s about heading in anticipate the pace of the the right direction. process are critical. Decisionmakers who can anticipate the pace of the process are critical. Simple changes can make a big difference Because of the Initiative, a local jail changed its discharge hours from the middle of the evening to standard working hours. Previously, the jail staff released 18 LESSONS LEARNED THROUGH THE CRIMINAL JUSTICE SYSTEM INITIATIVE

19 inmates at a time that was convenient for the jail staff, not taking into account the challenges this placed on case managers who needed to pick up clients from the jail and link them to services upon release. One of the funded programs made this reality apparent and the jail staff changed its discharge hours. There are other points of opportunity that are easily changed but could have a substantial impact on the overall effectiveness and efficiency of services. Develop data-sharing agreements As many Health Foundation grantees discussed, there were challenges to gathering data to document outcomes that were contained in other systems. Although datasharing agreements take time to produce, given the multiplicity of issues facing the clients, agreements across agencies are invaluable to documenting impacts. Without them, programs run the risk of incorrectly estimating the benefits of their programs. Work together toward shared problems By the nature of the population with behavioral challenges and criminal justice histories, systems that work together do far better to serve this population than those working alone Lessons for Researchers Practitioners have time and resource constraints, so make sure data collection is worthwhile to them Many practitioners priority is to deliver services to their clients. They may not view data collection as a resource-efficient endeavor. Thus, researchers should make sure the data collection process is designed in a way to benefit practitioners. For example, the data collection and analysis process should be used to help practitioners obtain new funding and evaluate how to serve their clients better. Researchers should work with practitioners early in the process to define specific outcome measures and methods Grantees had challenges defining their outcome measurements, selecting appropriate time intervals for data collection, reporting the magnitude of changes, using comparison groups, contacting clients who had left their programs, obtaining data from other agencies, and developing appropriate data systems. Researchers can provide guidance and help grantees develop data-collection protocols before implementation begins Lessons for Funders Grantees needed help with developing data-collection plans and research designs Grantees reported needing help with developing realistic evaluation plans with meaningful measures. Challenges included reporting on the magnitude of client outcomes, measuring outcomes at set intervals, and using comparison groups. Additionally, some grantees planned to gather data that they later realized would be too difficult to obtain. And even with data, databases sometimes lacked the ability to extract results. Funders must recognize that the ability to track outcomes will vary widely across behavioral health and criminal justice practitioners for a variety of reasons. Grantees appreciate the opportunity to learn from each other The Health Foundation held meetings in which grantees came together to discuss their experiences, challenges, and solutions. One grantee acknowledged that practices could not be directly transferred from one place to another, but said, I ll bring it back and try to make it work here and that the bringing together of multiple grantees was useful One grantee acknowledged that practices could not be directly transferred from one place to another, but said, I ll bring it back and try to make it work here and that the bringing multiple grantees together was useful because very few of us are as creative as 10 of us put together. because very few of us are as creative as 10 of us put together. The grantee meetings THE HEALTH FOUNDATION OF GREATER CINCINNATI 19

20 allowed stakeholders across counties and systems to share best practices and discuss pressing issues facing all of them. Use trained evaluators The Health Foundation expressed a strong interest in not using trained evaluators for its grantees. Because of this, the data-collection process and reported outcomes lacked consistency. Trained evaluators could be internal to the program, such as program analysts who work closely with the program manager; or external to the program, such as staff from a local university that partners with the program. The validity and reliability of the data collection methods are likely to be limited if evaluation is done by program practitioners who are untrained or unknowledgeable about data collection methods and their various strengths and weaknesses. More robust findings on the program s impact on participant outcomes would have been possible if data-collection efforts within the Initiative were more rigorous. Further, a more rigorous assessment of the overall Initiative would have been possible with better data-collection methods. Understand the tension between being prescriptive and standardized versus grantee-centric and flexible The Health Foundation staff worked with the grantees to develop their own outcome measures and data-collection processes to gather those data. The logic was that more meaningful information could be gathered from grantees if they were developing their own datacollection systems and outcomes. Further, the Initiative grantees were a diverse group, including public administrators, nonprofits and community-based service providers, and standardized outcomes would have likely meant that some meaningful outcomes would have been lost by using a standardized protocol. This decision, however, severely limits the Foundation s ability to look across programs and understand the impact of the Initiative as a whole. Consider requiring grantees to measure model fidelity Although model fidelity is critical to determining effectiveness, few grantees mentioned measuring fidelity in their program reports. Measuring model fidelity and quality assurance, particularly for programs with a strong evidence base, can help grantees and funders determine whether outcomes can be attributed to a particular model or to the implementation process. Measures of program fidelity would also aid in the assessment of the outcomes across participants being served by similar programs. Be hands-on The Health Foundation worked with grantees throughout the grant process, from submitting and refining their request for funding to developing and refining their evaluation and sustainability plans through to implementation. Given the Foundation s role on the projects and the flexibility given to grantees to refine their evaluation and sustainability plans, they engendered feelings of trust among grantees who felt they could rely on Foundation staff members to help them troubleshoot issues. One grantee described One grantee described the Foundation s helpful hands-on approach: Once we got our team together, the Health Foundation explained to us that we weren t ready. They were right. the Foundation s helpful hands-on approach: Once we got our team together, the Health Foundation explained to us that we weren t ready. They were right. We wanted to partner with a community mental health provider, and they were at the table but not really. The Health Foundation picked up on that. A couple months later, the grantee found a stronger partner and was approved by the Foundation. Because grantees felt that Foundation staff members were approachable and flexible, the grantees were more comfortable in setting realistic service targets and goals. 20 LESSONS LEARNED THROUGH THE CRIMINAL JUSTICE SYSTEM INITIATIVE

21 Use planning grants The use of planning grants by the Health Foundation allowed grantees time to develop collaborative partnerships, develop communication systems across partners, develop data-sharing tools and measurement plans, and set more realistic service targets and goals before needing to actually serve clients. Without the planning grants, the need to serve clients while developing the systems would need to happen simultaneously. This is less than ideal, given that criminal justice and behavioral health practitioners are often not engaged with each other or communicating routinely. Planning grants are tremendously useful in bringing diverse groups together and allowing partnerships to develop. Expect implementation issues Even with the use of planning grants, implementation issues are likely to arise. Issues external to the actual program, such as cuts in Medicaid funding or staff turnover in a correctional facility, for example, are likely to have an impact on any program services. Funders should anticipate implementation issues and be flexible in working with grantees to troubleshoot these issues as they arise. Do your homework Since the Health Foundation was previously not directly engaged in serving or funding services for the criminal justice population, staff members did a lot of upfront research, planning, and training internally to understand the justice system and the special needs of people involved in the criminal justice system before launching the Initiative. These planning activities likely led to them having a better understanding of where the Initiative could make a meaningful impact within the local Greater Cincinnati context. Given the planning, the Foundation understood the policy levers, what meaningful change could happen within the Greater Cincinnati region, and which local players would be able to work within and across the behavioral health and criminal justice systems. THE HEALTH FOUNDATION OF GREATER CINCINNATI 21

22 5.0. Conclusions Overall, the Health Foundation can point to an array of positive outcomes associated with the Initiative. As evidenced in the literature, the Initiative was focused squarely on an under- and unserved population and can point to several benefits of program participation for this population. Further, the Initiative has generated lessons for future behavioral health and criminal justice programming as well as the different system stakeholders who have the ability to foster such programming. In highlighting these lessons, we also shined a light on the ways in which the Health Foundation and other agencies like it might improve its processes and outcomes and better serve the people with mental health and substance use disorders in the criminal justice system. Endnotes 1 See The Health Foundation of Greater Cincinnati Substance Use Disorders and Severe Mental Illnesses: Interim Grantmaking Report. 2 See Fazel, S. & N. Langstrom Mental disorders among adolescents in juvenile detention and correctional facilities: A systematic review and metaregression analysis of 25 surveys. Journal of the American Academy of Child & Adolescent Psychiatry, 47, ; Steadman, H.J., F.C. Osher, P.C. Robbins, B. Case & S. Samuels Prevalence of serious mental illness among jail inmates. Psychiatric Services Sabol, W.J. & H. Couture Prison Inmates at Midyear Washington, DC: Bureau of Justice Statistics, U.S. Department of Justice; and Sabol, W.J. & T.D. Minton Jail Inmates at Midyear Washington, DC: Bureau of Justice Statistics, U.S. Department of Justice. 3 Silver, E., R.B. Felson & M. Vaneseltine The relationship between mental health problems and violence among criminal offenders. Criminal Justice and Behavior, 35, ; and Bennett, T., K. Holloway & D. Farrington The statistical association between drug misuse and crime: A metaanalysis. Aggression and Violent Behavior, 13, Chandler, R.K., B.W. Fletcher & N.D. Valkow. Treating drug abuse and addiction in the criminal justice system. Journal of the American Medical Association, 301, Golembeski, C. & R. Fullilove Criminal (in)justice in the city and its associated health consequences. American Journal of Public Health, 95, Morrissey, J.P., J.A. Fagan & J.J. Cocozza New models of collaboration between criminal justice and mental health systems. American Journal of Psychiatry, 166, See Morrissey, J., & Meyer, P. (2008, August). Extending assertive community treatment to criminal justice settings. Retrieved from 8 See National Alliance on Mental Illness CIT Resource Center cfm?section=cit2. 9 Osher, F., Steadman, H. J., & Barr, H. (2002, September). A best practice approach to community re-entry from jails for inmates with co-occurring disorders: the APIC model. Retrieved from gainscenter.samhsa.gov/pdfs/reentry/apic.pdf 22 LESSONS LEARNED THROUGH THE CRIMINAL JUSTICE SYSTEM INITIATIVE

23 Appendix: Data Limitations, Methods, and Results Since the Initiative began, the Health Foundation has been collecting outcome measures from grantee programs on changes in mental health, substance use, criminal justice involvement, and cost savings and cost shifting. In addition, as a condition of funding, each grantee developed a program logic model, process and outcome evaluation plan, and final report. UI worked with the Foundation to collect copies of all available planning, process, and outcome data for each program funded under the Initiative to assess outcomes, including cost outcomes. Our analysis of the outcome data was intended to quantify and compare key outcomes associated with Foundation grants to answer the following research questions: What are the outcomes associated with the funded programs? What population characteristics or combination of characteristics (e.g., age, race and mental health status) are associated with program completion and program success? Are there differences in outcomes associated with funded diversion programs and reentry programs; substance abuse and mental health programs? What is the relative efficacy of the various mental health and substance abuse programs and treatment strategies employed by Foundation-funded programs? A.1. Data Limitations After reviewing the data at length, we encountered multiple challenges to conducting quantitative analyses for comparison purposes. Each of these challenges limited our ability to make strong, valid and reliable conclusions about the overall impact of the Initiative on participant outcomes. Grantees defined their outcomes in various ways. Some grantees defined recidivism as the number of new jail days while others defined recidivism as the number of rearrests. While grantees measures of recidivism and other outcomes were likely appropriate for their respective programs, the variation across outcomes limited our ability to compare outcomes across programs. Grantees used different outcome periods. Some programs reported outcomes of participants after the programs had ended (e.g., six months following program completion), while some reported outcomes when the participants completed the program (e.g., at discharge). Others reported outcomes of participants while they were still active (e.g., three months after the start of their participation in the program). Given the different outcome periods, it would not have been meaningful to compare outcomes for participants with different dosages of the programs. Grantees did not report outcomes for all participants. Some grantees reported outcomes for only a portion of their program participants, which likely skewed their overall outcomes in an unknown way. Further, it is likely that the participants whose outcomes were known by the grantee organizations were those participants whose outcomes were most favorable (i.e., they were still attached to the program). Given the limited reporting, we were left with a less than full picture of the overall programs outcomes. Grantees reported aggregate data without individual-level data on participants. In general, the grantees did not provide enough information about individual-level outcomes. Aggregate information masks the variation across THE HEALTH FOUNDATION OF GREATER CINCINNATI 23

24 relevant variables, such as a participant s age, gender and risk level, which may explain why participant outcomes were different from others. Without such data, it would be inappropriate to compare programs because some grantees may have worked with more high-need or high-risk participants. Because of the lack of data, we were also unable to determine which population characteristics were associated with program success. Range of program interventions used within the Initiative. By the nature of the Initiative, grantees used a range of intervention models. As shown in Table 2, some of the models funded by the Initiative were employed by only one grantee. Further, some grantees did not serve very many participants, while others served several dozens. This made it difficult to conclude whether one model was more effective than another. Table 2: Number of Implementation Grants by Model or Approach Model Diversion Courts ACT/MH Court (1) Drug Court (1) DUI Court; Matrix Model (1) Mental Health Court (2) SAMI Court (1) SAMI Court (IDDT) (2) Assertive Community Treatment (ACT) 5 Crisis Intervention Team (CIT) 4 Assessment, Planning, Identifying, Coordinating (APIC) 2 Boundary Spanner 2 Drug Treatment 2 Assessment and aftercare treatment 1 Criminal Justice Resource Center 1 Functional Family Therapy 1 IDDT 1 IDDT/Maryland Community CJ Treatment Program 1 Jail Diversion Team 1 Matrix Model 1 Mobile Crisis Team 1 Multi-Systemic Therapy 1 Recovery Coach (Jail Diversion) 1 Reformers Unanimous 1 SAGE Model 1 Screening, case management, short-term therapy, crisis intervention 1 Screening, in-depth evaluations, intensive day treatment, community integration services 1 TAMAR 1 Teaching Family Model 1 Training and treatment 1 Transitional Housing 1 Count Total LESSONS LEARNED THROUGH THE CRIMINAL JUSTICE SYSTEM INITIATIVE

25 Grantees did not use comparison groups. Without an appropriate comparison group to use as a benchmark for participants outcomes, our ability to draw meaningful conclusions regarding the programs impacts were not possible. We had no way of knowing whether the outcomes reported were caused by the grantees or if there was an unmeasured factor that accounted for changes in participant outcomes. In 2007, the Health Foundation began to be more prescriptive about outcome measures, which appeared to improve outcome consistency across the grantees. However, many of the data challenges remained. Even though we were limited in our ability to compare grantee outcomes reliably, we nevertheless summarized outcomes and lessons from grantee reports and documents to give an overall picture of the successes and lessons from the Initiative. This included reading several hundred documents within the grantee reports to extract both qualitative and quantitative information. The following sections describe our methodology for conducting the analysis and our findings. A.2. Methods Of the 99 grants funded, UI received reports and materials for 79 grants. Of the reports and materials provided, we used the following documents to explore outcomes: Grantee proposal summaries and recommendations (GPS): The GPS documents described the grantee s initial request for funding, including its organization history, proposed action plan, and an assessment of grantee s capacity and capabilities. Site visit reports: The Health Foundation staff members conducted periodic site visits to each grantee and these reports detailed the progress of the grantee in achieving its goals at the time of the site visit. Evaluation plan: Each grantee had an evaluation plan that included its plans to evaluate its implementation processes and the outcomes of its programs. Annual reports: These reports detailed the progress of the grantee in achieving its goals at the time of the annual report. Some grantees were funded for one year while others were funded for several years. Final reports and close-out reports: These reports included an overall narrative for each grant. These reports provided information on grantee outcomes, barriers and facilitators to implementation, and whether grantees were sustained after the grant period ended. The GPS was used as the primary document for creating a database to document outcomes because every grantee had a GPS in the files that were provided to UI. Not all of the grantee files included the other types of documents. Given that the GPS was written before grantees began their activities, we reviewed the other types of documents for outcomes, while recognizing the limitations of these data for reporting outcomes as previously mentioned. The documents were used to develop a coding protocol and database to explore the outcomes of the funded programs systematically. Data were summarized into the following 10 main categories, with the possible information included within those categories in parentheses: Grant purpose (planning, implementation, technical assistance, operations support, or research purposes) Program type (reentry or diversion) Target population served (gender, youth/juvenile, probationers/parolees, children/families, mental health clients, substance abuse clients) Treatment model (Assertive Community Treatment, mental health court, drug THE HEALTH FOUNDATION OF GREATER CINCINNATI 25

26 court, etc.) Mental health outcomes (percentage of clients reporting improvements in mental health symptoms, percentage reduction in symptom assessment scores, etc.) Substance use outcomes (percentage of clients reporting reduced substance use during the program, percentage of clients reporting reduced substance use after program completion, etc.) Recidivism outcomes (percentage of clients with no new charges, percentage reduction in number of arrests after admission, etc.) Sustainability (whether the program continued after the grant period) Lessons (open-ended field) Cost-benefit outcomes (cost savings, cost shifting) As mentioned previously, grantees used different definitions for mental health, substance abuse and recidivism outcomes and different methods of data collection. Therefore, more systematic analyses of outcomes across grantees could not be conducted with the data. Instead, the proportion of programs that reported a positive impact of more than 50 percent was calculated for each type of outcome: substance use, mental health, and recidivism. Where reported by the grantees, cost-benefit calculations were also summarized. A.3. Findings Recidivism, Mental Health, and Substance Use Outcomes The descriptive results discussed below are exploratory due to the very limited nature of the data that was provided to the research team. The findings listed here should not be overstated as positive or negative effects of the various planning processes and intervention programs used by the grantees under the Initiative. Rather, the compilation of grants reflects a modest start at systematically planning and evaluating interventions for individuals involved in the criminal justice system with behavioral health issues. Table 3: Number of Grantees Reporting Outcomes, by Outcome Type Outcome Type N Reported Percentage Mental Health Substance Use Criminal Justice % % % Cost Benefit % In the subsequent section, we describe the implementation grantees and their outcomes. As shown in Table 3, a majority of grantees reported outcomes for mental health, substance use, and criminal justice, where applicable. All 41 implementation grantees were supposed to measure criminal justice outcomes and system cost savings since the Initiative s purpose was to reduce involvement with the criminal justice system by focusing on behavioral health issues, thereby producing cost savings. To determine whether grantees should have reported substance use and/or mental health outcomes, we examined the grantee s treatment model or approach and determined whether the model intended to address substance use, mental health, or both. Most grantees reported criminal justice outcomes and far fewer grantees reported cost benefit outcomes. For the most part, grantees reported positive outcomes. However, because each grantee collected and defined outcomes differently, their outcomes could not be averaged. As shown in Table 4, grantees measured mental health, substance use, and criminal justice outcomes and their results in various ways. 26 LESSONS LEARNED THROUGH THE CRIMINAL JUSTICE SYSTEM INITIATIVE

27 Table 4: Criminal Justice, Mental Health and Substance Use Disorder Outcomes by Grantee Organization Hamilton County Community Mental Health Board Greater Cincinnati Behavioral Health Services Forensic and Mental Health Services, Inc Transitional Living, Inc. Scioto Paint Valley Mental Health Center Kentucky Department of Corrections Model ACT ACT ACT Criminal Justice Outcome 91% of clients were not incarcerated during the program 79% of clients reduced the number of jail days served a 63% of clients had no new charges after program admission through the end of the grant period N Mental Health Outcome 64 59% of clients reported improvement in symptoms a, b 14 75% of clients reported improvement in symptoms one or more years after admission b 16 19% reduction in average Symptoms of Distress Score on the Ohio Mental Health Consumer Outcomes Instrument at one year after admission ACT NR NR 71% of clients had an improved BASIS-24 score measured at various times ACT/IDDT NR NR 3% increase in average quality of life score six months after admission APIC 91% of clients did not receive a new conviction resulting in incarceration while participating in the program d Talbert House APIC 50% of clients were not incarcerated 12 months after release c NorthKey Community Care Assessment and Aftercare Treatment 91% of clients did not re-enter the Grant County Detention Center during the 2007 fiscal year 61 52% of clients had an improved BASIS-24 score a 13% reduction in average BASIS-24 score a N Substance Use Disorder Outcome 64 NA NA 16 57% of clients had reduced substance use a,b 14 NR NA NA 31 NA NA % increase in number of clients who abstained from substance use six months after admission (increase from 38% to 46%) THE HEALTH FOUNDATION OF GREATER CINCINNATI 27 N NR NR NR NR NR 58% of clients remained drug-free throughout their period of post-release control or parole according to urine testing, parole officer reports, and case manager reports 35 23% reduction in average BASIS-24 score at three months after admission NR NR 36% of the clients who began receiving services in the 2010 fiscal year reported a reduction in substance use in the 2010 fiscal year according to client self-report during treatment sessions Note: NR means not reported and NA means not applicable a b Grantee did not specify when it measured this outcome. Grantee did not specify how this was defined. c Grantee reported that less than 50 percent of program completers were reincarcerated, but did not give a precise number. d 25 percent of its clients (15 of 61) received a parole violation

28 Organization Community Mental Health Center, Inc. Community Behavioral Health Horizon Services Shawnee Mental Health Center Brown County Community Board of Alcohol, Drug Addictions and Mental Health Services Mental Health Recovery Services of Warren & Clinton Counties NorthKey Community Care Central Clinic Model Boundary Spanner Chemical Dependency Boundary Spanner CIT CIT Criminal Justice Outcome 84% of clients showed reduced involvement b in the criminal justice system a 94% of clients had no new arrests during the fiscal year of admission 99% of encounters with CIT did not result in arrest 93% of mental health cases handled by the sheriff s office did not result in arrest N Mental Health Outcome % of clients had improved mental health assessment scores six months after assessment N Substance Use Disorder Outcome 74 60% of clients reporting a reduction of alcohol or other drugs by six months after assessment according to selfreports 170 NA NA 80% reduction in the number of total days of substance use for all clients 30 days after admission according to drug screens and selfreports NR 100% of clients showed improved functioning in the Ohio Mental Health Consumer Outcomes Instrument a % of mental health cases were linked to community treatment services N NA NA 260 NA NA CIT NR NR NR NR NA NA CIT Criminal justice resource center 98% of encounters with CIT did not result in arrest 37% reduction in number of incarcerations in the 12 months after contact with CIT compared with 12 months before contact 64% of clients had no new arrest 12 months after release from jail Note: NR means not reported and NA means not applicable a Grantee did not specify when it measured this outcome. b Grantee did not specify how this was defined % of mental health cases resulted in a referral to mental health NR services 77 86% of clients reported an improvement in symptoms on the Ohio Mental Health Consumer Outcomes Instrument at one year after admission 342 NA NA 18 61% of clients reported a reduction in substance use 12 months after release according to the ADAS Clinical Outcomes Measure LESSONS LEARNED THROUGH THE CRIMINAL JUSTICE SYSTEM INITIATIVE

29 Organization Dearborn and Ohio Counties Prosecutor's Office Kentucky Department of Corrections Transitions, Inc Clermont County Mental Health and Recovery Board Lifespan Clermont County Mental Health & Recovery Board Scioto Paint Valley Mental Health Center Clermont County Mental Health & Recovery Board Clermont County Mental Health and Recovery Board Model Drug Court (juvenile) Drug Treatment Drug Treatment DUI Court (Matrix Model) Functional Family Therapy Criminal Justice Outcome N Mental Health Outcome N Substance Use Disorder Outcome NR NR NA NA NR NR 54% of clients had not committed a new offense or been incarcerated six months after release from jail or prison 75% of clients had no new arrests six months after completing program 174 NA NA 88% of clients were substance-free six months after discharge 145 NA NA 100% of clients had a reduction in substance use during treatment b 88% of clients had no new charges a 195 NA NA 90% of days with no substance use during treatment according to self-reports 76% of clients did not have involvement with the juvenile court during the grant period % of clients showed a positive change in functioning as measured by the Youth Outcome Questionnaire at discharge IDDT NR NR 56% of clients reported an improved BASIS-24 score six months after admission IDDT/ Maryland Community Criminal Justice Treatment Jail Diversion Team Matrix Model 92% of clients did not return to permanent confinement b within one year a 88% reduction in average days of incarceration per person for three years after the program compared with three years after the program 90% of clients had no new convictions after program admission during the grant period Note: NR means not reported and NA means not applicable a Grantee did not specify when it measured this outcome. b Grantee did not specify how this was defined. 60 7% improvement in average score on the Ohio Department of Mental Health Quality of Life Survey a % of clients had a negative urinalysis at FFT program completion 9 100% of clients self-reported reduced substance use at discharge 37 75% of clients with a reduction in substance use 12 months after release compared to pre-incarceration use according to self-reports 142 NR NR NA NA % of clients had an improved BASIS-24 score at discharge compared with admission NR 73% of clients had no substance use 30 days before discharge according to an SOQIC Assessment N THE HEALTH FOUNDATION OF GREATER CINCINNATI 29

30 Organization NKU Research Foundation Butler County Treatment Alternative Court Demonstration Hamilton County Community Mental Health Board Hamilton County Community Mental Health Board Butler County Educational Service Center The Counseling Center Scioto County Prosecuting Attorneys' Office Cincinnati Union Bethel Hamilton County Court of Common Pleas Model Mental Health Court Mental Health Court Mental Health Court (ACT) Mobile Crisis Team Multisystemic Therapy Recovery Coach Reformers Unanimous SAGE Model SAMI Court Criminal Justice Outcome 84% of clients were not reincarcerated on new charges while active with the program during 2010 N Mental Health Outcome 69 81% of clients had an improved BASIS-24 score at six months after admission N Substance Use Disorder Outcome 11 NA NA 90% of clients were not 31 NR NR NA NA reincarcerated a 69% of clients had decreased arrests and convictions during one year in the program compared with one year before admission 96% of calls involving mentally ill individuals resulted in transport of the individual to the hospital rather than arrest. 65% of clients did not commit new juvenile offenses after admission a 71% of clients did not reoffend b one year after completing program Five clients were not incarcerated on new charges or failure to comply with probation a 95% of clients did not have a new prostitution charge six or 12 months after program completion 100% of clients were not charged, convicted, and incarcerated of a new offense while active in the program e 26 59% of clients reported improvement in clinical symptoms on the Ohio Mental Health Consumer Outcomes Instrument at six months or 12 months after admission or upon discharge 81 NA NA 355 NR NR NA NA NR 48% average reduction in Child and Adolescent Functional Attainment Scale score a NR NR NR 71 NA NA 56% of clients reported reduced use during 2009 or 2010 according to drug screens or self-report NR NA NA 90% of clients with reduced drug use in the first or second year of the program according to self-reports NR 77% of clients had an improved BASIS-24 score three months after program completion compared with admission 30 76% of clients had an improved BASIS-24 score at three or more months after admission NR 94% of clients with a reduction in number of days of use per month according to self-report a 25 73% of clients had a significant reduction b in use compared to use prior to admission according to urine testing and self-reports a Note: NR means not reported and NA means not applicable a Grantee did not specify when it measured this outcome. b Grantee did not specify how this was defined. e One client of the 30 included in the measure was sentenced to prison for a probation violation that was not related to a new charge. f The grantee did not specify how many clients were included in this measure. It reported that 513 participants said they had no drug use, which the grantee said was more than 90 percent of the total sample. N f NR LESSONS LEARNED THROUGH THE CRIMINAL JUSTICE SYSTEM INITIATIVE

31 Organization Community Behavioral Health Horizon Services Butler County Alcohol and Drug Addiction Services ADAS Board Clermont County Mental Health and Recovery Board Central Clinic Court Clinic Cincinnati Union Bethel Talbert House The Vision of Hope Outreach Center TAPP House/ TC, Inc Model SAMI Court SAMI Court (IDDT) Screening, case management, short-term therapy, crisis intervention Screening, in-depth evaluations, intensive day treatment, community integration services TAMAR Teaching Family Training and treatment Transitional Housing Criminal Justice Outcome 50% of clients had fewer convictions 90 days after discharge compared with 90 days before admission 57% of clients were not arrested or charged with an offense during involvement with the court 65% of clients did not reoffend per year 87% of clients were not convicted of new offenses a 81% of clients were not convicted of a new criminal charge six months after release from jail 63% of clients who did not have new adjudications 12 months after discharge Note: NR means not reported and NA means not applicable a Grantee did not specify when they measured this outcome. b Grantee did not specify how this was defined. N Mental Health Outcome 48 33% of clients had an improved BASIS-24 score at discharge NR 80% of clients were not hospitalized during court involvement to 71% of clients reported a 20% improvement on the Ohio Scales each year of the grant period % of clients had reduced the level of symptom distress a NR 67% of clients showed a decrease in trauma symptoms a,b 76 69% of successfully discharged clients demonstrated a reduction in problem severity as measured by the Ohio Scales N Substance Use Disorder Outcome 48 35% of clients had a reduced number of positive drug screens NR 27% of clients did not experience a relapse N 48 NR NR NA NA 16 94% of clients reduced substance abuse a,b 16 NR 84% of clients showed a decrease in substance use a,b 53 76% of clients remained substance-free 30 days post-discharge from treatment according to self-reports NR NR NA NA 40% of clients were substance-free upon completion according to urine testing NR NR NA NA 0% of clients reduced substance use up to three months after intake compared with 30 days before intake. No clients had substance use 30 days before intake and four reported substance use at data collection NR THE HEALTH FOUNDATION OF GREATER CINCINNATI 31

32 A.4. Findings Cost Benefit Outcomes Of the 41 implementation grantees, 17 reported cost outcomes in their evaluation documents. Table 5 shows the granteecalculated costs and benefits of their programs. Because each grantee measured cost savings over different periods of time, each cost savings figure was averaged over its measurement period. The reported per month costs saved ranged from a loss of $1,517 per client to a savings of $45,316 per client, with only one grantee reporting an overall loss. Although these cost findings are encouraging, they should be taken with caution. None of the grantees compared participant outcomes to a comparison group; therefore, the cost effectiveness of the grantee programs relative to no intervention, another intervention, or businessas-usual services is unknown. Table 5: System Savings Generated by Grantees Organization Diversion Programs Transitional Living, Inc. NorthKey Community Care NKU Research Foundation Cincinnati Union Bethel Shawnee Mental Health Center, Inc. Butler County Alcohol and Drug Addiction Services Board Model Savings Reported n Savings per Client Method ACT (-)$87, (-)$1,517 Cost of sentenced days (Cost of days served + Cost of services) over the grant period of 42 months CIT $20, $60 Cost of three jail days for each potential charge per diverted person over the grant period of 39 months Mental Health Court SAGE Model $88, $993 Costs of incarceration saved by diversion a (Cost of jail days served + Cost to operate program) for 12 months $484, $1,204 Costs of jail days not served a over 43 months CIT $105, $1,382 Cost of jail days avoided a over the grant period of 44 months SAMI Court (IDDT) a Grantee did not define this figure. $76, $2,547 Cost of hospital days in the year before admission + Costs of jail time before admission + Costs of prison days avoided a (Cost of hospital days in year of treatment + Cost of jail days during program) over the grant period of 24 months Did not subtract program costs x x x x 32 LESSONS LEARNED THROUGH THE CRIMINAL JUSTICE SYSTEM INITIATIVE

33 Organization Clermont County Mental Health & Recovery Board Community Behavioral Health Horizon Services Butler County Educational Service Center Hamilton County Court of Common Pleas Clermont County Mental Health & Recovery Board Greater Cincinnati Behavioral Health Services Clermont County Mental Health & Recovery Board Clermont County Mental Health & Recovery Board Reentry Programs NorthKey Community Care Model DUI Court (Matrix Model) Chemical Dependency Boundary Spanner Multisystemic therapy SAMI Court Jail Diversion Team Savings Reported n Savings per Client Method $623, $3,247 Cost of jail days suspended (Cost of jail days served + Costs of treatment) over 36 months $650, $3,827 Cost of the average percentage of days served of sentence (Cost of days served for sanctions) over 24 months between $5,045 and $23,100 per person 63 between $5,045 and $23,100 Cost of 157 days of incarceration (Cost of 157 days of service) $152, $5,095 Cost of prison days sentenced and not served in the first year of service (Cost of jail days served for sanctions + Costs of treatment) over 12 months $1,400, $9,859 Cost of jail days suspended (Costs of treatment) over 24 months ACT $340, $24,349 Cost of jail days avoided + Cost of hospital days avoided b for 18 months Matrix Model $4,947, $34,601 Cost of sentenced jail or prison days (Cost of jail or prison days served + Costs of treatment) over 33 months IDDT $2,401, $45,316 Cost of jail days suspended (Costs of jail days served + Costs of treatment) over 12 months Assessment and aftercare treatment $27, $785 Cost of incarceration days prevented a (Cost of days served + Cost of treatment) over the reporting period of 12 months Did not subtract program costs a Grantee did not define this figure. b This calculation assumes that their clients would have been immediately reincarcerated for the remaining length of the grant period after returning to the community. x x THE HEALTH FOUNDATION OF GREATER CINCINNATI 33

34 Organization Kentucky Department of Corrections Model Diversion/Reentry Programs Scioto County Prosecuting Attorneys Office Savings Reported n Savings per Client Method APIC $2,690,404 b 159 $16,921 b Cost of the number of days not served after release from incarceration during the grant period (Costs of treatment) over 36 months Reformers Unanimous Program $2,675, $7,040 Cost of jail or prison days prevented (Costs of sanction days served + Costs of treatment) over 24 months Did not subtract program costs b This calculation assumes that its clients would have been immediately reincarcerated for the remaining length of the grant period after returning to the community. 34 LESSONS LEARNED THROUGH THE CRIMINAL JUSTICE SYSTEM INITIATIVE

35 THE HEALTH FOUNDATION OF GREATER CINCINNATI 35

36 Rookwood Tower 3805 Edwards Road, Suite 500 Cincinnati, OH Toll-free