Table of Contents Table of Contents Table of Contents... 1 Acknowledgements... 4 Definitions and Abbreviations... 5 Executive Summary... 6 1.0 Background Information... 8 Introduction... 8 Overview of FSP Program... 9 History of the FSP DCR... 9 Data Collection and Reporting... 10 2.0 Project Overview... 11 Engaging Counties in the Project... 11 Table 1: List of Counties Participating in Project... 13 County Meeting Format... 14 Project Deliverables... 15 Table 2: Project Deliverables... 15 DCR Data Quality Preliminary Results and Discussion... 16 Table 3: Data Quality Preliminary Indicator Comparison of Participating and Non- Participating Counties... 18 3.0 County Summary Reports... 19 3.00 Summary Report Key for All Counties... 20 3.01 Alameda County... 21 3.02 Alpine County... 22 3.03 Amador County... 23 3.04 Butte County... 24 3.05 Calaveras County... 25 3.06 Colusa County... 26 3.07 Contra Costa County... 27 3.08 Del Norte County... 28 3.09 El Dorado County... 29 3.10 Fresno County... 30 3.11 Glenn County... 31 3.12 Humboldt County... 32 3.13 Imperial County... 33 3.14 Inyo County... 34 3.15 Kern County... 35 FSP Data Correction Project Page 1
Table of Contents 3.16 Kings County... 36 3.17 Lake County... 37 3.18 Lassen County... 38 3.19 Los Angeles County... 39 3.20 Madera County... 40 3.21 Marin County... 41 3.22 Mariposa County... 42 3.23 Mendocino County... 43 3.24 Merced County... 44 3.25 Modoc County... 45 3.26 Mono County... 46 3.27 Monterey County... 47 3.28 Napa County... 48 3.29 Nevada County... 49 3.30 Orange County... 50 3.31 Placer County... 51 3.32 Plumas County... 52 3.33 Riverside County... 53 3.34 Sacramento County... 54 3.35 San Benito County... 55 3.36 San Bernardino County... 56 3.37 San Diego County... 57 3.38 San Francisco County... 58 3.39 San Joaquin County... 59 3.40 San Luis Obispo County... 60 3.41 San Mateo County... 61 3.42 Santa Barbara County... 62 3.43 Santa Clara County... 63 3.44 Santa Cruz County... 64 3.45 Shasta County... 65 3.46 Sierra County... 66 3.47 Siskiyou County... 67 3.48 Solano County... 68 3.49 Sonoma County... 69 3.50 Stanislaus County... 70 3.52 Tehama County... 71 3.53 Trinity County... 72 3.54 Tulare County... 73 FSP Data Correction Project Page 2
Table of Contents 3.55 Tuolumne County... 74 3.56 Ventura County... 75 3.57 Yolo County... 76 3.63 Sutter-Yuba Counties... 77 3.65 Berkeley City... 78 3.66 Tri-City... 79 Appendix A... Appendix B... FSP Data Correction Project Page 3
Acknowledgements Acknowledgements We would like to thank and acknowledge the input and materials from many people and organizations. We value the participation, input and feedback from State, county, provider, vendor and additional stakeholder staff who participated in this project. We would especially like to thank counties who submitted example FSP data reports to share with this project and with other counties: Los Angeles, Orange, Sacramento, San Diego, San Francisco, Shasta, and Stanislaus. FSP Data Correction Project Page 4
Definitions and Abbreviations Definitions and Abbreviations Term Meaning Deliv. 16 State Data Correction 3M Quarterly Assessment ADL Activities of Daily Living Adult Adults of Ages 26-59 CCR California Code of Regulations Child/Youth Children of Ages 0-15 CIMH California Institute of Mental Health CSI Client Services Information DCR Data Collection and Reporting DMH California Department of Mental Health DHCS Department of Health Care Services EHR Electronic Health Record EPLD Enhanced Partner-Level Data FSP Full Service Partnership IADL Instrumental Activities of Daily Living ITWS Information Technology Web Services KET Key Event Tracking MH Mental Health MHSA Mental Health Services Act MHBG Mental Health Services Block Grant MHP Mental Health Plan MHSIP Mental Health Statistics Improvement Program Older Adult Older Adults of Ages 60+ PAF Partnership Assessment Form Partner A Partner of the Full Service Partnership PC Penal Code PLD Partner-Level Data MHSA Performance Outcomes and Quality Improvement PSC Partnership Service Coordinator SED Seriously Emotionally Disturbed State The State of California TAY Transitional Age Youth (Ages 16 25) TOC Table of Contents WIC California Welfare and Institution Code XML Extensible Markup Language FSP Data Correction Project Page 5
Executive Summary Executive Summary Project Scope: The Full Service Partnership (FSP) Data Collection & Reporting (DCR) State Data Correction & Cleaning Project was sponsored by the Mental Health Services Oversight and Accountability Commission (MHSOAC) to provide data correction and cleaning assistance for improved FSP DCR data quality. This included developing data quality measures for FSP DCR data. This project focused on engaging staff from the 59 California County Mental Health Plans ( Counties ) responsible for data collection and submission. It helped staff to identify barriers to quality data and to evaluate data quality through indicator summaries. The project provided each county with individualized assistance with and recommendations for improving data quality. In addition, tools and custom training were provided to enable Counties to more efficiently identify and resolve data quality issues. Project Deliverables: The State Data Correction & Cleaning Project consists of five primary deliverables. The 5 Deliverables of the State Data Correction & Cleaning Project: 1. Draft County DCR Data Quality Metrics Reports & Individual County Meetings 2. County Summary and Final Project Report 3. Enhanced Partner-Level Data Templates 4. Electronically Fillable FSP Forms 5. Catalog Summary of County-used FSP Indicators Project Outcome Measures: The Data Quality Preliminary Results presented in this document provide aggregate data quality indicators for all Counties for two time periods (taken from before, and during the State Data Correction & Cleaning Project). These indicators provide a quantitative metric to measure the early trends toward improvement of FSP DCR data quality. FSP Data Correction Project Page 6
Executive Summary Preliminary Project Results: Due to communication with all Counties regarding the goals of the project, both Counties who participated and those who did not directly participate became engaged in data quality improvement efforts. Out of 59 Counties, 37 Counties participated in the project while 22 did not participate. For Counties who currently have data in the FSP DCR, a total of 86.2% of Counties who participated in the project and 80.0% of Counties who did not participate showed early trends toward improvements in at least one of the data quality indicators analyzed. Counties who participated in the project improved more often across all five indicators as compared to Counties who did not participate in the project, ranging from +3.3% to +22.1% more improvement across the indicators measured. Note: The preliminary analysis was performed using FSP DCR data submitted before the end of the project. In addition, since the data quality metrics evaluate the entire partnership history for continuously active partners, the metrics may require several months or more to change considerably. Thus, this preliminary analysis simply identifies the trend of whether the data was improving in a county during the project; the analysis does not represent the full effects of the project or the final data quality improvements realized. Post-project analyses to measure the overall improvements were not performed due to time constraints of this project. Individual County Summaries: The County Summary Reports presented in this document provide a brief summary for each county of information gathered during the State Data Correction & Cleaning Project including each county s specific FSP DCR data collection process, county-specific action recommendations for improving data quality, and a set of county-specific information compiled during the project (including a reporting of key data quality indicators over the time period analyzed by the Data Quality Preliminary Results report). FSP Data Correction Project Page 7
1.0 Background Information 1.0 Background Information Introduction The Mental Health Services Oversight and Accountability Commission (MHSOAC) is responsible for providing oversight of the Mental Health Services Act (MHSA) and its components, as well as the broader community mental health system in California. Within this role, the MHSOAC ensures accountability to taxpayers and the public. As one of its oversight approaches, the MHSOAC has adopted a commitment to pursuing meaningful evaluation of the MHSA and greater community mental health system. The ability to successfully use evaluation methods to provide oversight and hold responsible entities accountable for their roles within the MHSA is dependent upon access to valid data that is reliably reported and made available to the MHSOAC on a regular basis. The MHSOAC has identified areas within the current county-level and statewide data collection and reporting systems that are problematic and in need of improvement. The MHSOAC has begun to directly address some of these issues, although it is not clear that the MHSOAC was intended to provide this function. Nonetheless, the MHSOAC is committed to advocating for improvement of the current data collection and reporting systems since the MHSOAC is dependent on the information that is made available via these systems in order to fulfill the statutory role in evaluation of the public mental health system. Improvements in data collection and reporting systems will increase confidence in the information obtained and conclusions drawn about the state of the MHSA and California community mental health system. Outcomes from the MHSA Full Service Partnership (FSP) are reported to the Data Collection and Reporting (DCR) system. Since August 2011, the MHSOAC has sponsored efforts to improve the quality of the FSP data. The State Data Correction & Cleaning Project initially focused on improving the quality of data in the FSP DCR through one-on-one meetings with each county to provide technical assistance. The meetings addressed specific barriers to high quality data for the county; and documentation, training, customized tools, and recommendations were provided to assist the county s data quality improvement efforts. Where possible, tools which addressed common issues raised by multiple Counties were developed, refined, and released to all Counties. FSP Data Correction Project Page 8
1.0 Background Information Overview of FSP Program There are 59 County Mental Health Plans (MHPs), including 56 Counties + Yuba/Sutter combined Counties + Berkeley City region + Tri-City region (hereafter Counties ), that receive state-based funding for mental health services as a result of California Proposition 63 and the MHSA which provides increased funding to support California s county mental health programs. The MHSA imposes a one percent income tax on annual personal income in excess of $1 million to address a broad continuum of prevention, early intervention and service needs. The MHSA provides the necessary infrastructure, technology and training elements that effectively support this system, with the purpose of promoting recovery for individuals with serious mental illness. Counties develop customized plans for mental health services in accordance with State requirements, including providing for significant local stakeholder input and involvement. To this end, MHSA funds a special program called the Full Service Partnership (FSP). FSP programs provide a full spectrum of mental health services to children/youth (Child, ages 0 15) and transition age youth (TAY, ages 16 25) and adults (Adult, ages 26 59) and older adults (Older Adult, ages 60+) who have a serious mental disorder. Individuals served by the FSP programs are referred to as partners in the program. Additional criteria, described in Welfare and Institution Code (WIC) 5600.3, must also be met. A basic principle of the program is its flexible funding, which assures that Counties may provide whatever services are necessary to help the individual access needed resources. Services offered by local programs include assessing the individual s needs; providing shelter/housing; establishing identification and legal assistance needs; and providing food, clothing, showers, dental care, medical, psychiatric, alcohol/drug treatment, and social rehabilitation. History of the FSP DCR The MHSA Section 5892(d) allowed investing a portion of the MHSA funds on administrative and technological needs to ensure adequate research and evaluation regarding the effectiveness of services being provided and the achievement of the outcome measures. In 2005, the Department of Mental Health (DMH) was permitted funding to develop the FSP Data Collection and Reporting (DCR) system. By January 1, 2006 the interim FSP DCR was available for county use, and 10 Counties used the interim FSP DCR. In June 2007, an enhanced version of the FSP DCR was made available, which allowed Counties to key-enter data or upload data via Extensible Markup Language (XML) batch file submission. In 2012, responsibility for the FSP DCR was transferred to the Department of Health Care Services (DHCS). DHCS maintains the FSP DCR and works with Counties to develop data validation rules, fixes and enhancements to the system. FSP Data Correction Project Page 9
1.0 Background Information Data Collection and Reporting Counties report partner information and outcomes of the FSP program directly to the FSP DCR system. Current regulations require Counties to collect partner outcome FSP data (CCR Title 9 3620.10) and submit it to the State within 90 days (CCR Title 9 3530.30). Counties submit data for three different types of partner assessments into the FSP DCR through an online user interface or by XML batch file submission. Through the FSP DCR system, the Partnership Assessment Form (PAF) gathers baseline information about the partner, while Key Event Tracking (KET) and Quarterly Assessment (3M) gather follow up information. The questions on each of the PAF, KET and 3M forms may differ slightly depending on the four age groups (Child/Youth, TAY, Adult and Older Adult). FSP Data Correction Project Page 10
2.0 Project Overview 2.0 Project Overview The purpose of the State Data Correction & Cleaning Project (hereafter The Project ) was to assist Counties in overcoming barriers to high quality FSP data in the FSP DCR. Each county was contacted and invited to participate in the Project. Of 59 Counties, 37 Counties directly participated. Each county attended one-on-one meetings to discuss barriers to FSP DCR data quality. Details about the participating Counties can be found in Table 1. While not every county participated in the Project, many of the non-participating Counties were aware of the Project, and some made efforts for data quality improvement external to direct Project efforts. At least five additional Counties (Berkeley City, Fresno, Glenn, Santa Cruz, and Siskiyou) made contact with the Project team to discuss the Project but county staffing, alternative priorities, or other county barriers prevented these Counties from participating in the Project. Engaging Counties in the Project From September, 2012 through March, 2013, several efforts were made to encourage Counties to participate in the Project. In addition, Counties who participated received a draft of the Data Quality Metrics Report with their county data from fiscal year 2010/2011. This incentive was designed to both give Counties feedback on the current status of county data quality and allow Counties to provide input on the final format of the Data Quality Metrics Report tool developed as part of the Project. Attempts to promote the Project and engage Counties are detailed in the following list. FSP Data Correction Project Page 11
2.0 Project Overview Dates and Activities for Project Promotion: September 10, 2012: Project announced and flyer for in-person visit schedule emailed through current list serve September 26, 2012: Project reminder and flyer for in-person visit schedule resent through current list serve September 28, 2012: Project discussed and questions answered during Statewide Data Correction and Best Practices Final Webinar October 31, 2012: Project reminder and flyer for in-person visit schedule resent through current list serve November 9, 2012: Project discussed and questions answered during Statewide DCR Data Quality Improvement Webinar December 16, 2012: Project reminder and flyer for in-person visit schedule resent through current list serve January 3, 2013: The MHSOAC and DHCS lists of MHSA email contacts were used to individually invite all remaining Counties to participate in the Project January 18, 2013: Project discussed and questions answered during Statewide DCR Data Quality Improvement Webinar March 15, 2013: Project update provided at Statewide DCR Data Quality Improvement Webinar FSP Data Correction Project Page 12
2.0 Project Overview Table 1: List of Counties Participating in Project Count County Date of Meeting Type of Meeting DCR County Type County Electronic Health Record System 1 Alameda 01/25/13 Web Meeting XML, Not Certified Clinician s Gateway 2 Butte 01/25/13 Web Meeting Avatar 3 Calaveras 05/21/13 Web Meeting Anasazi 4 Del Norte 03/01/13 Web Meeting Anasazi 5 Humboldt 01/08/13 In-Person Avatar 6 Kern 02/21/13 Web Meeting Anasazi 7 Kings 01/07/13 In-Person Anasazi 8 Lake 12/19/12 In-Person Anasazi 9 Lassen 02/07/13 Web Meeting Share Care 10 Los Angeles 12/10/12 In-Person XML, Certified Internal 11 Madera 02/08/13 Web Meeting Anasazi 12 Marin 03/15/13 Web Meeting XML, Not Certified Clinician s Gateway / Share Care 13 Mendocino 02/14/13 Web Meeting XML, Not Certified Avatar 14 Merced 01/07/13 In-Person Anasazi 15 Mono 02/21/13 Web Meeting Share Care 16 Monterey 01/15/13 Web Meeting XML, Not Certified Avatar 17 Napa 01/09/13 In-Person Anasazi 18 Nevada 12/19/12 In-Person Anasazi 19 Orange 12/10/12 In-Person XML, Certified Caminar / Building Internal 20 Placer 04/05/13 Web Meeting XML, Certified Avatar 21 Riverside 01/10/13 Web Meeting XML, Not Certified Avatar / Internal 22 Sacramento 12/19/12 In-Person Avatar 23 San Benito 02/14/13 Web Meeting Anasazi 24 San Bernardino 12/11/12 In-Person XML, Certified Internal 25 San Diego 12/20/12 Web Meeting Anasazi 26 San Francisco 11/02/12 In-Person Avatar 27 San Luis Obispo 12/12/12 In-Person Anasazi 28 Santa Barbara 12/12/12 In-Person XML, Not Certified Clinician s Gateway / Share Care 29 Santa Clara 12/14/12 In-Person Epic Care 30 Shasta 12/17/12 In-Person Anasazi 31 Sierra 01/16/13 Web Meeting None 32 Stanislaus 02/21/13 Web Meeting Anasazi and Caminar (for FSP) 33 Sutter-Yuba 03/14/13 Web Meeting Anasazi 34 Trinity 01/16/13 Web Meeting Anasazi 35 Tulare 01/07/13 In-Person Avatar 36 Tuolumne 04/05/13 Web Meeting Anasazi 37 Yolo 01/09/13 In-Person Avatar FSP Data Correction Project Page 13
2.0 Project Overview County Meeting Format After a county expressed interest in participating in the Project, the county was sent a draft of its own DCR Data Quality Metrics Report for fiscal year 2011/2012. An example of the report can be found in Appendix A. In addition, the county was asked to complete a questionnaire, found in Appendix B, about perceptions of data quality and barriers to data quality. During the meetings, each county described their DCR data collection and submission process. The Counties and the Project team worked together to identify potential barriers to data quality. Once potential barriers were identified, suggestions and recommendations were provided for addressing those barriers. Specific solutions were offered when possible. Some challenges were common among a number of Counties. These common issues often included difficulty summarizing data from the DCR to support various decision-making and tracking processes at the county. Suggestions for helpful summary reports were identified from several Counties. As a result, some of the resource allotted for the Project was used to enhance the MHSOAC Partner-Level Data (PLD) Templates which were previously developed within the contract. The Enhanced Partner-Level Data (EPLD) Templates included additional Partner Alert Reports and Summary Reports. The seven additional reports in the EPLD Templates are described in the next section, Project Deliverables. Some recommendations were given to all Counties. These universal recommendations appear in the following list. Since these recommendations were provided during nearly every meeting with the Counties, they will not appear in the individual county reports, for brevity. Universal Recommendations Provided to All Counties: Provide partner summary reports to Partnership Service Coordinators (PSCs) to assist in case management and increase the value of the FSP data to the primary data collectors thereby increasing their investment in data quality Provide summary reports of FSP indicators together with data quality indicators to programs, providers and directors; due to the nature of the FSP data collection processes, FSP indicators cannot be interpreted without knowledge of data quality Perform internal cross-checks of DCR data to internal EHR or other systems on a scheduled basis as resources permit Provide training to staff on filling out the FSP forms; due to the layout of the discontinuation reasons on the form, this section requires special consideration during the training as busy staff may simply select the first option, which may not be the best option FSP Data Correction Project Page 14
2.0 Project Overview Project Deliverables The five main project deliverables are described in Table 2. Project Deliverable 1 was delivered directly to participating Counties for county use. Project Deliverables 2-5 are available through the MHSOAC as resources for all Counties. Table 2: Project Deliverables Deliverable Name Description 1 Draft County DCR Data Quality Metrics Reports Individual draft reports delivered to Counties who participated in the Project 2 County Summary and Final Project Report This document includes the summary of each county data meeting and data quality metrics provided in Section 3.0 of this report 3 Enhanced Partner-Level Data (EPLD) Templates The following 7 reports were added to the EPLD Templates: Partner Alert Reports: 1. 3Ms Due by Month Report 2. All Active Partners Current Status Report 3. Active Partners Residential Alerts Report 4. Stagnant Partners Alert Report 5. Duplicate KETs Report Summary Reports: 6. Data Quality Metrics Report 7. Client Served Demographics Report (Exhibit 6 Data Facilitator) 4 Electronically Fillable FSP Forms FSP forms modified to have electronically fillable fields so that filled forms can be saved or shared electronically 5 Catalog Summary of County-used FSP Indicators Document of example FSP indicators used by stakeholders or Counties FSP Data Correction Project Page 15
2.0 Project Overview DCR Data Quality Preliminary Results and Discussion FSP DCR data quality evaluations were provided to all Counties who participated in the Project. Counties were advised on ways to improve their current workflow and processes to collect and submit FSP DCR data. Counties were encouraged to correct data for all currently active partners and, as resources permitted, year by year for previous partners. However, most Counties expressed a lack of resources for historical data review. While the impact of the efforts cannot be measured immediately, this section presents a preliminary analysis of data quality for a time period ( Timeframe 1 ) before the Project and for a time period ( Timeframe 2 ) during the Project. This preliminary analysis compares the data metrics during the entire partnership among continuous partners who were actively served during a 6-month time period before (1/1/2011-6/30/2011) and during (10/1/2012-3/31/2013) the Project. However, as the Project extended to June, 2013, and as many county meetings did not occur until after the second time period, this preliminary analysis cannot show the full benefit of data quality improvements related to this Project. Four areas of data quality were analyzed as described in the following list. Data Quality Metrics Analyzed: Counties who Improved 3Ms Includes Counties who increased the percent of 3Ms submitted which were due for continuously active partnerships as of Timeframe 2 as compared to Timeframe 1 Counties who Reduced Partial 3Ms Includes Counties who reduced the percent of 3Ms submitted which were incomplete for continuously active partnerships as of Timeframe 2 as compared to Timeframe 1 Counties who Improved KETs for Partners at 180 Days Includes Counties who increased the percent of continuously active partnerships with at least 180 days of service which had at least one KET assessment submitted as of Timeframe 2 as compared to Timeframe 1 Counties who Improved KETs for Partners at 365 Days Includes Counties who increased the percent of continuously active partnerships with at least 365 days of service which had at least one KET assessment submitted as of Timeframe 2 as compared to Timeframe 1 Counties who Improved DCR Data in Any Area Includes Counties who improved in at least one of the four metrics listed above as of Timeframe 2 as compared to Timeframe 1 FSP Data Correction Project Page 16
2.0 Project Overview Note: There are several limitations to these metrics. The data quality metrics analyzed are representative data quality indicators. These measures only identify whether data was submitted, but they do not evaluate the accuracy of the data submitted. Since the data quality metrics evaluate the entire partnership history for continuously active partners who were active during the time period identified, the metrics may require several months or more to change considerably. Thus, this analysis simply identifies the trend of whether the data was improving in a county during the Project; this analysis does not represent the full effects of the Project or the final improvements realized. As seen in Table 3, not all Counties have data in the FSP DCR which could be evaluated for the Project. Of the 37 Counties who participated in the Project, only 29 Counties had data in the FSP DCR which could be analyzed in this preliminary analysis. Many XML Counties were not yet certified and therefore had not yet been able to submit data to the FSP DCR. However, these Counties worked to improve data collection processes and data submission to their EHR systems to improve the quality of local data and thus improve the quality of partner data for future submissions to the FSP DCR. Counties who participated in the Project improved more often across all five indicators as compared to Counties who did not participate in the Project, ranging from 3.3% to 22.1% more improvement per indicator than Counties who did not participate in the Project. However, as discussed previously, many Counties who did not participate in the Project also actively worked to improve data quality. Thus, both Counties who participated and those who did not experienced data quality improvements in relation to the Project. A total of 86.2% of Counties who participated in the Project and 80.0% of Counties who did not participate had improvements in at least one of the data quality indicators analyzed. FSP Data Correction Project Page 17
2.0 Project Overview Table 3: Data Quality Preliminary Indicator Comparison of Participating and Non-Participating Counties Participating Counties Participating Counties Participating County Improvement over Non-Participating Counties Data Quality Indicator Counties 22 37 n/a Counties with Data in DCR (for evaluation) 20 29 n/a % Counties who Improved 3Ms 45.0% 48.3% +3.3% % Counties who Reduced Partial 3Ms 40.0% 62.1% +22.1% % Counties who Improved KETs for Partners at 180 Days 35.0% 51.7% +16.7% % Counties who Improved KETs for Partners at 365 Days 45.0% 51.7% +6.7% % Counties who Improved DCR Data in Any Area 80.0% 86.2% +6.2% FSP Data Correction Project Page 18
3.0 County Summary Reports The County Summary Reports provide a summary of each county for information gathered during the Project, including each county s specific FSP DCR data collection process, countyspecific action recommendations for improving data quality, and a set of county-specific information compiled during the Project. The County Summary Reports are divided into three subsections. The Three Sections of the County Summary Reports: 1. County DCR Data Collection Process Summary 2. County-Specific Recommended Actions 3. The provides the information compiled regarding each county and is divided into ten elements. The Ten Elements: 1. FSP Data Quality Meeting Date 2. or XML Submission 3. Active Partners Serving on June 30, 2011 4. County Electronic Health Record 5. Contracted Providers 6. Completeness of 3Ms Before Project 7. Change in % 3Ms Submitted 8. Change in Incomplete 3Ms Submitted 9. Change in Partners >180 Days with KET 10. Change in Partners >365 Days with KET Refer to Section 3.00 for the Summary Report Key which shows the format of the report provided for each county and includes a detailed description of each section and element of the report. FSP Data Correction Project Page 19
3.00 Summary Report Key for All Counties The first paragraph of each county report provides a summary of the county FSP DCR data collection processes. The two digits after the section number (i.e., 3.xx) represent the county identifier. A summary of county-specific recommended actions communicated to the county appears in this part of the report All Counties received some general recommendations as listed in 2.0 Project Overview. FSP Data Quality The date of the meeting to discuss improving FSP DCR data processes Meeting Date: and quality Identifies if the county was an online or an XML county at the time of the Project (11/01/12 /2013) Active Partners Serving Provides a daily caseload snapshot on a common day for all Counties on June 30, 2011: Electronic Health Identifies the electronic system used to collect general mental health Record: partner data Identifies if the county contracts to providers to serve FSP clients Completeness of 3Ms Identifies the completeness of 3Ms submitted compared to 3Ms due Before Project: during the entire partnership among continuous partners who were actively served during a 6-month time period before (1/1/2011- Change in % 3Ms Submitted: Change in Incomplete 3Ms Submitted: Change in Partners >180 Days with KET: Change in Partners >365 Days with KET: 6/30/2011) the Project. Compares the % of 3Ms submitted which were due during the entire partnership among continuous partners who were actively served during a 6-month time period before (1/1/2011-6/30/2011) and during (10/1/2012-3/31/2013) the Project. An increase represents an improvement in data quality. Compares the % of 3Ms submitted which were submitted incomplete during the entire partnership among continuous partners who were actively served during a 6-month time period before (1/1/2011-6/30/2011) and during (10/1/2012-3/31/2013) the Project. A decrease represents an improvement in data quality. Compares the % of partners who were served for at least 180 days who had at least one KET submitted among continuous partners who were actively being served during a 6-month time period before (1/1/2011-6/30/2011) and during (10/1/2012-3/31/2013) the Project. An increase represents an improvement in data quality. Compares the % of partners who were served for at least 365 days who had at least one KET submitted among continuous partners who were actively being served during a 6-month time period before (1/1/2011-6/30/2011) and during (10/1/2012-3/31/2013) the Project. An increase represents an improvement in data quality. FSP Data Correction Project Page 20
3.01 Alameda County Alameda County is in the process of becoming a certified XML submitting county for the DCR but has not yet been able to submit FSP data to the DCR. Alameda County has an experienced analytical team and uses SPSS as the preferred package to analyze county mental health data. However, the team has not yet developed many internal reports on the FSP data. Alameda County is interested in using the MHSOAC EPLD Templates for their data quality needs. Participate in the FSP DCR XML county workgroup Continue to work with DHCS to complete the certification process to submit data to the FSP DCR If the certification process does not progress quickly, consider formatting data extracts from EHR for direct upload to the EPLD templates Implement the EPLD Templates to one of the contracted providers as a pilot to identify its utility for the county, and extend to other providers as needed FSP Data Quality Meeting Date: January 25, 2013 Active Partners Serving on June 30, 2012: Completeness of 3Ms Before Project Change in % 3Ms Submitted: Change in Incomplete 3Ms Submitted: Change in Partners >180 Days with KET: Change in Partners >365 Days with KET: XML, not certified No Data in FSP DCR Clinician s Gateway Yes No Data in FSP DCR No Data in FSP DCR No Data in FSP DCR No Data in FSP DCR No Data in FSP DCR FSP Data Correction Project Page 21
3.02 Alpine County Alpine County did not participate in the Project, and Alpine has not submitted FSP data to the DCR. Review process to collect and submit data to the FSP DCR to identify and address barriers to submitting PAF, 3M, and KET assessments Utilize the online FSP DCR to submit FSP data Utilize the MHSOAC provided FSP DCR training materials (user manuals, curriculums and videos) FSP Data Quality Meeting Date: Active Partners Serving on June 30, 2012: Completeness of 3Ms Before Project Change in % 3Ms Submitted: Change in Incomplete 3Ms Submitted: Change in Partners >180 Days with KET: Change in Partners >365 Days with KET: None Neither, recommend online No Data in FSP DCR No Data in FSP DCR No Data in FSP DCR No Data in FSP DCR No Data in FSP DCR No Data in FSP DCR FSP Data Correction Project Page 22
3.03 Amador County Amador County did not participate in the Project. Amador submits FSP data using the online DCR. Review process to collect and submit data to the FSP DCR to identify and address barriers to KET and 3M submission Consider implementing the EPLD Templates to provide feedback to PSCs and monitor FSP data quality Utilize the MHSOAC provided FSP DCR training materials (user manuals, curriculums and videos) FSP Data Quality Meeting Date: None Active Partners Serving on June 30, 2012: 48 Completeness of 3Ms Before Project > 0% Change in % 3Ms Submitted: + 38.9% Change in Incomplete 3Ms Submitted: + 1.3% Change in Partners >180 Days with KET: + 27.1% Change in Partners >365 Days with KET: + 11.5% FSP Data Correction Project Page 23
3.04 Butte County Butte County uses the online FSP DCR with its approximately five contracted providers. Data is sometimes collected on the FSP paper forms. Other times, data is entered into the Avatar EHR system and manually transferred over to the online DCR. Butte has modified Avatar to include the FSP module in the development environment but has not rolled this module out into production yet. Butte County uses Crystal Reports to generate customized FSP reports through Avatar. Butte County is interested in becoming an XML submitting county for the DCR. Have providers enter provider number into the FSP DCR Update FSP DCR program names to reflect current programs Formulate a county-specified operational definition of target criteria for FSPs Contact nearby FSP Counties who are listed as interested in joining a regional FSP workgroup for mentoring Contact DHCS to investigate becoming an XML submitting county Participate in the FSP DCR XML county workgroup FSP Data Quality Meeting Date: January 25, 2013 Active Partners Serving on June 30, 2012: 338 Avatar Yes Completeness of 3Ms Before Project > 45% Change in % 3Ms Submitted: + 1.6% Change in Incomplete 3Ms Submitted: + 1.2% Change in Partners >180 Days with KET: + 27.5% Change in Partners >365 Days with KET: + 24.9% FSP Data Correction Project Page 24
3.05 Calaveras County Calaveras County utilizes the online DCR to submit FSP data. County staff serve all FSP clients and no providers are contracted for the FSP program. A lead staff member enters most data into the online FSP DCR and sends reminders to PSC for 3M assessments. The county expressed issues with CSI number not matching between CSI and the DCR for new FSP client enrollments. The county recently moved from the Avatar EHR to Anasazi EHR in the spring of 2013, and the county is now turning more attention to FSP data quality. The county recently implemented the EPLD Templates. The county has high quality FSP DCR indicators with excellent assessment completion rates. Run reports from EPLD Template quarterly to provide feedback to PSCs and monitor FSP data quality Utilize the existing weekly case management meetings to remind staff about 3M and KET assessments FSP Data Quality Meeting Date: May 21, 2013 Active Partners Serving on June 30, 2012: 21 Anasazi No Completeness of 3Ms Before Project > 90% Change in % 3Ms Submitted: - 2.2% Change in Incomplete 3Ms Submitted: + 9.8% Change in Partners >180 Days with KET: + 6.4% Change in Partners >365 Days with KET: -16.7% FSP Data Correction Project Page 25
3.06 Colusa County Colusa County did not participate in the Project. The county submits FSP data using the online DCR. Review process to collect and submit data to the FSP DCR to identify and address barriers to KET and 3M submission Consider implementing the EPLD Templates to provide feedback to PSCs and monitor FSP data quality Utilize the MHSOAC provided FSP DCR training materials (user manuals, curriculums and videos) FSP Data Quality Meeting Date: None Active Partners Serving on June 30, 2012: 22 Completeness of 3Ms Before Project > 5% Change in % 3Ms Submitted: - 4.2% Change in Incomplete 3Ms Submitted: 0% Change in Partners >180 Days with KET: - 5.2% Change in Partners >365 Days with KET: - 16.9% FSP Data Correction Project Page 26
3.07 Contra Costa County Contra Costa County did not participate in the Project. The county submits FSP data using the online DCR. The county has high quality FSP DCR data. Review process to collect and submit data to the FSP DCR to identify and address barriers to KET and 3M submission Consider implementing the EPLD Templates to provide feedback to PSCs and monitor FSP data quality FSP Data Quality Meeting Date: None Active Partners Serving on June 30, 2012: 225 Completeness of 3Ms Before Project: > 85% Change in % 3Ms Submitted: + 7.3% Change in Incomplete 3Ms Submitted: - 0.1% Change in Partners >180 Days with KET: + 1.7% Change in Partners >365 Days with KET: + 0.6% FSP Data Correction Project Page 27
3.08 Del Norte County Del Norte uses the online DCR, but has been unable to access the FSP DCR since spring of 2011. Thus, while Del Norte had very good FSP data quality in 2010, the current data quality is poor as the DCR data has not been updated in two years. The county is working on revising its operational definition of who qualifies for an FSP, and would like to know how other Counties are implementing their FSP program. Submit a ticket with DHCS to resolve the FSP DCR access issues and enter all missing data from the last two years Review process to collect and submit data to the FSP DCR to identify and address barriers to submitting PAF, 3M, and KET assessments Consider implementing the EPLD Templates to provide feedback to PSCs and monitor FSP data quality Contact nearby FSP Counties who are listed as interested in joining a regional FSP workgroup for mentoring FSP Data Quality Meeting Date: March 1, 2013 Active Partners Serving on June 30, 2012: 18 Anasazi No Completeness of 3Ms Before Project: > 80% when last able to access FSP DCR (2010) Change in % 3Ms Submitted: Unable to access FSP DCR since 2011 Change in Incomplete 3Ms Submitted: Unable to access FSP DCR since 2011 Change in Partners >180 Days with KET: Unable to access FSP DCR since 2011 Change in Partners >365 Days with KET: Unable to access FSP DCR since 2011 FSP Data Correction Project Page 28
3.09 El Dorado County El Dorado County did not participate in the Project. The county submits FSP data using the online DCR. The county had good quality 3M DCR data in 2011 and has improved the KET submission rate in 2013 to be excellent. Review process to collect and submit data to the FSP DCR to identify and address barriers to PAF completion, and KET & 3M submission Consider implementing the EPLD Templates to provide feedback to PSCs and monitor FSP data quality FSP Data Quality Meeting Date: None Active Partners Serving on June 30, 2012: 57 Completeness of 3Ms Before Project: > 85% Change in % 3Ms Submitted: - 0.7% Change in Incomplete 3Ms Submitted: + 0.2% Change in Partners >180 Days with KET: + 22.9% Change in Partners >365 Days with KET: + 7.6% FSP Data Correction Project Page 29
3.10 Fresno County Fresno County did not participate in the Project. The county submits FSP data using the online DCR. Review process to collect and submit data to the FSP DCR to identify and address barriers to PAF completion, and KET & 3M submission Consider implementing the EPLD Templates to provide feedback to PSCs and monitor FSP data quality Utilize the MHSOAC provided FSP DCR training materials (user manuals, curriculums and videos) FSP Data Quality Meeting Date: None Active Partners Serving on June 30, 2012: 644 Completeness of 3Ms Before Project: > 60% Change in % 3Ms Submitted: - 1.8% Change in Incomplete 3Ms Submitted: - 1.0% Change in Partners >180 Days with KET: - 6.6% Change in Partners >365 Days with KET: - 3.9% FSP Data Correction Project Page 30
3.11 Glenn County Glenn County did not participate in the Project. The county submits FSP data using the online DCR. The county has excellent 3M submission rates and lower quality KET submission rates. Review process to collect and submit data to the FSP DCR to identify and address barriers to KET submission Consider implementing the EPLD Templates to provide feedback to PSCs and monitor FSP data quality FSP Data Quality Meeting Date: None Active Partners Serving on June 30, 2012: 52 Completeness of 3Ms Before Project: > 85% Change in % 3Ms Submitted: + 5.7% Change in Incomplete 3Ms Submitted: + 0.2% Change in Partners >180 Days with KET: - 14.2% Change in Partners >365 Days with KET: - 2.5% FSP Data Correction Project Page 31
3.12 Humboldt County Humboldt County uses the online DCR for FSP data submission. The county holds a staff meeting every morning, and lead case managers assist other case managers with the FSP data collection and submission. The county has expressed difficulty collecting FSP data from partners who are served in group settings and the county desires a modified FSP data collection tool. The county is getting caught up on a backlog of FSP data which needs to be submitted to the FSP DCR. Humboldt has had a history of a very good KET submission rates and cross-checks legal events with the Sheriff s postings. The county is very interested in creating outcomes reports from the FSP data. Review process to collect and submit data to the FSP DCR to identify and address barriers 3M submissions; use the EPLD Template tool to run 3M due reports Formalize FSP and DCR training process for new staff Utilize the MHSOAC provided FSP DCR training materials (user manuals, curriculums and videos) Consider implementing the EPLD Templates to provide feedback to PSCs and monitor FSP data quality FSP Data Quality Meeting Date: January 8, 2013 Active Partners Serving on June 30, 2012: 150 None, but migrating to Avatar No Completeness of 3Ms Before Project: > 65% Change in % 3Ms Submitted: + 3.9% Change in Incomplete 3Ms Submitted: + 2.2% Change in Partners >180 Days with KET: - 13.9% Change in Partners >365 Days with KET: - 9.5% FSP Data Correction Project Page 32
3.13 Imperial County Imperial County did not participate in the Project. The county submits FSP data using the online DCR. The county has high quality 3M submission rates and has improved the rate of KET submission. Review process to collect and submit data to the FSP DCR to identify and address barriers to KET submission Consider implementing the EPLD Templates to provide feedback to PSCs and monitor FSP data quality FSP Data Quality Meeting Date: None Active Partners Serving on June 30, 2012: 322 Completeness of 3Ms Before Project: > 85% Change in % 3Ms Submitted: - 3.3% Change in Incomplete 3Ms Submitted: + 0.4% Change in Partners >180 Days with KET: + 6.8% Change in Partners >365 Days with KET: + 14.6% FSP Data Correction Project Page 33
3.14 Inyo County Inyo County did not participate in the Project. The county submits FSP data using the online DCR. The county has high KET submission rates. The county also has high 3M completion rates, but many of the 3Ms submitted are incomplete. Review process to collect and submit data to the FSP DCR to identify and address barriers to PAF and 3M completion Consider implementing the EPLD Templates to provide feedback to PSCs and monitor FSP data quality FSP Data Quality Meeting Date: None Active Partners Serving on June 30, 2012: 26 Completeness of 3Ms Before Project: > 85% Change in % 3Ms Submitted: + 6.6% Change in Incomplete 3Ms Submitted: + 19.9% Change in Partners >180 Days with KET: - 21.5% Change in Partners >365 Days with KET: - 7.4% FSP Data Correction Project Page 34
3.15 Kern County Kern County is one of the larger Counties using the online DCR. The county is interested in modifying their EHR (Anasazi) to collect FSP data and switching to XML submission. The county experienced an issue with DCR user account s ability to view partners, which surfaced after the FSP DCR was moved from the Department of State Hospitals (DSH) to DHCS in January, 2013. The county investigated the ability to change user groups and roles to fix the problem, but no solution has been found to date. Despite these barriers, the county worked with staff to improve FSP DCR data quality. Continue to work with DHCS regarding the issues related to FSP DCR user account s ability to view partners Contact DHCS to investigate becoming an XML submitting county Review process to collect and submit data to the FSP DCR to identify and address barriers to 3M submission rates Utilize the FSP DCR QA environment for testing and training Utilize the MHSOAC provided FSP DCR training materials (user manuals, curriculums and videos) FSP Data Quality Meeting Date: February 21, 2013 Active Partners Serving on June 30, 2012: 464 Anasazi Yes Completeness of 3Ms Before Project: > 65% Change in % 3Ms Submitted: + 7.2% Change in Incomplete 3Ms Submitted: - 0.3% Change in Partners >180 Days with KET: + 2.6% Change in Partners >365 Days with KET: + 7.7% FSP Data Correction Project Page 35
3.16 Kings County Kings County uses the online DCR for FSP submission. Each PSC submits their FSP data to the DCR, and training has been a barrier for this county. The county serves many veterans and who may not have a CSI number, which is an issue preventing completion of PAFs. The county substantially improved workflow for 3M and KET submission during the Project. Continue to review the process of submitting data to the FSP DCR to identify and address barriers to PAF completion and KET and 3M submission Consider implementing the EPLD Template tool to run 3Ms Due by Month reports, monitor data quality, and provide partner-level reports to PSCs on a regular basis Utilize the MHSOAC provided FSP DCR training materials (user manuals, curriculums and videos) to train staff Meet with PSCs to identify additional ways to support their work Identify and address partner engagement barriers FSP Data Quality Meeting Date: December 18, 2012 Active Partners Serving on June 30, 2012: 48 Anasazi No Completeness of 3Ms Before Project: > 5% Change in % 3Ms Submitted: + 44.5% Change in Incomplete 3Ms Submitted: - 30.6% Change in Partners >180 Days with KET: + 25.3% Change in Partners >365 Days with KET: + 35.9% FSP Data Correction Project Page 36
3.17 Lake County Lake County uses the online DCR for FSP submission. Staff work together to initiate and review partnerships. KETs are generally completed after monthly utilization review meetings, but the county expressed a need to review workflow to identify barriers to KET submissions. The county expressed issues with inactivating PSCs through the online FSP DCR system, and a resolution has not yet been found. The county made several suggestions for improving the proposed DCR data metrics reports and plans to use the EPLD Templates. The county would like to review data metrics by PSC instead of by provider. The county was provided a revised custom metrics report organized by PSC as part of the Project. The county had very good PAF completion and 3M & KET submission rates before the Project, but was able to improve to excellent levels during the Project. Submit a ticket to DHCS through medccc@dhcs.ca.gov to document and address the PSC deactivation issues Implement the EPLD Templates and provide partner-level reports to PSCs on a regular basis Require PSCs to complete the currently unused Provider field in the PAF so that EPLD Templates metrics reports will display metrics by PSC for the county FSP Data Quality Meeting Date: December 19, 2012 Active Partners Serving on June 30, 2012: 57 Anasazi No Completeness of 3Ms Before Project: > 80% Change in % 3Ms Submitted: + 7.9% Change in Incomplete 3Ms Submitted: + 4.5% Change in Partners >180 Days with KET: + 10.3% Change in Partners >365 Days with KET: + 9.0% FSP Data Correction Project Page 37
3.18 Lassen County Lassen County has modified their EHR (Share Care) to collect all FSP data. Clinicians enter all FSP data into the EHR, and then one administrative staff member manually rekeys data from the EHR to the DCR at intervals. The county has implemented the PLD Templates to provide partner-level reports to clinicians. The county expressed a barrier to DCR data quality due to an inability to inactivate virtual PSCs in the DCR. The county had excellent PAF completion, and KET and 3M submission rates at the start of the Project. The decreases in quality noted in this report are likely due to a lag in the manual data transfer between the Share Care EHR and the FSP DCR for the second time period evaluated. The FSP data in this county s EHR is likely complete and awaiting transfer to the DCR by the administrative staff member. The county does not have technical resources to pursue XML submission at this time. Rename inactive virtual PSCs with a zz_ in front of their name in the FSP DCR so that they sort to the bottom of the drop-down list; do not delete inactive PSCs as this will cause a validation issue with the FSP DCR data which tracks all PSCs to which a partner was ever assigned Set a monthly schedule to transfer data from the EHR to the FSP DCR such that no more than a 45 day lag appears in the FSP DCR data Implement the updated EPLD Templates Consider XML submission for future implementation FSP Data Quality Meeting Date: February 7, 2013 Active Partners Serving on June 30, 2012: 19 Share Care No Completeness of 3Ms Before Project: > 85% Change in % 3Ms Submitted: - 16.2% Change in Incomplete 3Ms Submitted: + 2% Change in Partners >180 Days with KET: - 25.0% Change in Partners >365 Days with KET: - 22.2% FSP Data Correction Project Page 38