Program Performance Review

Transcription

1 Program Performance Review Broward Addiction Recovery Division (BARC) April 15, 2008 Report No Office of the County Auditor Evan A. Lukic, CPA County Auditor

2 Table of Contents Topic Page Executive Summary... 3 Purpose and Scope... 5 Background... 6 Section 1: Assessment of BARC s Accountability System Section 2: Assessment of BARC s Performance Section 3: Assessment of BARC Program Alternatives Appendix A...45 Appendix B

3 Executive Summary In this Program Performance Review, we (1) assess the quality of BARC s accountability system, (2) evaluate its performance by analyzing available data, and (3) assess potential alternatives that could increase program revenues and reduce program costs. Section 1: Assessment of BARC s Accountability System (See pages 13-27) Prior to Fiscal Year 2008, BARC lacked an effective accountability system In Fiscal Year 2008 BARC made the following improvements o Added efficiency measures to each section s set of performance measures, thus meeting Office of Management & Budget requirements, and o Collected reasonably accurate unit cost data, which enables a comparison of BARC s service costs to similar agencies and setting appropriate client fees However, BARC should further improve its accountability system by o Fixing problems with its Client Management Information System in order to limit the need for time-consuming manual data collection, which is prone to error, o Routinely benchmarking against high-performing agencies to compare its performance in areas such as staff productivity and service costs, o Developing and implementing data reliability controls to correct data reporting problems regarding staff productivity, client counts, and client outcomes, and o Collecting additional client outcome data, as identified by program supervisors, to better evaluate program effectiveness Section 2: Assessment of BARC s Performance (See pages 28-38) Due to the data reliability problems discussed in Section 1, we could not evaluate BARC s effectiveness o As of April 2008, BARC did not have reliable data on two important outcome measures required by DCF: (1) whether clients successfully complete treatment and (2) whether clients are employed upon treatment discharge However, BARC s treatment model contains many elements advocated in addiction recovery literature o BARC has adopted several recommended practices, including continuum of care, duration of treatment, and provision of culturally and gender specific services 3

4 Our analysis of BARC s unit cost data indicates that inpatient treatment costs are significantly higher than those reported in a 2005 national study o For example, BARC s Fiscal Year 2007 daily per client detoxification cost of $458 is 83% higher than the daily per client national average of $250, o Similarly, BARC s daily per client residential treatment cost of $261 for Fiscal Year 2007 is 153% higher than the national average of $103 Two factors contribute to BARC s relatively high inpatient treatment costs: low bed utilization and high staffing levels o BARC s inpatient facilities are relatively underutilized; for example, the detox bed utilization rate of 80% and residential bed utilization rate of 67% are both lower than the 91% national average inpatient bed utilization rate o BARC s staffing levels are high; for example, BARC s residential treatment program bed to staff ratio of 1.6 is 106% higher than Miami-Dade County s ratio of 3.3 beds per staff Therefore, to reduce costs to meet the daily per client national average detox and residential treatment costs, BARC would need to increase bed utilization and decrease staffing; these actions would save the County approximately $4.7 million annually Section 3: Alternatives to Current Service Delivery Model (See pages 39-44) To reduce the amount of general and capital funds used for BARC services, the Board of County Commissioners could consider the following alternatives: Outsource some or all of BARC s programs o Only 3.4% of Florida s substance abuse treatment facilities are operated directly by local governments, o Most large Florida counties contract with community providers for the provision of substance abuse treatment services o If the County contracted with providers that could meet the national average cost for detoxification and residential treatment, it would save the County $4.7 million annually o In addition, the County could avoid a capital outlay of $21.3 million if it did not construct a new detox facility, as currently planned Not fund some BARC programs such as detoxification or outpatient services o Some other large Florida counties, such as Hillsborough and Orange, do not fund detoxification because local public hospitals provide these services; for Fiscal Year 2007, BARC s detox program cost $4.8 million o Miami-Dade and Orange counties do not fund outpatient treatment programs, instead relying on community providers for these services; the cost of BARC s outpatient program totaled $3.4 million in Fiscal Year

5 Offset the need for general funds by improving client fee collection and Medicaid reimbursement rates o BARC s client service fee collection rate is historically low; the average collection rate for the past two fiscal years is 4.9% o BARC did not collect available Medicaid reimbursement for the past three fiscal years; annual revenue projections indicate that uncollected Medicaid revenues total approximately $450,000 during this time Recommendations To address program deficiencies and identify the feasibility of alternative practices we make several recommendations including: o Providing a detailed justification of the need for a proposed $21.3 million detoxification facility expansion in light of existing bed underutilization (see page 33) o Conducting further comparative analysis with similar detoxification and residential programs, including Miami-Dade County s residential program, to analyze service cost, bed utilization, staffing ratios/composition and developing a plan to reduce treatment costs in accordance with results (see page 38) o Reporting to the Board of County Commissioners within 60 days the potential effects of implementing various service delivery models used in other large Florida counties, such as contracting with community providers for the provision of services, providing less intensive residential treatment options and not funding the existing level of treatment services (see page 41) 5

6 Purpose and Scope This report presents the results of a Program Performance Review of the Broward Addiction Recovery Division (BARC) within the Human Services Department. The purpose of this review is to assess the quality of BARC s accountability system, evaluate its performance and identify options for improving services and reducing operating costs. To accomplish our objectives, we: Reviewed pertinent performance measurement, general management, and addiction and recovery literature; Analyzed available performance data for BARC s operations; Analyzed information provided by BARC s managers and employees relative to specific management processes and controls; Reviewed applicable BARC records and documents, including relevant policies and procedures; Reviewed applicable Florida Statutes and Broward County Ordinances pertaining to substance abuse treatment; Interviewed BARC managers and employees; and Interviewed substance abuse treatment program officials in other jurisdictions and Florida Department of Children and Families staff Background Established in 1973, BARC provides medical and clinical treatment, substance abuse and nutrition education and support services to Broward County residents and homeless individuals who are chemically dependent and 18 years or older. In accordance with Section , Florida Statutes, BARC is licensed and regulated by the State of Florida as a substance abuse treatment provider. BARC provides a range of substance abuse treatment services intended to meet client needs, including: Access Services: At the time an individual seeks treatment BARC completes a comprehensive assessment, which involves gathering information about the individual s physical and mental health and history of drug use, and in conjunction with the individual, makes a determination of the appropriate treatment and level of care needed to meet individual and programmatic goals. Detoxification 1 (Detox) Services: A medically supervised 35-bed facility operating 24 hours per day, 7 days per week, for clients who need medical stabilization before receiving other program services or referral to other agencies. During detox, clients also receive individualized assessments and addiction and recovery education, and 1 The U.S. Substance Abuse and Mental Health Administration defines detoxification as the process through which a person who is physically dependent on alcohol, illegal drugs, prescription drugs or a combination, is withdrawn from the substance of dependence. 6

7 may participate in Alcoholics Anonymous (AA) and Narcotics Anonymous (NA) meetings. Residential Treatment Services: BARC operates a 92-bed, short-term inpatient program aimed at providing a supportive environment to establish the principle of recovery. Clients are provided counseling, substance abuse and nutrition education and family support and may participate in AA and NA meetings. Typically, residential treatment lasts for 28 days, after which clients may be referred to other BARC programs or community resources. 2 Non-Residential Day Treatment: A three to five week program operating Monday through Friday that generally provides intensive group and individual counseling and support services to clients upon their release from residential treatment and before placement in a less intensive program such as outpatient services. Outpatient Services: A three to six month program in which clients participate in individual and group counseling, are taught basic life management and interpersonal skills, receive psychiatric evaluations as needed and HIV testing, education and counseling. Outpatient services are provided mornings and evenings to enable clients to maintain employment while receiving treatment. Resources for Recovery: A case management and advocacy program for clients who have had difficulty staying sober in traditional settings and have a greater need for assistance and follow-up. Specialized Services: BARC also has programs that are designed to meet the needs of a diverse Broward County population, including: Co-Occurring Disorders: substance abuse treatment for individuals with both substance abuse and mental health issues; Family Involvement Program: educational services for family members of current clients as deemed appropriate; Hispanic Track: outpatient services provided in Spanish that take language and culture into consideration; Mature Adult Program: specialized treatment for clients who are 55 years of age and older; and Perinatal Addiction Services: residential, day treatment and outpatient services for pregnant women or women with children 7 years or younger. 2 However, pregnant women may stay in residential treatment until their babies are delivered and a 90-day residential treatment program is available to women with children 7 years of age and younger. 7

8 Service Locations BARC provides services at four facilities located throughout the County: BARC Central at 1000 S.W. 2 nd Street in Ft. Lauderdale, BARC Lauderhill at 4200 N.W. 16 th Street in Lauderhill, Stephen R. Booher Building at 3275 N.W. 99 th Way in Coral Springs, and BARC South at 5701 Hollywood Boulevard in Hollywood. Exhibit 1 below shows the types of services provided at each of these four facilities. Exhibit 1: Availability of BARC services by facility Type of Service BARC BARC Booher BARC Central Lauderhill Building South Access x Detoxification x Residential x Non-Residential Day x x Outpatient x x x Case Management x Specialized Services x x x x Source: BARC Client Admissions Exhibit 2 below shows the number of client admissions, by type, during Fiscal Year 2007; clients are often admitted to more than one type of service during their treatment. Exhibit 2: Most BARC admissions were for Access and Detoxification services Number of Type of Service Admissions Access 3,430 Detoxification 1,594 Residential 870 Non-Residential Day 1,036 Outpatient 893 Case Management 259 Specialized Services 846 Source: BARC While most clients seek treatment on their own, some clients are referred to BARC by the Florida Department of Children and Families, the judicial system or community health providers. Clients are charged fees for services, on a sliding scale, based on their annual income and family size. Section (2), Florida Statutes, requires clients receiving treatment to contribute as much of the cost of services as they can. 8

9 Exhibit 3 below shows an example of BARC s sliding fee scale for several services. Exhibits 4 thru 7 illustrate the gender, race, age, and addiction type(s) of Fiscal Year 2007 clients. Exhibit 3: BARC s Fiscal Year 2008 sliding fee scale is based on each client s annual income and family size Client Income Access Detoxification Residential Outpatient (Group) Level 3 Fee/Hour Fee/Day Fee/Day Fee/Hour $0 - $7,499 $7.50 $18.50 $13.00 $3.00 $7,500 - $14,999 $15.00 $37.00 $26.00 $6.00 $15,000 - $22,499 $22.50 $55.50 $39.00 $9.00 $22,500 - $29,999 $30.00 $74.00 $52.00 $12.00 $30,000 - $37,499 $37.50 $92.50 $65.00 $15.00 $37,500 - $44,999 $45.00 $ $78.00 $18.00 $45,000 - $52,499 $52.50 $ $91.00 $21.00 $52,500 - $59,999 $60.00 $ $ $24.00 $60,000 - $67,499 $67.50 $ $ $27.00 $67,500 and over $75.00 $ $ $30.00 Source: Broward County Administrative Code Exhibit 4: A significant majority of BARC s Fiscal Year 2007 clients were male Male 3,529 (70%) Source: BARC Female 1,525 (30%) 3 In this example, the client s family size is one. 9

10 Exhibit 5: Most of BARC s Fiscal Year 2007 clients were white 4 White 3,172 (63%) Multi-Racial 219 (4%) Other 38 (1%) Black 1,625 (32%) Source: BARC Exhibit 6: BARC served a range of client age groups in Fiscal Year (27%) 65 and over (1%) (41%) (31%) Source: BARC of BARC s 5,054 (9.8%) Fiscal Year 2007 clients were Hispanic. 10

11 Exhibit 7: BARC clients were most frequently admitted for alcohol dependence treatment in Fiscal Year Cocaine 1,743 Cannabis 890 Opioid 1,447 Alcohol 2,862 Source: BARC Program Resources As shown in Exhibit 8 below, BARC s Fiscal Year 2008 operating budget of $19 million is primarily funded by the general fund and a grant from the Florida Department of Children and Families (DCF). Exhibit 8: Most of BARC s Fiscal Year 2008 funding comes from the general fund Funding Source Revenue % of Budget General Fund $13,701, % DCF Grant 6 $3,745, % Other State Grants $701, % Federal Grant (HUD) $406, % Kids in Distress Grant $262, % Medicaid Revenue $150,000 <1% Revenue from Operations $55,000 <1% Miscellaneous Revenue $6,000 <1% Total $19,029, % Source: Office of Management & Budget 5 Per BARC, a client may be diagnosed with multiple addictions when entering treatment. Therefore, the number of admissions per addiction type is larger than the number of clients served in Fiscal Year Opioids are a class of drugs derived from the opium poppy plant or produced synthetically that are used to relieve pain, dull the senses or induce sleep. Examples of opioids include heroin, codeine and methadone. 6 DCF annually provides BARC with negotiated grant funding for substance abuse treatment services via a three-year contract approved by the Board of County Commissioners; BARC s current contract with DCF expires after Fiscal Year

12 Division Organization BARC is organized into 3 sections: Administration/Support Services; Outpatient Services; and Residential Services. Exhibit 9 below summarizes BARC s Fiscal Year 2008 operating budget, full-time equivalent (FTE) positions and major services, by section. Exhibit 9: In Fiscal Year 2008, BARC was appropriated $19 million and had 226 FTE positions Section Budget FTE Major Services Administration/ Support Services $2,124, Overall direction, oversight, management and administrative support of BARC s programs and operations Outpatient Services $6,137, Access Services --Comprehensive physical and mental health assessment Outpatient Services --Individual and group counseling --Basic life management and interpersonal skills education --Psychiatric evaluations --HIV testing, education and counseling Non-Residential Day Treatment --Intensive individual and group counseling --Support services Resources for Recovery --Case management and peer mentoring Specialized Services --Day treatment and outpatient services for previously mentioned target populations Other Services --Partner with local law enforcement and community agencies to provide drug screening Residential Services $10,767, Detoxification Services --Medical stabilization --Individualized assessments --Addiction and recovery education Residential Treatment --Individual and group counseling --Substance abuse and nutrition education --Family support services Specialized Services -- Residential treatment for previously mentioned target populations TOTAL $19,029, Source: Office of Management & Budget 12

13 Section 1: Assessment of BARC s Accountability System This section presents our assessment of BARC s accountability system. Establishing an effective accountability system is essential to ensure public funds are spent in a manner that achieves desired results and improves the provision of public services. An effective accountability system provides quality information to enable: Citizens to hold elected officials, managers, employees and private contractors accountable for the efficient and effective use of public funds; Elected officials and other policymakers to make informed budget and policy decisions; and Managers to detect and correct operational deficiencies and improve program results As a framework for evaluating accountability, we assessed BARC s performance measurement and reporting system relative to six best practices. Specifically, we have examined whether the Division has: Established a complete set of performance measures, as required by the Office of Management & Budget (OMB); Collected and analyzed unit cost data for each major service and activity; Collected and analyzed data to assess operational efficiency and effectiveness; Benchmarked its performance against other agencies that provide similar services or generally accepted industry standards; Established processes to ensure performance data reliability; and Used performance data to modify current practices or change operational processes in order to achieve better program results Appendix A on page 45 summarizes our Section 1 findings. Best Practice 1: Establish a complete set of performance measures According to performance measurement literature, a comprehensive set of performance measures provides information to enable policymakers and managers to assess each agency s workload, operating efficiency, effectiveness, and societal impacts. The Office of Management and Budget Services (OMB) requires County agencies to collect and report information relative to four types of performance measures: work output, efficiency, client benefit/effectiveness, and strategic outcome. 7 7 Work output focuses on the quantity of service provided; efficiency measures the ratio of output per input; client benefit/effectiveness assesses the quality of the service from the clients perspective; and strategic outcome states the consequences of the program in a big picture sense. 13

14 BARC established a complete set of performance measures during Fiscal Year 2008 We found that BARC has not historically met OMB s performance measure reporting requirement due to a lack of efficiency measures for its three sections. However, in January 2008, BARC worked with OMB officials to develop efficiency measures for each of its sections. Specifically, BARC s Administration/Support Services Section will begin reporting the efficiency measure, Number of purchasing forms and credit card purchases processed per certified agency buyer during Fiscal Year Similarly, BARC s Residential Services Section and its Outpatient Services Section will report the Number of direct service and contact hours per clinical staff per week. With these new efficiency measures, BARC meets OMB requirements. Appendix B on page 46 lists BARC s Fiscal Year 2008 performance measures. Best Practice 2: Collect and analyze unit cost data for major services Unit cost refers to the cost of producing an output or outcome. 8 unit cost data is important because it can be used to: Collecting and analyzing Compare an agency s performance to similar service providers, Predict how changes in the demand for services will affect an agency s budget, Assess how changes in operations could affect costs, Identify wasteful processes, Set appropriate service fees, and Estimate the impact of budget decisions BARC has established a goal of providing the most cost effective methods of treatment For Fiscal Year 2008, BARC established a goal of providing the most cost effective method of treatment to clients. According to substance abuse treatment literature, measuring cost effectiveness is increasingly important as demands to justify service costs have heightened due to pressures on public funding. Industry research indicates that other things equal, the lower the treatment cost, the more cost effective a program or intervention will be. 9 To determine its treatment costs, BARC should collect and analyze unit cost data. For example, BARC should routinely monitor the cost per client per day of providing its primary services, such as detoxification and residential treatment to ensure these services are provided in a cost effective manner. Additionally, BARC should use unit cost data to make certain that client service fees (generally set on a per day or per hour basis) are 8 Source: Florida Legislature s Office of Program Policy Analysis and Government Accountability (OPPAGA) 9 Source: Economic Benefits of Drug Treatment: A Critical Review of the Evidence for Policy Makers Treatment Research Institute at the University of Pennsylvania. February

15 appropriate and to target reasonable reimbursable rates during contract negotiations with Florida s Department of Children and Families (DCF). 10 BARC has not historically collected and analyzed unit cost data, but the Division began collecting this data in February 2008 We found that BARC has not historically collected unit cost data. Therefore, we could not evaluate the Division s cost effectiveness across time. However, in February 2008 the Division developed unit cost data for the following services: Detoxification, Day treatment, Residential treatment, Access (intake and assessment), and Outpatient treatment BARC s development of unit cost data in Fiscal Year 2008 is important for two reasons. 11 First, it enables managers and program evaluators to compare BARC s service costs with similar agencies and/or industry standards (in Section 2 of this report, beginning on page 28, we compare BARC s service delivery costs for detoxification and residential service to national averages). This type of analysis generally provides an indication of whether treatment is provided in an efficient manner. Additionally, BARC s unit cost data provides justification for client service fees. BARC s February 2008 unit cost data raises questions about the appropriateness of Fiscal Year 2008 client service fees BARC s unit cost data suggests that its client service fees bear no relationship to the actual cost of providing service. For example, BARC s maximum fee for Access services (i.e. intake and assessment) is $75 per hour, while BARC s actual cost to provide these services is reported to be $60 per hour. As BARC officials told us that many clients are indigent, it is critical that they are not overburdened by service fees. However, as of Fiscal Year 2008, BARC has not established criteria defining the relationship that should exist between service cost and client fees DCF annually provides BARC with negotiated grant funding for substance abuse treatment services via a three-year contract approved by the Board of County Commissioners; BARC s current contract with DCF provides $3.7 million of annual program funding and expires after Fiscal Year While BARC s unit cost methodology captures its base operating cost of providing services, it does not capture the County s full cost of providing services because it excludes capital budgeted expenditures, such as building depreciation. 12 For example, the County could pass 100% of its actual cost of providing substance abuse treatment services to clients, or it could establish an appropriate level of subsidy for each service. Regardless of the fee criteria chosen, BARC s existing sliding fee scale would provide a level of subsidy to low income clients. 15

16 We also noted that BARC has not developed separate unit cost data for its two types of outpatient service, (1) individual and (2) group counseling. Rather, the Division developed one aggregate unit cost for both services. As BARC has a different client service fee for individual outpatient counseling than it does for group outpatient counseling, it should develop unit cost data for both services to assess the appropriateness of its fees. Exhibit 10 below shows BARC s cost per service and its Fiscal Year 2008 client service fees. Exhibit 10: BARC s cost of providing services bears no apparent relationship to its Fiscal Year 2008 client service fees Service BARC Cost Per Service BARC Client Service Fee Range 13 Detoxification $458 per day $ $185 per day Residential $261 per day $13 - $130 per day Day Treatment $248 per day $7 - $70 per day Access Outpatient $60 per hour $60 per hour $ $75 per hour $9 - $90 per hour (individual) $3 - $30 per hour (group) Source: BARC and Broward County Administrative Code, Chapter 36.1 Recommendation 1: We recommend that the Board of County Commissioners direct the County Administrator to take the following actions: Establish criteria defining the appropriate relationship between client service fees and BARC s actual cost of providing services by September 30, 2008 Develop unit cost data for both individual and group outpatient treatment to ensure client service fees for these services are appropriate; any proposed client service fee changes should be submitted to the Board of County Commissioners for approval by September 30, 2008 Best Practice 3: Collect and analyze internal operational data To effectively manage County programs and operations, managers need more data than what is reported in the annual budget. The data reported in the annual budget generally relates to an entity s overall performance rather than specific functions and activities. While the budget data is useful for accountability purposes, managers commonly need additional information to effectively supervise daily operations. 13 As depicted in more detail on page 9, BARC has implemented a sliding fee scale based on each client s annual income. Therefore, we provide the range of fees that can be charged for each BARC service. 16

17 BARC routinely collects internal operational data BARC officials provided several examples of collecting internal operational data to monitor program activities. For example, BARC provided two reports which are regularly used to monitor employee productivity. In one report, the Division monitors the caseload per clinician on a weekly basis to ensure that staff has been assigned the appropriate number of clients. Similarly, the Division uses another weekly report to monitor all activities performed by each clinician, including the number of counseling sessions and client drug screenings conducted. This data, while not reported to policy makers, is necessary to ensure the efficient delivery of treatment services. Client management information systems (CMIS) provide consistent and timely internal operational data To be optimally useful, internal operational data needs to be obtained in a timely manner. Data that is time-consuming to collect and analyze impedes clinical employees from focusing on their primary function of providing direct treatment services to clients. Additionally, supervisors need access to real-time data to monitor productivity and assess the outcomes of new initiatives. To help reduce the burden of data collection, many agencies use client management information systems (CMIS) as their primary source of data collection and management reports. According to CMIS literature, human services agencies often use these systems to collect client information and track the services they have received. As CMIS data is stored in a central database, management and other staff can easily access it to track progress toward desired outcomes. We found that effective CMIS provide two primary advantages over manual data collection: Improved data reliability. An effective CMIS requires staff to complete designated fields during client intake and subsequent counseling. This provides assurance that critical data is consistently captured for every client served. Also, by removing the human element from data computations, a CMIS decreases the opportunity for errors. Real-time data analysis. As CMIS data is stored in a central database, any authorized user can access reports that provide up-to-the minute assessments of agency performance. This is superior to manually collected data, which is often collected by staff for a designated period (usually one week or month) and then tallied into a summary management report. Also, because CMIS data is in electronic form, managers can easily use it to forecast the effect of adding or removing resources from particular service areas. 17

18 Despite investment in a CMIS, BARC relies on manually collected internal operational data In September 2002, BARC purchased a client management information system (hereafter called ECHO ) from The Echo Group, Inc. for $461,482. According to company literature, ECHO provides users with the ability to create electronic client records, treatment plans and management reports. However, we found that many internal operational data reports regularly used by BARC supervisors are generated through manual data collection processes. Specifically, clinical staff uses Excel spreadsheets or standardized paper forms to track basic client data, such as the number of clients seen each week. Additionally, more detailed information about the specific services provided by each clinician, such as the number of individual and group counseling sessions conducted, is also captured on a weekly basis through manual reports. Manually collected data is aggregated on a weekly or monthly basis for supervisory review. BARC officials told us that the Division continues to rely on manually collected data due to its slow implementation of ECHO and its lack of technical resources to develop desired reports within the system. For example, ECHO has not been configured to provide supervisors with key productivity reports, such as the amount of time each counselor has spent providing group counseling. While each clinician inputs this data into ECHO on a daily basis, clinicians have also been instructed to manually collect this data in Excel spreadsheets for supervisory review. Supervisors stressed the importance of monitoring this data because it is necessary to determine if BARC is meeting established goals for direct service hours. 14 Better utilization of BARC s CMIS should improve program efficiency BARC officials told us that they believe fully realizing ECHO s capabilities would increase the number of direct service hours available to clients. This would increase the number of clients that each clinician could counsel on a weekly basis, or allow each clinician to provide more intensive counseling services to existing clients. We believe this is important because increasing the number of clients treated or the effectiveness of existing services within existing resources would improve BARC s cost effectiveness. Recommendation 2: We recommend that the Board of County Commissioners direct the County Administrator to take the following action: Discontinue the use of time-consuming manual data collection, where possible, by developing reports within ECHO that will allow supervisors to efficiently and effectively monitor staff productivity and client outcomes by September 30, BARC defines direct service hours as the actual time spent on activities directly associated with a single client, including case staffing. Time may include travel if the travel is integral to a service event or otherwise billable. 18

19 Best Practice 4: Contact outside sources, such as industry and other government agencies, to benchmark performance The Government Finance Officers Association recommends benchmarking as a method of obtaining information that can be used to identify inefficient practices and strategies to achieve better results. Benchmarking refers to performance comparisons of organizational business processes against an internal or external standard of recognized leaders. County managers should systematically and routinely compare policies, practices, and performance measures against those of other local governments, private contractors that provide similar services and industry standards. BARC has historically conducted limited benchmarking In response to our request to provide examples of benchmarking, BARC cited its participation in a study conducted by the Joint Commission in In this study, the performance of BARC and two other South Florida substance abuse treatment providers was compared. Specifically, BARC was compared to Hanley-Hazelden of West Palm Beach, Florida and Transitions Recovery Program of North Miami Beach, Florida. The study evaluated several measures of performance, including the average length of stay (per client) and the percent (of clients) with a satisfactory discharge rating, which is a common measure of treatment completion. Several factors limit the usefulness of BARC s previous benchmarking effort Our review disclosed several concerns with the 2001 benchmarking effort which severely limit its value in assessing BARC s performance. These include: Peer comparability concerns. BARC officials expressed concern that Hanley- Hazelden and Transitions Recovery Program were not appropriate BARC peers. These officials advised that BARC typically serves lower income Broward County residents, while Hanley-Hazelden and Transitions Recovery Program generally market their services to a more affluent clientele. As an example of the difference in client demographics served by these agencies, BARC officials surveyed these agencies in Fiscal Year 2008 and found that Hanley-Hazelden charges $25,000 per month for its residential treatment program, while Transitions Recovery Program charges $12,000 per month. BARC s highest monthly fee for residential treatment in Fiscal Year 2008 is $3,640. Lack of efficiency or outcome comparisons. Our review of BARC s benchmarking study revealed that it provided very limited insight about the efficiency and effectiveness of each agency. For example, the study did not include any comparisons of service costs, staff productivity (i.e. cases per clinician) or the percentage of clients that remained substance free after treatment. Without this data, cost effectiveness comparisons cannot be performed. 15 The Joint Commission is an independent, non-profit organization that has accredited and certified more than 15,000 health care agencies in the United States, including BARC. 19

20 Age of study. As the Joint Commission study was conducted in 2001, its value has diminished over time. For example, the expected average length of stay (per client) may change through the years as new treatment modalities are researched and adopted in the substance abuse treatment industry. Accordingly, it is important for administrative and treatment personnel to make reasonably current comparisons of performance. BARC took initial steps in Fiscal Year 2008 to identify appropriate peer agencies and benchmark agency performance; however, as of April 2008 BARC has not identified peers During our review, BARC officials expressed frustration in identifying appropriate peer agencies. These officials said that providing direct substance abuse treatment services makes BARC a unique entity with the State of Florida (in Section 3 of this report beginning on page 39, we examine the substance abuse treatment delivery methods used by other Florida counties). However, BARC took several steps in Fiscal Year 2008 to identify appropriate peers for benchmarking. For example, in November 2007, BARC officials attended a half-day benchmarking training sponsored by the OMB. Additionally, as of March 2008, the Division reports that it continues to research benchmarking opportunities and strategies with the Joint Commission and the U.S. Substance Abuse and Mental Health Services Administration (SAMHSA). Despite these efforts, as of April 2008 BARC had not identified peers for benchmarking. Recommendation 3: We recommend that the Board of County Commissioners direct the County Administrator to take the following actions: Identify appropriate peer agencies (or industry standards) that can be used as reasonable benchmarks of BARC s performance in future evaluations Identify appropriate performance measures to benchmark, including service cost and staff productivity comparisons Report the results of Division s initial benchmarking analysis to the Board of County Commissioners by September 30, 2008 Best Practice 5: Establish a process to ensure performance data reliability According to the U.S. Government Accountability Office (GAO), performance data should be reasonably timely, complete and accurate to be useful for budget allocation and policymaking purposes. To ensure reliable performance data, government agencies should implement internal controls, such as establishing clear and unambiguous performance measure definitions, having front line and middle managers independently review performance data, and testing a sample of performance data at least quarterly. 20

21 BARC has historically lacked internal controls to ensure reliable client counts are reported to the Office of Management & Budget BARC s historical client counts, as reported to the OMB each fiscal year, show a significant decrease in the number of clients served beginning in Fiscal Year For example, in Fiscal Year 2005 BARC reported 6,231 outpatient client admissions. However, in Fiscal Year 2006, BARC reported only 1,295 outpatient client admissions, a 79.2% decrease. Exhibit 11 below shows the number of outpatient client admissions BARC has reported to the OMB since Fiscal Year Exhibit 11: The number of client admissions to BARC s outpatient program, as reported to OMB, declined by 79.2% between Fiscal Year 2005 and ,000 6,000 7,005 Number of Outpatient Client Admissions 6,231 4,000 2,000 1,295 1,696 0 FY04 FY05 FY06 FY07 Source: Office of Management & Budget We also observed an apparent decrease in clients served in BARC s residential program during this same period. Specifically, in Fiscal Year 2005 BARC reported admitting 1,742 clients to its residential program. However, in Fiscal Year 2006, BARC reported admitting only 735 residential clients, a 57.8% decrease. Exhibit 12 below shows the number of residential client admissions BARC has reported to the OMB since Fiscal Year Exhibit 12: The number of client admissions to BARC s residential program, as reported to OMB, declined by 57.8% between Fiscal Year 2005 and 2006 Number of Residential Client Admissions 2,000 1,500 1,512 1,742 1, FY04 FY05 FY06 FY07 Source: Office of Management & Budget 21

22 BARC officials told us that they believe the number of clients served between Fiscal Year 2005 and 2006 did not decline at the levels suggested by data historically reported to OMB. Rather, we were told that BARC has historically struggled to obtain accurate client counts due to its reliance on manually collected data, which is often vulnerable to misinterpretation and of limited value. In Fiscal Year 2006, Division officials made a concerted effort to ensure that reasonably accurate client counts were reported to the OMB from that point forward. This effort included having supervisory staff review data accuracy during monthly meetings. Division officials told us that data reported prior to Fiscal Year 2006 is inaccurate; however, data reported after this period presents a truer picture of the Division s annual workload. 16 Consequently, we could not use historical client count data to evaluate the Division s past performance. We identified four primary concerns with BARC s Fiscal Year 2007 data which suggest additional steps should be taken to ensure data reliability Despite actions taken in Fiscal Year 2006 to improve data reliability, we identified four primary data reliability issues with BARC s Fiscal Year 2007 data. These issues suggest additional steps should be taken to ensure that accurate data is reported to the OMB, policymakers and other entities in Fiscal Year Discrepancies in Client Counts As described above, BARC has historically relied upon manually collected data to determine annual client counts. However, annual client count data is also collected within the Division s CMIS, called ECHO. To validate the client count data reported to the OMB in Fiscal Year 2007, we compared it to client count data generated from the ECHO system that was provided to us by BARC officials. Our comparison suggests that client count data reported to OMB in Fiscal Year 2007 is inaccurate. For example, BARC reported to OMB that it admitted 703 clients to its Day Treatment program in Fiscal Year Conversely, BARC s ECHO system indicates that the Division admitted 1,036 clients to the Day Treatment program. As shown in Exhibit 13 on the next page, other BARC services, such as residential, outpatient and detoxification showed smaller variances between these two reports. 16 Some of the decline in client admissions in Fiscal Year 2006 may have been due to the closing of one of BARC s facilities after Hurricane Wilma. However, Fiscal Year 2007 client admissions, which were not affected by Hurricane Wilma, are fairly consistent with Fiscal Year This indicates that inaccurate data reporting was the primary reason for the apparent decline in admissions observed from Fiscal Year 2005 to Fiscal Year

23 Exhibit 13: Data provided by BARC indicates that client counts reported to OMB are incorrect Service OMB Report ECHO Report % Difference Day Treatment 703 1, % Access 3,956 3, % Residential % Outpatient 1,696 1, % Detoxification 1,559 1, % Source: BARC & Office of Management & Budget Significant Understatement of Service Costs in Results Team Proposal Form During the development of the Fiscal Year 2008 budget, each Broward County agency was directed by the OMB to complete a Results Team Proposal Form. These forms required each agency to indicate if alternative providers outside Broward County Government could provide the same or similar services to residents. In its March 2007 response, BARC identified several private substance abuse treatment facilities that operate in Broward County, but indicated that privatization would lead to greater expense to the community. As evidence, BARC compared its unit cost of residential/outpatient service to an industry benchmark and reported that it already operates in a highly efficient manner and that further funding cuts were likely to impact the quality of client care and its effectiveness. However, as previously described on page 15, we found that BARC did not develop unit cost data until February Further analysis of data reported in the Fiscal Year 2008 Results Team Proposal Form revealed that BARC used a flawed unit cost methodology in its comparison, causing the Division to significantly understate its service costs. For example, BARC reported a cost of $10.49 per client per day for its residential treatment program. However, BARC s February 2008 unit cost analysis indicates a cost of $261 per client per day for residential treatment, which we believe is a more realistic figure. Consequently, the cost comparison provided by BARC in its Fiscal Year 2008 Results Team Proposal Form should be considered invalid. As County officials use the information and data provided in the Results Team Proposal Forms to make funding allocation decisions, it is critical for this data to be reliable. Exhibit 14 below compares BARC s per day service costs for residential and outpatient services to the costs it reported in its Fiscal Year 2008 Results Team Proposal Form. Exhibit 14: BARC significantly understated service costs in its Fiscal Year 2008 Results Team Proposal Form Service Type FY08 Results Team Proposal Form February 2008 BARC Unit Cost Analysis Residential $10.49 per day $261 per day Outpatient $6.48 per day $60 per hour Source: BARC & Office of Management & Budget (OMB) 23

24 Incomplete Data Submitted to the Florida Department of Children and Families (DCF) As part of its contractual requirements for grant funding, BARC must collect and submit monthly client treatment outcome data to DCF. Specifically, BARC s contract with DCF contains two client outcome performance measures that are used to assess the effectiveness of BARC s treatment. While additional client outcome measures exist in substance abuse treatment literature, we found that BARC relies on these two measures as the primary indicators of treatment success. Exhibit 15 below lists the two substance abuse treatment performance measures in BARC s DCF contract, and the performance standards that BARC must meet to remain in contract compliance. Exhibit 15: BARC s DCF contract contains performance measurement requirements Performance DCF Performance Measure Target Percent of discharged adults successfully completing treatment 69% (no alcohol or other drug use during the month prior to discharge) Percent of adults employed upon treatment discharge 76% Source: Florida Department of Children and Families Performance Contract No. JD220 As we desired to use data reported by BARC to DCF to evaluate the Division s performance, we met with BARC officials to discuss how this data is collected and reported. Our principle concern was the number of Fiscal Year 2007 client discharges reported by BARC to DCF. Specifically, BARC officials provided an internal report that indicated the Division had discharged 2,540 clients in Fiscal Year However, data provided by DCF showed BARC reported discharging only 546 clients in Fiscal Year BARC officials told us that data reported to DCF has historically been incomplete. Specifically, Division officials told us of continued technical difficulties uploading data to DCF s database. As of the conclusion of our fieldwork in March 2008, BARC had not provided us with a corrective action plan to ensure that complete data is uploaded to DCF in future reporting periods. However, it is important for BARC to correct this data reporting problem because DCF officials indicated to us that future grant funding decisions may be more heavily influenced by agency performance. Incomparability of Outpatient Staff Productivity Data In each Program Performance Review, we evaluate employee productivity data to assess program efficiency. In response to our request, BARC provided Fiscal Year 2007 productivity data for its outpatient treatment units. Specifically, this data appeared to indicate the average caseload per clinician. We met with BARC s four outpatient supervisors to review Fiscal Year 2007 productivity data. Our primary concern was the disparate level of productivity that we 24

25 observed across outpatient locations. For example, in March 2007, BARC s outpatient unit at its Lauderhill location reported a caseload of 17 clients per clinician, while BARC s outpatient unit at its Fort Lauderdale location reported a caseload of 94 clients per clinician. BARC s outpatient supervisors told us that each unit collected productivity data through slightly different methodologies in Fiscal Year 2007; therefore productivity data is not comparable. For example, some supervisors reported the aggregate caseload of their unit, while others reported the average caseload per clinician. Consequently, we could not assess the efficiency of BARC s outpatient program. Recommendation 4: To address the four data reliability issues we identified, we recommend that the Board of County Commissioners direct the County Administrator to take the following actions: Develop data reliability controls, such as written measure definitions and supervisory review, to ensure that accurate client count data is reported to the OMB as part of the County s performance measurement and reporting system, Ensure the use of reliable unit cost data in future service cost comparisons submitted during the budget development process, Work with DCF to ensure that complete and accurate performance data is reported as contractually required, Standardize employee productivity reporting for outpatient services to ensure the accuracy and comparability of data across units Best Practice 6: Use performance data to modify practices or change operational processes to achieve better results According to the U.S. Government Accountability Office (GAO), the benefit of collecting performance information is only fully realized when the information is actually used by managers to make decisions directed towards improving results. Managers can use performance information to identify operational problems and their causes, and to develop corrective actions; to effectively plan and prioritize workload and resources; and to identify more effective approaches to program implementation. BARC has established a structured, organization-wide process to focus agency resources BARC annually updates a document known as its Performance Improvement Plan. Similar to a strategic plan, BARC s Performance Improvement Plan helps to focus resources on meeting Division objectives, such as reducing the risk of clients acquiring and transmitting infectious diseases. To meet its Performance Improvement Plan objectives, BARC has established interdepartmental committees to assess and improve specific objectives that have been 25