FLORIDA DEPARTMENT OF CHILDREN AND FAMILIES

FLORIDA DEPARTMENT OF CHILDREN AND FAMILIES QUARTERLY PROGRESS REPORT ON ROA IMPLEMENTATION - JUNE 2016 DELIVERABLE 2B Date: 08/23/2016 Version: 3.00

TABLE OF CONTENTS 1.1 ROA Program Plan Implementation Timeline and Initiatives... 7 1.2 ROA Program Implementation Team... 10 1.3 ROA Program Governance... 14 1.3.1 Roles and Responsibilities... 16 2.1 Status of ROA Program Implementation (SFY 2015-16)... 17 2.2 ROA Program Implementation Risks... 30 2.3 ROA Program Implementation Issues... 30 2.4 Looking Forward Planned SFY 2016-17 ROA Implementation Activities... 31 TABLE OF EXHIBITS Exhibit 1: ROA Program Initiatives... 10 Exhibit 2: DCF, OCW and PQMU Organization... 12 Exhibit 3: ROA Program Implementation Governance... 15 Exhibit 4: ROA Program Governance Roles and Responsibilities... 16 Exhibit 5: Status of ROA Program Implementation Initiatives... 28 Exhibit 6: ROA Program Implementation Risks... 30 Exhibit 7: ROA Program Implementation Issues... 31 Exhibit 8: ROA Program Implementation Activities (SFY 2016-17)... 33 Florida Department of Children and Families - Office of Child Welfare Page i

Revision History DATE AUTHOR VERSION CHANGE REFERENCE 06/22/2016 North Highland.001 Prepared initial draft 06/30/2016 North Highland 1.00 07/27/2016 North Highland 1.01 07/28/2016 North Highland 2.00 Updated report to prepare for submission to DCF for review Updated report based on DCF feedback and submitted for informal review Updated report based on DCF feedback and prepared for submission to DCF for review 08/05/2016 North Highland 2.01 Updated report based on DCF review feedback 08/15/2016 North Highland 2.02 08/23/2016 North Highland 3.00 Updated report based on DCF feedback and prepared draft for DCF review Updated report based on DCF feedback and prepared for DCF review Quality Review NAME ROLE DATE Tina Worley NH Client Lead / Project Manager 06/28/2016 Tina Worley NH Client Lead / Project Manager 06/30/2016 Tina Worley NH Client Lead / Project Manager 08/01/2016 Florida Department of Children and Families - Office of Child Welfare Page ii

EXECUTIVE SUMMARY Creating positive change for Florida s children and families is only possible when all the organizations involved with child welfare recognize their individual and collective roles in enhancing the safety, permanency, and well-being of those served. In Florida, the key child welfare stakeholders and partners include the Department of Children and Families (DCF, Department), Community-Based Care (CBC) lead agencies, contracted community providers, other state agencies, tribes, and the legal and judicial systems. Collectively, these stakeholders represent Florida s Child Welfare Community. The unique partnerships within the Child Welfare Community create opportunities for long-term improvement by bringing together many perspectives and experiences with a singular focus on improving the lives and safety of each child in Florida. The Florida Legislature passed Florida Statute (F.S.) 409.997 in 2014. The actions of the Legislature allowed the creation of a platform for extensive advancement of the child welfare system through establishment of the Results-Oriented Accountability Program (ROA Program, Program). The goal of the ROA Program is to develop an integrated, research-informed framework designed to inform communities, the child welfare system, and legislators on essential elements of child protection (Chapters 20, 39, and 409, F.S.). This document is the Progress Report for the implementation of the ROA Program for State Fiscal Year (SFY) 2015-16, reflecting status as of June 2016. The Progress Report references F.S. 409.997 and the implementation initiatives and roadmap outlined in the ROA Program Plan created by the Department in February 2015, to assess the progress of the Program implementation. The report also discusses circumstances resulting in deviation from the implementation roadmap and how guidance from Department Secretary Mike Carrol and the ROA Governance Committee establishes the strategic direction and focus for the ROA Program implementation efforts on an annual basis. The status reported in this document is based primarily on discussions with Office of Child Welfare (OCW) management and staff, participation in ROA Program meetings and workgroups, and a review of relevant documentation and includes the following sections: Program Background: This section includes an overview of the ROA Program background and purpose, implementation team, and governance. Program Implementation Status: This section includes information detailing the status of the Program implementation to date, including implementation activities, risks, and issues. A summary of the ROA Program objectives and major Program implementation activities planned for the upcoming state fiscal year is also provided. Recommendations: This section presents recommendations for the Department s consideration related to the implementation activities planned for the upcoming state fiscal year. Florida Department of Children and Families - Office of Child Welfare Page 1

Key Accomplishments Over the past year the Department has made a number of significant accomplishments with respect to the ROA Program implementation, as highlighted below. Additional detail on these accomplishments and other implementation activities is provided in Section 2 of this report. A Child Welfare Performance and Quality Management Unit (PQMU) was created within the OCW and charged with the responsibility for planning and managing the ROA Program implementation. A ROA Project Manager has been identified to serve as a dedicated resource for the ROA Program implementation and will be moving into this role in early SFY 2016-17. The PQMU expended over $100,000 in ROA-related hardware and software during SFY 2015-16. As part of this investment, the PQMU secured licenses for and implemented Tableau data visualization software and SPSS predictive analytics software. Basic training on Tableau and SPSS was conducted for PQMU staff, and advanced training in both applications is planned for SFY 2016-17. Additionally, three PQMU Performance Management Unit (PMU) staff obtained their Six Sigma Yellow Belt and Green Belt certifications in SFY 2015-16. Twenty-nine interactive dashboards for priority measures were developed in Tableau for child welfare stakeholders. A subset of those dashboards for CBC Scorecard measures was released in May 2016 to Program stakeholders and the general public via the DCF website and Tableau Public. The PQMU also developed a Child Welfare Key Indicators Report which is distributed monthly to child welfare partners. The report presents a number of measures the Department has identified as critical to assessing the overall health of the child welfare system. The PQMU established a monthly process to review the Key Indicators Report with DCF management to discuss trends and prioritize items for deeper analysis and further action. The ROA Program Technical Advisory Panel, which was established to serve in an advisory capacity during the development of the ROA Program Plan, was remobilized to continue to perform in this role throughout the Program implementation. A Technical Advisory Panel Charter was created describing the role and responsibilities of the panel. The ROA Program Governance Committee was established, and members were appointed in March 2016. A Governance Committee Charter was developed describing the role and responsibilities of the committee. The initial meeting of the committee was held on April 7, 2016. During this meeting the Governance Committee Charter, ROA Program implementation status, and plan for the upcoming SFY were presented and discussed. An introduction to Tableau and demonstration of the dashboards were presented to the committee. The feedback received from the ROA Governance Committee on the status of the Program implementation was very favorable, and the next meeting is planned for July 2016. Subject matter experts from ROA stakeholder groups statewide collaborated to develop detailed process diagrams and narratives for the five activities that drive the ROA cycle of accountability: Outcomes Monitoring, Data Analysis, Research Review, Evaluation, and Quality Improvement (QI). Process documentation for the identification Florida Department of Children and Families - Office of Child Welfare Page 2

and validation of ROA outcome measures was also created. The process documentation describes the activities required to complete the ROA Program processes, the resources responsible for executing those activities, as well as critical hand-off points between the processes. Additional external resources were secured through competitive procurement to support Program implementation efforts in the areas of advanced data analytics and data strategy and governance. Under the PQMU s direction, a Florida Safe Families Network (FSFN) Business User Data Dictionary and Data Governance and Strategy were developed. The FSFN Business User Data Dictionary identifies, documents, and analyzes FSFN data critical to supporting ROA measures and reporting and advanced analytics initiatives. The Data Governance and Strategy document provides an overarching data strategy linked to business strategy, data governance standards, data standardization and stewardship responsibilities, standard procedures for data acquisition, and master data management (MDM) procedures and constructs. Additionally, an advanced analytics project was conducted to study re-maltreatment and re-reporting to better understand risk factors influencing outcomes for children and to identify needs for policy and practice changes. A key finding from this analysis was breaking the cycle of maltreatment requires policies and practices designed to counter perpetrator recidivism over a period of five to ten years or more. As a result of monitoring out-of-home care (OOHC) data and identifying an increase in the number of removals and placement of children in OOHC, the Department partnered with Casey Family Programs and The Ounce of Prevention Fund of Florida to conduct an analysis of this trend, utilizing data analysis techniques such as multivariate analysis. The OOHC Analysis Project, which is representative of the outcomes monitoring and data analysis stages of the ROA Program s cycle of accountability, was completed in February 2016 with a final report providing countermeasures and recommendations. The FSFN System Adoption Project was initiated which involves a collaboration between the Department and each CBC lead agency to identify CBC information and technology requirements and develop an individualized System Adoption Plan that achieves full adoption of FSFN while supporting CBC business processes. This project will help to identify training and policy needs and ensure complete, accurate, and consistent data resides in FSFN for the ROA Program and other DCF initiatives. The Department engaged the Florida Institute for Child Welfare (FICW) to conduct a feasibility study to inform the design of a planned study to assess the inter-rater reliability and validity of new family assessment tools. The feasibility study is scheduled to be completed in December 2016. Key Challenges Key challenges the Department has faced in the initial year of the ROA Program implementation include: Annual funding for the ROA Program implementation for SFY 2015-16 was very limited in comparison to the estimated funding requirements described in the ROA Program Florida Department of Children and Families - Office of Child Welfare Page 3

Plan. This constraint impacted the OCW s ability to complete all activities outlined for SFY 2015-16 in the ROA Program Plan while maintaining business as usual. As a result, OCW resources were allocated between the ROA Program implementation activities and day-to-day operational responsibilities and demands. The ROA Program Plan recommended a much larger number of dedicated resources to perform implementation activities, including DCF, FICW, and contractor resources. The shift in available resources likely will impact not only the duration of the implementation plan, but also the overall scope and approach. As a result, the implementation roadmap and the actual implementation efforts are not aligned, which makes planning for the multi-year implementation difficult. Recommendations for SFY 2016-17 For SFY 2016-17, under the guidance and leadership of the DCF Secretary and ROA Governance Committee, the OCW has established six objectives for the ROA Program which serve to focus implementation activities. The PQMU has identified the tasks required to achieve these objectives which form the foundation of the ROA Program SFY 2016-17 implementation schedule. Based on the PQMU s planned implementation activities for SFY 2016-17, North Highland has developed recommendations for the Department s consideration, which align with three of the ROA Program objectives and are centered on continuing to build the infrastructure (people, processes, and technology) necessary for the Program s success. These recommendations are summarized below and presented in more detail in Section 3 of this report. Objective 3: Operationalize Year One foundational components of OCW Data Governance and Master Data Management Strategy Recommendation: North Highland recommends the Department build executive management and program area leadership buy-in and support for formal data governance through establishing a: Well-executed communications strategy Strong business case Comprehensive framework and charter documents Once the objectives of data governance are agreed upon and approved, the Department should designate data governance roles and responsibilities, document policies and procedures under which data governance will operate, define the supporting IT and business processes, and evaluate the necessary tools. Recommendation: North Highland recommends the OCW develop a person-centric view of data, with the child data as a sub-set. This effort builds on language in the ROA Program Plan specific to creating a data viewpoint from the perspective of a child in care and should be coordinated with the Department s Client Data Link project Florida Department of Children and Families - Office of Child Welfare Page 4

already underway. Planning, processes, procedures, and tool evaluation specific to MDM should leverage the foundation built for data governance. Objective 5: Identify and validate key drivers to service delivery, financial stability, and outcomes to be monitored monthly utilizing individual CBC dashboard views Recommendation: North Highland recommends the Department conduct an additional inventory and study of services and practice data maintained at each CBC, as noted in the ROA Program Plan. Based on the results, a plan and roadmap can be developed to integrate and assimilate this information to enrich FSFN. Recommendation: North Highland recommends the Department leverage an algorithm already being used for an existing Master Person Index (MPI) effort and create a process to apply the algorithm to FSFN at the source to begin consolidating the multiple person identities correctly. In the longer term, the Department may consider a more sophisticated commercial solution for potential improvement. Objective 6: Develop a statewide data analytics network that inventories, shares, and leverages insights from the various data analytics projects underway in the State Recommendation: In addition to the ROA communication strategy to be executed in SFY 2016-17, North Highland recommends the PQMU create an Organizational Change Management (OCM) Plan for the ROA Program implementation and begin to establish a change network across the Child Welfare Community stakeholders impacted by the initial phase of the Program deployment. Recommendation: North Highland recommends the OCW designate a point person for all OCW advanced analytics projects and initiatives. Furthermore, it is recommended the OCW be the business owner of child welfare analytics for the State to facilitate opportunities for collaboration while ensuring consistency in approach and execution. Florida Department of Children and Families - Office of Child Welfare Page 5

SECTION 1 PROGRAM BACKGROUND Improving outcomes for Florida s vulnerable children and families is dependent on the concept the entire community is responsible for child safety, permanency, and well-being. Florida s child welfare system consists of key stakeholders and partners that include the Department, CBC lead agencies, contracted community providers, other state agencies, tribes, and the legal and judicial systems. These stakeholders, which comprise the state s Child Welfare Community, each play an important role in and have a shared accountability for the safety, permanency, and well-being of Florida s children. With the support of state policymakers and legislators, DCF and its partners are implementing the ROA Program to deliver greater innovation and reform. The Program will address existing and future challenges and drive positive change in Florida s child welfare system. To do this, the Program will establish a robust and continuous cycle of monitoring, data analysis, research, evaluation, and quality improvement to further advance the system s efforts to improve outcomes, identify new programs and services impacting those outcomes, and enable research and evidence-informed practice, policy, and decision-making. In 2014, the Florida Legislature passed F.S. 409.997, directing the Department to contract with a qualified consultant or organization to develop a plan for the implementation of the ROA Program. As stated in F.S. 409.997, the purpose of the ROA Program is to monitor and measure: the use of resources the quality and amount of services provided child and family outcomes The Program conceived by the Legislature and whose design and implementation is laid out in detail in the ROA Program Plan submitted by the Department in February 2015 is focused on improving the outcomes of the children and families served by the OCW through evidence-informed interventions, practices, and policies. The Program is based on the cycle of accountability framework presented by Mark Testa and John Poertner in their book, Fostering Accountability, Using Evidence to Guide and Improve Child Welfare Policy. The cycle of accountability relies on operationalizing five key activities to further advance the efforts of state child welfare systems to evaluate performance on outcomes, identify new or promising interventions and strategies, review the validity of programs, and conduct continuous quality improvement to ensure the system as a whole is learning and moving toward the accomplishment of goals which positively impact children and their families. Florida Department of Children and Families - Office of Child Welfare Page 6

The five activities included in the cycle of accountability are: 1. Outcomes Monitoring, activities to define, validate, implement, and monitor outcome measures throughout the Child Welfare Community 2. Data Analysis, approaches and procedures to critically study performance results to determine if variances discovered are in fact issues which should be explored further 3. Research Review, a series of activities to gather and validate evidence to support the development and implementation of interventions to address areas for improvement 4. Evaluation, activities to assess promising interventions for children and families to determine if deployment to a larger population is warranted 5. Quality Improvement, a series of actions to implement interventions across new domains, or to challenge, change, and test new assumptions about the underlying goals supporting the child welfare practice model The ROA creates accountability and transparency by incorporating processes and tools for the dissemination of performance results to the Child Welfare Community and general public. The goal of Program accountability and transparency is to present accurate and timely performance information, along with an analysis of factors influencing trends, enabling stakeholders to view a true picture of the system and potential needs for improvement. 1.1 ROA PROGRAM PLAN IMPLEMENTATION TIMELINE AND INITIATIVES The ROA Program Plan provides a five-year roadmap of initiatives for the implementation and ongoing operation of the Program. Initial implementation activities occur over two years to establish the ROA Program, including an initial set of measures and the associated reporting and monitoring for key stakeholder groups. After the initial implementation, there are three years of ongoing implementation activities to continue to incorporate new data, measures, reporting, and monitoring for additional stakeholder groups. In total, the Program Plan identified 15 implementation initiatives and their associated activities, which are summarized below in Exhibit 1: ROA Program Initiatives. NO. INITIATIVE DESCRIPTION IMPLEMENTATION ACTIVITIES 1 Results-Oriented Accountability Implementation Project Team This initiative creates the management structure and processes required to manage and oversee Program implementation. Establish ROA Implementation Project Team Create program management processes Create governance processes and procedures Develop implementation master schedule Engage project teams and execute prioritized initiatives Manage and respond to risks affecting program implementation PLANNED DURATION Initial 07/2015-12/2015 Ongoing 01/2016-06/2020 Florida Department of Children and Families - Office of Child Welfare Page 7

NO. INITIATIVE DESCRIPTION IMPLEMENTATION ACTIVITIES 2 Measure Development and Validation 3 Master Data Management 4 Data Lab and Tools 5 Data System Updates for Initial Measurement Gaps 6 Accountability Reports 7 Quality Assurance (QA) / Compliance Resource Analysis This initiative develops reliable and valid outcome measures, sets outcome measure baselines and targets, and identifies outcome measure drivers. This initiative develops and implements effective governance processes and procedures to enable data collaboration, quality, and standardization while safe guarding confidentiality. This initiative establishes the ROA analytics hardware and software environment. This initiative resolves gaps between the selected ROA outcome measures and FSFN data required to calculate the measures. This initiative supports the development of ROA reports for Program stakeholders, creating transparency and accountability by facilitating timely and meaningful access to Program results. This initiative assesses and provides recommendations on the current utilization of internal and external QA/QI resources and the QI resources required to support the ROA Program. Identify ROA measures Confirm measure validity and identify unintended consequences through research, data analysis, and logic models Develop calculation algorithms Set performance baselines and targets Pilot measures and make necessary adjustments based on results Update relevant documentation Define MDM procedures and documentation Establish data sharing agreements with stakeholders Coordinate updates to data sharing agreements as needed Implement MDM across the Child Welfare Community Install and configure analytics software Test software installation and configuration Conduct knowledge transfer Resolve gaps between initial set of ROA outcome measures and the availability of FSFN data required to calculate the measures Develop report requirements Design reports for stakeholder review and approval Develop report extracts, data transformation, and presentation mechanisms Test reports Develop and conduct end user training Deploy reports Conduct ongoing annual updates Identify internal and external units conducting QA/QI and contract compliance activities Assess capacity to conduct case reviews and other QI activities Assess external capacity/cost to conduct contract compliance reviews Develop QI staffing recommendations PLANNED DURATION Initial 01/2016-03/2016 Ongoing 04/2016-06/2020 Initial 01/2016-06/2016 Ongoing 07/2016-06/2020 01/2016-06/2016 07/2016-06/2017 10/2015-03/2016 Ongoing 07/2016-06/2020 10/2015-01/2016 Florida Department of Children and Families - Office of Child Welfare Page 8

NO. INITIATIVE DESCRIPTION IMPLEMENTATION ACTIVITIES 8 Quality Improvement Organization 9 Results-Oriented Accountability Reporting System 10 Child Welfare Community Data 11 Institutional Review Policy (IRB) Updates 12 Research Standards This initiative includes assessing the QI needs of the ROA Program, securing the staff to meet the requirements, and developing and delivering a QI focused training for managers, staff, and other stakeholders to instill and reinforce a learning organization mindset. This initiative establishes the operational reporting system to monitor and improve results and accountability across Child Welfare Community. This initiative develops non-fsfn data interfaces to incorporate quality data from the Child Welfare Community into the ROA reporting system and addresses data quality issues in FSFN. This initiative provides an IRB policy and process for the ROA Program which meets federal and other requirements. This initiative develops ROA Program research standards, a Levels of Evidence construct, methodologies, protocols, and tools to ensure research reviews are conducted consistently and produce reliable results. Assess QI needs associated with the Program Develop/Modify a QI Program Plan and procedures to include ROA functions Develop a QI Staffing Plan Obtain QI staff to support Program operations, research, evaluation, and QI functions Develop and deliver training on the QI/Learning Organization for managers, line staff, and stakeholders Incorporate stakeholder data and screens into the ROA Reporting System Complete typical development life cycle Complete FSFN System Adoption initiative Map data sources to target fields Provide data details for data sharing agreements Develop interfaces and data extraction and transformation tools Configure data cleansing procedures and tools Facilitate the review of IRB processes employed by FICW and affiliated institutions Determine the need for updates to FICW or DCF IRB policy to ensure all federal and other requirements are met Develop Program-specific procedures for engaging the IRB process Research and develop levels of evidence construct specific to Florida and child welfare Research, test, and implement a meta-analysis protocol for research reviews requiring meta-analysis of large numbers of target studies Develop a process for estimating research review budgets PLANNED DURATION 01/2016-06/2016 Initial 01/2016-03/2016 Ongoing 04/2016-06/2020 Initial 01/2016-03/2016 Ongoing 04/2016-06/2020 04/2016-05/2016 04/2016-06/2016 Florida Department of Children and Families - Office of Child Welfare Page 9

NO. INITIATIVE DESCRIPTION IMPLEMENTATION ACTIVITIES 13 Pilot Study Standards 14 Research and Evidence- Informed Practice Training Development 15 Results Oriented Accountability FICW Support This initiative develops pilot procedures to determine all of the critical elements, approvals, and considerations to be addressed prior to implementing a pilot. This initiative includes conducting a training needs assessment and developing research and evidence-informed practice training to drive a culture shift within the system to that of a learning organization in which data is used in decision-making. This initiative represents FICW activities required to support the ROA Program, including providing a ROA center of learning, acting as ROA advisors, leading research and evaluation pilots, and optimizing the ROA implementation. Develop pilot study procedures Assess and update CBC contracts to allow for pilots Create standards for project management of pilot studies Create protocols for staffing pilots Conduct a training needs assessment Design a training strategy Develop training materials Implement training materials through FICW Conduct training evaluation to determine effectiveness Integrate course into new hire curriculum Coordinate database administration Advise ROA Program Implementation Team Perform data analysis Perform data extracts, transformation, and loads Perform program research Develop ROA policy, strategic design, and thought leadership Support ROA implementation and training PLANNED DURATION 04/2016-08/2016 07/2016-11/2016 07/2016-06/2020 Exhibit 1: ROA Program Initiatives The ROA Program Plan estimated the cost for the first year of the Program implementation (SFY 2015-16) between $8.5 and $10.6 million and recommended 46 to 55 full time equivalents (FTEs) working across the Department, FICW, contracted providers, and external vendors. However, the Department has received $500,000 in annual funding to implement the Program, an amount which represents only a small fraction of the original cost estimates. Each year, with guidance from the DCF Secretary and ROA Governance Committee, the OCW establishes a set of objectives for the ROA Program which serve to focus the PQMU s implementation efforts based on the original ROA Program initiatives. The PQMU has worked diligently over the past year towards completing the planned SFY 2015-16 Program implementation activities which spanned across ten of the fifteen ROA Program initiatives. 1.2 ROA PROGRAM IMPLEMENTATION TEAM The Department s OCW is primarily responsible for the implementation and operation of the ROA Program. The OCW is committed to the safety, well-being, and timely permanency of Florida Department of Children and Families - Office of Child Welfare Page 10

Florida s children and families and is responsible for a wide range of services, including assistance to help families stay intact or be reunified, out-of-home care, adoption, and independent living skills for foster care youth transitioning to adulthood. To do this, the OCW works with six DCF Region Offices, 17 CBC lead agencies, and six Sheriff s Offices to execute policy and practice for child protective investigations and case management services. The ROA Program Plan recommended a reorganization of the OCW to create a structure to better support the implementation and operation of the Program. In 2015, the OCW established the PQMU based on the Program Plan s recommended design to oversee the implementation and ongoing operation of the Program. The PQMU is led by the PQMU Director who reports directly to the Assistant Secretary for Child Welfare. The PQMU includes three sub-units: Data and Reporting Unit (DRU): This unit supports the Department s internal and external reporting needs, including developing and producing federal reports, management reports, operations reports, ad hoc reports, and performance dashboards, as well as assisting the Department in responding to media, legislature, and general public requests for information. The DRU also provides data sets and SQL queries to the Performance Management Unit to be used for data analysis. Performance Management Unit: This unit monitors key performance indicators, produces performance summaries, and disseminates information through publications and presentations. These products contain initial stages of data analysis in support of Root Cause Analysis, a critical part of the ROA Program s cycle of accountability that ties data reporting, monitoring, and analysis together with continuous quality improvement. The PMU also supports data-driven management through the provision of ad hoc data analysis services and support of research and evaluation projects. Quality Improvement / Quality Assurance (QI/QA) Unit: This unit performs ongoing case reviews of children served by DCF, including subjects of child protective investigations or children who are/were recently in out-of-home or in-home care. The unit also interviews parties involved in the child s case to understand how day-to-day practice influences child and family functioning and outcomes. Quality management (QM) data is regularly reported to DCF leadership, Region Offices, and CBC lead agencies to ensure operations are engaged in activities to improve practice. Exhibit 2: DCF, OCW and PQMU Organization depicts the placement of the PQMU in the Department s organizational structure and highlights the detailed organization of the PQMU. Florida Department of Children and Families - Office of Child Welfare Page 11

The People Governor Secretary Inspector General Children s Legal Services Director General Counsel Deputy Secretary Chief of Staff Legislative Affairs Director Communications Director Planning Director Assistant Secretary for Economic Self-Sufficiency Assistant Secretary for Operations Assistant Secretary for Administration Assistant Secretary for Sub. Abuse & MH Assistant Secretary for Child Welfare Economic Self-Sufficiency Director Regional Managing Director Central Region Human Resources Director Sub. Abuse & MH Director Child Welfare Practice Director Office of Public Benefits Integrity Director Regional Managing Director Northeast Region Chief Financial Officer Sexually Violent Predator Program Director Child Welfare Operations Director Refugee Services Director Regional Managing Director Northwest Region Chief Information Officer Homelessness Director Child Welfare Planning & Strategic Projects Director Regional Managing Director Southeast Region General Services Director State MH Treatment Facilities Director Child Welfare Performance & Quality Management Director Regional Managing Director Southern Region Contract Services Director NE FL St. Hosp. & N FL Eval. Trmt. Ctr. Administrator Child Welfare Performance and Quality Management Unit Regional Managing Director Suncoast Region Florida State Hospital Administrator Child Welfare Performance and Quality Management Unit Director Adult Protective Services Director Data and Reporting Manager QI/QA Manager Performance Management Manager OCW Hotline Investigations / Reporting Federal Liaison Case Management / CCL Ongoing Services / Special Projects Hotline QA Investigations / Data Analytics / FSFN Adoption Federal Reporting QA Specialist / System Admin Investigations / Child Fatalities Finance / Independent Living Reporting Title IV-E Stipend Specialist Hotline /DV /IL /HT Special Analyses / All Areas Exhibit 2: DCF, OCW and PQMU Organization Florida Department of Children and Families - Office of Child Welfare Page 12

Due to limited funding, the OCW implemented the ROA Program Plan s recommended functional organizational changes with the exception of the ROA Program Implementation and enhanced training functions described below. The temporary ROA Program Implementation function was to include approximately seven program management staff responsible for and dedicated to the development of the ROA Program and its related processes, including the management of initiatives within the Department and across key stakeholders (See Initiative 1). Because the ROA Program Implementation function was not included and staffed as part of the PQMU s initial organizational structure, the workload originally planned for this unit was reallocated across the other three areas of the PQMU. During the first year of implementation, the PQMU Director and the three PQMU managers formed the core ROA Program Implementation Team. The enhanced training function proposed in the ROA Program Plan was to be responsible for developing longer-term system-wide training strategies, coordinating with partners, such as FICW, to develop and deliver Program training across the Child Welfare Community, identifying training needs resulting from ROA Program activities, and identifying new and effective ways to enhance the skills of all ROA Program stakeholders (see Initiative 14). The ROA Program is intended to be a Department-University partnership; therefore, FICW plays a key role in the Program implementation, particularly in the areas of research, advanced data analysis, and predictive analytics. FICW was established in 2014 through F.S. 1004.615 as a consortium of public and private universities offering degrees in social work with the purpose of advancing the well-being of children and families by improving the performance of child protection and child welfare services through research, policy analysis, evaluation, and leadership development. FICW s Director serves as a member of the ROA Governance Committee, ROA Technical Advisory Panel, and ROA Program Implementation Team. While FICW has been actively involved in many Program implementation activities to date, similar to the Department, resource constraints in SFY 2015-16 limited their ability to execute the implementation activities originally intended for FICW in the ROA Program Plan. In addition to the OCW and FICW, the PQMU routinely calls on subject matter experts from key stakeholder groups, including the DCF Region Offices, CBC lead agencies, community service providers, and advocates, to perform activities in support of the Program implementation. For example, many of these stakeholders participated on workgroups to lead the design and documentation of the ROA Program processes. In SFY 2016-17, the PQMU Director will appoint a ROA Project Manager as a dedicated resource to drive implementation efforts with both internal and external stakeholders and perform the activities described in Initiative 1. The ROA Project Manager has been identified and will transition into this new role in early SFY 2016-17. In addition to the ROA Project Manager, a dedicated FICW resource with advanced research and data analysis skills will be collocated with the PQMU in SFY 2016-17 to assist with a number of the ROA implementation initiatives. Florida Department of Children and Families - Office of Child Welfare Page 13

1.3 ROA PROGRAM GOVERNANCE The ROA Program governance structure provides a framework for how decisions will be made about the Program s implementation. Well-defined governance is critical to the Program s success in that it establishes the roles and responsibilities of all parties involved in the implementation and determines, in advance, clear decision-making processes and authority. The ROA Program governance structure includes the following components: ROA Program Implementation Team, which performs the activities necessary to implement the ROA Program and is currently comprised of the PQMU managers and staff and representatives from the OCW, DCF Region Offices, CBC lead agencies, community providers, university schools of social work (FICW and University of South Florida (USF)), and child welfare advocates and subject matter experts PQMU Director, who oversees, coordinates, and manages the day-to-day activities of the ROA Program Implementation Team with the support of a ROA Program Implementation function ROA Program Implementation function, which will be led by the ROA Project Manager to assist the PQMU Director with planning, coordinating, and managing ROA Program implementation activities FICW, which was established by F.S. 1004.615 for the purpose advancing the wellbeing of children and families by improving the performance of child protection and child welfare services through research, policy analysis, evaluation, and leadership development, and as described in the ROA Program Plan, plays an integral role in advising the Department on the planning and implementation of the ROA Program ROA Technical Advisory Panel, which was established by F.S. 409.997 to advise the Department during the planning and implementation of the ROA Program, and consistent with F.S. 409.997, includes representatives from FICW, the Senate, the House of Representatives, CBC lead agencies, family representatives, child welfare advocates, CBC providers, other contracted providers, community alliances, and family representatives, as well as the Department and Children s Legal Services (CLS) ROA Governance Committee, which serves as the decision-making body for the implementation and operation of the ROA Program and, as recommended in the ROA Program Plan, is currently comprised of representatives from the Department, Sheriff s Offices, CBC lead agencies, FICW, child welfare advocates, CLS, and the court system The ROA Program governance structure is depicted graphically in Exhibit 3: ROA Program Implementation Governance. Florida Department of Children and Families - Office of Child Welfare Page 14

Exhibit 3: ROA Program Implementation Governance Florida Department of Children and Families - Office of Child Welfare Page 15

1.3.1 ROLES AND RESPONSIBILITIES The roles and responsibilities of the key ROA Program governance structure are described in Exhibit 4: ROA Program Governance Roles and Responsibilities. ROLE ROA Governance Committee PQMU Director ROA Program Implementation (Project Management) ROA Implementation Team FICW ROA Technical Advisory Panel RESPONSIBILITIES Represents the Child Welfare Community stakeholders Provides leadership and serves as the decision-making body for the implementation of the Program Sets Program strategies and priorities Provides authority for use of resources to implement defined strategies Oversees the Program implementation, including but not limited to: staffing, assignments, stakeholder communications, schedule management, status reporting, risk and issue management and mitigation/resolution Coordinates Program implementation activities across Program stakeholder groups Provides overall leadership and guidance in support of the Program implementation Secures funding and resources for the Program implementation To be led by a dedicated ROA Project Manager beginning in SFY 2016-17 Supports ROA Program Sponsor/Director Establishes and maintains the Program implementation schedule Implements and manages project management processes Assists in coordination and support of implementation activities Performs activities to implement the ROA Program according to the Program implementation schedule Reports progress to the ROA Program Director Advises the ROA Program Director and ROA Implementation Team on implementation activities and resources Recommends improvements to proposed Program processes and procedures to ensure the implemented Program is based on best practice and meets goals and objectives Represents the Child Welfare Community stakeholders Provides technical assistance and guidance throughout the ROA Program implementation Assists in the establishment and assessment of ROA Program implementation benchmarks and milestones Monitors progress toward the achievement of ROA Program implementation benchmarks and milestones Exhibit 4: ROA Program Governance Roles and Responsibilities In November 2015, the PQMU reconvened the ROA Technical Advisory Panel to advise on Program implementation activities. In March 2016, the PQMU established the members of the ROA Governance Committee and held an initial meeting date on April 7, 2016. Florida Department of Children and Families - Office of Child Welfare Page 16

SECTION 2 PROGRAM IMPLEMENTATION STATUS Each year, the DCF Secretary, ROA Governance Committee, and the OCW establish a set of objectives for the ROA Program which serves to guide the PQMU s implementation efforts based on the original ROA Program initiatives and available resources. As detailed below in Section 2.1, the PQMU has worked diligently over the past year towards completing the planned SFY 2015-16 Program implementation activities which spanned across ten of the fifteen ROA Program initiatives. As described in Section 1.1, many of the implementation initiatives require ongoing effort over the course of the multi-year Program implementation. In SFY 2015-16 activities focused on building the infrastructure necessary to support further implementation tasks as well as the successful operation of the ROA Program. In the upcoming year, the Department s work to establish these foundational elements will continue. Section 2.4 presents an overview of the SFY 2016-17 ROA Program objectives and planned implementation activities. 2.1 STATUS OF ROA PROGRAM IMPLEMENTATION (SFY 2015-16) The current status of the ROA implementation initiatives is presented in Exhibit 5: Status of ROA Program Implementation Initiatives. The information provided includes: Status: : Work on the activity has not started In Progress: Work on the activity has started but is not complete Complete: Work on the activity is complete Ongoing: Work on the activity has started and will continue for the duration of the implementation Actual Completion: For completed activities, the date activity was completed Estimated Completion: For activities not yet started, in progress, or ongoing, the estimated completion date Comments: Additional information regarding the status of the implementation activity Florida Department of Children and Families - Office of Child Welfare Page 17

INITIATIVE/ACTIVITY Establish Results- Oriented Accountability Implementation Project Team STATUS ACTUAL ESTIMATED COMMENTS Initiative 1: ROA Program Implementation Project Team: In SFY 2015-16, these activities were supported by North Highland under the ROA Program and Data Analytics contract. In SFY 2015-16, the OCW was reorganized to create the PQMU, referencing the organization structure described in the ROA Program Plan. However, the PQMU structure does not include the recommended ROA Implementation Project Team with dedicated Complete July 2016 n/a project management resources. Create Program Management Processes Create Program Governance Processes / Procedures Complete June 2016 n/a Complete April 2016 n/a In SFY 2016-17, a dedicated ROA Project Manager position will be established within the PQMU to lead the planning and execution of ROA Program implementation activities. In SFY 2015-16, key project management plans/processes/tools were developed for schedule, communications, and risk management. The ROA Project Manager will be responsible for executing and maintaining the project management plans/processes/tools for the duration of the ROA Program implementation. In SFY 2015-16, the PQMU established ROA Program governance processes and procedures. The ROA Technical Advisory Panel was reconvened in November 2015 and met in November 2015, February 2016, March 2016, and June 2016. The Technical Advisory Panel will continue to meet at least quarterly for the duration of the Program implementation. The ROA Governance Committee membership has been established, and the first meeting was held on April 7, 2016. The ROA Governance Committee will continue to meet quarterly over the life of the Program. Pursuant to Section 409.997(2)(g), F.S., the Department must submit an annual ROA performance report to the Governor and Florida Legislature. The report summarizes performance across key safety, permanency, and well-being indicators. The OCW has submitted the annual report for the past two years, and the report covering the period of SFY 2015-16 is currently being prepared for submission on or before October 1, 2016. The ROA Project Manager will be responsible for executing and maintaining ROA Program governance processes and procedures for the duration of the ROA Program implementation. Florida Department of Children and Families - Office of Child Welfare Page 18

INITIATIVE/ACTIVITY Develop Implementation Master Schedule Engage Project Teams and Execute Prioritized Initiatives Manage and Respond to Risks Affecting Program Implementation STATUS ACTUAL ESTIMATED Ongoing n/a SFY 2019-20 Ongoing n/a SFY 2019-20 Ongoing n/a SFY 2019-20 Initiative 2: Measure Development and Validation COMMENTS This is an ongoing annual task completed by the ROA Project Manager for the duration of the ROA Program implementation. The Master Implementation Schedule for SFY 2015-16 was completed in April 2016, and a draft schedule has been prepared for SFY 2016-17 based on the ROA Program objectives. As additional task-level detail becomes known for SFY 2016-17, the ROA Project Manager will update the schedule accordingly. In SFY 2015-16, five workgroups were established to design and document ROA Program processes to prepare for the implementation of the processes in SFY 2016-17. The PQMU engaged both internal departmental stakeholders as well as external Child Welfare Community stakeholders and advisors to serve on the workgroups. The workgroups conducted regular meetings from late February through May. The final draft ROA process documentation was presented to the ROA Technical Advisory Panel on June 28, 2016. In SFY 2015-16, ROA Program implementation risks were managed by the North Highland Project Manager. In SFY 2016-17, ROA Program implementation risks will be managed by the PQMU s ROA Project Manager. Florida Department of Children and Families - Office of Child Welfare Page 19

INITIATIVE/ACTIVITY STATUS ACTUAL ESTIMATED Identify ROA Measures Ongoing n/a SFY 2019-20 Confirm Measure Validity and Identify and Address Unintended Consequences Through Research, Data Analysis, and Logic Models Develop Calculation Algorithms Set Performance Baselines and Targets Pilot Measures and Make Necessary Adjustments Based on Results n/a SFY 2019-20 n/a SFY 2019-20 n/a SFY 2019-20 n/a SFY 2019-20 COMMENTS In SFY 2014-15, a proposed set of initial measures for the ROA Program was provided in the ROA Program Plan. These measures were developed by a workgroup of Child Welfare Community stakeholders and primarily apply to the Department, CBC lead agencies and providers, and Sheriff s Offices. The measures include several of the existing federal outcome measures, CBC Scorecard measures, CPI Scorecard measures, and Priority of Effort (POE) measures. In SFY 2015-16, DCF and other key ROA Program stakeholders developed and documented a process to identify and validate ROA Program outcome measures. In SFY 2016-17, an Outcome Measures Workgroup comprised of representatives from the Child Welfare Community stakeholder groups will be established to oversee this process and the development of a validation methodology. The Outcome Measures Workgroup will review the proposed set of initial measures included in the ROA Program Plan and make updates as necessary to finalize the initial set of measures. Once the initial set of measures is finalized, validated, and deployed, additional measures will be identified and validated for other Child Welfare Community stakeholder groups, such as the judicial system, state partner agencies, etc., throughout the Program implementation. Florida Department of Children and Families - Office of Child Welfare Page 20