Replicating Home Visiting Programs With Fidelity: A Useful Pathway For Improving Quality And Maximizing Outcomes December 15, 2010
Participants Moderator Melissa Brodowski, Children s Bureau/ACF Presenters Deborah Daro, Chapin Hall at the University of Chicago Charity Eames and Susan Zaid, DePelchin Children's Center Lindy Kaemming and Christi Bergin, St Vincent Mercy Medical Center 2
Replicating Home Visiting Programs With Fidelity: A Useful Pathway For Improving Quality And Maximizing Outcomes December 15, 2010 Deborah Daro Chapin Hall at the University of Chicago
Acknowledgements Sponsoring agency: the Children s Bureau (CB) within the Administration for Children and Families, U.S. Department of Health and Human Services Federal project officer: Melissa Lim Brodowski My colleagues on the work underlying this presentation: Kim Boller, Debra Strong, Kirsten Barrett, Cay Bradley, Diane Paulsell, Patricia Del Grosso, and others at Mathematica 4
Key Objectives Provide background on EB home visiting initiative and need for a common fidelity system Introduce our definition and overall approach to monitoring program fidelity Review the core domains captured through our fidelity system Provide a set of specific indicators states might use to craft their own systems for monitoring their diverse home visitation program investments Share approaches to using fidelity data from two programs: DePelchin Children's Center and St Vincent Mercy Medical Center 5
The Supporting EBHV Program In 2008 ACF s Children s Bureau funded 17 grantees in 15 states to: Select home visiting program models that were evidence-based (as defined for purposes of the grant) Leverage the grant funds to build infrastructure to implement, scale up, and sustain their selected programs with fidelity Participate in local and cross-site evaluations The grantees will engage partner organizations to build infrastructure and implement and sustain home visiting programs over a 5-year period 6
Grantees Selected Several Home Visiting Models Home Visiting Program Model Nurse-Family Partnership Target Population First-time pregnant women < 28 weeks gestation Number of Grantees Selecting Model 11 Healthy Families America Parents as Teachers Pregnant women or new 5 parents within two weeks of infant s birth Birth or prenatal to age 5 3 SafeCare Birth to age 5 3 Triple P Birth to age 12 1 Source: Koball et al. (2009). Grantee plan updates. 7
Cross-Site Evaluation Overview Mathematica and Chapin Hall at the University of Chicago funded to conduct a six-year cross-site evaluation Goal: identify successful strategies for adopting, implementing, and sustaining high quality home visiting programs 8
Fidelity Research Questions Were the evidence-based home visitation programs selected by the grantees implemented and delivered with fidelity? To what extent t did the grantees modify these national models to fit their target populations and local service delivery context? What contextual factors were associated with fidelity of implementation? 9
Why is Fidelity Important? Contributes to programs being consistently delivered in a manner that has been proven or believed to be effective Facilitates the ability of program managers to identify gaps in a model s current capacity to reach a specific target population or operate within a given context Contributes to enhanced practice and promotes continuous program improvement 10
The Initative s Definition of Fidelity Fidelity refers to the extent t to which h an intervention ti is implemented as intended by the designers of the intervention. Fidelity refers not only to whether or not all the intervention components or activities were actually implemented but whether they were implemented in the proper manner. 11
Interest in Two Aspects of Fidelity Structural (Implementation fidelity) Nature of the provider participant relationship Manner of service delivery Hiring qualified staff/providing sufficient training and supervision Engaging the target population Achieving recommended dosage and duration Maintaining caseload levels Dynamic (Intervention fidelity) 12
Steps in Developing the System Partnered with a small team of selected grantees to identify candidate indicators Coordinated multiple calls with national model representatives to identify their common fidelity criteria and concerns Reviewed draft system with relevant stakeholders Developed eloped specific data collection protocols and system to obtain participant level data from implementing agencies 13
Capturing Initial Fidelity Organizational readiness Compliance with a model s staffing requirements (qualifications, staffing levels, initial training) Demonstrate appropriate fit for community Sufficient participants and ability to enroll them Sufficient service linkages and resources Ability to monitor ongoing implementation 14
Capturing Ongoing Fidelity Documenting consistent practice on key indicators Caseloads Supervision Staff stability Participant engagement rates Documenting consistent and appropriate service dosage/duration as defined by each model Documenting appropriate participant-provider provider relationships and delivering services as intended 15
The Fidelity Framework Program Level: Information about service delivery locations collected monthly. Staff Level: Home visitor and supervisor characteristics collected at baseline and updated as necessary; if a home visitor or supervisor leaves, obtain termination data. Participant Level: Participant demographics and characteristic at baseline and updated as necessary; if a participant leaves the program, obtain termination data. Service Delivery: Information about the home visitor- participant relationship, the content of each home visit, and the provision of services on an ongoing bases. Source: Adapted from Barrett et al. 2010. 16
Fidelity Indicators Program Level Number of families referred to the program Number of families successfully enrolled Any significant changes in the pace of referrals or enrollments Any planned adaptations ti made in the service approach or content in response to local conditions Any significant changes in funding levels or sources of support 17
Fidelity Indicators Staff Level Demographic characteristics (gender, age, race, education) Employment experience (duration in current job, role, prior experience in delivering home based services) Supervisor s monthly supervisory caseloads Home visitor s monthly caseloads Termination i information i (date staff left) lf) 18
Fidelity Indicators Participant Level Referral information (source and time between referral and firsthome visit) Relationship of primary program participant to target child Demographic characteristics (age, race, marital status, employment, education, etc.) Pregnancy history (number of children) 19
Fidelity Indicators Service Delivery Number of visits offered and proportion completed Visit location, duration, and content covered Perception and quality of the provider participant relationship Working Alliance Inventory (WAI) Overall duration and reason for service termination 20
Fidelity Data Collected To Date Current sample covers service operations from October 2009 - September 2010 Current sample size 39 implementing agencies 236 home visitors and supervisors 1,585 participants Almost 14,000 home visits Additional information will be obtained in early 2011 reflecting program operations through h the end of 2010 21
Summary Comments Monitoring fidelity is important for all programs, not just home-based interventions or those that are evidence-based Fidelity data has multiple benefits: Provides documentation that services are being delivered as intended and that quality is being sustained Provides evidence to inform program managers on the need for model adaptations and how best to structure these efforts Provides evidence state planners can use to assess the implementation of their current investments and guide new investments in order to maximize outcomes 22
Healthy Solutions Local Fidelity Assessment Plan December 15, 2010 Charity Eames and Susan Zaid DePelchin Children's Center
Is Healthy Solutions modeling Triple P with fidelity? Does the program adhere to the Triple P framework? (Structural fidelity) Does the program adhere to the dynamic guidelines or principles of implementation? (Process fidelity) Both structural & process fidelity ensure quality & consistency of program delivery* *Mowbray, C.T., et al. (2003). Fidelity Criteria: Development, Measurement, and Validation. American Journal of Evaluation, 24 (3): 315 340. *Daro, D., et al. (2009, March 26). Proposed Evaluation Design for the Fidelity Domain (HVM 060). Memorandum to the EBHV Peer Learning Network. 24
Healthy Solutions Fidelity Measures Triple P training and accreditation (Structural) Program-level training (Structural) Content checklists (Structural) Case-specific specific logs (Dynamic) Client satisfaction questionnaires (Dynamic) Program activity reports (Structural) Weekly supervision (Structural) 25
Strategies for Monitoring Fidelity Training at both the national model and local service delivery levels Content tracking and case-specific logs Client satisfaction forms Program activity reports Weekly supervision 26
National Triple P Training Practitioners attend 5 days of Standard & Pathways Triple P training The accreditation process requires practitioners to demonstrate proficiency in program delivery and principles Accreditation occurs after 2 to 3 months Access to Triple P Provider Network Peer support network of other Triple P providers (monthly) Benefits to participation include: Ongoing consultative advice and support Mechanism for quality control of professional services Foster teamwork and collaboration Helps prevent of stress & disillusionment amongst staff 27
Additional Healthy Solutions Training Program level training Training in protocol and technical requirements necessary to properly implement and monitor the program Includes training on: Eligibility criteria Referral process Completion and submission of forms Data collection protocols Booster training sessions Occur at least semi annually Serve as a refresher Encourages communication between evaluation & clinical lstaff Deals with issues before they become problems 28
Content Checklists Tracks adherence to Triple Ti P curriculum Checklists monitor: Parental participation in sessions Key issues discussed in sessions Completion of required sessions 29
Case Specific Logs Developed dto ensure adherence to specific best practice principles identified by the program as significant to maximizing the effectiveness of the intervention Parenting interventions ti should empower families Parenting interventions should build on existing strengths Intervention services should be designed to facilitate access Parenting interventions should be culturally appropriate Parenting interventions should be both child and parent centered 30
Case Specific Logs (cont d) Supervisor documents if there is sufficient i evidence (through direct or taped observation, representation in the case record, or in supervision dialogue) that the principles were practiced for each family Case specific logs assess fidelity to these core best practice principles 31
Client Satisfaction Questionnaires Collects information on the quality of services from the participant s perspective Questions include: How would you rate the quality of service you and your child received? Did you receive the type of help you wanted from the program? To what extent has the program met your child s needs? Additional questions were developed by DePelchin and included in the questionnaire to capture whether practitioners utilized the best practices identified by Triple P 32
Program Activity Report Program Activity Reports (PARs) provide tracking on time and costs associated with program operation These reports chronicle daily dil service activities iii related to the client whether direct or indirect service activities (e.g., counseling, supporting, and teaching clients or locating resources for clients) Also tracks administrative activities related to the client both direct and indirect (e.g., weekly supervision with practitioners orreferral referral screeningpotentialclients) clients) 33
Program Activity Report Program Activity Reports (PARs) chronicle daily direct and indirect service activities related to the client (e.g. counseling, supporting, and teaching clients or locating resources for clients) Also tracks direct and indirect administrative activities related to the client, but occurred without the client present (e.g weekly supervision with practitioners or referral screening potentialclients) These reports provide tracking on time and costs associated itdwith program operation 34
Program Activity Report (cont d) For purposes p of fidelity, this document can capture the number of Triple P and Healthy Solutions trainings attended; the number of sessions that occurred between the practitioner and client; and the frequency of supervision sessions Alsotracks clinician caseload, frequency and length of client visits, and no shows Key information i from PARs is pulled by the evaluator and reported to the management team monthly Helps management team assess program implementation make changes as necessary 35
Weekly Supervision Weekly one to one supervision ii and team meetings with the Program Coordinator and clinical staff occur to discuss cases or specific service delivery issues, clinical or administrative Supervisor explores and discusses ways in which Healthy Solutions practitioners have incorporated best practices into the clinician client relationship 36
For More Information Contact information: Susan Zaid, MA Evaluator szaid@depelchin.org Charity Eames, LPC, LMFT, ACPS Program Coordinator ceames@depelchin.org 37
Healthy Connections Home Visitation Mercy St. Vincent Medical Center Lindy Kaemming, Program Coordinator Christi Bergin, Program Evaluator December 15, 2010
Overview Model Healthy Families America Enroll prenatally Enrollment based on standardized assessment at intake Level of services varies based on family needs HV caseload determined by frequency of home visits Maximum caseload of 15 families at most intense level of frequency Select curriculum (limited list) Curriculum Growing Great Kids & Growing Great Families Content includes: Basic Care; Social-Emotional development; Cues and Communication; Physical and Brain development; Play and Stimulation; and Family (or) Parent Support Caregiver determines when a module is presented 39
Overview (continued) Modified to fit context: Enhancement: Staff expertise (substance abuse, domestic violence, mental health) Target Population Referred from Lucas County Help Me Grow Prenatal through 3 months of age Standardized assessment does not dictate eligibility 40
Aspects of Fidelity Structural (Implementation fidelity) Staff demographics & credentials Staff training Staff supervision Number of clients referred, enrolled, & receiving visits Staff caseload Client t demographics Dynamic (Intervention fidelity) Provider participant relationship (WAI) Service Delivery
Staff Training Staff BS with variety of expertise Healthy Families America model 141 hours for home visitors & 151 hours for supervisors Roughly 2 months to complete Curriculum - Growing Great Kids & Growing Great Families Involves both content and skills Tier 1 5 days with GKI trainer Tier 2 6 months with supervisor Tier 3 15 assignments at own pace with supervisor Roughly 1 year to complete all tiers 42
Staff Supervision Weekly one-one-one supervision for each home visitor it 1.5-2 hours/week Parallel Process Productivity is reviewed with home visitors on the 1 st & 15 th of each month using HFA formula Weekly 2-hour staff team meetings Supervisor to staff ratio 1:7 Supervision is good for my soul (Home Visitor) 43
Service Delivery Documentation Contact Form (using Access) Module Form (from GGK) Completing both forms requires 5-45 minutes per contact with family Discussed during supervision meetings Home Visit Shadowing (video) Frequency determined by HV and supervisor Evaluated separately by the HV and supervisor, and then jointly 44
Serendipitous Lesson Delayed start-up for 4 months (due to funding disruption, cross-site evaluation planning time) Detailed training and curriculum orientation Community collaboration 45
For More Information Read the research brief Replicating Evidence-Based Home Visiting Models: A Framework for Assessing Fidelity http://www.supportingebhv.org/crossite Contact us Deborah Daro, ddaro@chapinhall.org Charity Eames, ceames@depelchin.org Susan Zaid, szaid@depelchin.org Lindy Kaemming, lindy_ kaemming@mhsnr.org g Christi Bergin, cbergin@mchsi.com 46
Questions or Comments? 47