Implementing EBPs with Fidelity

Transcription

1 MACMHB Institute 25 Implementing EBPs with Fidelity Dean L. Fixsen, Ph.D. National Implementation Research Network Florida Mental Health Institute Goals for Today Help you become wise consumers of evidence-based practices and programs Help you see what it takes to put the products of science into service in your communities and provider agencies We Have Problems Homebuilders program developed, evaluated, and replicated (e.g. Utah and Michigan) with good effect Federal government invested $5,, over 5 years for national implementation James Bell Associates evaluation: adopted philosophy, not procedures We Have Problems Change initiatives that are heavily dependent on people (reengineering, TQM, culture change) fail 8-9% of the time About 1% of what is taught in training gets transferred to the job "Up to 7% of the failures in business are not due to poor strategy or a lack of good ideas, but to flawed execution." R. W. Rogers, 22 1

2 We Have Problems Review of TQM adoptions in 5,492 hospitals "If organizations can minimize evaluation and inspection of their internal operations by external constituents through adoption alone, they may neglect implementation altogether, decoupling operational routines from formally adopted programs." Westphal, Gulati, & Shortell (1997) We Have Problems Effective meds to cure TB but unable to use effectively with TB patients: Inexperienced local medical staff Lack of cultural competence Lack of affordable drug supply Lack of infrastructure Patients don t take pills as prescribed Khatri & Frieden, 22 We Have Problems We Have Problems Each year, more research is done More attention is directed to evidence-based practices and programs in journals and meetings Few benefits are realized by consumers of our services Human services are characterized by highly variable, often ineffective, and sometimes harmful services to consumers (Institute of Medicine, 21; U.S. Department of Health and Human Services, 1999; 21). 2

3 We Have Problems Practices seem to be "influenced by fads and fashions that are adopted overenthusiastically, implemented inadequately, then discarded prematurely in favor of the latest trend. Walshe & Rundall, 21 We Have Problems Science to Service Gap What is known is not what is adopted to help children, families, and adults Implementation Gap What is adopted is not used with fidelity and good effect What is implemented disappears with time and staff turnover. Unlike other economic resources, knowledge is not depleted but gains in value with use. Knowledge-based economies include: computer software applications, the internet, franchised operations, and evidence-based human service programs Syntheses of the intervention evaluation literature by a variety of authors Statistical meta-analyses within domains by a variety of authors Websites that publish results of assessments of evidence for particular practices or programs 3

4 Synthesis of the implementation evaluation literature (funded by the William T. Grant Foundation) Full text reviews of 1,54 articles Any domain (e.g., agriculture, business, child welfare, engineering, health, juvenile justice, manufacturing, medicine, mental health, nursing, and social services) Intensive review of implementation practices purveyor activities and implementer activities (funded by SAMHSA) ACT, CASELS, DBT, FFT, IDDT, Incredible Years, MST, NFP, nutrition for the aged, MDTFC, PBS, Supported Employment, Children s MH, adult MH, education, juvenile justice, child welfare, health, prevention/ early intervention Implementation researchers Phyllis Panzano & Dee Roth (OH), Kay Hodges & Jim Wotring (MI), Gary Bond & Bob Drake, Howard Edelman & Linda Taylor, Robert Felner, Sharon Mihalic, Jaideep Motwani, Robert Rosenheck, Sonja Schoenwald, Sidney Winter, and others Old Knowledge Developing and nationally implementing the Teaching-Family Model (treatment in group homes, foster homes, family homes, classrooms) Sharing experiential knowledge among Fairweather Lodges, National Follow- Through school models, ACT, Homebuilders, PRYDE program, etc. Longer-term view of the processes involved in useful implementation 4

5 TEACHING-FAMILY MODEL Teaching Family Replications 9 Fixsen, Blase, Timbers, & Wolf (21) Lonnie Phillips, Elaine Phillips-Stork, Montrose Wolf, Dean Fixsen, Barbara Timbers, Gary Timbers, Karen Blase Cumulative Homes Present (38 Years and Counting) Cumulative Couples Teaching and Motivation Systems Timbers, et al. (1972) Rational Problem Solving (Fixsen, Phillips, Phillips, & Wolf, 1973) Appropriate Response to Criticism Yes Linda No Yes Karen No Yes Robin No Yes Kate No Baseline Treatment Percentage of Boys Trial-Set Consequence Pre-Set Consequence Trial-Set Unfair Consequence Probe Pre-Set Consequence Trial-Set Preference Probe Opportunities Family Meetings 5

6 Leadership (Phillips, Phillips, Fixsen, & Wolf, 1973) Classroom Improvement (Bailey, Wolf, & Phillips, 197) Number of Items Completed Baseline Purchased Manager ±5 1 2 Days Indiv. Assign. Indiv. Conseq. ±5 Purchased Manager ±5 Indiv. Assign. Indiv. Conseq. ±5 Per Cent Study Baseline Daily No Report Card Card Days Daily Report Card Replication of a Program Replication is the heart of science (Sidman, et al.) We don t know until we replicate it -- repeatedly (Baer) Failure to Replicate A.P. Lots of data, but: Inadequate staff selection (degrees) Inadequate staff preparation (conceptual, idealized) Inadequate definition of core intervention components Inadequate feedback loops (slow and person dependent) 6

7 Correcting Problems Cooperation Serving Community Needs Correcting Problems Cooperation Serving Community Needs Home Environment Correcting Problems Cooperation Follow Guidelines Community Comments Cooperation Correcting Problems Communication Cooperation Effective In Helping Child Communication Fairness of T-Ps Concern of T-Ps Effectiveness of T-Ps In Correcting Problems Pleasantness of T-Ps Helpfulness of T-Ps Good Treatment Program Social Skills of Youths Teaching Skills Self-government Motivation System Family-Style Living Condition of Home (T-P Res.) Record Keeping Condition of Home (Board Res.) Court Social Service Board Schools Parents T-Ps 7=Completely Satisfied 6=Satisfied Address: 5=Slightly Satisfied Evaluation Type: 4= Neither Satisfied nor Dissatisfied 3=Slightly Dissatisfied 2=Dissatisfied 1=Completely Dissatisfied Youths Shaded area represents acceptable ange of performance Professional Evaluation Current License Yes No First Evaluation (Post Hoc) 1/72 Core Intervention Components Teaching Relationship Development Motivation Systems Rational Problem Solving Skills Curriculum Counseling Advocacy Generalization Procedures Integration of Components Practitioner Preparation Six-day preservice workshop Ongoing consultation coaching 2 Regular consumer, stakeholder, & performance evaluations Certification as Practitioners 1 Correcting Problems Cooperation Serving Community Needs Correcting Problems Cooperation Serving Community Needs Home Environment Correcting Problems Cooperation Follow Guidelines Community Comments Cooperation Correcting Problems Communication Cooperation Effective In Helping Child Communication Court Social Service Board Schools Parents Fairness of T-Ps Concern of T-Ps Effectiveness of T-Ps In Correcting Problems Pleasantness of T-Ps Helpfulness of T-Ps Good Treatment Program Youths Social Skills of Youths Teaching Skills Self-government Motivation System Family-Style Living Condition of Home (T-P Res.) Record Keeping Condition of Home (Board Res.) Professional Evaluation T-Ps Address: Evaluation Type: 7=Completely Satisfied 6=Satisfied 5=Slightly Satisfied 4= Neither Satisfied nor Dissatisfied 3=Slightly Dissatisfied 2=Dissatisfied 1=Completely Dissatisfied Shaded area represents acceptable range of performance Current License Yes No Certified Teaching-Parents 7

8 Replication Success First couple that failed went through the new training systems and were very successful Teaching-Parents Clearly, our failure, not theirs Mission-Oriented Research Practice issues define research agenda. Not a research interest but a program need. Research on staff selection methods Research on staff training methods Research on staff evaluation methods (fidelity) Implementation and Delinquency Bedlington, et al., 1982 Implementation and Youth Satisfaction with TPs Bedlington, et al., 1982 Percent Parental-Teaching With Youths 5% 4% 3% 2% 1% % r s =.94 Delinquency Teaching Homes Mean Self Reported Delinquency Ratings Percent Parental-Teaching With Youths 5% 4% 3% 2% 1% % r s =.94 Youth Ratings Teaching Homes Mean Youth Satisfaction Ratings 8

9 Mission-Oriented Research Proximity Analysis Inadequate sustainability what is implemented with fidelity disappears with changes in leadership, staff, or system supports % Sustained 5+ Years years to generate 6 attempted replications 5 years follow-up data 1 year time frame to discover proximity Far Kansas N = 6 Near (3 hrs) Near North Carolina Intermediary Organization Group Home Sustainability Regional Services Staff Selection Staff Training Staff Consultation Staff Evaluation Program Evaluation Facilitative Administration System Interventions Teaching-Family Homes Consumer Services Consumer Services Consumer Services % Homes Open 6+ Yrs Fixsen, Blase, Timbers, & Wolf (21) attempted implementations 6 years follow up 18 year time frame N = 84 N = 219 Group Home Strategy Org. Development Strategy 9

10 Mission-Oriented Research Attempted Org. Replications Research on intermediary organization development % Achieve Certification KU Ph.D. N=9 N=1 Fixsen, Blase, Timbers, & Wolf (21) Others Pre (N=19) 6.4 Yrs Org. Development Services Active Purveyor Role Manage org. change process Train trainers, coaches, evaluators, administrators Coach trainers, coaches, evaluators, administrators Evaluate trainers, coaches, evaluators, administrators Develop a community of practice Active Purveyor Role Simultaneous, Multi-Level Interventions Purveyor Practitioner Organization Management (leadership, policy) Administration (HR, structure) Supervision (nature, content) System of care State policies 1

11 Org. Implementation Phillips, Baron, Blase, Fixsen, and Maloney (1975) Attempted Org. Replications Fixsen, Blase, Timbers, & Wolf (21) % Achieve Certification attempted organizational implementations 4 years to a certification decision 2 year time frame for data collection KU Ph.D. Others Pre (N=19) 6.4 Yrs KU Ph.D. Others N=9 N=1 N=3 N=37 Post (N=4) 3.7 Yrs Org. Development Services Teaching-Family Association Treatment standards Ethical standards Annual evaluation of practitioners Tri-annual evaluation of organizations (org. fidelity measures) Forum for discussing evaluation & research findings Forum for identifying & incorporating innovations into the Model Service Delivery Systems Community-Based Therapeutic Group Homes Campus-Based Therapeutic Group Homes Treatment Foster Homes Homebased Treatment Supported Independent Living Apartments Public and Private Schools 11

12 Service Delivery Settings using Teaching-Family Model: Populations Delinquent Emotionally Disturbed Behaviorally Disordered Abused and Neglected Foster Care Home Based Residential Group Independent Living Emergency Shelter Day Treatment Developmentally Disabled Autistic Populations Served by Teaching-Family Model: Abuse & Neglect 4 48 Adult Psychiatric Autistic 229 Delinquent 236 Developmentally Delayed 19 Dual-Diagnosis 1153 Emotionally Disturbed 114 Mentally Retarded 79 Other Sex Offender Substance Abuse The amount and quality of research concerning a practice or program is not related to its usefulness in clinical practice The most common methods used to translate science to service are the least effective Effective implementation methods have been identified and evaluated 12

13 Over 16, RCTs re: pain management published in over 9 medical journals over 2 years Less than 2% use of effective pain management procedures to benefit patients experiencing pain Jadad, 1999 The usability of a program has little to do with the weight of the evidence regarding that program Evidence on effectiveness helps us select what we might choose to implement Evidence on outcomes does not help implement the program successfully Science to Service Science to Service Effective intervention practices + Effective implementation practices = Good outcomes for consumers No other combination of factors reliably produces desired outcomes for consumers. INTERVENTION Effective NOT Effective Effective Some EBPs Project DARE Scared Straight Stop Smoking IMPLEMENTATION NOT Effective Most Research Transactional Interventions 13

14 Science to Service IMPLEMENTATION Science to Service IMPLEMENTATION Effective NOT Effective Effective NOT Effective INTERVENTION EBP NOT Effective High Fidelity Low Fidelity INTERVENTION Functional Family Therapy NOT Effective 47% FFT Therapists 75% < Control Recid 53% FFT Therapists 38% < Control Recid Washington State Institute for Public Policy (22) A Way to Think Science to Service INTERVENTION Effective NOT Effective Effective Consumer Benefits IMPLEMENTATION NOT Effective In the past 3 years: Probably 6,+ experimental studies of effective interventions in physical health and behavioral health (e.g., over 15, RCTs just for pain mgmt) Perhaps 4 experimental studies concerning the factors important to the implementation of those treatments 14

15 Science to Service About 99% of the annual NIH research budget is spent on understanding etiology and developing new treatments About 1% is spent on finding ways to effectively use those treatments (up from ¼ of 1% in the 197s) Developing evidence-bases for practices and programs is important Knowing how to implement an evidence-based program is equally important One without the other leads us nowhere Excellent experimental evidence for what does not work Dissemination of information by itself does not lead to successful implementation (research literature, mailings, promulgation of practice guidelines) Training alone, no matter how well done, does not lead to successful implementation With various designs and measures, these two widely used strategies repeatedly have been shown to be ineffective in human services, education, health, business, and manufacturing 15

16 Successful implementation on a useful scale requires a purveyor An individual or group of individuals representing a program or practice who actively work to implement that practice or program with fidelity and good effect Purveyors accumulate data & experiential knowledge, more effective and efficient over time Active Purveyor Role Simultaneous, Multi-Level Interventions Purveyor Practitioner Organization Management (leadership, policy) Administration (HR, structure) Supervision (nature, content) System of care State policies Implementation strategies are common across many domains (e.g. adult MH, children s MH, justice, medicine, education) Knowledge of successful implementation strategies will benefit all human services Implementation is not an event A mission-oriented process involving multiple decisions, actions, and corrections Implementing an evidencebased program takes 2 to 4 years 16

17 Implementation occurs in stages: Exploration Installation Initial Implementation Full Implementation Innovation Sustainability Exploration Stage Awareness, preplanning, initiation (stakeholders, leaders, champions) Community-Purveyor information exchange, mutual assessment, trust Purveyor clear & explicit re: the program & its rationales & the effort required over time Assess/develop motivation & buy in Perceived risk, ability to manage risk Match local needs and EBP inclusion exclusion criteria and benefits Availability of funding Installation Stage Structural and instrumental changes (hire/redeploy staff, cell phones, HR policies, job descriptions, funding and referrals, space) Resources consumed but no consumers seen (start up may add 1-2% to first year costs) Purveyor & leaders establish clear plans and expectations Establish infrastructure and groups to carry out implementation efforts Prepare organizational staff Prepare stakeholders Initial Implementation Stage Begin to change practices, provide services to consumers Put core components in place Change organizational structures & culture, overcome fear & inertia Purveyor's implementation strategy Manage organizational change Practitioner related issues and activities Fidelity measures, process measures 17

18 Implementation Drivers CONSULTATION CONSULTATION & COACHING COACHING PRESERVICE PRESERVICE TRAINING TRAINING STAFF STAFF EVALUATION EVALUATION INTEGRATED & COMPENSATORY RECRUITMENT RECRUITMENT AND AND SELECTION SELECTION DECISION DECISION SUPPORT SUPPORT DATA DATA SYSTEMS SYSTEMS FACILITATIVE FACILITATIVE ADMINISTRATIVE ADMINISTRATIVE SUPPORTS SUPPORTS SYSTEMS SYSTEMS INTERVENTIONS INTERVENTIONS TRAINING COMPONENTS Theory and Discussion..+Demonstration in Training + Practice & Feedback in Training + Coaching in Classroom OUTCOMES (% of Participants who Demonstrate Knowledge, Demonstrate new Skills in a Training Setting, and Use new Skills in the Classroom) Knowledge Skill Demonstration Use in the Classroom 1% 5% % 3% 2% % 6% 6% 5% 95% 95% 95% Joyce and Showers, 22 Installation and Implementation: Non-Linear Processes Installation Implementation Initial Implementation Frequency of mention by implementers and purveyors (N = 43 items) Selection 3% Training 6% Coaching 15% Staff Evaluation 5% Prog Evaluation 7% Facilitative Adm 47% Systems Int 17% 18

19 Full Implementation Stage Implementation Components integrated, fully functioning New implementation site ready to be evaluated re: consumer outcomes Initial Implementation - expansion from without = import enabling talent and expertise Full Implementation - expansion from within = homegrown talent and expertise Sources of Impl. Drivers Build into your organization (i.e. train your staff to do selection, training, coaching, staff evaluation, decision support system data providers, facilitative administrators, system intervenors) Hybrid (i.e. continue to contract for outside groups to do one or more of the implementation drivers from now on) Consider implications for integrated & compensatory benefits & for sustainability Stages of Implementation Innovation First do it right (high fidelity) Then do it differently (evaluate changes, improvement/drift) Sustainability Starts during exploration, never stops Information and trust, good outcomes, expanding support base during all stages Ability to retain function while changing form given turnover, changing needs and context 19

20 Stages of Implementation: Research Findings (Panzano, Seffrin, Chaney-Jones, Roth, Crane-Ross, Massatti, et al, in press). Vertical Integration Implementation requires a sustained effort in order to produce desired outcomes at each stage Practitioner Organization System of care State policies Federal policies Facilitate Hinder Active alignment of policies and coordination of efforts in support of practitioner s use of effective practices to benefit consumers New Freedom Commission, System Transformation Work of Implementation Changing the behavior of adult human service professionals Changing organizational structures, cultures, and climates Changing the thinking of system directors and policy makers Successful and sustainable implementation of evidence-based practices and programs always requires organizational change. What This Means for You Put implementation knowledge to work in your community and provider organization Have evidence-based practices and programs come to life in your organizations Improve consistency, reduce errors, and increase benefits to consumers and their families 2

21 A Scale of Usefulness Evidence-based programs have demonstrated effectiveness through research Promising practices have some evidence for benefits to consumers (Jim Wotring & Kay Hodges in Michigan). Common practices have no data to support their use but over many years they have been built into a series of laws, regulations, funding mechanisms, professional and organizational accreditation standards, etc. that sustain them. Harmful practices have evidence indicating harm to clients but often still are supported as a common practice Move Up the Scale of Usefulness Systems need to address barriers to adoption, implementation, and sustainability and facilitate more useful strategies Organizations need to have clear theories of change and promote consistent practices and high levels of accountability for consumers, program fidelity, and outcomes Practitioners need skill sets tailored to the context, populations, and situations that will be encountered in using programs Thank You We thank the following organizations for their support re: the data and other information presented in these slides: National Institute of Mental Health (research and training grants) Juvenile Justice and Delinquency Prevention (program development and evaluation grants) Annie E. Casey Foundation (cultural competence and EBPs grant) William T. Grant Foundation (implementation literature review grant) Substance Abuse and Mental Health Administration (ORC Macro sub-contract for SOC implementation analyses; implementation strategies grant) For More Information Karen A. Blase Dean L. Fixsen kblase@fmhi.usf.edu dfixsen@fmhi.usf.edu Louis de la Parte Florida Mental Health Institute University of South Florida 21