SUD Outcomes Research: Potential Methods of Outcomes Analysis in SUD Treatment Programs Deborah Harkness, MSC, LAADC, CATC Dale White, MBA, CADC-CAS
Outcome Definition and Measurement: This session is meant to discuss the development of outcome measures for substance use disorder entities. What are the factors of most concern? Rates of success, failure, graduation, drop out, length of stay, completion of treatment plan goals, post-discharge accomplishments?
Types of Outcomes 1. Subjective/Qualitative: Based on or influenced by personal feelings, tastes, or opinions. These are statements of personal opinion in relation to an event or experience. 2. Objective/Quantitative: Not influenced by personal feelings or opinions and representing mathematical facts. These are answers of a numeric nature. They can be analyzed as an average, a logarithmic average, or using other statistical methods.
Subjective/Qualitative Response: What did you like about the event? What did you dislike about the event? What would you change based on the information presented?
Objective/Quantitative Response: How often did the client engage in treatment? What was the client s length of stay? What was the client s progress, using drug tests, analytics, and/or discharge disposition?
Likert scale The most widely used approach to scaling responses in survey research, such that the term (or more accurately the Likert-type scale) is often used interchangeably with rating scale, although there are other types of rating scales. The scale is named after its inventor, psychologist Rensis Likert. On a scale of one to ten, with one being least in agreement with the statement, and ten being most in agreement with the statement, what would you score this activity?
These are the deliverables from this session: 1. Outcomes measurement for a single event. Please look at the evaluation card you have been given for this session. 2. Outcomes measurement scoring for an episodic event. Please see the CCR Likert Scale evaluation form. This highlights the Likert form of data capture, and was used for our prevention program. All line items mirror the weekly curriculum topics.
These are the deliverables from this session (cont.): 3. Outcomes measurement for an episodic event. Please see the Family Focused Prevention outcomes report. This is an actual report to Fresno County. It includes: Curriculum information; Grant amount and budget information; Implementation of funding; Calculation of individuals served; Calculation of encounters; Calculation of individual average cost; Calculation of cost per encounter; Calculation of post-episode benefit; Strengths, improvements, and recommendations.
These are the deliverables from this session (cont.): 4. A single calendar period of treatment encounters based on diagnosis code, and cross tabbed by age group; and a single calendar period of treatment encounters based on payor, and cross tabbed by age group. 5. A treatment analysis comparing two times frames against each other.
These are the deliverables from this session (cont.): And finally, our entity policy and procedure to follow up with clients post-discharge, in order to gather qualitative information with regard to the discharge plan. Post-discharge information can take many forms. Potential queries may be: Last use of drug of choice; Adherence to the discharge plan; Connection to outside recovery entities.
Outcomes Measurement serves two purposes. One, funding entities (DHCS, County, Private) want to see that their dollars are making a positive impact on the community. However Every opportunity to measure outcomes is an opportunity to market your business. SUD treatment providers can and should embrace outcomes measurement as a tool to differentiate your program from those around your community.
Single Session Activities From a SUD Entity: How do we measure the impact and/or effectiveness of a single session event produced or enacted by a SUD treatment provider?
Types of Single Session Events That May Come From a SUD Entity: Community Service: Neighborhood Events Recovery Activities Public Service Clubs/Organizations Continuing Education Unit Trainings: AOD Counselors Therapists Attorneys Physicians
Outcomes Measurement Techniques for Single Session Event From a SUD Entity: The foundation of outcomes measurement, regardless of the activity, is the sign in sheet. When the provider facilitates, have a sign in sheet available. When the provider aids in the process of another entity, either ask for their sign in sheet, or provide your own.
Community service events might include: School-based education Recovery event presentations Booth or table exhibits at local functions
Any SUD treatment provider can become a provider of Continuing Education Units: Typical affiliations might be: CCAPP for counselors CAADE for counselors CAMFT for therapists State Bar of California for attorneys American Academy of Family Physicians for doctors
Potential Financial Benefit From Outcomes Measurement Techniques for Single Session Event: Single event activities can be resourced as data for: Public proposal development Grant funding Invitation to community activities
Multiple-Event Activities From a SUD Entity: How do we measure the impact and/ or effectiveness of a multiple- event episode produced or enacted by a SUD treatment provider?
Types of Multiple Event Activities That May Come From a SUD Entity: Anger Management Parenting Batterer s Intervention Co-parenting Counseling Extended Education
Outcomes Measurement Techniques for Multiple Event Activities From a SUD Entity: Again, sign in sheets are paramount. Determine attendance for every session provided. Satisfaction surveys on a multiplesession episode are appropriate. Design the satisfaction survey based on each session s topic, and use Likert scales on all aspects of the educational regimen.
Real world example #1: The satisfaction survey Real world example #2: The outcomes report
Potential Financial Benefit From Outcomes Measurement Techniques for Multiple Event Activities: As with single session events, multiple event activities can be resourced as data for: Public proposal development Grant funding Invitation to community activities
Treatment Activities at a SUD Entity: How do we measure the impact and/or effectiveness of treatment episodes produced or enacted by a SUD treatment provider?
Types of Treatment Activities That May Come From a SUD Entity: Intake Intake Assessment Treatment Planning Group Counseling Individual Counseling Crisis Counseling Collaborative Counseling Discharge Planning
Outcomes Measurement Techniques for Treatment Activities in a SUD Entity: Attendance: Sign-in sheets Length of Stay: Duration of treatment Discharge disposition: Success vs dropout Pivot Tables for all fields captured
Potential Financial Benefit From Outcomes Measurement Techniques for Treatment Activities: Funding initiation or enhancement Marketing Community Awareness, i.e. Prop. 36
The first step in analyzing data for treatment episodes comes with the development of data. Data can be gathered: Manually (eck!) Spreadsheets Databases Purchased software But regardless of the storage and recording medium, it is crucial that we be able to import, export, copy, paste, manipulate, and calculate.
As an example, I built a spreadsheet with data from the last fiscal year of our outpatient treatment entity. My data dictionary for this demonstration is thus: First Name Counselor Last Name Minutes Date Payment Number in Group Client ID Number Group Number Primary Diagnosis Full Name Date of Birth Topic or Event ICD-10 Age Time of Event Two more calculated fields are Fiscal Year Half and Age Range.
The accompanying pivot tables provide information such as: Total Encounters for Diagnosis Codes cross tabbed by age range; Total Encounters For Funding Source cross tabbed by age range. What else can be measured, cross tabbed, and analyzed across time periods with a data set like this?
Real world example #3: The outcomes crosstabs for July-December 2016 Real world example #4: The comparative analysis of the first half of the fiscal year and the last half of the fiscal year.
Your imagination is the limit: Number of intakes Intakes that resulted in certificates of completion Intakes that resulted in certificates of completion, cross tabbed by counselor Intakes that resulted in certificates of completion, cross tabbed by counselor and by age group Intakes with discharges by CalOMS category Intakes with discharges by CalOMS category cross tabbed by counselor Intakes with discharges by CalOMS category cross tabbed by counselor and by age group Number of group sessions per client per episode Length of time from intake to discharge, face-to-face or administrative Length of time from intake to last face-to-face Average length of stay per client Participatory average length of stay, excluding intake only with no sessions attended Logarithm-normalized average length of stay per client Participation as a ratio of the first 90-day treatment plan Participation as a ration through the most recent 90-day treatment plan Discharges with relation to drug testing results
Why Measure Outcomes? Demonstrates Quality Ensures Effectiveness and Efficiency Adherence to evidence supported treatments Use resources most efficiently Right person, right care, right time Develop appropriate levels of care
Builds Trust in the System Public Patients Providers Payors Allows for a Systematic Approach to Improvement Andrew Bertagnolli, PhD, Kaiser Permanente, 2013
Follow Up data and qualitative reporting: Most of what we have been discussing relates to quantitative outcomes measurement. However, many prefer to engage in qualitative feedback and post-discharge reports. Real world example #5: The handout has what we use to obtain permission to gather qualitative information post-discharge.
In Psychiatry Journal (2014), Steven L. Procter, et al, concluded: findings also suggest that residential programs may be best suited to place a high priority in designing and implementing systems that emphasize patient contact immediately following discharge and work to increase attendance at available continuing care options, particularly in the first 12 months after discharge. Maintaining contact post-discharge is supported by the literature, AND is therefore an excellent mechanism to capture post-discharge data.
In conclusion: Ours is an industry that longs for more and better data analysis. Ours is an industry that has benefited from what meager research and data analysis we have thus far. We all, as treatment providers and as entities, have it within our power to become outcome measurement and analytical superstars. We are like gold miners standing above a freshly discovered and yet untouched Mother Lode. We wish all of you the best of luck, and we offer our knowledge and support at any time.