2 Cross-Site Data Reporting and PFS 2014 Grantee Meeting March 24 25, 2015 Evaluation Phillip W. Graham, DrPH PFS Cross-site Evaluation Project Director Pamela Roddy, PhD CSAP, PEPC COR
WELCOME AND PEP-C OVERVIEW (PAMELA RODDY) 3
Agenda/Topics 4 1 2 3 Cross-site Evaluation Design Logic model, evaluation questions, key analytic features, grantee evaluation plans Data Collection Cross-site requirements at grantee level, cross-site requirements at community level, federal reporting requirements Data Quality Needs, accessibility, community outcomes issues, community surveys, substitute data requests, CLI-R issues, reporting interventions
Agenda/Topics 5 4 Data Entry (MRT) and Cleaning Key features, structure, quarterly progress report, outcome data, cross-site instruments, data cleaning 5 Access to Training and Technical Assistance Key resources, TA requests, Knowledge Base, training
Logic Model, Evaluation Questions, Key Analytic Features, Grantee Evaluation Plans EVALUATION DESIGN 6
PFS Cross-Site Evaluation Logic Model 7 The Logic Model is included in the cross-site evaluation grantee binder.
PFS Cross-Site Evaluation Questions 8 EQ1 Was the implementation of PFS programs associated with a reduction in underage drinking and/or prescription drug misuse and abuse? EQ2 Did variability in the total level of funding from all sources relate to outcomes? Did variability in the total level of PFS funding relate to outcomes, above and beyond other funding available to communities? EQ3 What intervention type, combinations of interventions, and dosages of interventions were related to outcomes at the grantee level? What intervention type, combinations of interventions, and dosages of interventions were related to outcomes at the community level?
PFS Cross-Site Evaluation Questions 9 EQ4 Were some types and combinations of interventions within communities more cost effective than others? EQ5 How does variability in factors (strategy selection and implementation, infrastructure, geography, demography, subrecipient selection, Training/TA, barriers to implementation) relate to outcomes across funded communities? Evaluation Questions are included in the cross-site evaluation grantee binder.
Key Analytic Features 10 Innovative Analytic Approaches EQ 1 EQ 2 EQ 3 EQ 4 EQ 5 Data Harmonization Qualitative Comparative Analysis (QCA) Cost Effectiveness Analysis Identification of epidemiological data Identification of matched comparison communities
Grantee Evaluation Plan Checklist The PEP-C evaluation plan checklist lists components to include in grantee-level evaluation 11 Checklist Sections PFS Evaluation Goals Evaluation Questions Required Performance Measures Measurement Behavioral Health Disparities Evaluation Plan Checklist included in cross-site evaluation grantee binder
Grantee Evaluation Plan: Review Process 12 1. Grantees should have submitted evaluation plans by December 31st 2. State Project Officers review evaluation plans using checklist 3. CSAP and PEP-C review evaluation plans for quality 4. CSAP and PEP-C meet to review, reconcile, and combine consensus comments 5. Grantees receive written feedback and guidance to address comments
Evaluation Design 13 Questions Comments Concerns
Cross-Site Requirements at Grantee Level, Cross-Site Requirements at Community Level, Federal Reporting Requirements DATA COLLECTION 14
Cross-Site Requirements: Grantee-Level Data Grantee-Level Process Data 1. Grantee Level Instrument (GLI) 2. Project Director (PD) Interview 3. Quarterly Progress Reports 15 Grantee-Level Outcome Data 1. PFS Selected Grantee-Level Outcomes
Revised Grantee Level Instrument 16 GLI-R Topic Areas Coordination of state efforts Use of strategic plans Access to data sources Data management Workforce development Cultural competence Sharing of evaluation data Sustainability The GLI-R & Guidance Manual are available for download on the PEP-C flash drive & Knowledge Base.
Project Director Interview 17 PD Interview Topic Areas Subrecipient selection process Capacity building Leveraging of resources Implementation of plans to address health disparities Selection of evidence-based programs, policies, and practices The PD interview guide is available for download on the PEP-C flash drive & Knowledge Base.
Quarterly Progress Report 18 The quarterly progress report is CSAP s grantee monitoring tool. PEP-C will extract select data to use in the cross-site evaluation. The quarterly progress report collects data on the five SPF-PFS steps and Health Disparities: 1. Assessment Accomplishments/barriers 2. Capacity Membership, meetings, funding resources, training and technical assistance, accomplishments/barriers 3. Planning Strategic plan, accomplishments/barriers
Quarterly Progress Report 19 4. Implementation Subrecipient progress, accomplishments/barriers 5. Evaluation Evaluation plan, targeted priority(ies), final evaluation report, other document upload, accomplishments/barriers 6. Health Disparities Accomplishments/barriers, health disparities documents upload The Quarterly Progress Report & Guidance Manual are available for download on the PEP-C flash drive & Knowledge Base.
Grantee-Level Outcome Data Standard grantee-level outcome data sources (e.g. NSDUH, UCR, FARS) will be pre-populated no need to submit Grantees without NSDUH data are encouraged to submit grantee-level outcome data 20
Cross-Site Requirements: Community-Level Data 21 Community-Level Process Data 1. Revised Community Level Instrument (CLI-R) o Submitted by subrecipients Community-Level Outcome Data 1. PFS Selected Community-Level Outcomes o Submitted by grantee for each community
Revised Community Level Instrument 22 CLI-R Topic Areas Subrecipient structure and capacity building Data availability and planning Stakeholders and partners Intervention implementation (incl. # reached, adaptations) Barriers and sustainability The CLI-R & Guidance Manual are available for download on the PEP-C flash drive & Knowledge Base.
Revised Community Level Instrument 23 Intervention Funding Data Includes PFS and non-pfs sources (leveraging) Intervention Cost Data Collected separately for start-up and ongoing implementation; in-kind resources Simplified SAMHSA cost template (Cost Template) o Personnel title of staff, FTEs, total costs ($) o Resources contracted services; training; supplies (equipment, curricula, fliers, etc.); incentives The CLI-R Cost Template is available for download on the PEP-C flash drive & Knowledge Base.
Community-Level Outcome Data 24 For each community, the grantee is required to submit relevant Intervening variables Consumption data Consequence data The Community Outcomes Guidance Manual & FAQ document are available for download on the PEP-C flash drive & Knowledge Base.
Community-Level Outcome Data 25 Community Outcomes Requirements-at-a-Glance are included in the crosssite evaluation grantee binder.
Community-Level Outcome Data 26 NSDUH sub-state report data cannot be used for community level outcomes due to Geographic mismatch Pooled 3-year estimates Wide confidence intervals Commonly Used Surveys YRBS PRIDE Statewide surveys Communities that Care BRFSS Community surveys
Non-Target Community Outcome Data 27 Why provide outcome data for prescription drug misuse/abuse and underage drinking if you are only targeting one or the other? Analysis of whether your efforts are associated with changes in the non-target outcome Comparisons between communities that do and do not target the outcome
Federal GPRA Reporting Requirements 28 From CLI-R Number of communities who report an increase in prevention activities that are supported by collaboration and leveraging of funding streams Number of EBPPPs implemented by subrecipient communities From Community Outcomes Number of subrecipient communities that improved on one or more PFS outcome measure
CSAP Performance Measures at the Grantee Level 29 From Quarterly Progress Report Number of training and technical assistance activities per funded community provided by the grantee Reach of training and technical assistance activities (numbers served) provided by the grantee Percentage of subrecipient communities that submit data to the grantee data system
CSAP Performance Measures at the Grantee Level 30 From CLI-R Percentage of subrecipient communities that have increased the number or percent of evidence-based programs, policies, and/or practices Percentage of subrecipient communities that report an increase in prevention activities supported by leveraging of funding stream
CSAP Performance Measures at the From CLI-R Community Level 31 Number of evidence-based programs, policies, and/or practices implemented by subrecipient communities Number of active collaborators/partners supporting the grantee s comprehensive prevention approach Number of prevention activities at the subrecipient level that are supported by collaboration and leveraging of resources Number of people reached by IOM category (universal, selected, indicated) and demographic group Number, type and duration of evidence-based interventions by prevention strategy implemented at the community level
PFS Required Outcome Measures: Grantee and Community Level 32 Consumption Measures 30 day alcohol use 30 day binge drinking 30 day or 12 month prescription drug misuse and abuse Intervening Variables Perception of parental disapproval or attitude Perception of peer disapproval or attitude Perceived risk/harm of use Family communication around drug use
PFS Required Outcome Measures: Grantee and Community Level 33 Consequences School attendance and enrollment School suspensions and expulsions Substance-related traffic crashes Substance-related traffic injuries Substance-related crime
PFS 2014 Data Collection Schedule 34 Data Collection Tool Baseline Submission Frequency of Collection Grantee Level Data Quarterly Progress Report January 31, 2015 Quarterly Grantee Level Instrument (GLI-R) April 2015* First and last year of project Project Director Phone Interview April 2015* Grantee Outcomes (for grantees without NSDUH data) Community Level Data November 1, 2015 Community Level Instrument (CLI-R) May 1, 2015* Community Outcomes (Submitted by grantee) November 1, 2015 First, third, and last years of project Annually May and November of each year Annually Data Collection Schedule is included in the cross-site evaluation grantee binder. *Pending OMB approval
Data Collection 35 Questions Comments Concerns
Needs, Accessibility, Community Outcomes Issues, Community Surveys, Substitute Data Requests, CLI-R Issues, Reporting Interventions DATA QUALITY 36
Data Quality Needs and Accessibility The cross-site evaluation aims to support and intersect with grantee evaluation. Grantees will have access to submitted data. Grantees have a shared interest in usable highquality data. 37
Data Quality Challenges 38 Challenging Data Source Community Outcomes data CLI-R Solution Clear communication between grantee evaluator and PEP-C team to ensure submitted Community Outcomes data are adequate (e.g. Substitute Data Source Requests) Grantee-level project director, project coordinator, and evaluator assist subrecipients with CLI-R data
Common Community Outcomes Issues Issue Examples Solution Outcome data need to be reported in the manner specified for the particular outcome measure Overly aggregated data can be difficult to use and interpret Specified outcome measure may require reporting by percentage, rate, etc. When communities target age 18-25, not able to include data in prescription drug misuse analyses unless submit disaggregated data (18-20/21-25) 39 Community Outcomes Guidance Manual outlines PFS required outcomes, reporting requirements, etc. Identify data sources that allow for data aggregation and disaggregation
Community Outcomes: Community Surveys Many PFS grantees are using community surveys, esp. to reach those aged 18-25. Grantees are asked to report sampling strategy and response rates, important indicators of data quality, in Evaluation Plan. Grantee Evaluation Plan should include evaluator s assessment of the generalizability of the community survey data submitted for Community Outcomes. Community Outcomes reporting asks for information on age-related sample parameters and other sample descriptors. 40
Don t Have an Exact PFS Required Outcome Measure? 41 Seek approval to substitute an existing measure that is similar to the relevant PFS Required Outcome Measure. Use the Substitute Data Source Request module in the MRT. If needed, contact PEP-C to set up a meeting with CSAP and PEP-C to discuss options.
Common CLI-R Issues 42 Subrecipients sometimes report their targeted consumption, consequence, and intervening variables incorrectly. Subrecipients may misunderstand how to report interventions. Subrecipients usually need assistance estimating targeted population size, numbers reached, and costs for interventions.
CLI R Data Quality 43 The more involved the grantee staff/evaluator, the more usable the CLI R data for the cross-site AND grantee evaluations!
CLI-R: Reporting Interventions 44 Accurate start dates for PFS interventions are needed to establish when implementation impacted community outcomes. Do subrecipients understand what constitutes an intervention? Example: Has the subrecipient appropriately listed only overall interventions rather than listing each individual activity/event separately?
CLI-R: Intervention Reporting Error To reduce reporting burden, list only the intervention name, rather than separate implementations/activities. 45 Correct Life Skills Training Social Norms Marketing Social Host Liability Ordinance Incorrect Life Skills Grade 8 Elm Middle School Life Skills Grade 8 Maple Middle School Life Skills Grade 8 Oak Middle School Radio Ads Billboard Posters Meet with Legislators Lobby Day Community Presentation
CLI-R: Population Targeted/Reached Grantees will need to help subrecipients with: Targeted population size estimates Numbers served/reached estimates (especially for environmental strategies) PEP-C uses these fields to calculate percent of population reached by intervention. PEP-C will provide guidance to Grantees on estimating numbers reached. Grantee evaluators will need to assist subrecipients with calculating these numbers. 46
CLI-R: Capturing Cost Data 47 Grantees will need to assist subrecipients with completing new cost data section of CLI-R. Sorting out PFS and leveraged funds Estimating in-kind labor and resource amounts In-Kind Contributions (non-labor) Total Estimated Value ($) Total In-kind Non-labor Contributions $0.00
Single Community Grantees 48 What do single community grantees report in the Community Outcomes and CLI-R? Guidance materials for single community grantees will be provided. PEP-C will also contact single community grantees individually to discuss.
Data Quality 49 Questions Comments Concerns
Key Features, Structure, Quarterly Progress Report, Outcome Data, Cross-Site Instruments, Data Cleaning DATA ENTRY AND CLEANING 50
Key Features of the MRT MRT designed for: Ease of use and reduced data entry burden Improved online validation and data revision processes Real-time reporting in most cases Help popups available to assist users TA request link Knowledge Base that houses instruments, manuals, FAQs 51 MRT Overview is included in the cross-site evaluation grantee binder.
Easy Access to the MRT 52
MRT Structure (4 sections) 53 Contact Information Grantee and subrecipient contact info Quarterly Progress Reporting 5 PFS steps and Health Disparities Outcome Data PFS Grantee and Community Outcome Data and Substitute Data Source Requests Cross-Site Instruments CLI-R and GLI-R
Contact Information Section 54 The Project Director (PD) will be sent MRT log-in. Grantee key staff information is pre-filled with known information. PD edits or adds grantee staff information and assigns system roles. Other grantee staff will automatically receive MRT log-ins. PD or grantee staff enters subrecipient staff information. Subrecipient staff will automatically receive MRT log-ins.
Quarterly Progress Report Process PD or Project Coordinator enters and submits Quarterly Progress Report. Grantee Evaluator and other grantee contacts can edit/enter Quarterly Progress Report but cannot submit. CSAP SPO reviews Quarterly Progress Report and requests revisions or accepts. Grantees cannot move on to their next Quarterly Progress Report until the previous report has been accepted by their CSAP SPO. 55
Outcome Data Section 56 Grantee/evaluator submits: PFS Selected Grantee-Level Outcomes (if applicable) Substitute Data Source Requests Community-Level Targeted Outcomes for each subrecipient
Cross-Site Instruments Section 57 GLI-R All grantee staff with MRT access are able to enter/edit GLI-R. Grantee evaluator can edit/enter, but not submit. Grantee PD or Project Coordinator submits.
Cross-Site Instruments Section CLI-R Subrecipients are responsible for entering CLI-R and submitting to grantee. Grantees (staff or evaluator) are able to edit and enter CLI-R data to assist subrecipients. must review and accept or request revisions to the CLI-R for all subrecipients. 58
Data Cleaning Process 59 As data are entered by the user, the system will use real-time validation to check for common errors at the time of data entry. The PEP-C Data Cleaning and Processing Team will review the data for quality and follow-up with grantees, if necessary, via a Data Feedback Form.
Cross-Site Instruments Data 60 Grantees GLI-R and CLI-R datasets will be made available in a form usable for grantee and local evaluations. Process is currently under development.
Data Entry and Cleaning 61 Questions Comments Concerns
TRAINING &TECHNICAL ASSISTANCE (TTA) 62
PEP-C TTA for Cross-Site Evaluation Core components of the PEP-C TTA: Web-based Technical Assistance request system Phone and email requests Trainings o Prior to data submission dates o Ongoing availability for training o Responsive to grantee and evaluator needs 63
Request Technical Assistance 64 If you have questions about cross-site evaluation: Click the Technical Assistance link within MRT to access the Knowledge base and/or the TA request form. o Help Desk will respond by email or phone Contact Help Desk by email or phone: o Email: PFS-PEPC@ccs.rti.org o Phone: 866-558-0724
Request Technical Assistance 65
Knowledge Base 66
Tentative Training Dates 67 Data Collection Tool Grantee Level Data Quarterly Report Grantee Level Instrument (GLI-R) Community Level Data Community Outcomes Community Level Instrument (CLI-R) Training Date November 2014 Baseline Submission January 31, 2015 Frequency of Collection Quarterly (January, April, July, October) March 2015 April 2015* First & last year of project October 2015 November 1, 2015 Annually Training Audience Grantee Staff and Evaluators Grantee Staff and Evaluators Grantee Staff and Evaluators April 2015 May 1, 2015* November and May Grantee Staff, Evaluators, and Subrecipients *Pending OMB approval
Training and Technical Assistance 68 Questions Comments Concerns