Overview of CBA Program Monitoring and Evaluation Miriam E. Phields, MA, PhD Lead, Systems and Evaluation Activity CBB, DHAP, NCHHSTP, CDC PS14-1403 Capacity Building Assistance (CBA) for High-Impact HIV Prevention Orientation Meeting Event August 27, 2014 National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention Division of HIV/AIDS Prevention
Presentation Overview Monitoring and evaluation of CBA services Highlights of CBA services delivered, including TA and training, as reported in CRIS and TEC How CRIS and TEC are used in reporting and monitoring of national CBA service delivery CBA Evaluation Project (contract) Data collection activities for CBA providers (14-1403s)
Highlights of CBA Service Delivery CBA service delivery in the first four months of PS# 14-1403 CRIS requests Information dissemination, training, and TA Trainings posted to TEC CBA trainings HIP training conducted by CBA providers and posted to TEC by Danya CDC trainings
CRIS Request s Data from CRIS application. Request data from 4/1/14 7/31/14 are included (Program Year 1). Requests that have been deferred, declined, withdrawn, or are still new, in triage, or creating have been excluded. Limitations: o Data may contain some user error. o There may be some under reporting in CRIS.
CRIS Requests in First Six Months # of Requests 450 400 350 300 250 200 150 100 50 0 by PS# Cohort 1st Six Months 326 110 PS #09-906 PS #14-1403
CRIS Request s by Category CBA for Health Departments 86 26% CBA for Healthcare Organizations 25 8% CBA for CBOs 201 62% CBA for HPGs 14 4% Tot al (7/31/14) = 326
CRIS Requests by Delivery Mechanism TBD 126 39% TA & Training 6 2% Information Dissem. 17 5% Training only 82 25% TA only 95 29% Tot al (7/31/14) = 326
CBA for Health Departments: PWP 35 41% Components PWN 11 13% Policy 5 6% ODM 15 17% HIV Testing 15 17% Condom Distribution 5 6% Total (7/31/14) = 86
CBA for CBOs: Components PWP 58 29% PWN 28 13% HIV Testing 31 16% Condom Distribution 2 1% ODM 82 41% Tot al (7/31/14) = 201
CBA for Healthcare Organizations: Components PWP 14 56% PWN 4 16% HIV Testing 7 28% Total (7/31/14)= 25
CRIS Request s: By State/Territory St ate # of Requests NY 53 CA 37 TX 32 St ate # of Requests ID 6 GA 5 LA 5 St ate # of Requests WI 2 CO 2 NC 2 7 8 % DC 19 PR* 19 OH 17 SC 16 FL 15 IL 15 VA 11 MO 11 MS 8 PA 5 CO 2 AZ 4 HI 4 TN 4 MA 4 ID 3 MO 3 CT 3 AL 3 WA 3 NJ 2 VI 2 AK 2 OR 2 GU* 1 AR 1 DE 1 MD 1 RI 1 Tot al (7/31/14) = 326
Training Events Calendar (TEC)
Notes TEC data from 4/1/14 7/31/14 are included (Program Year 1). Includes trainings conducted by 14-1403 grantees and the CDC staff. Limitations: Not all training sessions have been posted to TEC. This may not represent all training conducted by CDC partners.
TEC Trainings by Components HIV Testing 14 21% PWN 8 12% HIV Planning Group-Policy 2 3% ODM 16 23% PWP 28 41% Total (7/31/14) = 68
Sources of Data for Monitoring CBA Service Delivery CRIS and TEC Progress reports CDC Program Consultant site visit reports Communication between grantees and their CDC Program Consultant TA and training evaluation tools (e.g., consumer satisfaction) HIP Training Coordination Center (DANYA)
How We Use CRIS and TEC Data Summarize and describe CBA service delivery Report on CBA service delivery to specific health department jurisdictions Report on CBA service delivery to internal and external partners Monitor CBA grantees To conduct consumer satisfaction monitoring To improve CBA service delivery and CRIS/TEC applications
CBA Evaluation Project Purpose: To assess customer satisfaction with and outcome of CBA services that support high impact HIV prevention (HIP). First comprehensive monitoring, evaluation, and quality assurance strategy to determine consumer satisfaction and outcome of CBA services. Began with PS# 09-906 cohort and continues with the PS# 14-1403 CBA provider grantees. The findings will be utilized to improve CBA services, increasing the capacity of CBOs directly and indirectly funded by CDC, health departments and HPGs, and HCOs to help achieve the HIV prevention goals set forth in the National HIV/AIDS Strategy (NHAS) and DHAP Strategic Plan.
Objectives of CBA Evaluation To describe CBA services (information, training and TA) provided by CBA providers What was provided, by whom, to whom, and where? To provide program monitoring and quality assurance of the CBA program To determine usefulness and satisfaction with CBA services To determine whether CBA services resulted in increased capacity and improved HIV prevention services To use data to continuously improve the CBA program and CDC/CBB s activities
Short -term Outcomes Increased accessibility and availability of CBA services by target population Increased utilization of CBA services by target population Improved capacity of CBA consumers to implement HIP and supporting activities E.g., changes in knowledge, skills, self-efficacy, and intended or actual use of capacity
Overarching Questions 1. To what extent are CBA recipients satisfied with the CBA services they receive? 2. How do CBA services impact CBA recipients (and their organizations ) capacity to implement HIP? 3. How do CBA recipients think that CBA services can be improved? 4. What factors correlate with CBA recipient satisfaction and capacity building outcomes? 5. What are PPB Project Officers and CBB Program Consultants perceptions of the CBA program (i.e., satisfaction, benefit, quality)? 6. To what extent do recipient satisfaction and capacity building outcomes improve over time for individual CBA providers and for the overall CBA program?
Data Collection Tools Health Professional Application for Training (HPAT) Post Course Evaluation (PCE) standard items Training Follow-up Survey TA Follow-up Survey CRIS and TEC databases Progress Reports SciMetrika Administered: CBA Recipient Key Informant Interviews Facilitated discussion with CDC staff (POs and PCs)
CBA Evaluation Data Collection Tools DRAFT * Data collection date to be determined. Do not collect data until CDC notification. HIP Training (Danya) Partner Services Trg (14-1408s) Danya Uploads Data to CRIS PTCs Upload Data to CRIS 1.HPAT 2.Training Roster 3.Post Course Evaluation 1.HPAT 2.Training Roster 3.Post Course Eval (TBD) 90-day Training Follow-up Sur vey 14-1403s Data Collection TA 14-1403s Upload Data to CRIS 1.HPAT 2.TA Roster 45-day TA Follow-up Sur vey Non-HIP Training (CBA specific training) 14-1403s Upload Data to CRIS 1. HPAT
CBA Evaluation Data Collection CBA providers will upload CBA evaluation data into CRIS for all TA and training (non-hip Dayna coordinated training). Data should be uploaded within 10 business days of the end of the TA or training. Data collection tools for CBA providers: Health Professional Application for Training (HPAT) Roster TA and training participant contact information Post Course Evaluation HIP training only Data collection start date TBD (est. mid-september) Phased data collection starting with Category A and B CRIS data collection training webinar scheduled for mid- September.
M&E Feedback Loop Post Course Evaluation summaries for HIP courses (Danya) Quality Assurance Reports for Training and TA (SciMetrika) Part I - Summary of overall findings and Part II - Grantee specific findings CRIS request data (grantee specific) TEC training data (grantee specific) Feedback from CBA providers
Miriam E. Phields, PhD Systems & Evaluation Activity, Lead Email: byn8@cdc.gov Telephone: (404) 639-4957 For more information please contact Centers for Disease Control and Prevention 1600 Clifton Road NE, Atlanta, GA 30333 Telephone: 1-800-CDC-INFO (232-4636)/TTY: 1-888-232-6348 Visit: www.cdc.gov Contact CDC at: 1-800-CDC-INFO or www.cdc.gov/info The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention. National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention Division of HIV/AIDS Prevention