Program Evaluation A 30,000 View Kenneth M. Portier, PhD. Director of Statistics American Cancer Society NHO Statistics & Evaluation Center (SEC) Who Invited the Statistician? 2
Program Evaluation is. One way to establish value. One way of measuring progress. Important to program success. Program Evaluation is more than. Monitoring progress Surveying recipients Creating stories Program Evaluation. Is political. Can promote change OR cause resistance and backlash. Doing a good job can extract a price. 3 What IS the Program? The Logic Model of a Program A picture (a flow chart) of how the program works including assumptions. A conscious process that creates an explicit understanding of program goals, challenges, resources, outputs, outcomes (impacts) and timelines. Helps keep a balanced focus on the big picture as well as on the component parts. Trying to perform an evaluation without a good understanding of the organization and its program will lead to confusion, conflict and failure. 4
Program Definition Logic Model Inputs Outputs Outcomes - Impact Activities Participation Short Term Medium Term Long Term Priorities What we do first. What we invest. What we do. Who we reach. What the short term results are. What the medium term effects are. What the ultimate impacts are. Assumptions What we believe. External Factors S0 S1 S2 S3 S4 S5 S6 S7 M1 M2 M3 M4 L1 L2 What affects us that is outside our direct control. 5 A systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan to do, and the changes or results you hope to achieve. July, 2007, Logic Model Presentation, York & Sim, TCC Group Impact in 1 year Impact in 3-5 years Impact in 7-10 years Version 3.0, 9/22/2008 ACS Research and Professional Training Programs Program Action Logic Model Priority To enable ACS to accomplish its mission of eliminating cancer as a major health problem by funding innovative, high impact research and training and communicating the results to our constituents, and in doing so, provide one cornerstone of scientific credibility to the organization. Assumptions Peer Review Process is core to the EG Program ACS is a prestigious funding partner for launching innovative and high impact cancer research ACS is a funding agency of choice for beginning investigators (any stage) Research program is integral to ACS mission More money invested implies more impact and progress Inputs Outputs Outcomes - Impact Activities Participation Short Term Medium Term Long Term What we What we Who Short term Medium Ultimate invest. do. participates. results term effects impacts Donor funds Solicit Nurses All Program Research Reducing the Sponsor reviewers Interns Areas Advances Burden of funds Solicit Residents Molecular Integrated Cancer Collaborator proposals Researchers Biology of Research Increasing funds Review Reviewers Cancer Findings Health Equity Staff time proposals Cancer Cancer Cell Clinician Staff expertise Fund research survivors Biology and Training in Staff effort Fund training Volunteers Metastasis Cancer Control Volunteer time Mentor Donors Translational/ Communication Volunteer grantees Volunteer Peer Preclinical Resources effort Market the Review Cancer Computing program Experts Research resources Plan Volunteer Peer Inflammation Program Collect Review and Immunity Directors Program Stakeholders in Cancer (Intellectual Results Division Staff Clinical Cancer Capital) Disseminate Social Workers Research and Provide Program Research and Epidemiology infrastructure Achievements training Carcinogenesis to support Develop institutions, Nutrition and grantees funding Collaborative the ACS mechanisms organizations Environment Reputation Strategic Professional Cancer Control (credibility) Planning Societies and Prevention Supporting Government Research evidence Other nonprofit Professional Health based programs in organizations Training in divisions Cancer Control Presentations at scientific meetings 6 External Factors Economy Competing Funding Agencies Healthcare Industry Biotech/ Pharma/ Insurance Changes in academic/research environment Federal Funding ACS Leadership Perception ACS Divisions Government Policies Changes to cancer practice Constituents: *Donors *Cancer Survivors *Academic Community *Cancer Clinicians Community *Peer Reviewers *Applicant Pool *Healthcare Professionals *Volunteer stakeholders *Divisions *General Public
ACS Extramural Grants - Short Term Program Outcomes Outcome Statements inform Metric Choice S0. All Program Areas S0.1 Increased impact of research outcomes through a balanced portfolio of leading edge projects in multiple disciplines. S0.2 More ACS grantees who demonstrate a continuing career commitment to the field of cancer. S0.3 Stronger relationships between the ACS Program Directors in EG and each program s reviewers and grantees. S0.4 Increased scientific/professional exchange among ACS grantees through ACS EG-supported activities. S0.5 Increased frequency and effectiveness of communications that would result in enhancing awareness and understanding by constituents of the role and value of ACS research and training programs S0.6 More donations to ACS dedicated to supporting the research program. S0.7 Increased opportunities for collaboration among, and between, ACS grantees and other funding agencies. S0.8 Provide evidence to demonstrate that EG Programs outcomes support ACS priorities and activities. S3. Translational/Preclinical Cancer Research S3.1 Increased numbers and streamlined processes for preclinical evaluations of cancer therapies and preventatives, and combination treatments. S3.2 Increased numbers of clinically validated vertebrate tumor models for spontaneously occurring pre-malignant and malignant lesions. S3.3 Enhanced knowledge of inherited and somatic gene mutations associated with cancer risk, development, progression and therapeutic response. S3.4 Increased discovery and validation of cancer biomarkers, diagnostic, and prognostic tools for early detection and disease management. 7 What is the focus of this evaluation? What are the key questions? Monitoring Evaluating Is the program achieving short-term outcomes? Is the program on track to impact long-term outcomes? Is the program having an impact now? What is keeping us from achieving program goals? 8
Who is interested in the evaluation? Stakeholders 1. Participants Researchers, Cancer Patients: How and will this program help me? 2. Participating Stakeholders Funders and senior management: Is our investment producing results? (Mission, Income) 3. Supporting Stakeholders Program designers and managers: Do my ideas have value? Are my ideas being implemented as I envisioned them? 9 Who is involved in the program? Program staff and administrators. Program collaborators and outside supporters. Participants/recipients of program efforts. Who do we need to talk to? 10
Collecting Information/Data? What do we need to know? What is the best way to get the information? surveys (web, e-mail, phone, paper), interviews, focus groups, secondary data analyses (database searches, text mining), social media (Facebook), etc. When is the best time to get the information? Will we be able to get the information? Carrot or Stick approach? Descriptive or Comparative? Natural or Constructed comparison groups? 11 The classical experimental and quasi-experimental designs may NOT be the most informative approaches to evaluation. ACS Extramural Grants Evaluation Conclusions Q1: Is the probability of successfully competing for NIH research funding different for researchers whose ACS RSG proposal was approved for funding compared to researchers whose ACS RSG proposal was not approved for funding? NIH funding during or after Jan 2001 No ACS approval for funding No 370 43.84 Yes 100 40.00 Total 470 Yes 474 56.16 150 60.00 624 Total 844 250 1094 12 Pr[NIH during and after ACS not approved] = Pr[NIH during and after ACS approved ] 474 150 = 844 250 P-value=0.28 Conclusion: No difference Chi-Square test for 2 x 2 table
ACS Extramural Grants Evaluation Conclusions Q2: Is the probability of successfully competing for NIH research funding different for researchers whose ACS RSG proposal was actually funded compared to researchers whose ACS RSG proposal was not funded? NIH funding during or after Jan 2001 ACS Funded 1 No Yes Total No 431 44.52 Yes 537 55.48 39 30.95 87 69.05 470 624 Total 968 126 1094 1 Cancelled assumed funded. Pr[NIH during and after ACS not funded] = Pr[NIH during and after ACS funded ] 537 87 = P-value=0.0038 968 126 13 Conclusion: Large difference Chi-Square test for 2 x 2 table Presenting Results? Executive Summary Findings Summary Presentations Full Full Report Results Results & FAQ FAQ 14
Implementing Change? Who? When? How? Evaluations can point the way to improvement. 15 THE OFFICIAL SPONSOR OF BIRTHDAYS 16