USING SECONDARY DATA TO HELP ASSESS AND EVALUATE COMMUNITY, ECONOMIC, AND HEALTH DEVELOPMENT PROGRAMS

Transcription

1 USING SECONDARY DATA TO HELP ASSESS AND EVALUATE COMMUNITY, ECONOMIC, AND HEALTH DEVELOPMENT PROGRAMS John J. Green The University of Mississippi Center for Population Studies State Data Center of Mississippi -- Annual Affiliates Meeting -- November 2012

2 OVERVIEW OF THE ISSUES Community, economic, and health development programs are often planned and implemented without systematic prior assessment Additionally, monitoring and evaluation of these programs is often lacking Insufficient data and limited resources pose problems This results in a knowledge gap concerning: what problems are in need of attention, the factors contributing to them, what strategies work/do not work in addressing the problems, and the improvements that need to be made Secondary data from public sources may help to fill some of these gaps Mixed-methods designs that combine secondary data with primary data offer a practical strategy for conducting efficient and useful assessments and evaluation In all cases, creative, careful, and logical design is of critical importance

3 ASSESSMENT AND EVALUATION DESIGN Case Design: Focuses on one case (group, organization, community, event) within its social context; can also do multiple case comparisons Cross-Sectional Design: Data are collected for all of the variables of interest using one sample at one time Longitudinal Design: Data are collected at two or more times, including trend, cohort, and panel studies Experimental Design: The independent variables are controlled in some way and subjects/participants are randomly assigned to experimental and control groups Quasi-Experimental Design: Models experimental design, but is missing one or more aspects of a true experimental study, most frequently random assignment into experimental and control groups Adapted from: Adler, Emily S. and Roger Clark How it s Done: An Invitation to Social Research. Belmont, CA: Thomson/Wadsworth.

4 THE DESIGN CHALLENGE In terms of arguing causality (that program X leads to outcome Y), experimental design is often considered the gold standard However, it is typically not possible to meet the requirements of traditional true experimental design for community, economic, and health development programs Furthermore, there are many nuances and complexities to social life that may be overlooked with traditional experimental design So, the challenge is to design assessment and evaluation studies that: Provide a logical basis to address influence/causality Can be conducted within real world constraints Take into account the complexities of social life A combination of logical reasoning based on quasi-experimental design and use of mixed-methods might help

5 PRIMARY AND SECONDARY DATA Primary data are collected first-hand by the researcher for the specific purpose of a particular project Observation Interviews Focus groups Surveys Secondary data are collected for a different and/or broader purpose (typically from administrative records, vital records, and surveys), but are available for other uses We sometimes use available data as a synonym for secondary data This presentation focuses on secondary data use, but in the context of program assessments and evaluations with mixed-method designs

6 COMMONLY USED SOURCES OF SECONDARY DATA Decennial Census, Economic Census, Census of Agriculture American Community Survey Vital health records Educational data Organizational/programmatic administrative records Most secondary sources involve periodic data collection, allowing for longitudinal analysis Both a benefit and challenge to secondary data sources like these are their ties to specific geographic spaces (states, counties, etc.)

7 EXAMPLE: EDUCATION, POVERTY, AND WORKFORCE DEVELOPMENT PROGRAMS AmeriCorps*VISTA The AmeriCorps*Volunteers In Service to America (VISTA) program is a national initiative to fight poverty with passion Funded by the Corporation for National and Community Service Volunteers partner with nonprofit agencies and educational organizations to help build capacity to better serve communities and, overtime, address the factors influencing poverty Delta State University and the University of Mississippi both operate VISTA programs, and it is imperative that leaders, community partners, and volunteer members understand the poverty levels and trends in the places where they are working Additionally, monitoring and evaluation efforts are necessary to track their understanding of poverty and how the work they are doing builds the capacity of local organizations to provide needed services

8 EDUCATION, POVERTY, AND WORKFORCE DEVELOPMENT PROGRAMS (CONT D) Tri-County Workforce Alliance A nonprofit focused on expansion and enhancement of educational and workforce development opportunities in Bolivar, Coahoma, and Quitman Counties of Mississippi, the Tri-County Workforce Alliance (TCWA) is also serving residents of Sunflower and Tallahatchie Counties Among its programs, TCWA is partnering with the Dreyfus Health Foundation and Mississippi Office of Nursing Workforce on the New Pathways program (with funding from the WK Kellogg Foundation) to offer educational opportunities to youth for health care careers Youth preparation and mentorship Health careers academy Professional and leadership development workshops Student nurse externships and dedicated educational units Problem Solving for Better Health (PSBH) workshops Education and advocacy in the core component of children s health and development in the context of disparities This work requires assessment to understand education, employment, and poverty trends; and there is need for monitoring and evaluation

9 EDUCATION, POVERTY, AND WORKFORCE DEVELOPMENT PROGRAMS (CONT D) Primary Data Sources: In-depth interviews (VISTA) Participatory group processes (VISTA and TCWA) Fish bowl World café Surveys (VISTA and TCWA) Secondary Data Sources: Decennial Census American Community Survey Mississippi Department of Education

10 Educational Attainment (% of Adults 25+ with a Bachelor s Degree or Higher) in Mississippi and the Delta Region ( ) % Bachelor s Degree or Higher Family Poverty in Mississippi and the Delta Region ( ) % at or Below Poverty Level Source: U.S. Census Bureau, American Community Survey. Figures constructed by The University of Mississippi Center for Population Studies. Margins of error are based on 90% confidence intervals.

11 Percent of Adults 25+ with Incomes Below the Poverty Line in Mississippi and Select Delta Region Counties ( ) Mississippi Bolivar Coahoma Quitman Sunflower Tallahatchie Total income below poverty Total income below poverty by sex 16.1 (+/- 0.2) 26.4 (+/- 2.0) 26.9 (+/- 2.8) 29.2 (+/- 4.2) 27.5 (+/-2.8) 26.6 (+/- 4.1) Male Female Male Female Male Female Male Female Male Female Male Female Percent of Adults 25+ with Incomes Below the Poverty Line by Sex and Educational Attainment in Mississippi and Select Delta Region Counties ( ) Mississippi Bolivar Coahoma Quitman Sunflower Tallahatchie Male Female Male Female Male Female Male Female Male Female Male Female < High school degree High school or GED Some college or associate s Bachelor s or above Source: U.S. Census Bureau, American Community Survey. Tables constructed by The University of Mississippi Center for Population Studies. Margins of error are based on 90% confidence intervals. Margins of error not shown here are available from the U.S. Census Bureau s American Factfinder website.

12 Average Composite ACT Scores in Mississippi, Selected Delta Region School Districts, and Program Participants (2011) 36 Average Composite ACT Scores Sources: Mississippi Department of Education, Statewide Assessment Information (2011) and Tri-County Workforce Alliance program documentation (2011). *Students involved in high school nurse mentorship program; participants are from Bolivar, Coahoma, Quitman, and Tallahatchie counties.

13 EXAMPLE: MATERNAL AND CHILD HEALTH PROGRAM The Right! from the Start program is focused on conducting research, building awareness, and providing education and training on the topic of poor birth outcomes, especially preterm births Coordinated by Women and Children s Health Initiatives and the Community Foundation of Northwest Mississippi, with support from the WK Kellogg Foundation Data sources: Primary Participatory community forums (world café sessions) Upcoming will include survey of workshop participants Secondary Decennial Census American Community Survey Mississippi Department of Health Vital Records and Statistics

14 Preterm Births by Race and Age, Pooled Data Preterm Births (<37 Weeks) Per 100 Live Births Preterm (<37 Weeks) Birth Rate Per 100 Live Births Preterm Birth Rates in Mississippi, 1980 to years years years years years White Black Total years years 40 years + Source: Mississippi Department of Health, Vital Records. Figures constructed by the University of Mississippi Center for Population Studies.

15 CHALLENGES There are challenges confronted in using secondary data as part of evaluation design Data limitations abound, as the secondary sources were not developed with the purposes of evaluating specific community-based initiatives Programs often serve a limited number of people (especially at their early stages) and population level changes are unlikely to be observed, even for the most successful efforts Programs often take place in towns, counties, and regions with persistent education and poverty problems, so even if a program is successful, it may be doing so in a worsening context It takes a long-term view to see marked population level changes around education, workforce development, poverty, and health problems, yet funders want to see shorter-term outcomes

16 MOVING FORWARD Program assessment and evaluation are of critical importance to effective decision making, especially in the face of limited resources Educational institutions, government agencies, and private foundations can work with nonprofit organizations to better assess the needs for programs, monitor progress, and evaluate processes and outcomes Methodological innovations are needed in regard to design, data collection, and analysis Expand use of quasi-experimental design Longitudinal focus (investigate patterns over time) Use of comparison groups (compare people, groups, and/or places with a program to those that were similar, with the exception of having the program) Mixed methods, including quantitative and qualitative approaches Statistical data are helpful for identifying and describing patterns Observation, interview, and focus group data are useful for interpreting patterns and informing propositions to explain patterns

17 SPECIAL THANKS Faculty, staff, and students affiliated with The University of Mississippi Center for Population Studies and the Department of Sociology and Anthropology Aaron E. Henry Community Health Services Center Community Foundation of Northwest Mississippi Dreyfus Health Foundation Mississippi Office of Nursing Workforce Tri-County Workforce Alliance Women and Children s Health Initiatives WK Kellogg Foundation