Indian Health Service (IHS) Health Promotion/Disease Prevention (HP/DP) Alberta Becenti, Project Officer

Transcription

1 Indian Health Service (IHS) Health Promotion/Disease Prevention (HP/DP) Alberta Becenti, Project Officer EVALUATION REPORT March 11, 2010 Laverne Morrow Carter, Ph.D., MPH Project Director-Lead Evaluator Natasha A. Brown, MPH Research Coordinator Research and Evaluation Solutions, Inc B Old Franconia Rd. Alexandria, VA

2 IHS Health Promotion/Disease Prevention Evaluation TABLE OF CONTENTS TABLES INDEX EXECUTIVE SUMMARY... 4 I. INTRODUCTION... 9 PURPOSE... 9 BACKGROUND... 9 METHODOLOGY...11 II. FINDINGS III. RECOMMENDATIONS Appendixes A. REESSI Data Tracking Form B. Record-Coding Sheets C. Interview Guide and Electronic Field Notes Form D. Site Visit Letter E. Site Visit Agenda F. Grantee Interviews Request Letter and Checklist G. Grantee Interview Data Quality Review Form H. Grantee Applications Summary I. IHS Grantee Reporting Form J. Grantee Reports Evaluation Summary Page 2

3 INDEX OF TABLES Table 1 Grantee Services Provided and Highlighted Outcomes 16 Table 2 Site Visit Interview Results 21 Table 3 Comparison of Mean Program Operations Responses 23 Table 4 Comparison of Mean Service Delivery Responses 24 Table 5 Comparison of Mean Program Outcomes Responses 25 Table 6 Summary of Goals and Outcomes for Grantees that Met and Exceeded Reporting and Outcomes Guidelines 27 Page 3

4 EXECUTIVE SUMMARY PURPOSE The Office of Management and Budget (OMB) made a request in 2008 for an external independent evaluation of the impact and effectiveness of the Indian Health Service (IHS) Health Promotion/Disease Prevention grantees. BACKGROUND Within the last several years, IHS has sought to move local behavioral health activities from a crisis mode to a structured and ecologically based prevention model. The agency now focuses on four strategic areas 1) mobilizing Tribes and Tribal programs to use evidence-based approaches that embrace tradition and culture, 2) promoting community level collaboration with state and federal agencies, 3) promoting leadership development within the communities, and 4) providing advocacy for behavioral health programs across multiple levels of community and government. The focus on health promotion and disease prevention was manifest in 2005 with the solicitation and award of 20 competitive grants to Tribal and Urban programs to implement innovative programs in various settings (i.e., community, worksite, school). The maximum FY2005 grant award was $64, In 2006, 13 additional competitive grants were awarded with a maximum grant amount of $100, The FY2006 projects were funded for a three-year budget period, while the FY2005 projects were for one year. In June 2008, Research and Evaluation Solutions, Inc. (REESSI) was awarded a contract to conduct an independent implementation evaluation of 33 grantees and an impact synthesis of the same grantees. The investigation focused on three questions for the implementation assessment: 1. Is the program reaching its population of focus? 2. Is the delivery of services consistent with the stated program design, goals, and objectives? 3. Do positive changes appear among the program participants? Additionally, this study was completed to fulfill the Office of Management and Budget (OMB) request for an external independent evaluator to conduct an impact and effectiveness evaluation of the IHS Health Promotion/Disease Prevention (HP/DP) grantees. The purpose of the contract was to provide an objective evaluation of the IHS HP/DP grant program through professional oversight using quantitative evaluation methods. The evaluation team was asked to make recommendations for improvement, identify best and promising Page 4

5 practices for expansion, and propose strategies to enhance the overall program. The content of the evaluation activities focused on criteria set forth by the contract and included assessments of: 1. The achievement of positive and measureable health outcomes 2. Health outcomes that are linked to IHS Clinical, GPRA, or PART performance measures 3. The extent to which the grantees met the HP/DP purpose and goals and reported their outcomes 4. Internal/External Collaborations 5. New Resources Secured 6. Barriers Identified 7. Needs related to program and operations 8. Evaluation needs METHODOLOGY To obtain the data and information needed to assess the impact and effectiveness of the HP/DP grantees, the evaluation team employed three strategies 1) a content analyses of the original grantee applications and associated progress reports, 2) site visits to four grantees where actual program documents were reviewed and structured interviews conducted and 3) collection of program data and phone interviews with a large sample of grantees. A triangulation of methods was used to determine points of convergence in the data reviewed and collected. The content analyses of the original applications and one progress report for the 33 FY2005 and FY2006 grantees were completed concurrently with four grantee site visits at the onset of the evaluation in July The interviews were conducted after a 13-month delay in evaluation activities due to the required OMB approval process. Two lead evaluators thoroughly reviewed all the coded files and assessed the data from the content analyses as well as the associated source documents, while listing recurrent themes and topics. After completing this process, the evaluators made comparisons, noting points of convergence across all data sources. The staff entered the quantitative data from the interviews into a statistical software package and descriptive statistics were used to review and analyze the outcomes. FINDINGS The overall findings offer responses to the primary evaluation questions proposed at the onset of the study. Is the program reaching its population of focus? Based on the data from all sources, the programs reached the populations of focus. The populations included children, youth, families, employees, and students. A large number of grantees focused on children, youth and families. Page 5

6 Several stated they faced challenges with getting parents involved and consequently shifted the focus to youth and children. Is the delivery of services consistent with the stated program design, goals, and objectives? This area presented a primary challenge for nearly all the grantees. A fair number of grantees shared that the applications were prepared without the input of the persons who actually implemented the program. Moreover, the program staff is not proficient and knowledgeable of how to structure program goals, objectives, and outcomes. The evaluation team reached a consensus based on the content analyses and the interviews that five grantees have an excellent command of program planning and evaluation practices as evidenced by their applications, progress reports/results, information submitted for the interviews and their actual interviews: 1) Choctaw Nation of Oklahoma Health Department, 2) Kodiak Area Native Association, 3) Menominee Indian Tribe of Wisconsin, 4) Southeast Alaska Regional Health Consortium and 5) South Central Foundation. These grantees could coach, train, and advise the other grantees on the process. Two grantees have a good command of program planning and evaluation practices: 1) Huron Potawatoni, Inc. and 2) Salish Kootenai College. Across all evaluation measures, they provided the required information and documentation. Do positive changes appear among the program participants? Was there an achievement of positive and measureable health outcomes? Seven grantees presented information and evidence that show positive changes and outcomes. Were health outcomes linked to IHS Clinical, GPRA, or PART performance measures? None of the site visit grantees were familiar with the GPRA and PART measures. Only three of the other grantees discussed these guidelines in detail during the interviews and it appears that many of the grantees are not fully informed and aware of these guidelines and measures. The seven model programs appear to be knowledgeable of the IHS clinical goals. To what extent did grantees meet the HP/DP purpose and goals and report their outcomes? All grantees linked their program activities to the HP/DP purpose and goals in their original applications. The reporting of outcomes connected to those goals appears problematic for all but seven of the grantees. During the site visits and interviews, the evaluators experienced the passion and commitment that most grantees have for their programs. Most of the activities were creative and comprehensive. However, it seems that a fairly significant number of program personnel did not know how to measure and capture their successes, quantitatively. Page 6

7 Internal/External Collaborations Most of the grantees built viable external collaborations. This area was a major strength of the HP/DP program across all grantees. The grantees constructed linkages with media outlets, school districts, mainstream youth organizations, and health clinics. One program was able to attract a national media outlet (NBC) and acquire donations from all over the country for their children and youth camp. New Resources Secured Most grantees reported that they secured additional resources, but only four grantees submitted quantifiable information. Several grantees received inkind office space and resources; volunteer efforts; and supplemental grants from state and local agencies. For those four organizations that submitted quantifiable information, the total dollar value of new resources was $292, over the FY2005-FY2006 project periods. It appears that most of the grantees did not understand how to track and quantify monetary values for the new resources they secured. The evaluators found that Southeast Alaska Regional Health Consortium and Menominee Indian Tribe of Wisconsin presented the best data on the procurement of additional resources both in-kind and financial. Barriers Identified The number one barrier identified by the grantees was staff retention and turnover. Many grantees struggled with maintaining the same staff over multiple program years. The second most discussed barrier is the disconnect between the tribal councils and program operations a number of grantees experienced delays in hiring and program start-up while awaiting approval from tribal councils. Needs related program and operations Grantees expressed a number of needs related to program administration and operations. They specifically cited the desire for additional funding for staff to maintain records and analyze data and the need for increased support and technical assistance from the IHS staff. Most grantees shared that they would like more contact and direction on their program operations. Moreover, they expressed a need for ongoing contact with other grantees to engage in information exchange and networking. Evaluation needs The majority of the grantees shared that they would benefit from technical assistance and support with their local evaluation efforts. Most grantees conceded that they do not have program planning expertise the personnel are better at implementation and interfacing with program participants. Major Page 7

8 challenges exist with developing instruments to capture their outcomes along with staff support and expertise for data analyses. RECOMMENDATIONS Observed Best Practices for Health Promotion and Disease Prevention The evaluators were impressed with the structure, creativity, and outcomes of five grantee programs 1) Choctaw Nation of Oklahoma Health Department s focus on healthy lifestyles with 5 th graders in schools; 2) Kodiak Area Native Association s community based cancer screening; 3) Menominee Indian Tribe of Wisconsin s comprehensive approach to worksite interventions; 4) Southeast Alaska Regional Health Consortium s intergenerational health promotion activities, and 5) South Central Foundation s policy and social marketing intervention. Overarching Key Recommendations 1. The IHS Health Promotion/Disease Prevention pre-application and application process should be improved to assure that applicants are familiar with the key elements of program planning and how to capture and measure program success. 2. The IHS Health Promotion/Disease Prevention post grant award support and technical assistance process should be significantly enhanced to offer structured program implementation support to grantees. 3. The IHS method for tracking grantee reports and outcomes must be enhanced and managed in a more systematic manner to assure that grantees are meeting their stated goals, objectives, and outcomes. Page 8

9 INTRODUCTION PURPOSE The Office of Management and Budget (OMB) made a request in 2008 for an external independent evaluation of the impact and effectiveness of the Indian Health Service (IHS) Health Promotion/Disease Prevention grantees. BACKGROUND With a budget of approximately $4.1 billion dollars, the Indian Health Service (IHS) within the U.S. Department of Health and Human Services provides comprehensive health services to nearly 1.9 million American Indians and Alaska Natives. Members of these populations have a lower life expectancy than all Americans and the infants die at higher rates than the general U.S. populations. American Indians and Alaska Natives also carry a higher burden of mortality than other Americans for a broad cross section of diseases including diabetes (190% higher), alcoholism (550% higher), tuberculosis (750% higher), and homicide (100% higher). 1 Within the last several years, IHS has sought to move local behavioral health activities from a crisis mode to a structured and ecologically based prevention model. The agency now focuses on four strategic areas 1) mobilizing Tribes and Tribal programs to use evidence-based approaches that embrace tradition and culture, 2) promoting community level collaboration with state and federal agencies, 3) promoting leadership development within the communities, and 4) providing advocacy for behavioral health programs across multiple levels of community and government. An IHS Prevention Task Force was established and the activities of the group are guided by Healthy People 2010 and consistent with the IHS Strategic Plan and the Government Performance and Results Act (GPRA) reporting system. 2 The focus on health promotion and disease prevention was manifest in 2005 with the solicitation and award of 20 competitive grants to Tribal and Urban programs to implement innovative programs in various settings (i.e., community, worksite, school). The maximum FY2005 award was $64, In 2006, 13 additional competitive grants were awarded for a maximum amount of $100, The 2006 projects were funded for a threeyear budget period, while the 2005 projects were for one year. 1 IHS Fact Sheet. (2008). Indian Health Disparities. Retrieved on May 15, 2008 at 2 IHS Health Promotion and Disease Prevention. Retrieved on May 15, 2008 at 3 HPDP Newsletter February 2008, Page 4. Page 9

10 Under the Office of Management and Budget, the Program Assessment Rating Tool (PART) was developed to assess the effectiveness of federal programs and help inform management actions, budget requests, and legislative proposals directed at achieving results. The PART examines various factors that contribute to the effectiveness of a program and requires that conclusions be explained and substantiated with evidence. The PART assesses if and how program evaluation is used to inform program planning and to corroborate program results. Federal agencies and programs are required to conduct nonbiased evaluations on a regular or as-needed basis to fill gaps in performance information. These evaluations should be of sufficient scope and quality to improve planning with respect to the effectiveness of the program. 4 The evaluations must be independent and determine if the program is effective and achieving results. In June 2008, Research and Evaluation Solutions, Inc. (REESSI) was awarded a contract to conduct an independent implementation evaluation of 33 grantees and an impact synthesis of the same grantees. The project was delayed nearly 13 months to secure OMB clearance for the interview process with grantees. The evaluation activities ceased in August 2008 and resumed in October The investigation focused on three questions for the implementation assessment: 1. Is the program reaching its population of focus? 2. Is the delivery of services consistent with the stated program design, goals, and objectives? 3. Do positive changes appear among the program participants? Additionally, this study was completed to fulfill the Office of Management and Budget (OMB) request for an external independent evaluator to conduct an impact and effectiveness evaluation of the IHS HP/DP grantees. The purpose of the contract was to provide an objective evaluation of the IHS HP/DP grant program through professional oversight using quantitative evaluation methods. The evaluation team was asked to make recommendations for improvement, identify best and promising practices for expansion, and propose strategies to enhance the overall program. The content of the evaluation activities focused on criteria set forth by the contract and included assessments of: 1. The achievement of positive and measureable health outcomes 2. Health outcomes that are linked to IHS Clinical, GPRA, or PART performance measures 3. The extent to which the grantees meet the HP/DP purpose and goals and reported their outcomes 4 OMB. (2004). What Constitutes Strong Evidence of a Program s Effectiveness? Retrieved on June 8, 2008 from Page 10

11 4. Internal/External Collaborations 5. New Resources Secured 6. Barriers Identified 7. Needs related to program and operations 8. Evaluation needs METHODOLOGY To obtain the data and information needed to assess the impact and effectiveness of the HP/DP grantees, the evaluation team employed three strategies 1) content analyses of the original grantee applications and associated progress reports, 2) site visits to four grantees where actual program documents were reviewed and structured interviews conducted and 3) collection of program data and phone interviews with a large sample of grantees. Content Analysis of Grants and Follow-up Reports The REESSI Project Director received electronic grant records from the IHS Project Officer that included the grant application and the grantee reports and contact information. For the 20 FY2005 grantees one grant and one report were received for each grantee. The records for the 13 FY2006 grantees included all grant applications, however the evaluator did not receive a significant number of reports. By July 2008, at least 2-3 reports should have been submitted by each grantee the documents transmitted included one report for nine grantees and four grantees had no reports. A data tracking form is included in Appendix A it shows the grantee for each funding period and what documents REESSI received and reviewed for each grantee. The evaluator conducted no reviews of progress reports submitted during the dormant period of the evaluation (August 2008 October 2009). Four REESSI evaluators were assigned grantees. Each evaluator reviewed the records of six to nine grantees. The reviewers used an electronic record-coding sheet that included standard items for both the application and the reports to capture the same data from all the records. Copies of the documents are included in Appendix B. Each grant application and associated reports were read through in their entirety by the evaluators. During the second review, the evaluator coded the documents using electronic coding sheets developed in a Microsoft Excel worksheet. Two evaluators reviewed all grantee documents and coded reports, extracting repeated themes and categories. Site Visits Methodology Two members of the evaluation team engaged in planning and conducting site visits to four sites Indian Health Care Resource Center (Tulsa, OK), Central Oklahoma American Indian Health Council (Oklahoma City, OK), Ramah Navajo School Board (Pine Hill, NM), and Pueblo San Felipe (San Felipe Pueblo, NM). The team finalized the interview guide and the document for recording electronic field notes. Both forms are attached in Appendix C. Page 11

12 Initial contact was made with the sites via phone and a follow-up letter was sent confirming the visit and setting forth the items to be reviewed on-site. A second call was made to each site one day prior to each visit. A copy of the site visit letter is attached in Appendix D. Two of REESSI s senior evaluators conducted the site visits from Tuesday August 5-Friday August 8, Upon arrival to each site, the evaluation team discussed the agenda for the visit with the project staff. A copy of the agenda is attached in Appendix E. The evaluators reviewed program documents provided by the grantee staff. After the review, the evaluation team interviewed the key staff and closed the visit. The results of the interviews were analyzed using descriptive statistics. Methodology for Grantee Interviews The evaluation staff requested materials representative of Program Operations, Service Delivery, and Program Outcomes from each of the FY 2005 and FY2006 grantees. A submission checklist accompanied the request for materials. Appendix F contains a sample request letter and the checklist, which instructed grantees to indicate whether each of the required back-up documents was included and provide additional comments, including explanations for materials not included. Grantees were asked to return the materials and checklist no later than November 6, 2009 and to schedule an evaluation interview with a REESSI staff member between November 9 and December 4, A major inhibitor to this approach was the time lapse for the 2005 grantees (4-5 years) and the 13-month delay in the project. Several grantees were challenged with gathering the requested data, because of staff turnover and changes. The evaluation team extended the deadline, collected grantee documents and conducted interviews through December 18, Upon receipt of each grantee s package, two senior members of the evaluation team reviewed the materials in their entirety. The purpose of the reviews was to determine the quality of the included materials and to verify that required items marked as Included on the checklist were actually sent by the grantee. Each of the required items was scored on a scale of 1 to 5, with 1 indicating Poor quality and 5 indicating Excellent quality. Any additional questions needed to clarify materials were also noted; these questions were asked at the beginning of the evaluation interview conference call. This quality review was conducted prior to each grantee s interview. A sample form used for the quality review is included in Appendix G. The interview responses were entered into a statistical software package and descriptive statistics were performed. Page 12

13 FINDINGS Findings from Content Analyses of Grants and Follow-up Reports The evaluators performed comprehensive reviews of the funded grant applications for FY2005 (N=20) and FY2006 (N=13). Interventions of this nature generally should be constructed with an overarching goal, administrative and implementation aims and objectives related to impact. Indian Health Service funded the FY2005 grantees for one year, consequently major changes in participant behaviors and population level outcomes were not anticipated. These changes generally require at least a five-year structured intervention. The agency funded the FY2006 grantees for three years, and more outcomes related to behavioral changes and even policy improvements should be expected. Program outcomes can be used to provide feedback on the program s overall performance, including effectiveness, efficiency, and ability to reach intended audiences 5. The standards to which these outcomes should be held are varied (i.e., arbitrary, experiential, utility, historical, scientific, normative, proprietary, feasibility, and model) 6. Emerging programs are often held to arbitrary or experiential standards (i.e., those that involve the perceived needs and priorities of the community). With time, programs should be held to stronger, historical standards, which are based on previous performance and data, require routinely accessible data, and the technical capacity to collect the data across time. Furthermore, the long-term sustainability of the program and its outcomes are partially grounded in the standardization of program components, as well as the establishment of organizational routines that can be followed even when there are changes within the organization 7. The review of the FY2005 and FY2006 applications yielded the following observations: 50% (N=10) of the FY2005 applicants presented well-defined interventions in their proposals with goals, related objectives, activities, and measures. About a third of the FY2005 grantees described the interventions, but included no concrete measures of success for the project in their applications. More than half of the FY2006 grantees ( 7) described the intervention but presented weak goals, objectives, and measures of success in their applications. 5 Handler, A., Issel, M., & Turnock, B. (2001). A conceptual framework to measure performance of the public health system. American Journal of Public Health, 91(8), Judd, J., Frankish, J., & Moulton, G. (2001). Setting standards in the evaluation of community-based health promotion programmes A unifying approach. Health Promotion International, 16(4), Pluye, P., Potvin, L., & Denis, J.L. (2004). Making public health programs last: Conceptualizing sustainability. Evaluation and Program Planning, 27(2), Page 13

14 Approximately 50% of the applicants in both rounds of funding seemed to focus more on following the application outline than presenting a cohesive project plan with specific objectives, activities, outcomes and measures that were linked. The structure and presentation of these applications were confusing. For most of the FY2006 grantees, the evaluators found a disconnect between what was proposed in the original application, the report on outcomes for the first year of activities, and what was proposed for the second year of activities (continuation applications). A summary document of each grantee s program goal and the evaluators comments on each application are presented in Appendix H. Findings from Grantee Reports The grantees were provided with a reporting form to submit to IHS at different time intervals. That form is attached in Appendix I. After a careful review of the grantee reports, the evaluators offer the following final observations: The evaluators received one report for each of the FY 2005 grantees. Two reports should have been completed and submitted for each grantee. In August 2008, the IHS project staff provided one report for nine of the FY2006 grantees. Reports were not provided for four of the grantees (Menominee, Norton Sound, Pueblo San Felipe, and Round Valley). We anticipated receipt of more progress reports (two per year). Based on the initial grant requirements, each grantee should have completed four progress reports by August Quality of the FY2005 Reports o Five out of the 20 grantees used the required progress report format and included comprehensive information on their program activities that linked back to what was proposed in the original grant application. The evaluators considered these reports to be of good content and quality. o Five grantees submitted what was requested in the report form and linked their activities back to the grant application. These grantees were considered exemplary in their reporting process. o Three out of the 20 grantees did not use the report format at all. Two reports offered cursory and deficient information that did not link back to what was proposed. One grantee did not use the report 8 Note: The evaluation activities ceased for 13 months. Additional reports were received for the FY2006 grantees, but were not assessed by the evaluators. The results reported are based on assessment of the reports received at one point in time in July Page 14

15 format, but offered a comprehensive project report that exceeded the requirements of the report form. o Seven of the grantees used the report format, but did not report on all the elements. The information offered was cursory and sketchy. They did not link activities back to their original goals and objectives. Quality of the FY2006 Reports o Two of the ten grantees reviewed provided detailed responses in the report format and offered comprehensive information on the program outcomes, linking the activities back to what was proposed. The reports were exemplary. (Note: IHS transmitted nine reports and one report was submitted directly by the grantee). o One of the ten grants provided what was requested and linked the activities back to what was proposed. o Two grantees completed portions of the report form and had partial links back to the activities proposed. o Four grantees offered only a few elements of what was requested on the report form and had weak linkages back to what was originally proposed. Those grantees with the most comprehensive reports showed the most favorable outcomes and quantitative results for both administrative and program operations. Common Barriers presented in the reports o The number one barrier was hiring and retaining staff these challenges were mentioned by nearly all the grantees. o Navigating the various systems and recruiting participants for the program activities were also presented as challenges. All grantees devoted significant efforts and time to program activities. Most grantees do not appear to have an understanding of program planning and how to measure program success. A summary document of the content analyses of the reports is included in Appendix J. The overviews in Table 1 below present the actual services provided, as well as highlighted outcomes described in the grantees reports received by the evaluator. Page 15

16 Table 1 Grantee Services Provided and Highlighted Outcomes 2005 Grantees Actual Services Provided Highlighted Outcomes 1-ABERDEEN AREA TRIBAL CHAIRMEN'S HEALTH BOARD(AATCHB) 2-ALAMO NAVAJO SCHOOL BOARD, INC. 3-CHOCTAW NATION OF OKLAHOMA HEALTH DEPARTMENT 4-EASTERN ALEUTIAN TRIBES 5-HO-CHUNK NATION 6-INDIAN HEALTH CARE RESOURCE CENTER, INC. 7-INTER-TRIBAL COUNCIL OF MICHIGAN, INC. Provided a health communication campaign and health education sessions to increase knowledge of the dangers of environmental tobacco smoke. Created outdoor fitness path at newly constructed community wellness center. This organization has been identified as a model program. Please refer to Table 6, p. 27. Provided training sessions for physical activity leaders (PALs) from 6 surrounding communities. Developed a community health representative (CHR) program to promote physical activity and proper nutrition among tribal members. Supported summer camp and after-school programs. Provided opportunities for women to improve their cardiovascular health through a variety of organized health and fitness activities. Received smoke-free homes & vehicles pledges from 2,383 individuals. Disseminated 4,000 posters for smoke-free home campaign. Trained 20+ staff members of partner organizations on providing tobacco prevention and cessation education. 143 children and adults took part in opening ceremonies. The fitness path became an integral part of summer camp (avg. 25 participants/day) and school-based activities (10-12 students/semester. Hosted walk/run event on Thanksgiving Day with 129 participants. This organization has been identified as a model program. Please refer to Table 6, p. 27. Training was provided for 9 PALs. All of the PALs reported various successful outreach program activities. Community members gave anecdotal feedback that pedometers provided helped increase physical activity. Increased program referrals by educating medical and behavioral staff on program activities. Developed Get Fit Strategic Plan. Distributed Get Fit Resource manual to 500+ tribal members. Completed 2 training sessions each on nutrition, exercise physiology, motivational interviewing, and human disease etiology. Provided homework help, healthy snacks, cultural and experiential activities, and fitness and nutrition education to youth in grades K-5. Developed t-shirts and distributed 1800 brochures for summer camp. Awarded the Valuable Investment Award from Tulsa Public Schools for work with youth at the school. Camp is now referred to as Wellness Camp within the community. 102 women participated in at least one event, with some women returning for up to six activities. 53 women set personal health goals, with 55% of them achieving their goals and 85% making progress on their goals after 2 months. Annual retreat data indicate improvements in physical activity, exposure to secondhand smoke, diabetes management, and health screenings. Page 16

17 Table 1-Continued Grantees Actual Services Provided Highlighted Outcomes 8-KALISPEL TRIBE OF INDIANS 9-KENAITZE INDIAN TRIBE 10-MIGIZI COMMUNICATION, INC. 11-NATIONAL INDIAN JUSTICE CTR. 12-NATIVE IMAGES, INC. Provided diabetes health education at the individual and community level. Provided targeted health education on nutrition, physical activity, and smoking cessation to heads of households. Provided family-level culture-based health and wellness programming. Developed culturally appropriate, community-based public education program incorporating a single session video prevention tool and a companion workbook. Provided for a physical fitness and a nutritional program for seniors. Conducted one healthy cooking class for tribal and community members. Prepared one monthly healthy dinner for tribal and community members. RN conducted screenings for blood sugar, blood pressure, and cholesterol for 52 tribal and community members. Established a permanent educational display on diabetes. Provided nutrition, food label reading, purchase, preparation and presentation class to 165+ individuals. Contracted with a private yoga instructor to provide for fee class to individuals. Provided berry picking and indigenous food processing and canning class to 12 individuals. Provided 4-week parenting class to reduce parent and child conflict and stress to 16 parents. Provided heads of households with stress reduction and lifestyle management classes 3 times/week; average of 10 attendees/class. Provided heads of households with traditional healing sessions 3 times/week; average of 10 attendees/class. Provided individual health behavior consultations for 238+ individuals. Completed gourmet healthy meal preparation and dinner with local chef for 18 individuals. Provided exercise activities led by an exercise physiologist to 47 individuals. Developed and distributed postcards on smoking cessation and prevention and secondhand smoke. Served 309 adults and youth from over 150 families in nutrition and fitness activities, including food buying trips and guidance on developing individual and family fitness plans. Provided 60 children at local school with nutrition and fitness programming, including summer camps on gathering traditional foods and on-site fitness activities before and after school. Duplicated video to tape, DVD, and organization website. Reproduced workbook and evaluation tools and made available on organization website. Trained 50+ Native youth to use video and workshop as public education tool. Distributed video and workbook to 10+ local programs serving at-risk Native youth. Garnered and maintained the support of collaborators, including local media outlets. Provided 2.5-hour wellness sessions 2 times/week for 26 elders; included physical activity, guided meditation, and nutritional education. Ninety-eight percent of elders reported completing 1 hour or more of exercise at home in between sessions. Ninety-five percent of elders tried healthy cooking tips at home with their families. Weight loss of 3% maintained by 75% of elders. Collaborated with local health department and state diabetes program to provide health fair, follow up services and address health care needs. Page 17

18 Table 1-Continued PETA WAKAN TIPI 2005 Grantees Actual Services Provided Highlighted Outcomes 14-RAMAH NAVAJO SCHOOL BOARD, INC. 15-ROCKY BOY HEALTH BOARD (CHIPPEWA-CREE TRIBE) 16-SALISH KOOTENAI COLLEGE 17-SAN DIEGO AMERICAN INDIAN HEALTH CENTER Provided health education focused on diabetes and obesity prevention for youth. Developed summer youth camp project to promote and support healthy lifestyles for youth. Provided physical activity and nutrition education to community youth. This organization has been identified as a model program. Please refer to Table 6, p. 27. Developed a walking club and a smoking cessation program Conducted 4 healthy eating and physical activity sessions; included garden work and preparation of healthy snacks using indigenous foods. 100% of children could name at least 2 indigenous foods by the end of the program. Fortytwo percent of the children learned that a balanced diet can prevent diabetes. At the end of the program, 47% reported that they played sports more often. Parents reported that children had chosen to eat fewer sweets and had increased physical activity. Established and maintained internal and external collaborations to plan and provide activities for youth. Completed 40-hour training with summer camp staff. Provided campers with physical activity self-monitoring log and heart rate monitors. Provided bike safety and basic bike repair education sessions for campers. All campers took part in more than 60 minutes/day of moderate-tovigorous exercise. Provided nutrition education session on healthy meal planning. Served campers meals that met Healthy People 2020 goals, with the exception of sodium limitations. Provided education sessions on the adverse health effects of drugs, tobacco and alcohol. Provided youth with weight training classes, swimming, and walking/running/hiking/jump roping clubs; 94 youth participated. Conducted nutrition education sessions on understanding the food pyramid and reading food labels. Facilitated elder/youth round dance with traditional healthy foods, education on the importance of physical activity, and drug and alcohol prevention; 341 adults and youth attended the event. Organized basketball leagues for 3-4 and 5-6 grade youth divisions with 4 games/day for 4 days/week; had 23 teams in 2 different leagues. Also provided basketball leagues for 7-8 and 9-12 grade youth divisions, soccer leagues with 3-4 and 5-6 grade youth divisions, and flag football leagues with 7-8 and 9-12 grade youth divisions. Wellness Center had an open gym everyday throughout the summer and averaged 40 youth/day using the facility. This organization has been identified as a model program. Please refer to Table 6, p. 27. Provided smoking cessation services to 161 individuals, with 147 of those receiving 1-hour counseling sessions and 146 receiving substance abuse treatment counseling services in conjunction with smoking cessation counseling; one individual was successful in quitting smoking. The walking club provided services to 8 individuals, with 2 graduating from the program. 18-SOUTH CENTRAL FOUNDATION This organization has been identified as a model program. Please refer to Table 6, p. 27. This organization has been identified as a model program. Please refer to Table 6, p. 27. Page 18

19 Table 1-Continued Grantees Actual Services Provided Highlighted Outcomes 19-SOUTHEAST ALASKA REGIONAL HEALTH CONSORTIUM 20-TULALIP TRIBES Table This organization has been identified as a model program. Please refer to Table 6, p. 28. Developed walking school bus program to increase physical activity among school children This organization has been identified as a model program. Please refer to Table 6, p. 28. Actively engaged several community members and partners in developing anticipated program Grantees Actual Services Provided Highlighted Outcomes 1-CENTRAL OKLAHOMA AMERICAN INDIAN HEALTH COUNCIL, INC. 2-CHIPPEWA CREE TRIBE 3-CONF. TRIBES OF WARM SPRINGS 4-HURON POTAWATOMI, INC. 5-INDIAN HEALTH CARE RESOURCE CENTER OF TULSA, INC. 6-INTER-TRIBAL COUNCIL OF MICHIGAN, INC. Facilitated six 3-day camps to educate youth about diabetes prevention, physical activity, nutrition, and other health related topics Provided physical activity and nutrition education to community youth Provided methamphetamine prevention project to local elementary and middle schools This organization has been identified as a model program. Please refer to Table 6, p. 27. Conducted diabetes and obesity prevention program for at-risk families Worked to improve and implement best practices interventions focused on tobacco, obesity, physical activity, and nutrition Ninety-three children attended the camp over the course of the 3 days. Educational sessions focused on diabetes, hand washing safety, nutrition awareness, healthy food preparation, and physical activity. Children were provided with healthy snacks and given the opportunity to engage in physical activity each day. Children increased their knowledge of diabetes prevention by 38%. In addition to FY2005 activities, hosted family food and fun night at 2 schools, with a total of 625 adults and children participating. On average, 75 youth/day utilized the weight room, table games, and open gym. Hosted large speaking engagement where 2 individuals spoke to 400 students about meth addiction. Developed several second public service announcements at local radio station, with a listening audience of 15,000. Designed and distributed 200 meth addiction pamphlets. Provided meth education sessions at 4-H culture camp, with an average of 10 children/session. Hosted community night out in conjunction with annual meth prevention conference; there were approximately 70 attendees. Hosted regional meth conference with 130+ attendees over 2 days. Created and implemented zero tolerance for meth policy. This organization has been identified as a model program. Please refer to Table 6, p. 27. Provided monthly educational demonstration food preparation classes to an average of 6 children and their caregivers/parents. Provided monthly nutrition education to an average of 20 individuals. Hosted spring break wellness quest with various activities including gardening skill-building, experiential learning games, and general health lessons; there were 65 participants. Provided group exercise class for children 5-16 years old, with an average of 6 attendees/week. Maintained partnership with Camp Fire USA to establish community family club with 15 families. Completed communication with each participating tribe on at least a quarterly basis. Page 19

20 Table 1-Continued Grantees Actual Services Provided Highlighted Outcomes 7-KODIAK AREA NATIVE ASSOCIATION 8-MENOMINEE INDIAN TRIBE OF WISCONSIN 9-NORTON SOUND HEALTH CORPORATION 10-PUEBLO SAN FELIPE 11-ROUND VALLEY INDIAN HEALTH CENTER, INC. 12-SOUTHCENTRAL FOUNDATION 13-SOUTHEAST ALASKA REGIONAL HEALTH CONSORTIUM This organization has been identified as a model program. Please refer to Table 6, p. 27. This organization has been identified as a model program. Please refer to Table 6, p. 27. Provided smoking prevention and cessation program to youth and adults Developed comprehensive health promotion program to reduce chronic disease among community members Provided Fit Teen program to address obesity, anger, and feelings of low self-esteem among youth This organization has been identified as a model program. Please refer to Table 6, p. 27. This organization has been identified as a model program. Please refer to Table 6, p. 28. This organization has been identified as a model program. Please refer to Table 6, p. 27. This organization has been identified as a model program. Please refer to Table 6, p. 27. Provided smoking prevention education campaign to over 500 individuals during a basketball tournament. Counseled 180+ patients on smoking and eye health. Collaborated with local organization and signed MOA to provide collaboration smoking cessation and prevention outreach for 3 years. Distributed 30+ educational tidbits to employees via . N/A Twenty-nine youths participated in health screening. Conduct weekly Fit Teen parent potluck meetings, with an average of 16 participants, who began to bring healthier food over time. Arranged option to earn college credit at local community college for youths and adults who participated in summer garden activity. This organization has been identified as a model program. Please refer to Table 6, p. 27. This organization has been identified as a model program. Please refer to Table 6, p. 28. Site Visits Results The review of the program documents and interviews at each site led the evaluators to the following conclusions: 1) In most cases the person who wrote the grant and developed the proposed program was not involved in implementation which often led to a disconnect between what was proposed and what was implemented. 2) The grantees lack knowledge and experience in linking program goals to measurable objectives, activities, and associated outcome measures. None of the site visit grantees had a strong command of this process. 3) The grantees lack knowledge and experience in program recordkeeping. The grantees were able to verbally discuss what they had done, but the back-up documents (sign-in sheets, participant records, pre-post test results and analyses) were weak. 4) The grantees lack experience and knowledge in establishing measures for program operations and tracking those measures. 5) The grantees lack the capacity to adequately collect program data and conduct analyses. They do not have staff and resources for these functions. 6) The tribal decision-making process often curtails program implementation on reservations. Page 20

21 7) Grantees could benefit from opportunities to network and exchange challenges, ideas, and solutions with each other. The mean scores of the interview responses are shown in column five in Table 2. These are the responses provided by the program staff. The evaluators asked the staff to respond honestly, based on what they could justify with documents and evidence. The scale ranged from 1(strongly disagree) to 7 (strongly agree). The weakest scores are in the program outcomes component, while the highest scores are in service delivery. The scores converge with the verbal responses from the grantees and the results of the content analyses. Table 2 Site Visit Interview Results Descriptive Statistics QUESTIONS N Minimum Maximum MEAN Std. Deviation PROGRAM OPERATIONS No difficulty in meeting administrative objectives Have the necessary resources to carry out program The goals we set forth were realistic We were able to form external linkages We were able to attract supplemental funding Staff is knowledgeable about all aspects of program SERVICE DELIVERY We are delivering the services we intended Clients are/were aware of the services we offered through our funded HPDP. Clients are satisfied with the services they received from our program. Clients would recommend other people for our HPDP program. Clients have had an opportunity to offer feedback on how to improve the program. Our program data show that our outcome goals and objectives are being met/were met. Our services for the funded HPDP program are reaching/reached the intended audience. Clients have an improved quality of life after participating in our program. Clients are more knowledgeable about their health and ways to improve their health since participating Our program evaluation included measureable health outcomes in communities tied to GPRA and PART PROGRAM OUTCOMES Page 21

22 Interview Results The evaluators received evaluation materials from 17 of the FY2005 (N=12) and FY2006 (N=9) grantees. Of those grantees submitting materials, 4 received HPDP funds in both FY2005 and FY2006. Seven grantees (FY2005 N=5; FY2006 N=4) submitted materials that were overall of Excellent of Very Good quality. These materials were presented in an organized manner that clearly demonstrated the presence and/or usage of most, if not all, of the required items in each of the three core areas of evaluation; there were few, if any, missing items. These grantees were also found to have applications and progress reports of good to excellent quality. Eight grantees (FY2005 applications, N=5; FY2006 applications, N=4) submitted materials that had an overall quality of Average. 9 These materials were presented in a less organized manner, and there were questions about how some materials represented the required items. There were a few missing items, but clear explanations were provided as to why the items were not included. Two grantees (FY2005 applications, N=2; FY2006 applications, N=1) submitted materials that had an overall quality of Fair or Poor. These materials were missing several items, some of which were without explanation, and it was unclear how some of the materials fulfilled the requirements. Most grantees were able to provide adequate information concerning Program Operations. The grantees had difficulty in providing justification documents for the Program Outcomes area. A primary weakness was the submission of pre-post samples and evidence of the number of participants who actually completed the program. Moreover, deficiencies existed in materials representing the Service Delivery of the programs. Several of the grantees were unable to provide samples of participant registration or feedback forms. Explanations provided for missing items varied for each grantee, but the most common answers were that the items no longer existed due to circumstances beyond their control, the items could not be located due to staff turnover, and/or the items were not used during the program. The information in Tables 3-5 provides comparisons of the mean responses for the interview items in each of the core areas of evaluation. 9 It should be noted that some grantees were awarded grants for both rounds of funding FY2005 and FY2006. Page 22

23 Table 3 Comparison of Mean Program Operations Responses Program Operations Evaluation Interview Items: 1. We have/had no difficulty in meeting administrative (hiring staff, training staff, planning recruitment, preparing materials, budgets) objectives for our HPDP project 2. We have the necessary resources (staff, money, equipment, facilities) etc. to carry out the program 3. The goals that we set forth in our proposed intervention are/were realistic for our audience of focus 4. We were able to form external linkages for support for our program operations 5. We have been able to attract supplemental funding and resources for the HPDP project 6. Staff is/were knowledgeable about all aspects of the program (timelines, reports, objectives) The FY2005 and FY2006 grantees provided similar responses to the Program Operations evaluation interview items. Attracting supplemental funding and meeting administrative objectives presented the greatest challenges for the grantees. Grantees were most successful with forming external linkages for program operations support. Page 23

24 Table 4 Comparison of Mean Service Delivery Responses Service Delivery Evaluation Interview Items: 1. We are delivering/delivered the services we proposed to the intended audience 2. Clients are/were aware of the services we offer(ed) through our funded HPDP program 3. Clients are/were satisfied with the services they received from the funded HPDP program we offer(ed) 4. Clients would recommend other people for our HPDP program 5. Clients have/had opportunities to offer feedback on how to improve the program (examples surveys, conversations with staff or a suggestion box) The grantees responses for the Service Delivery items were more varied between FY2005 and FY2006, as well as between each interview item, as compared to the other core evaluation areas. Overall, particularly for FY2006, grantees had the most difficulty with providing their clients with opportunities to offer feedback on how to improve their programs. FY2005 grantees were more challenged with ensuring that their clients were aware of the services offered through their programs. Grantees were least challenged with delivering the proposed services to the intended audience. Page 24

25 Table 5 Comparison of Mean Program Outcomes Responses Program Outcomes Evaluation Interview Items: 1. Our program data show that our outcome goals and objectives are being met / were met 2. Our services for the funded HPDP program are reaching/reached the intended audience 3. Clients have/had an improved quality of life after participating in our HPDP program 4. Clients are more knowledgeable about their health and ways to improve their health since receiving services from our HPDP program 5. Our program evaluation included measurable health outcomes in communities tied to Government Performance & Results Act (GPRA) or Performance Assessment Rating Tool (PART) performance measures Overall, the FY2006 grantees gave stronger responses for the Program Outcomes items. Both the FY2005 and FY2006 grantees were challenged with developing a program evaluation plan that included measurable health outcomes tied to GPRA or PART performance measures. The grantees were most confident in knowing that their services reached the intended audience and resulted in clients being more knowledgeable about their health and ways to improve their health. Content Analysis of Open-ended Item Responses There were several commonalities in the grantees comments within each of the core areas of evaluation. A review of the completed FY2005 and FY2006 evaluation interview forms yielded the following observations: Page 25

26 Approximately 60% of the grantees discussed a need for additional funding to improve program operations, service delivery, and/or program outcomes. Grantees indicated that longer funding cycles would be helpful in producing better program outcomes by enabling them to expand, extend, and keep better records of program activities and participation, as well as hire and maintain staff with key skill sets. High turnover rates for staff were a challenge discussed by 35% of the grantees. Frequent changes in staff resulted in delayed implementation of program activities; the expenditure of time and human resources on training new staff; and, in some cases, lack of leadership for the program and/or overall organization. Nearly 50% of the grantees directly discussed and/or made statements that indicated the need or desire for additional contact with Indian Health Service. More specifically, there was a need for technical assistance, particularly for activities related to conducting local evaluations, and the desire to attend joint, in-person grantee meetings and trainings over the course of the funding cycle(s). While most of the grantees discussed generally positive experiences with their respective communities and local partners, there were four grantees that explicitly described interactions that indicate long-term acceptance of their programs or program components. These interactions include explicit requests from community members for the continuation and/or expansion of program activities; the integration of the program into the larger mission and health-related goals of the community; and participation and assistance from local partners beyond what was initially expected or agreed upon. The interview results also converge with the results of the interviews with grantees during the on-site visits. Responses to the Overarching Evaluation Questions Is the program reaching its population of focus? Based on the data from all sources, the programs reached the populations of focus. The populations included children, youth, families, employees, and students. A large number of grantees focused on children, youth and families. Several stated that they met challenges with getting parents involved and consequently shifted the focus to youth and children. Is the delivery of services consistent with the stated program design, goals, and objectives? This area presented a primary challenge for nearly all the grantees. A fair number of grantees shared that the applications were prepared without the input of the persons who actually implemented the program. Moreover, the program staff is not proficient and knowledgeable of how to structure program goals, Page 26

27 objectives, and outcomes. The evaluation team reached a consensus based on the content analyses and the interviews that five grantees have an excellent command of program planning and evaluation practices as evidenced by their applications, progress reports/results and information submitted for the interviews and their actual interviews: 1) Choctaw Nation of Oklahoma Health Department, 2) Kodiak Area Native Association, 3) Menominee Indian Tribe of Wisconsin, and 4) Southeast Alaska Regional Health Consortium and 5) South Central Foundation. These grantees could coach, train, and advise the other grantees on the process. Two grantees have a good command of the program planning and evaluation practices: 1) Huron Potawatoni, Inc. and 2) Salish Kootenai College. Across all evaluation measures, they provided the required information and documentation. Do positive changes appear among the program participants? Was there an achievement of positive and measureable health outcomes? Seven grantees presented information and evidence that show positive changes and outcomes. The information in Table 6 summarizes the results gleaned from reports and interview data. Table 6 Summary of Goals and Outcomes for Grantees that Met and Exceeded Reporting and Outcomes Guidelines Grantee Funding Year Goal Select Positive Outcomes Promote healthy lifestyles for 5 th Increase in physical 3-4 days level Increase in length of time exercising FY2005 grade students. They reached 221 Decrease in time with TV & Video students in five schools. Increase in fruit/vegetable consumption Choctaw Nation of Oklahoma Health Department Huron Potawatomi, Inc Kodiak Area Native Association Menominee Indian Tribe of Wisconsin Salish Kootenai College South Central Foundation FY2006 FY2006 FY2006 FY2005 FY2005 and FY2006 To decrease modifiable risk factors that lead to chronic diseases. The focus was on families. They reached 206 participants. To implement a community based screening program that increases cancer screening among Alaskan natives on Kodiak Island. To implement a work-site based program to change unhealthy behaviors. Focused on employees of the tribe, the college, and the casino-hotel. N=1342 To provide group fitness classes on the reservation. For both awards, to implement a community based project that leads to a tobacco-free environment in all South Central facilities. Increase in fruit/vegetable consumption Slight increase in physical activity Provided education sessions to all 206 participants 112 Breast Screenings 283 Cervical Cancer Screenings 68 Colorectal Screenings Identified all KANA female patients over 21 (N=2228). Sent letters re: screening to all females Significant knowledge increases from educational sessions Increase in number of employees engaging in health coaching Significant decrease in tobacco use Significant increase in physical activity Conducted 381 workout sessions in four sites over a one-year period A full tobacco-free policy was developed and approved by the board. A work-group was established Marketing and publicity materials were developed and posted 1331 patients enrolled in the smoking cessation program FY 2005 Page 27

28 Grantee Funding Year Goal Select Positive Outcomes Southeast Alaska Regional Health Consortium FY2005 and FY2006 For both awards, to initiate a culturally based community intervention to increase healthy life styles. Offered screening to 55 adults and 10 children in FY2005 Planned, conducted, and completed all proposed camps and classes but one in FY2005 FY2006, conducted 93 health screenings and 40 persons participated in the knowledge camps FY2006 saw minor improvements in blood pressure and cholesterol The outcomes for the FY2005 grantees (Choctaw Nation of Oklahoma Health Department and Salish Kootenai College) with one year of operation and less than $65, in IHS funding are commendable and more than justify the expenditures. The outcomes for the grantees with two years of funding are strong for South Central Foundation the efforts led to major health-related policy changes and formed the foundation for a continuation of a well-constructed social marketing campaign with the FY2006 funds. The evaluators were provided with only one progress report for the FY2006 grantees, consequently the team was unable to see results past the first six months of operations. As previously mentioned, the progress reports for the other grantees provided descriptions of programs and activities, but very little on actual measurable outcomes. Were health outcomes linked to IHS Clinical, GPRA, or PART performance measures? None of the grantees on the site visits were familiar with the GPRA and PART measures. Only three grantees discussed these guidelines in detail during the interviews and it appears that many of the grantees are not fully informed and aware of these guidelines and measures. The seven model programs appear to be knowledgeable of the IHS clinical goals. To what extent did grantees meet the HP/DP purpose and goals and report their outcomes? All grantees linked their program activities to the HP/DP purpose and goals in their original applications. The reporting of outcomes connected to those goals appears problematic for all but seven of the grantees. During the site visits and interviews, the evaluators experienced the passion and commitment that most grantees have for their programs. Most of the activities were creative and comprehensive. It seems that a fairly significant number of programs did not know how to measure and capture their successes, quantitatively. Internal/External Collaborations Most of the grantees built strong external collaborations. This area was a major strength of the HP/DP program across all grantees. The grantees constructed linkages with media outlets, school districts, mainstream youth organizations, and health clinics. One clinic was able to attract a national media outlet (NBC) and acquire donations from all over the country for their children and youth camp. Page 28

29 New Resources Secured Most grantees reported that they secured additional resources, but only four grantees submitted quantifiable information. Several grantees received inkind office space and resources; volunteer efforts; and supplemental grants from state and local agencies. For those four organizations that submitted quantifiable information the total dollar value of new resources is $292, over the FY2005-FY2006 project periods. It appears that most of the grantees did not understand how to track, quantify and attach monetary values to the new resources they secured. The evaluators found that Southeast Alaska Regional Health Consortium and Menominee Indian Tribe of Wisconsin presented the best data on the procurement of additional resources both in-kind and financial. Barriers Identified The number one barrier identified by the grantees was staff retention and turnover. Many grantees struggled with maintaining the same staff over multiple program years. The second most discussed barrier is the disconnect between the tribal councils and program operations a number of grantees experienced delays in hiring and program start-up while awaiting approval from tribal councils. Needs related program and operations Grantees expressed a wide range of needs related to program and operations additional funding for staff to maintain records and analyze data; increased support and technical assistance from the IHS staff-most grantees shared that they would like more contact and direction on their program operations. Moreover, they expressed a need for ongoing contact with other grantees to engage in information exchange and networking. Evaluation needs The majority of the grantees shared that they need technical assistance and support with their local evaluation efforts. Most grantees conceded that they do not have program planning expertise the personnel are better at implementation and interfacing with program participants. Major challenges exist around developing instruments to capture their outcomes and staff support and expertise for data analyses. RECOMMENDATIONS Observed Best Practices for Health Promotion and Disease Prevention The evaluators were impressed with the structure, creativity, and outcomes of five grantee programs 1) Choctaw Nation of Oklahoma Health Department s focus on healthy lifestyles with 5 th graders in schools; 2) Kodiak Area Native Association s community based cancer screening; 3) Menominee Indian Tribe of Wisconsin s comprehensive approach to worksite interventions; 4) Southeast Alaska Regional Health Consortium s intergenerational health Page 29

30 promotion activities, and 5) South Central Foundation s policy and social marketing intervention. Choctaw Nation of Oklahoma Health Department s (Improving the Health of Children) The Choctaw Nation of Oklahoma Health Department s " Get Movin' " Project enhanced and expanded the existing programs for fifth grade students offered by the Healthy Lifestyles and Diabetes Wellness Center programs. These programs provided culturally relevant diabetes prevention education and promoted increased physical activity with the use of a pedometer. The Choctaw Nation of Oklahoma service area comprises 10.5 counties of southeastern Oklahoma. The mass media campaign covered the entire service area. The interventions for fifth grade students focused on the communities of Hugo, Clayton, Coalgate, Atoka, and Boswell and reached 206 participants. The Kodiak Area Native Association (Community Based Cancer Screening) The Kodiak Area Native Association initiated Securing Our Future, Preserving Our Past Cancer Screening Initiative. The project increased the capacity of the local agency to identify and screen residents of the Alaska Native population on Kodiak Island. The program personnel engaged in a structured and intensive effort to identify all females over the age of 21. Once the persons were identified, they mailed letters to over 2000 women and initiated the process of screening those who responded. Menominee Indian Tribe of Wisconsin (Worksite Health Education and Screening) The project focused on worksite health education and wellness. The employees are reachable and natural audience for the intervention. The program personnel conducted health risk assessments with more than 50% of the employees. Using the outcomes, they constructed multiple educational and lifestyle interventions focused on the three categories of employees. The planning, assessment, and implementation strategies of this project are commendable. Southeast Alaska Regional Health Consortium (Intergenerational Health Promotion) The project focused on using community-driven programs and alliances to promote and support healthy lifestyles through increasing traditional knowledge and to contribute to the Chilkat Indian Village social and economic development strategic plan. This was an excellent example of channeling health promotion through cultural awareness. This program was successful in bringing together multiple generations within the community to participate in various cultural activities and the engagement of Traditional Knowledge Camps to promote health. The ability to raise support through collaborations and external funding sources was exceptional. South Central Foundation (Policy and Social Marketing) This project focused on the use of changes in policies related to tobacco use to drive associated changes in behaviors. During the project period the Page 30

31 program witnessed major policy changes. The project offers a template for how other organizations can initiate and be successful in changing local policies to improve health outcomes. The grantee also engaged in a planned and deliberate strategy to create culturally appropriate materials for a social marketing campaign. The organization witnessed a large number of patients who enrolled in the smoking cessation sessions. The evaluation team believes other tribal organizations can benefit from an in-depth review and understanding of how these interventions were structured, the outcomes, and the lessons learned. Recommendations Pre-application and Application Process The evaluators observed that several applications were weak and missing key program elements (overarching goals, objectives with activities and outcomes). The program planning process should start prior to the submission of an application for funding. Additionally, according to many grantees, persons who were not the program implementers prepared applications. Specific Recommendations: IHS should employ the cooperative agreement procurement mechanism. This mechanism will allow the agency to offer support and technical assistance to grantees as they initiate and implement their projects. IHS should conduct pre-application briefing and information sessions to guide potential applicants through the program requirements, expectations, and required components of the program. For applicants with well-constructed program ideas and weak methods for determining program success, IHS should make awards with contingencies that focus on the grantee providing a comprehensive plan for determining the success of the program before start-up. IHS should include a method for determining the authoring of proposals and the involvement the author(s) will have in the implementation of the program and/or management of the grant. IHS should consider more long-term awards (five years) with increased funding levels that support improved long-term outcomes. Post Grant Award Support and Technical Assistance The evaluators found the lack of technical assistance to be the area of greatest concern for most of the grantees. Nearly all of the grantee expressed a need for more technical support from IHS. Page 31

32 Specific Recommendations: IHS staff should conduct a multi-day, face-to-face orientation for all new grantees. This meeting should include presentations from IHS staff, program and evaluation experts, and allow time for grantees to network and exchange information and ideas. The orientation should clarify expectations and performance norms. IHS should require that all grantees submit a final work plan that includes the program goal(s), objectives, activities linked to those objectives, methods for measuring the objectives, methods for collecting and analyzing the data, a final staffing plan, and timelines. Additionally, the grantees should submit a program logic model consisting of the inputs, activities, outputs, as well as short-term, intermediate, and long-term outcomes. The IHS staff should conduct monthly teleconferences with grantees to track progress and manage challenges within the programs. The IHS program staff should determine a method to engage the staff of the best practices organizations in a meeting with current grantees to share their approaches and lessons. Tracking Progress Reports and Outcomes The evaluators noted that many grantees did not use the required progress report form, which made it impossible to identify and compare program outcomes and results across grantees. Specific Recommendations: IHS should consider collecting program data from grantees through a web-based database and reporting system, which controls the data entered and returns error messages when fields are incomplete. With the manual submission of reports, IHS personnel should be designated to conduct ongoing reviews of reports to determine if grantees are submitting reports on schedule and if the content of the reports conforms to the required and needed documentation. Immediate feedback should be provided to grantees, particularly those that are not on track with stated goals and objectives. Page 32

33 Page 33 APPENDIX A REESSI DATA TRACKING FORM

34 2005 Grantees ABERDEEN AREA TRIBAL CHAIRMEN'S HEALTH BOARD Evaluator Grant Application Progress Report Notes NAmutah Received 1 Received Coded ALAMO NAVAJO SCHOOL BOARD, INC. SAJackson Received 1 Received Coded CHOCTAW NATION OF OKLAHOMA HEALTH DADeRoin Received 1 Received Coded DEPARTMENT EASTERN ALEUTIAN TRIBES, INC. MWGlover Received 1 Received Coded HO-CHUNK NATION DADeRoin Received 1 Received Coded INDIAN HEALTH CARE RESOURCE CENTER, INC DADeRoin Received 1 Received Coded INTER-TRIBAL COUNCIL OF MICHIGAN, INC MWGlover Received 1 Received Coded KALISPEL TRIBE OF INDIANS MWGlover Received 1 Received Coded KENAITZE INDIAN TRIBE DADeRoin Received 1 Received Coded MIGIZI COMMUNICATION, INC MWGlover Received 1 Received Coded NATIONAL INDIAN JUSTICE CTR DADeRoin Received 1 Received Coded NATIVE IMAGES INC DADeRoin Received 1 Received Coded PETA WAKAN TIPI DADeRoin Received 1 Received Coded RAMAH NAVAJO SCHOOL BOARD, INC DADeRoin Received 1 Received Coded ROCKY BOY HEALTH BOARD NAmutah Received 1 Received Check SALISH KOOTENAI COLLEGE NAmutah Received 1 Received Coded SAN DIEGO AMERICAN INDIAN HEALTH CENTER DADeRoin Received 1 Received Coded SOUTH CENTRAL FOUNDATION DADeRoin Received 1 Received Coded SOUTHEAST ALASKA REGIONAL HEALTH CONSORTIUM MWGlover Received 1 Received Coded TULALIP TRIBES SAJackson Received 1 Received Coded Page 34

35 2006 Grantees CENTRAL OKLAHOMA AMERICAN INDIAN HEALTH NAmutah Received 1 Report 1-2 Missing COUNCIL, INC CHIPPEWA CREE TRIBE NAmutah Received 1 Report 1-2 Missing CONF. TRIBES OF WARM SPRINGS NAmutah Received 1 Report 1-2 Missing HURON POTAWATOMI, INC. SAJackson Received 1 Report 1-2 Missing INDIAN HEALTH CARE RESOURCE CENTER OF TULSA, DADeRoin Received 1 Report 1-2 Missing INC. INTER-TRIBAL COUNCIL OF MICHIGAN, INC. SAJackson Received 1 Report 1-2 Missing KODIAK AREA NATIVE ASSOCIATION MWGlover Received 1 Report 1-2 Missing MENOMINEE INDIAN TRIBE OF WISCONSIN NAmutah Received No Reports Need All NORTON SOUND HEALTH CORPORATION MWGlover Received No Reports Need All PUEBLO SAN FELIPE DADeRoin Received No Reports Need All ROUND VALLEY INDIAN HEALTH CENTER INC. SAJackson Received No Reports Need All SOUTHCENTRAL FOUNDATION MWGlover Received 1 Report 1-2 Missing SOUTHEAST ALASKA REGIONAL HEALTH CONSORTIUM MWGlover Received 1 Report 1-2 Missing Page 35

36 Page 36 APPENDIX B RECORD CODING SHEETS

37 Page 37

38 Page 38

39 Page 39 APPENDIX C SITE VISIT INTERVIEW GUIDE AND FIELD NOTES FORM

40 Page 40

41 Page 41

42 Page 42 APPENDIX D SITE VISIT LETTERS

43 July 29, 2008 Marline Martinez Administrative Services Director Ramah Navajo PO Box 10 Pinehill, NM Dear Ms. Martinez: Research and Evaluation Solutions, Inc. (REESSI) has been selected by the Indian Health Services (IHS) to conduct an independent evaluation of the Health Promotion Disease Prevention (HPDP) grantees for FY 2005 and FY This evaluation fulfills the Office of Management and Budget (OMB) request for an external independent evaluator to assess the impact and effectiveness of the program grantees. Our firm has a 23 year history in program planning and evaluation activities in communities of color. REESSI is a team-oriented solution center that brings together a transdisciplinary group of professionals who share the corporate mission. Our mission is to provide research-based support and services to government and non-government organizations, which translate into action and solutions that improve lives, and reduce lifelong poverty, premature death and disability in communities of color. We are excited to have Dr. DeeAnn DeRoin as part of our evaluation team. She made the initial contact with your organization to set-up the site visit. During the week of August 4, Dr. DeRoin and I will conduct one-day (6 hours) site visits in Oklahoma and New Mexico. Our visit to Ramah School Board will be Friday, August 8, We will start at 9:00 a.m. and the visit will conclude around 3:00 p.m. A schedule for the week is attached. We know that most organizations are generally nervous about evaluations, however our purpose is to determine how you structured and implemented your program; what worked and was effective; and what did not work. Our agenda will include a face-toface interview with 1-2 staff that were/are directly involved with the program; a review of program records (sign-in sheets, curricula, handouts, materials, and other items); and a discussion about your on-site evaluation tools and activities. If you have any questions, please feel free to contact me at or Dr. DeeAnn DeRoin at We are looking forward to our visit. Sincerely, Laverne Morrow Carter, PhD, MPH President/Managing Principal July 29, 2008 Page 43

44 Nancy O Bannion Project Director 550 South Peoria Ave. Tulsa, OK Dear Ms. O Bannion: Research and Evaluation Solutions, Inc. (REESSI) has been selected by the Indian Health Services (IHS) to conduct an independent evaluation of the Health Promotion Disease Prevention (HPDP) grantees for FY 2005 and FY This evaluation fulfills the Office of Management and Budget (OMB) request for an external independent evaluator to assess the impact and effectiveness of the program grantees. Our firm has a 23 year history in program planning and evaluation activities in communities of color. REESSI is a team-oriented solution center that brings together a transdisciplinary group of professionals who share the corporate mission. Our mission is to provide research-based support and services to government and non-government organizations, which translate into action and solutions that improve lives, and reduce lifelong poverty, premature death and disability in communities of color. We are excited to have Dr. DeeAnn DeRoin as part of our evaluation team. She made the initial contact with your organization to set-up the site visit. During the week of August 4, Dr. DeRoin and I will conduct one-day (6 hours) site visits in Oklahoma and New Mexico. Our visit to the Tulsa Urban Indian Clinic will be Tuesday, August 5, We will start at 9:00 a.m. and the visit will conclude around 3:00 p.m. A schedule for the week is attached. We know that most organizations are generally nervous about evaluations, however our purpose is to determine how you structured and implemented your program; what worked and was effective; and what did not work. Our agenda will include a face-to-face interview with 1-2 staff that were/are directly involved with the program; a review of program records (sign-in sheets, curricula, handouts, materials, and other items); and a discussion about your on-site evaluation tools and activities. If you have any questions, please feel free to contact me at or Dr. DeeAnn DeRoin at We are looking forward to our visit. Sincerely, Laverne Morrow Carter, PhD, MPH President/Managing Principal Page 44

45 July 29, 2008 Dear Mr. Swalwell: Research and Evaluation Solutions, Inc. (REESSI) has been selected by the Indian Health Services (IHS) to conduct an independent evaluation of the Health Promotion Disease Prevention (HPDP) grantees for FY 2005 and FY This evaluation fulfills the Office of Management and Budget (OMB) request for an external independent evaluator to assess the impact and effectiveness of the program grantees. Our firm has a 23 year history in program planning and evaluation activities in communities of color. REESSI is a team-oriented solution center that brings together a transdisciplinary group of professionals who share the corporate mission. Our mission is to provide research-based support and services to government and non-government organizations, which translate into action and solutions that improve lives, and reduce lifelong poverty, premature death and disability in communities of color. We are excited to have Dr. DeeAnn DeRoin as part of our evaluation team. She made the initial contact with your organization to set-up the site visit. During the week of August 4, Dr. DeRoin and I will conduct one-day (6 hours) site visits in Oklahoma and New Mexico. Our visit to the Oklahoma City Urban Indian Clinic will be Wednesday, August 6, We will start at 9:00 a.m. and the visit will conclude around 3:00 p.m. A schedule for the week is attached. We know that most organizations are generally nervous about evaluations, however our purpose is to determine how you structured and implemented your program; what worked and was effective; and what did not work. Our agenda will include a face-to-face interview with 1-2 staff that were/are directly involved with the program; a review of program records (sign-in sheets, curricula, handouts, materials, and other items); and a discussion about your on-site evaluation tools and activities. If you have any questions, please feel free to contact me at or Dr. DeeAnn DeRoin at We are looking forward to our visit. Sincerely, Laverne Morrow Carter, PhD, MPH President/Managing Principal Page 45

46 July 29, 2008 Dear Daytona Raye: Research and Evaluation Solutions, Inc. (REESSI) has been selected by the Indian Health Services (IHS) to conduct an independent evaluation of the Health Promotion Disease Prevention (HPDP) grantees for FY 2005 and FY This evaluation fulfills the Office of Management and Budget (OMB) request for an external independent evaluator to assess the impact and effectiveness of the program grantees. Our firm has a 23 year history in program planning and evaluation activities in communities of color. REESSI is a team-oriented solution center that brings together a transdisciplinary group of professionals who share the corporate mission. Our mission is to provide research-based support and services to government and non-government organizations, which translate into action and solutions that improve lives, and reduce lifelong poverty, premature death and disability in communities of color. We are excited to have Dr. DeeAnn DeRoin as part of our evaluation team. She made the initial contact with your organization to set-up the site visit. During the week of August 4, Dr. DeRoin and I will conduct one-day (6 hours) site visits in Oklahoma and New Mexico. Our visit to Pueblo of San Felipe will be Thursday, August 7, We will start at 9:00 a.m. and the visit will conclude around 3:00 p.m. A schedule for the week is attached. We know that most organizations are generally nervous about evaluations, however our purpose is to determine how you structured and implemented your program; what worked and was effective; and what did not work. Our agenda will include a face-toface interview with 1-2 staff that were/are directly involved with the program; a review of program records (sign-in sheets, curricula, handouts, materials, and other items); and a discussion about your on-site evaluation tools and activities. If you have any questions, please feel free to contact me at or Dr. DeeAnn DeRoin at We are looking forward to our visit. Sincerely, Laverne Morrow Carter, PhD, MPH President/Managing Principal Page 46

47 Page 47 APPENDIX E SITE VISIT AGENDA

48 Page 48

49 APPENDIX F GRANTEE INTERVIEWS REQUEST LETTER & CHECKLIST Page 49

50 Page 50

51 Page 51

52 APPENDIX G GRANTEE INTERVIEW DATA QUALITY REVIEW FORM Page 52

53 Page 53