CENTER TRT EVALUATION PLAN North Carolina Maternity Center Breastfeeding-Friendly (NC MCBF) Designation Program Evaluation Plan: Purpose: The purpose of this evaluation plan is to determine the extent of implementation and acceptability, and effectiveness, of a systems approach to implementing the Baby-Friendly Hospital Initiative (BFHI), modeled after the approach taken by the North Carolina Maternity Center Breastfeeding-Friendly (NC MCBF) Designation Program, a project of the North Carolina Division of Public Health, Nutrition Services Branch. This evaluation plan provides a menu of options for evaluation questions covering several dimensions commonly included in program evaluations, including: reach, adoption, extent of program implementation, and effectiveness in addressing targeted outcomes. Please note that this suggested evaluation plan focuses on program implementation; a list of other relevant evaluation questions is available in the Evaluation section of the Center TRT website. Evaluation Questions: This evaluation plan likely includes many more evaluation questions than will be feasible to answer. Similarly, it may include questions that are less important for your particular context or lack questions that should be prioritized for your context. Center TRT recommends working with your stakeholder group to prioritize the evaluation questions you will seek to answer. Design: The evaluation is a pre- and post-test design with no control group. Data collection: A variety of data collection tools are referenced throughout the evaluation plan. One of the process data collection tools was developed by evaluators of the NC MCBF Designation Program and is available for your use. This tool is asterisked in this evaluation plan. Please note that Center TRT has not reviewed this tool. Most tools suggested will need to be created to apply to your context. These same tools also appear in the lower section of the Center TRT Logic Model. Process evaluation tools: Maintained by individual o Training log - to track which, when, and how many maternity staff have been trained o - to assess knowledge of and attitudes towards BFHI Ten Steps; could look at example from the Connecticut Breastfeeding Initiative for ideas of questions: http://centertrt.org/content/docs/intervention_documents/intervention_materials/ct_breastfeeding_initiative/evaluation_tool_3_- _CBI_Maternity_Staff_Web_Survey_Final.pdf o patient - to assess patient attitudes towards BFHI Ten Steps Maintained by coordinating agency (ie, state health department) o Criteria/assessment tool for reviewing maternity center applications* o - to assess number and characteristics of that are designated under the NC MCBF Designation Program, and which of the BFHI Ten Steps are being adopted o Materials development and distribution log to track what type and how many materials were developed and distributed Center TRT, UNC-HPDP Evaluation Plan for NC MCBF; posted May 2014 Page 1 of 6
o Training and technical assistance log to track number of contacts or hours of training and technical assistance provided to o Interviews with key staff at statewide coordinating agency o Budget and other recorded costs Maintained by State Hospital Association o State Hospital Association data on - to assess demographic and other characteristics of that are designated under the NC MCBF Designation Program o State Hospital Association patient data and/or patient data collected from - to assess demographic and other characteristics of mothers served by that are designated under the NC MCBF Designation Program Short-term (1-3 year) outcome evaluation tools: Baby-Friendly USA Self-Appraisal Tool http://www.babyfriendlyusa.org/get-started/d1-discovery (click on link to self-appraisal tool for PDF download) - to assess knowledge of and attitudes towards BFHI Ten Steps; could look at example from the Connecticut Breastfeeding Initiative, for ideas of questions relating to knowledge and attitudes (other questions will need to be added accordingly): http://centertrt.org/content/docs/intervention_documents/intervention_materials/ct_breastfeeding_initiative/evaluation_tool_3_- _CBI_Maternity_Staff_Web_Survey_Final.pdf - to assess number and characteristics of that are designated under the NC MCBF Designation Program, and which of the BFHI Ten Steps are being adopted Tools that are asterisked were used by the NC MCBF Designation Program and are available for your use. These tools can be accessed on our website within the Evaluation Materials section of the NC MCBF Designation Program Intervention Package. Contact information for the developers is available on the Center TRT website within the NC MCBF Designation Program Intervention Package. Please contact the intervention developers for questions about the intervention itself. Finally, measuring breastfeeding practices among mothers and babies is not addressed in this evaluation plan. Here are some approaches to measuring these outcomes: http://www.babyfriendlyusa.org/get-started/the-guidelines-evaluation-criteria. PROCESS EVALUATION This section should address the reach of the intervention into the intended population; the adoption or uptake of the intervention by setting; and the fidelity of implementation of the intervention components and core elements. Evaluation Questions Data to be Collected Data Collection Method Center TRT, UNC-HPDP Evaluation Plan for NC MCBF; posted May 2014 Page 2 of 6
Reach How many and what were the characteristics of the maternity center staff and administration reached by the NC MCBF Designation Program? # of staff trained Demographics (maternity center where staff work, race, training level, position title, etc.) Training log kept by How many and what were the characteristics of the mothers and babies reached by the NC MCBF Designation Program? Adoption How many/what proportion of within the state applied for the NC MCBF Designation? Demographics of women giving birth in across the state compared to demographics of women giving birth in with a NC MCBF designation # and proportion of maternity centers that submitted applications to the NC MCBF Designation Program State Hospital Association patient data and/or patient data collected from State Hospital Association data on What were the characteristics of within the state that applied for the NC MCBF Designation? Implementation Statewide Coordinating Agency What was the application/designation process that the statewide coordinating agency developed and used for the NC MCBF Designation Program? Characteristics of that submitted applications to the NC MCBF Designation Program Program application # and type of materials developed for the designation packet State Hospital Association data on Materials development and distribution log Center TRT, UNC-HPDP Evaluation Plan for NC MCBF; posted May 2014 Page 3 of 6
What criteria or assessment tools did the statewide coordinating agency develop for assessment of the NC MCBF Designation Program applications? # and type of criteria and/or assessment tools created Criteria/assessment tool for reviewing maternity center applications* Materials development and distribution log What training and technical assistance was provided to participating maternity centers by the coordinating agency? Training content # of training and technical assistance contact hours Training and technical assistance log Were the perceived benefits of the NC MCBF Designation Program worth the costs from the perspective of the statewide coordinating agency? Perceived benefits of the NC MCBF Designation Program (reported by staff at the statewide coordinating agency) Perceived and monetary costs of participation in the NC MCBF Designation Program (reported by staff at the statewide coordinating agency) Interviews with key staff at statewide coordinating agency Budget and other recorded costs Implementation Maternity Center level Was the NC MCBF Designation Program acceptable to maternity center level stakeholders? Perceived level of acceptability of the NC MCBF Designation Program reported by maternity center level stakeholders (key contact, maternity staff) What were maternity staff perceptions of the NC MCBF Designation Program? Perceptions of NC MCBF Designation Program reported by maternity staff Center TRT, UNC-HPDP Evaluation Plan for NC MCBF; posted May 2014 Page 4 of 6
What were mother/parent perceptions of the NC MCBF Designation Program? Perceptions reported by mothers/parents patient Was the NC MCBF Designation Program implementation feasible for maternity centers? Reports of feasibility (or lack thereof) by maternity staff and key maternity center contacts Were the perceived benefits of the NC MCBF Designation Program worth the financial and other costs for the? Reports of perceived benefits (by maternity center contacts) Reports of perceived and monetary costs (by maternity center contacts) OUTCOME EVALUATION This section should address the effect of the intervention on the intended short-term outcomes (those you can measure at the end of a 1-3 year project period). Evaluation Questions Data to be Collected Data Collection Method Short-term Outcomes Outcome 1: Increased positive attitudes about implementing the Ten Steps from BFHI How have attitudes about the BFHI Ten Steps improved among maternity center staff and administration, from pre to post-implementation of the NC MCBF Designation Program? % change in favorable attitudes toward BFHI Ten Steps among maternity center staff and administration Outcome 2: Increased positive knowledge about implementing the Ten Steps from BFHI How has knowledge about the BFHI Ten Steps improved among maternity center % change in knowledge of BFHI Ten staff and administration, from pre to post-implementation of the NC MCBF Steps among maternity center staff and Designation Program? administration Outcome 3: Increased # of BFHI steps adopted by How has the number of BFHI steps adopted by changed over time, from pre to post-implementation of the NC MCBF Designation Program? # of Baby-Friendly practices (measured at baseline and follow-up) designation applications Center TRT, UNC-HPDP Evaluation Plan for NC MCBF; posted May 2014 Page 5 of 6
Outcome 4: Increased number of hospitals/ designated under NC MCBF Designation Program By what percentage has the number of hospitals/ with any of the # of centers designated under the NC star-level designations (1, 2, 3, 4, or 5 stars) under the NC MCBF Designation MCBF Designation Program (measured Program increased? at baseline and follow-up or could be measured yearly to show change over time) Possible Unintended Consequences Parent/family dislike of or disagreement/frustration with Baby-Friendly approach. Increased focus and resources spent on achieving BFHI steps, and reduced focus and funding for other areas. Though NC MCBF Designation Program intends to help achieve BFHI status, hospitals and could choose to fulfill the NC MCBF Designation Program rather than work directly with the BFHI. Center TRT, UNC-HPDP Evaluation Plan for NC MCBF; posted May 2014 Page 6 of 6