FUTURE VISION PLAN. Global Grant Monitoring and Evaluation Toolkit

Transcription

1 ENGLISH (EN) FUTURE VISION PLAN Global Grant Monitoring and Evaluation Toolkit Dear Project Sponsors, Congratulations on your decision to apply for a Rotary Foundation Global Grant! This is an exciting time, and we know you have worked hard to develop your project. One of the key elements to a successful project is monitoring and evaluation. In addition to strengthening individual projects, monitoring and evaluation allows Rotary to tell its story to the world. We know that you re eager to tell the story of Rotary s efforts in your community as well, and we hope that the information and tools included in this toolkit will help you accomplish that goal. If you have questions about monitoring and evaluation or need assistance in developing communication materials, please contact us at futurevision@rotary.org. Regards, Future Vision Pilot Operations Division The Rotary Foundation Global Grants Monitoring and Evaluation Toolkit (April 2012) 1

2 Table of Contents Introduction... 3 How to Use This Toolkit... 3 Monitoring and Evaluation Plan Overview... 4 Standard terminology... 5 Setting Project Goals... 6 Data Collection... 7 Hazards of Double Counting... 9 Sustainability Selecting from TRF Standard Measures Basic Education and Literacy Measures Disease Prevention and Treatment Measures Economic and Community Development Measures Maternal and Child Health Measures Peace and Conflict Prevention/Resolution Measures Water and Sanitation Measures Creating Additional Project Specific Measures Creating a Monitoring and Evaluation Plan Monitoring and Evaluation Plan Explanation Sample Monitoring and Evaluation Plan Resources Rotary Resources External Resources Glossary Global Grants Monitoring and Evaluation Toolkit (April 2012) 2

3 INTRODUCTION Monitoring and evaluation has many benefits, for project sponsors and for The Rotary Foundation: Project sponsors can track activities and make necessary adjustments to the current project and to future project plans. Project sponsors can assess the scope and impact of their projects. The Rotary Foundation can track worldwide Rotarian efforts and accomplishments. The Rotary Foundation and the project sponsors can accurately publicize and describe achievements. In order to do these things, using standard terminology and methodology is important. Standards ensure that when Rotarians in Guatemala count the number of entrepreneurs who have been trained on business development, they are using the same definitions and methods as Rotarians in Indonesia or Ethiopia. This standardization also allows The Rotary Foundation to publish articles on the number of people with access to improved sanitation as a result of Rotarian efforts using the same definitions for improved as other organizations. HOW TO USE THIS TOOLKIT Global grant projects connect to one or more of Rotary s six areas of focus and are designed to be measurable and sustainable. This toolkit is meant to assist global grant project sponsors in creating a monitoring and evaluation plan for their projects. Specifically, this document will help you: Learn basic monitoring and evaluation concepts Identify resources for setting project goals Understand the process of creating a monitoring and evaluation plan, including: Selecting data collection methods Avoiding double counting Incorporating sustainable activities into a project design Determining when to scale up or replicate activities Selecting appropriate measures (see list of official TRF measures, pages 11-22) Creating additional measures to fit the specific needs of the project Identify additional resources Global Grants Monitoring and Evaluation Toolkit (April 2012) 3

4 An overview of the process of creating a plan, which is required to complete a global grant application, is shown below. A monitoring and evaluation plan should be ed to your global grant senior coordinator when submitting a global grant application. When reporting on the grant, project sponsors are asked to provide information on the results of their monitoring and evaluation activities and to indicate the extent to which measurement targets were met. This can be done using the monitoring and evaluation results document and ed to the senior coordinator when submitting the report. MONITORING AND EVALUATION PLAN OVERVIEW Action Project Stage Refer to... Learn about TRF s monitoring and evaluation expectations. Proposal Global Grant Monitoring and Evaluation Toolkit, Principles of Sustainability, and other resources Establish clear project goals. Proposal and/or Setting Project Goals, page 6 Application Identify applicable TRF standard measures. Proposal and/or Application Selecting from TRF Standard Measures, page 11 Identify additional project specific Proposal and/or Creating Additional Project specific measures. Complete a monitoring and evaluation plan as an addendum to the online application. the plan to the global grant senior coordinator. Collect data in accordance with the monitoring and evaluation plan. Complete the monitoring and evaluation results document. the document to the global grant senior coordinator. Application Application Measures, page 23 Creating a Monitoring and Evaluation Plan, page 24 Implementation Data Collection, page 7 Reporting Creating a Monitoring and Evaluation Plan, page 24 Global Grants Monitoring and Evaluation Toolkit (April 2012) 4

5 STANDARD TERMINOLOGY Monitoring and evaluation is the process of collecting and analyzing data on project activities. The purpose of monitoring and evaluation is to answer the questions how effective is your project? or how well did the project meet its goals? Monitoring involves the routine process of collecting specific data as agreed upon before project implementation. It allows for the collection of documentation about project activities and any improvements to beneficiaries that occurred as a result of the project. Examples of monitoring include tracking student enrollment at a school, incidence of disease in a community, or persons with access to clean water. Evaluation generally involves a post-project assessment of the degree to which the project goals were met and the impact project activities had on beneficiaries. While monitoring takes place regularly throughout the project implementation, evaluation typically takes place once the project is completed. Evaluations can also take place regularly for years after the project is completed in order to assess the lasting impact to the community. Monitoring and evaluation typically involves the collection of both quantitative data (numerical) and qualitative data (stories, statements, or opinions). Both in monitoring and evaluation there is a need for baseline data collection. Baseline data simply refers to facts about individuals or a community. It is collected before project implementation and provides a clear picture of the situation at the beginning of a project. Data collected through monitoring and evaluation is then compared to the baseline data to better assess project achievements. For a glossary of monitoring and evaluation terms used in this document, see pages Global Grants Monitoring and Evaluation Toolkit (April 2012) 5

6 SETTING PROJECT GOALS Project goals reflect what the project sponsors hope to accomplish through the project and the desired long-term outcomes. Both the project plan and the monitoring and evaluation plan should be based on specific global grant project goals. In order to create appropriate goals, you should conduct an assessment of the community you wish to serve. The purpose of the assessment is to obtain information about resources and needs, whether and how issues are currently being addressed, and actions likely to result in community improvements. Community assessments can be conducted in a number of ways. See Community Assessment Tools for ideas on how to gather information. When setting project goals, you may wish to begin by reviewing the goals of the selected area of focus. Consider how these goals match those of the project and how the project aims to have a direct impact on the goal. The distinction between direct impact and indirect impact is important: as it differentiates between projects that have possible tangential connections to an area of focus and projects that precisely align with one. Take, for example, a water and sanitation project that aims to provide clean water to a community. The project aligns with the water and sanitation goal to increase equitable community access to safe drinking water and sanitation; this is a direct impact. The project could also have indirect impacts in other areas of focus. For example, access to clean water may lead to A decrease in diarrheal disease, which improves health (disease prevention and treatment) and increases the likelihood that children can attend school more often (basic education and literacy) and adults can take part in income-generating activities more regularly (economic and community development) Safe deliveries and healthier children (maternal and child health) Fewer conflicts over water sources (peace and conflict prevention/resolution) However, each of these connections is a potential by-product of the direct impact of access to clean water and do not provide a direct fit with these areas of focus. In addition to the area of focus goals, consider developing additional goals specific to the project. When developing project goals, choose goals that are specific and measurable. For instance, rather than setting a goal to improve health in the beneficiary community, you could make your goal increase the number of children under the age of five vaccinated against measles and tuberculosis. Global Grants Monitoring and Evaluation Toolkit (April 2012) 6

7 DATA COLLECTION Once you ve set project goals, you can begin developing a monitoring and evaluation plan to align with those goals. As part of the plan, you ll need to determine the best way to collect data on the specified measures. Consider the following questions: What data do we want to collect? How will we collect it? When/how often should we collect data? Who will collect the data? In determining how to collect data, consider a wide range of options (described on the following page) and combine multiple options to best address the measures. Data sources generally fall into two main categories: routine and non-routine. Routine data is collected continuously, typically by a non-rotary entity. For example, the government may collect ongoing data on certain services provided, or a school may collect ongoing data on student attendance. This routine data can often be accessed and used for project purposes. However, because the data isn t project specific, you ll need to combine it with other measures to provide a richer picture of the project. Nonroutine data is collected less frequently and is typically gathered by Rotarians or the cooperating organization for the purposes of monitoring and evaluating their activities. Data may include surveys conducted with beneficiary communities or testing the water quality in filters provided through the project. With nonroutine data, time and budget considerations can influence on the specific tool and data collection method used. Realistically, you should budget 5 percent to 10 percent of project funds for monitoring and evaluation activities to cover expenses such as travel, services rendered by external persons/agencies, and supplies. Monitoring and evaluation expenses are eligible and expected in global grant budgets. Global Grants Monitoring and Evaluation Toolkit (April 2012) 7

8 The following table lists basic tools for developing a plan for data collection. Sponsors will often need to use multiple tools in order to address all information needs. Method/Tool Description Useful for... Direct Data collected through directly observing Providing or observing direct Observation activities, outputs, or impacts during a services provision, such as leading a Focus Groups/ Interviews Grant Records & Reports Public Records Questionnaires & Surveys Satellite Imagery Testing project site visit In depth conversations with beneficiaries one individual (interview) or multiple individuals (focus group) in which the interviewer/facilitator poses pre set questions in order to collect specific information Documents kept by the project sponsors on project participation, activities, and impacts for the use of Rotarians, RI/TRF, and others Documents held by the government, private entities, or other organizations that are available to the public Specific questions asked of individuals in a beneficiary population in an effort to gather data on attitudes, impressions, opinions, satisfaction level, etc. Digital data obtained from sensors carried in satellites or global positioning devices Activity to determine a specific result, including water quality, retention of information, or effect of a medical treatment training or treating a patient Digging deeper into the impacts of a project and the experiences of the beneficiaries; soliciting qualitative data. Can be customized to sponsors needs. Documenting basic information (e.g., outputs, counts) required by TRF Quickly and inexpensively accessing standard information (often quantitative) collected by an outside entity. Can supplement information collected by the sponsors. Gathering quantitative and qualitative data from a beneficiary group. Can be customized to the sponsors needs. Recording the location of activities implemented in multiple places, such as boreholes, health clinics, schools, etc. Determining the result of a specific activity; providing quantitative data Global Grants Monitoring and Evaluation Toolkit (April 2012) 8

9 HAZARDS OF DOUBLE COUNTING The Rotary Foundation asks global grant project sponsors to incorporate into their monitoring and evaluation plan a measure of total number of direct beneficiaries. It is essential that project sponsors understand the issue of double counting in order to avoid this common monitoring mistake. Double counting occurs when an individual or group is counted more than once when documenting beneficiaries of project activities. It can happen with data from in a single organization and across multiple organizations, for example: One person visits a health clinic on multiple occasions to receive treatment and is counted each time, reflecting a higher number of beneficiaries than is accurate. Two organizations hold hygiene education classes at a local elementary school, and each counts the students separately for their own reporting purposes. Rotarians hold several conflict resolution trainings in different parts of the community, and one participant attends multiple trainings and is counted separately at each training. To decrease the risk of double counting, create a documentation system that includes multiple distinguishing factors of each beneficiary that can be cross-referenced across records. For instance, record the full name, date of birth, and gender of each participant, along with service received, date, and location. Before finalizing your record or report, compare lists and remove duplicates. If you re reporting to partners/government entities, you ll need to develop a system to ensure that double counting is not taking place between each partner reporting. Global Grants Monitoring and Evaluation Toolkit (April 2012) 9

10 SUSTAINABILITY The Rotary Foundation defines sustainability as the capacity for maintaining long-term outcomes to serve the ongoing need of a community after grant funds have been expended. TRF expects global grant funding to serve as an investment in long-lasting change in a community. By creating sustainable projects, Rotarians maximize the impact of their efforts and Rotary funding. The most effective time to incorporate sustainability into any project is during the initial planning phase. A sustainable project design should encourage beneficiaries to continue to implement the change into the future, even after the grant has ended and all grant funds have been spent. Sustainability can be achieved in many ways, such as: Providing a strong training component that increases the knowledge and abilities of community members to continue addressing an issue Establishing a permanent community-led leadership committee that will manage the ongoing aspects of the project Incorporating fees for service to collect the funding needed for ongoing maintenance and activities Focusing on income generation to allow community members to increase their personal finances and financial security Sustainability should not be incorporated after the project design and activities have been determined. Trying to add sustainable components to a project already designed is likely to delay TRF acceptance of the global grant request. To learn more about TRF s approach to sustainability for global grant projects, take these next steps: Review the Sustainability E-learning Module. Read the Principles of Sustainability. Ask a district or club officer to share the sustainability case studies found in the Grant Management Manual and Grant Management Seminar Leaders Guide. Contact a Future Vision staff member to discuss the project design. Most projects start small. A modest initial scope provides a valuable opportunity to test a project design and make adjustments to maximize results. After a project achieves and documents positive impact, it may be time to consider scaling up. Scaling up refers to increasing the scope of a project by replication, or implementing the project in additional locations or expanding the number of participants in the existing location. The decision to replicate or expand is not always clear. When considering replication, ensure that the project design is appropriate to the conditions in the new location and that there is strong local support for the project. You may also wish to consider whether additional support from other clubs, districts, or cooperating organizations would make the Global Grants Monitoring and Evaluation Toolkit (April 2012) 10

11 project more effective. While scaling up may be an effective way to increase the impact of a successful project, it is important to revisit the issue of sustainability before making modifications to an existing project design. SELECTING FROM TRF STANDARD MEASURES The TRF standard measures for each area of focus are listed on the following pages. All global grant project sponsors are required to select and collect data on the measure (or measures) that aligns with their project activities. Each table lists standard measures, definitions, and appropriate options for collecting data. BASIC EDUCATION AND LITERACY MEASURES The Rotary Foundation promotes education and literacy for all by Ensuring that children have access to quality basic education Reducing gender disparity in education Increasing adult literacy Strengthening the capacity of communities to support basic education and literacy Supporting studies related to basic education and literacy Measure Definition Measurement Options Total number of direct beneficiaries Direct beneficiaries = individuals who receive a clear and immediate benefit from project activities benefiting schoolaged children new school aged children new female school aged children Do not simply report on the number of individuals living in a specific community or region. School aged children = students 18 or under attending primary or secondary schools. Count each student once. School aged children = students 18 or under attending primary or secondary schools. New = students who were not previously enrolled in school and are now attending as a direct result of the project School aged children = students 18 or under attending primary or secondary schools. New = students who were not previously enrolled in school and are now attending as a direct result of the project Public records Public records Public records Global Grants Monitoring and Evaluation Toolkit (April 2012) 11

12 Measure Definition Measurement Options teachers receiving training in adult education Training = instruction focused on a specific topic or curriculum; may be conducted multiple times. Adult education = training in basic skills, such as reading, writing, and proficiency in local language, for students 18 and older adults receiving literacy training institutions participating in program adults using new literacy skills new teaching positions created Count each participant once, regardless of how many times the training is offered or how many trainings a participant attends. Training = instruction focused on a specific topic or curriculum; may be conducted multiple times Adult literacy training = training that either specifically targets reading and writing skills to students 18 and older or provides skills for use by teachers in educating adult students Count each participant once, regardless of how many times the training is offered or how many trainings a participant attends. Institution = entity involved in the direct provision of basic education (grades 1 6) and/or adult education (students 18 and older) Program = training in basic skills such as reading, writing, and proficiency in local language Using = utilizing the skills provided through the project in essential daily interactions, such as employment Adults = individuals over 18 Count each beneficiary once. New = teaching position that didn t previously exist and directly results from the project Teaching position = positions located in schools or formal community based education facilities Focus groups and interviews Questionnaires and surveys Public records Global Grants Monitoring and Evaluation Toolkit (April 2012) 12

13 DISEASE PREVENTION AND TREATMENT MEASURES The Rotary Foundation reduces the causes and effects of disease by Improving the capacity of local health care professionals Combating the spread of HIV/AIDS, malaria, and other major diseases Enhancing the health infrastructure of local communities Educating and mobilizing communities to help prevent the spread of major diseases Supporting studies related to disease prevention and treatment Measure Definition Measurement Options Total number of direct beneficiaries Direct beneficiaries = individuals who receive a clear and immediate benefit from project activities medical and health professionals trained individuals reporting better quality of health care services recipients of disease prevention intervention Do not simply report on the number of individuals living in a specific community or region. Medical and health professionals = doctors, nurses, midwives, medical students, technicians, community health workers, and health educators Training = instruction focused on a specific topic or curriculum; may be conducted multiple times Count each participant once, regardless of how many times the training is offered or how many trainings a participant attends. Health care services = preventive measures, such as education or screening, and treatments intended to cure or alleviate the effects of a disease, such as the provision of drugs (e.g., antiretrovirals) Better quality = more affordable, of a higher quality, more efficient, more effective, or greater variety of treatment options Recipients = individuals who received services as part of the activity intended to prevent and/or treat a specific disease Do NOT simply report on the number of individuals living in a specific community or region. Public records Questionnaires and surveys Questionnaires and surveys Global Grants Monitoring and Evaluation Toolkit (April 2012) 13

14 Measure Definition Measurement Options communities reporting decrease in cases of disease(s) targeted Communities = individual villages, geographic areas, or medical institutions. Cases of disease = the number of new disease incidences within a specific time period. Public records benefiting health facilities Communities reporting an increase in access to local health facilities healthfocused events health educational campaigns Health facilities = locations where health care is provided, including hospitals, clinics, laboratories, etc. Communities = individual villages, geographic areas, or medical institutions Health facilities = locations where health care is provided, including hospitals, clinics, laboratories, etc. Health focused event = event (health fair, training, disease screening, etc.) that provides attendees with information or services intended to increase their awareness about how to prevent and/or treat disease Health educational campaigns involve disseminating reliable, fact based information intended to increase awareness of how to prevent and/or treat a disease. Such campaigns may consist of information sessions or involve a training curriculum, but others may involve use of posters, billboards, radio, and other media to educate community members. Public records Questionnaires and surveys Global Grants Monitoring and Evaluation Toolkit (April 2012) 14

15 ECONOMIC AND COMMUNITY DEVELOPMENT MEASURES The Rotary Foundation invests in people to create measurable and enduring economic improvement in their lives and communities by Strengthening the development of local entrepreneurs and community leaders, particularly women, in impoverished communities Developing opportunities for decent and productive work, particularly for youth Building the capacity of local organizations and community networks to support economic development Supporting studies related to economic and community development Measure Definition Measurement Options Total number of direct beneficiaries Direct beneficiaries = individuals who receive a clear and immediate benefit from project activities entrepreneurs supported businesses supported youth trained Do not simply report on the number of individuals living in a specific community or region. Entrepreneurs = current or potential owners of small businesses or those engaged in informal income generating activities Support indicates that entrepreneurs actively participated in project activities. Businesses = income generating activities, small businesses, cooperatives, and social enterprises Support = active participation of business owners or workers in project activities Youth = children ages Training = instruction focused on a specific topic or curriculum; may be conducted multiple times Training should focus on economic topics such as income generation, financial literacy, vocational training, entrepreneurship, etc. Count each participant once, regardless of how many times the training is offered or how many training sessions a participant attends. Global Grants Monitoring and Evaluation Toolkit (April 2012) 15

16 Measure Definition Measurement Options people Training = instruction focused on a specific topic trained or curriculum; may be conducted multiple times Training should focus on economic topics such as income generation, financial literacy, vocational training, entrepreneurship, etc. training programs implemented jobs created youth employed in income generating activities Count each participant once, regardless of how many times the training is offered or how many training sessions a participant attends. A training program focused on a specific topic or curriculum may be conducted multiple times. Count the program once, regardless of how many times it is offered. Count only positions created (did not exist before the project and can be directly attributed to it). Employment = formal positions as well as entrepreneurial and informal incomegenerating activities Youth = children ages Count only youth who participated in the project and as a result started new employment activities. Employer surveys/ questionnaires Survey Questionnaire Global Grants Monitoring and Evaluation Toolkit (April 2012) 16

17 MATERNAL AND CHILD HEALTH MEASURES The Rotary Foundation improves the lives of mothers and their children by Reducing the mortality rate for children under the age of five Reducing the maternal mortality rate Improving access to essential medical services and trained health care providers for mothers and their children Supporting studies related to maternal and child health Measure Definition Measurement Options Total number of direct beneficiaries Direct beneficiaries = individuals who receive a clear and immediate benefit from project activities children under age 5 receiving medical treatment mothers receiving prenatal care Do not simply report on the number of individuals living in a specific community or region. Medical treatment = the treatment of existing medical conditions, and preventive services Examples of treatment include: Immunizations Micronutrient supplements Oral rehydration therapy for diarrhea Antibiotic treatment for pneumonia and other respiratory infections Antimalarial drug treatment (e.g., chloroquine, artemisinin) Antiretroviral drug treatment Prenatal care = at least four visits with a trained health care worker over the course of a woman s pregnancy Each visit can include:1 Monitoring of progress of pregnancy and assessment of maternal and fetal well being Detection of problems complicating pregnancy Birth planning, advice on danger signs and emergency preparedness Public records Public records Questionnaires and surveys 1 WHO Recommended Interventions for Improving Maternal and Newborn Health Global Grants Monitoring and Evaluation Toolkit (April 2012) 17

18 Measure Definition Measurement Options maternal and child health professionals trained Training = instruction focused on a specific topic or curriculum; may be conducted multiple times Count the program and each participant once, regardless of how many times the training is offered or how many trainings a participant communities that report decreased mortality rates of children under age 5 communities that report decreased mortality rates of mothers attends. Communities = individual villages, geographic areas, or medical institutions Child mortality rate = number of children who die before reaching age 5 per 1,000 live births2 Communities = individual villages, geographic areas, or medical institutions. Maternal mortality rate = number of maternal deaths per 1,000 live births Public records Public records benefiting health facilities Maternal death = death of a woman while pregnant or within 42 days after pregnancy ends3 Health facilities = locations where health care is provided, including hospitals, clinics, birthing centers, etc. Improvements can include training of medical personnel and patients, provision of medical equipment and/or education materials, addition of exam rooms, supply of medical consumables, etc. 2 United Nations Millennium Development Goals 3 World Health Organization Global Grants Monitoring and Evaluation Toolkit (April 2012) 18

19 PEACE AND CONFLICT PREVENTION/RESOLUTION MEASURES The Rotary Foundation promotes the practice of peace and conflict prevention/resolution by Strengthening local peace efforts Training local leaders to prevent and mediate conflict Supporting long-term peace-building in areas affected by conflict Assisting vulnerable populations affected by conflict, particularly children and youth Supporting studies related to peace and conflict resolution Measure Definition Measurement Options Total number of direct beneficiaries Direct beneficiaries = individuals who receive a clear and immediate benefit from project activities groups and organizations participating in peace building efforts communities that reported reduction in level of conflict training programs implemented Do not simply report on the number of individuals living in a specific community or region. Group or organization = formal or informal collection of individuals who come together for a specific purpose Peace building efforts = activities to prevent and/or resolve conflicts Conflict = dispute that has, or may potentially have, broad negative impacts within a community or region Conflicts often arise due to political, religious, or ethnic differences and can lead to the marginalization of certain peoples or groups. Training = instruction focused on a specific topic or curriculum; may be conducted multiple times Count each participant once, regardless of how many times the training is offered or how many trainings a participant attends. Questionnaires and surveys Interviews and focus groups Questionnaires and surveys Global Grants Monitoring and Evaluation Toolkit (April 2012) 19

20 Measure Definition Measurement Options local leaders or community groups who received training Local leader = person who occupies place of status within the community, either in formal or informal roles Community groups = formal entities that come together for a specific purpose Training = instruction focused on a specific topic or curriculum; may be conducted multiple times conflicts mediated Count each participant once, regardless of how many times the training is offered or how many trainings a participant attends. Conflict = dispute that has, or has the potential to have, broad negative impacts within a community or region Mediation = process in which an impartial third party helps people resolve a dispute or plan a transaction Conflicts often arise due to political, religious, or ethnic differences and can lead to the marginalization of certain peoples or groups. Interviews and focus groups Questionnaires and surveys Global Grants Monitoring and Evaluation Toolkit (April 2012) 20

21 WATER AND SANITATION MEASURES The Rotary Foundation ensures that people have sustainable access to water and sanitation by Increasing equitable community access to safe drinking water and basic sanitation Strengthening the ability of communities to develop and maintain sustainable water and sanitation systems Educating communities about safe water, sanitation, and hygiene Supporting studies related to water and sanitation Measure Definition Measurement Options Total number of direct beneficiaries Direct beneficiaries = individuals who receive a clear and immediate benefit from project activities Number AND percentage change of people with access to improved drinking water supply Do not simply report on the number of individuals living in a specific community or region. Improved sources of drinking water include: Piped water into dwelling Piped water to yard/plot Public tap or standpipe Tubewell or borehole Protected dug well Protected spring Rainwater Unimproved sources of drinking water include: Unprotected spring Unprotected dug well Tanker truck Car with small tank/drum Surface water Bottled water (considered to be improved only when the household uses drinking water from an improved source for cooking and personal hygiene)4 Questionnaires and surveys Public records 4 WHO/UNICEF Joint Monitoring Programme for Water Supply and Sanitation Global Grants Monitoring and Evaluation Toolkit (April 2012) 21

22 Measure Definition Measurement Options Number AND percentage change of people with access to improved sanitation facilities Improved sanitation includes: Flush or pour/flush facilities connected to a piped sewer system or septic system Pit latrines with a slab Ventilated improved pit latrines Composting toilets Unimproved sanitation includes: Flush or pour/flush toilets without a sewer connection Pit latrines without slab Bucket latrines Hanging toilets/latrines Questionnaires and surveys Public records Number AND percentage increase of people with access to disinfected water through point of use technologies Number AND percentage of water facilities in use 1 3 years after project completion people trained No facilities; open defecation5 Point of use technology = household water treatment options, such as boiling, chlorination, filtration, and solar disinfection Eligible treatment options are those proven to be effective in disinfecting water and decreasing diarrheal diseases. Water facility = a system that provides water to one or more persons, either at the household or community level In use = still functioning properly, being used regularly and appropriately, and maintained consistently by the beneficiary population Training = instruction focused on a specific topic or curriculum; may be conducted multiple times Count the program and each participant once, regardless of how many times the training is offered or how many trainings a participant attends. Questionnaires and surveys Testing.. Questionnaires and surveys. Public records... 5 WHO/UNICEF Joint Monitoring Programme for Water Supply and Sanitation Global Grants Monitoring and Evaluation Toolkit (April 2012) 22

23 CREATING ADDITIONAL PROJECT-SPECIFIC MEASURES TRF standard measures provide The Rotary Foundation with a variety of valuable information, but they do not provide sufficient information to represent a project s impact. Most of the standard measures involve the counting of participants and activities, or outputs; which is an important first step in showing the extent of a project s scope. Few standard TRF measures, however, describe a project s actual impact on participant s lives, knowledge, or health. Global grant project sponsors must develop and incorporate additional projectspecific measures. After you have chosen TRF measures that best represent core project activities, create additional measures to collect the necessary quantitative and qualitative data to demonstrate the project s success. Quantitative (numeric) data can be counted, like hours of instruction in a vocational training project, the number of graduating students, and the number of students placed in jobs. Qualitative (descriptive) data provides a view into the effectiveness of project activities. Using a vocational training team funded by a global grantas an example, descriptive data may include: Demonstrating the level of knowledge gained by students through pre- and posttesting Tracking the increase in income of students following completion of the training program Using a yearly survey to show an increase over time in student satisfaction with the instruction provided When choosing TRF measures and creating additional project-specific measures, project sponsors should consider which measures relate most strongly to the project s highest priority goals. It will be most effective to incorporate into the monitoring and evaluation plan only a modest number of measures that clearly link to specific goals. By carefully selecting only the most important measures, sponsors will ensure that monitoring and evaluation activities are conducted in a meaningful and efficient manner throughout the life of the project. Global Grants Monitoring and Evaluation Toolkit (April 2012) 23

24 CREATING A MONITORING AND EVALUATION PLAN The following chart shows what information project sponsors should include in their own monitoring and evaluation plan. A blank template of a monitoring and evaluation plan is available online. The sponsors can begin by identifying the list of project measures, both by selecting from the standard measures on the previous pages and by creating project-specific measures. For each measure, the specific definitions, targets, descriptions and schedule will need to be determined. MONITORING AND EVALUATION PLAN EXPLANATION Measure Definition Target Description of Method Schedule Identify and include the applicable standard measures developed by TRF. Include the definitions for the standard measures found in the monitoring and evaluation Establish a specific numeric value that represents the expected, unduplicated target for the Describe how measurement data will be collected. Use the measurement options listed in the monitoring and evaluation toolkit as a guide. Indicate the specific timeline you will use to collect measurement data. Identify and include additional projectspecific measures toolkit. Clearly define terms used in the projectspecific measures. Use the definitions of standard measures as a guide. full grant period. Establish a specific numeric value that represents the expected unduplicated target for the full grant period. Describe how measurement data will be collected. Use the measurement options listed in the monitoring and evaluation toolkit as a guide. Indicate the specific timeline you will use to collect measurement data. A sample of a completed plan, which incorporates both TRF measures and additional projectspecific measures, is also provided for reference: Global Grants Monitoring and Evaluation Toolkit (April 2012) 24

25 SAMPLE MONITORING AND EVALUATION PLAN Brief project description: To prepare youth for formal and informal employment and increase incomes of the participating youth. Measure Definition Target Description of Method Schedule Total number of direct beneficiaries youth trained youth employed in incomegenerating activities Direct beneficiaries are individuals who receive a clear and immediate benefit from project activities. Training will focus on job preparedness, job seeking skills, and entrepreneursh ip. Each participant will be counted once, but will participate in a yearlong training program. Youth are defined as ages Employment includes both formal employment positions as well as incomegenerating activities initiated by entrepreneurs. Youth are defined as ages (60 new, unduplicated youth involved in project activities each year for three years) 180 (60 new, unduplicated youth involved in training each year for three years) 60 (20 youth employed in formal positions and 40 youth engaged in entrepreneurial incomegenerating activities) Participant records will be maintained. Training will occur over a sixmonth period, after which participating youth will receive ongoing mentoring. Attendance records will be maintained for all training sessions. Active and inactive participants will be contacted via survey to determine their employment status. Only those youth who indicate formal or informal employment in more than one survey will be counted. Monthly compilation of participant records related to training. Quarterly mentoring reports. Attendance records will be compiled on a rolling basis throughout the six months of training conducted each year. Surveys will be conducted for all participants every six months. Global Grants Monitoring and Evaluation Toolkit (April 2012) 25

26 Measure Definition Target Description of Method Schedule youth receiving completion certificates from training program youth who benefited from a mentoring relationship Completion requirements include participating in a minimum of 190 training hours and a passing grade for each training module. Youth are defined as ages Mentoring involves an assigned professional who will spend one hour per week supporting a participating youth. Youth are defined as ages (total training hours = 240) Eighty percent of participants will attend at least 190 hours of training and receive a passing grade for each training module. 90 (mentoring hours necessary to reach minimum requirements = 24) Attendance records will be maintained for all training sessions. Pretests will be administered and results used to tailor training content. Post tests will be administered and graded. Satisfaction surveys will be administered for qualitycontrol purposes. Mentoring reports will be collected from both mentors and youth. Reports will include hours of interaction, mentoring topics, and participant satisfaction data. Attendance records will be compiled on a rolling basis throughout the six months of training conducted each year. Pre and post tests will be administered at the beginning and end of each monthly training module. Satisfaction surveys will be administered quarterly. Mentoring reports will be collected quarterly. Global Grants Monitoring and Evaluation Toolkit (April 2012) 26

27 RESOURCES ROTARY RESOURCES Areas of Focus Statements of Purpose and Goals /AreasofFocus.aspx Rotary s Areas of Focus (965) Principles of Sustainability n.pdf Global Grants Overview _global_grants.aspx Grant Management Manual EXTERNAL RESOURCES Monitoring and Evaluation Fundamentals: A Self-Guided Mini-Course Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP) Global Grants Monitoring and Evaluation Toolkit (April 2012) 27

28 GLOSSARY Area of focus goals have been established by TRF to help define the funding guidelines for global grants. Business is broadly defined as an income-generating activity, small business, cooperative, or social enterprise. Cases of disease can be measured by cumulative incidence (the number of new cases within a specific time period). Community is broadly defined as an individual village, geographic area, or medical institution. Community assessment is a method of collecting information on resources and needs, whether and how issues are being addressed, and actions likely to affect community improvements. Data includes facts, statistics, or items of information. Quantitative data (numeric) provides a view into the effectiveness of project activities. Qualitative data (descriptive) is gathered through stories, statements, or opinions. Baseline data is collected before project implementation to provide a clear picture of the situation before any intervention. Routine data is collected continuously, typically by a non-rotary entity. Nonroutine data is collected less frequently and is typically initiated by Rotarians for the purposes of monitoring and evaluating a project. Data collection is the process of identifying and gathering data from sources directly related to project activities; collection methods include direct observation of activities, outputs, or impacts during a project site visit. Direct beneficiaries are individuals who receive a clear and immediate benefit from project activities. See data collection. Double counting occurs when an individual or group is counted more than once when documenting beneficiaries of project activities. Employment can include both formal positions as well as entrepreneurial and informal income-generating activities. Entrepreneur is a current or potential small-business owner or a person engaged in informal income-generating activities. Evaluation is the assessment of the impact that project activities have on the beneficiaries and/or the degree to which project goals were met. Global Grants Monitoring and Evaluation Toolkit (April 2012) 28

29 Focus group/interview are data collection tools involving in-depth conversation with individual (interview) or multiple (focus group) beneficiaries where specific questions are posed by the interviewer/facilitator in order to collect specific information. are documents kept by the project sponsors on project participation, activities, and impacts for Rotarian, RI/TRF, and other s uses. Health care services include preventive measures, such as education or screening, and treatments, such as the provision of drugs (e.g., antiretrovirals) intended to cure or alleviate the effects of a disease. Health education campaigns involve disseminating reliable, fact-based information intended to increase awareness of how to prevent and/or treat a disease. Campaigns may consist of information sessions or involve a training curriculum, but others may involve use of posters, billboards, radio, and other media to educate community members. Health-focused events include events where attendees receive information or services intended to increase their awareness about how to prevent and/or treat disease (health fairs, trainings, disease screenings, etc.). Health facilities are locations where health care is provided and can include hospitals, clinics, laboratories, etc. Impact is the direct effect that a project or activity has on a person, community, etc. Jobs created are positions that did not exist before the project and that can be directly attributed to the project. Literacy trainings specifically target adult students reading and writing skills. Measure is a description of a project s actual impact on participant s lives, knowledge, or health. TRF standard measures, established by The Rotary Foundation, provide standardized information about global grant activities for each area of focus. In addition, project-specific measures are established to complete the data necessary to demonstrate the project s success, and will likely combine numeric and descriptive data. Mediation is a process in which an impartial third party helps people resolve a dispute or plan a transaction. Medical and health professionals include doctors, nurses, midwives, medical students, technicians, community health workers, and health educators. Monitoring is the routine process of collecting specific data as agreed upon before project implementation. Global Grants Monitoring and Evaluation Toolkit (April 2012) 29

30 Mortality rates See survival rates. Nonroutine data See data. Outcomes are the expected positive changes in the knowledge, behaviors, or circumstances of project beneficiaries as a result of the project. They are established before the project is implemented. Outputs are the activities conducted during the implementation of a project in order to reach its expected outcomes. Point of use technology includes household water treatment options proven to be effective in decreasing diarrheal diseases, such as boiling, chlorination, filtration, and solar disinfection. Prenatal care is at least four visits with a trained health care worker over the course of a woman s pregnancy. Each visit can include:6 Monitoring of progress of pregnancy and assessment of maternal and fetal well-being Detection of problems complicating pregnancy Birth planning, advice on danger signs and emergency preparedness Preventive medical treatment can include immunizations, micronutrient supplements, oral rehydration therapy for diarrhea, antibiotic treatment for pneumonia and other respiratory infections, administration of antimalarial (e.g., chloroquine, artemisinin, etc.) or antiretroviral drugs. Project goals are statements of what the project sponsors hope to accomplish through the project and the desired long term outcomes. Public records are documents containing data available to the public from governments, private entities, or other organizations. Questionnaires and surveys are assessment tools that use specific questions to gather data on attitudes, impressions, opinions, satisfaction level, etc., from individuals in a beneficiary population. Replication occurs when a project is repeated with a similar design in the same or a new location. Satellite imagery is digital data obtained by using a global positioning device. Scaling up is the process of increasing proportionally the size and scope of a project or activity to serve a wider region or population. 6 WHO Recommended Interventions for Improving Maternal and Newborn Health Global Grants Monitoring and Evaluation Toolkit (April 2012) 30