An evaluation of Big Lottery Fund s International Funding Programmes. Janice Needham Ann Sanders Catherine Sexton

An evaluation of Big Lottery Fund s International Funding Programmes Janice Needham Ann Sanders Catherine Sexton January 2013

CONTENTS Executive summary i 1 Introduction 1 2 BIG s approach to international funding 2 3 Context 5 4 An overview of impact of BIG s international funding 12 5 Impact on health 15 6 Impact on human rights 22 7 Impact on livelihoods 25 8 Impact on natural resources 30 9 Impact on education 34 10 Programme effectiveness 39 11 Learning 42 12 Targeting future investment 46 13 Conclusions and recommendations 48 Appendices A Interviewees B Summary of projects funded C Outcomes for International Communities grants D Outcomes for International Strategic grants E Examples of impact F Measures of success G Comic Relief overview

EXECUTIVE SUMMARY This evaluation is a component of an overall review of the impact and future direction of Big Lottery Fund (BIG) s international funding. BIG is currently delivering two international funding programmes - International Communities and International Strategic - as part of its UK funding portfolio. The objectives for the evaluation are to: Assess the impact and effectiveness of BIG s international funding against the programmes outcomes. Identify as far as possible whether there have been other impacts not anticipated in the project or programme outcomes which have been beneficial or detrimental e.g. in terms of process, capability-building, learning between projects, influence etc. Analyse the distinct role that BIG s international programmes currently play within the wider funding landscape. Identify learning from international projects about impact, delivery, process or sharing of results which could be relevant for BIG s UK projects. Suggest how BIG could target any future investment in international programmes in light of the analysis of impact and learning. The evaluation reviewed: 90 International Communities (IC) grants totalling 39,049,734 in 38 countries 6 International Strategic (IS) grants totalling 26,583,254 in 14 countries Conclusions and recommendations from the evaluation are summarised below: Impact and effectiveness It is estimated that 1.7m people have benefited from 67 completed grants on International Communities programme, with the impact related to programme outcomes estimated as follows: 1.3m individuals have benefited from projects to improve mother and child, reproductive and mental health and to improve measures to prevent, control and mitigate impact from HIV/AIDS and other diseases 477,000 individuals have benefited from an improved environment in which to exercise their civil and political rights or being supported to campaign peacefully for civil and political rights 47,500 individuals/households have improved livelihoods 30,000 individuals/households have benefited from improved access to safe water and sanitation or more sustainable use of land 28,500 students have improved access to better and more relevant primary education and to date 4,389 girls have enrolled in project schools covered by the strategic grant (with this expected to rise to 14,000 by the end of the project) There is evidence of impact at an individual level, community or organisational level and strategic (policy and practice) level for all IC outcomes. In general the impact is more evident at an individual level. There is evidence of strategic influence at local, national, regional and international level within all IS grants, demonstrated most successfully by the grant to University College London s Institute of Child Health. The IC programme is highly effective at delivering change particularly for individuals and communities, whereas the IS programme is i

able to focus on all aspects of an issue, contributing to longer-term systemic change. Most projects target disadvantage: For health, about 1 in 4 IC projects target those living with HIV/AIDS. Other disadvantaged groups targeted include disabled people, pastoralists and migrant workers. Several projects are based in some of the poorest countries in Africa (Sierra Leone, Ethiopia, Zimbabwe and Zambia) while other target communities are those affected by the Asian Tsunami of 2004. Human rights projects are aimed at disadvantage within communities with more than half targeting including women or disabled people. Livelihoods and natural resources projects are based in some of the poorest countries in Africa (the Gambia, Malawi, Ethiopia, Zambia) plus countries those affected by the Tsunami. About a third of projects targeted women. Other groups supported include displaced families affected by conflict, poor pastoralists, migrant workers and indigenous people. Within each theme, there are clusters of projects that could inform future programming and provide opportunities for leaning and communication activities: Health - maternal and child health, HIV and AIDS, prevention and control of diseases Human rights - improving disabled people s rights; women s civil and political rights and reducing violence against women and girls. Livelihoods - projects which develop and strengthen trade federations, craft associations, producer groups and co-operatives Natural resources - related to sustainable use of land and natural resources and increased access to clean water Education projects which focus on inclusion There are various examples where there may be the potential for replication or scaling up, including a project in Rwanda setting up a fair trade network and strengthening producer groups and a project in Democratic Republic of Congo supporting children with disabilities. A project in Tajikistan focusing on trafficking and sexual exploitation may have some relevance to BIG s UK programmes. Factors that have contributed to the impact and effectiveness of the programme include: Learning cross-cutting themes length of funding International Grants Team performance The learning opportunities between partners in portfolio grants are highly valued by strategic grant holders and have improved the effectiveness of projects in many cases. There is no systematic approach to learning between IC projects. There is limited evidence of systematic cross-programme learning between UK and international programmes, although there is considerable potential for more. BIG s involvement in the Improve It Framework offers potential to support significant learning across the international sector. Comic Relief and BIG have many areas in common and are planning actively to explore options for joint working. ii J Needham, A Sanders and C Sexton January 2013

Comic Relief s learning questions offer a good model for a more structured approach to learning within BIG s international programmes. Cross-cutting themes are useful principles, contributing to effectiveness and there may be potential for wider use of these in UK programmes. Assessment of impact There is a wealth of information to support the assessment of impact, although this could be collected and held in a more structured and systematic form. There are various issues relating to information and methodology: BIG s role There is no framework for recording information within BIG supporting evaluation other than a case studies approach. An increased focus on outcomes when collecting and reviewing information is required, by grant holders and BIG staff. There is some evidence of difficulties in relation to data collection by organisations, in particular in relation to outcomes. Guidance on external evaluations would be of benefit to increase quality and usefulness to BIG. There is a methodological issue to resolve about counting households/individuals if this approach is to be used in the future. Recent research shows that BIG lies 4 th out of 90 UK based international development funding foundations. BIG is engaged with the principal stakeholders in the sector and there is strong support for the International Programme, but there is limited awareness of its impact and strategic focus. Previous research found that the key value of BIG s funding is seen to be in its broadbased approach; a reduced budget might prioritise marginalised communities; potential for BIG to support partnerships and work in collaboration with other funders; potential for BIG s role to develop in relation to linking UK grant holders and issues facing the UK and international communities. Future investment BIG should develop a clear rationale for its international programmes and a theory of change, incorporating cross cutting themes. BIG should develop its future programmes around target groups, identifying desired outcomes for each group. BIG should continue to run a mixed portfolio with both reactive and proactive funding strands. The reactive strand should have a greater focus with grants lasting up to 5 years and possibly up to 750,000. The proactive strand could include strategic initiatives such as collaboration with other funders such as Comic Relief, piloting social investment and increasing engagement with the private sector, as well as possibly replicating successful approaches identified in the reactive programme. BIG should develop its capacity to assess programme impact with a clear evaluation framework and identified learning questions. A greater focus on outcomes data and guidance on external evaluations would be of value. J Needham, A Sanders and C Sexton January 2013 iii

1 INTRODUCTION 1.1. This evaluation is a component of an overall review of the impact and future direction of Big Lottery Fund (BIG) s international funding. BIG is currently delivering two international funding programmes - International Communities and International Strategic - as part of the UK funding portfolio. 1.2. The objectives for the evaluation are to: Assess the impact and effectiveness of BIG s international funding against the programmes outcomes. Identify as far as possible whether there have been other impacts not anticipated in the project or programme outcomes which have been beneficial or detrimental e.g. in terms of process, capability-building, learning between projects, influence etc. Analyse the distinct role that BIG s international programmes currently play within the wider funding landscape. Identify learning from international projects about impact, delivery, process or sharing of results which could be relevant for BIG s UK projects. Suggest how BIG could target any future investment in international programmes in light of the analysis of impact and learning. 1.3. The evaluation methodology comprised: A desk review of relevant literature A desk review of project documentation for 90 grants from the International Communities programme and 6 grants from the International Strategic programme awarded between 2006 and 2009 (details on this methodology can be found in Section 5). Interviews with grant holders for each of the International Strategic programme grants including some national partners, BIG staff, a member of BIG s International Grants Committee and staff from Comic Relief (as a comparator grant maker). A list of interviewees is provided at Appendix A. 1

2 BIG S APPROACH TO INTERNATIONAL FUNDING Funding context 2.1. BIG s Mission and Values state that it is: committed to bringing real improvements to communities, and to the lives of people most in need. To achieve this, we have three values that underpin all our work: making the best use of Lottery money using knowledge and evidence being supportive and helpful 2.2. Prior to the merger of Community Fund and New Opportunities Fund, international funding was one component of Community Fund s remit and subsequently BIG has been encouraged, through its policy directions, to continue to fund international projects along broadly similar lines. Current policy directions state that organisations funded should include those with a base in the United Kingdom and working overseas. 2.3. The overall aim for the International Grant Programme is to fund effective projects that tackle the causes of poverty and deprivation and bring about a long-term difference to the lives of the most disadvantaged people. 2.4. A theory of change underpins the development of BIG s other funding programmes and is standard practice in international funding but is not explicitly documented for this programme. This evaluation assembled the key elements that would normally contribute to a theory of change from available documentation, i.e. the overall aims, programme aims and cross-cutting themes. 2.5. The International Communities (IC) programme outcomes are closely aligned with United Nations Millennium Development Goals (MDGs) and focus on improved: primary education for the most disadvantaged girls and boys health, specifically: mother and child health; mental health; reproductive health; prevention of HIV/AIDS and the mitigation of its impact, and prevention and control of major local diseases access to and the use of natural resources to benefit the most disadvantaged people ability for the most disadvantaged people to exercise their human rights livelihoods in both rural and urban settings to reduce poverty in a sustainable way 2.6. The IC programme funds projects in Africa, Asia (including the Pacific and Central Asian countries), the Middle East, Central and South America, the Caribbean, and parts of Eastern Europe. 2.7. Within the IC programme there is also a strand of funding for the longer-term reconstruction of areas affected by the Asian Tsunami of 2004. This funding prioritises projects that: restore livelihoods and empower communities J Needham, A Sanders and C Sexton January 2013 2

rebuild capacity of local organisations and communities to deliver services or advocacy serve the needs of particularly vulnerable groups, such as older people, disabled people or orphans. 2.8. The rationale for BIG s overall approach to its International Strategic (IS) funding was agreed by the Board in 2006. It was agreed that the strategic element should focus on particular MDGs: improved mother and child health improved education with a specific focus on girls improved measures to prevent HIV/AIDS and mitigate their impact and be targeted in areas that are unlikely to meet the MDGs: Sub-Saharan Africa, Bangladesh, Nepal, Pakistan and certain Indian states. 2.9. The rationale for basing funding priorities on the MDGs is that they provide an internationally recognised framework of priority development issues. Within this framework, focusing on health and education underpins all other efforts to increase economic and social wellbeing, and on making progress towards MDG One, the eradication of extreme poverty and hunger. These are also the areas where BIG feels its funding could have the greatest impact. Funding is also intended to build upon the learning derived from the legacy Community Fund programme and would have a greater potential for sharing learning, both overseas and with UK NGOs. 2.10. On both programmes, applicants are required to demonstrate the following crosscutting themes in their proposals: diversity participation influencing opinion capacity building collaboration sharing learning (IS programme only) These themes are used to assess the strength of project proposals but are not systematically monitored post-award. 2.11. In addition to BIG s current review of the future of its international funding, there has also been useful research conducted over the past few years. In June and July 2009 BIG staff, with the assistance of an academic intern, compiled research on issues worthy of consideration when planning future international funding. 1 The study highlighted key points from 14 areas of research, many of which are still relevant including: results of consultation: the key value of BIG s funding is seen to be in its broadbased approach; a reduced budget might prioritise marginalised communities; potential exists for BIG supporting partnerships and working in collaboration with other funders; potential for BIG s role in relation to linking UK grant holders and issues facing the UK and international communities the status of MDGs 1 Themes emerging from international future funding research - paper prepared for BIG International Committee. Emma Wethey July 2009 J Needham, A Sanders and C Sexton January 2013 3

the impact of climate change on disadvantaged communities sustainable development the complexity of the issue in relation to BIG s requirement to fund UK-based organisations arguments for and against projects applying for solely economic outcomes how BIG might make provision for people with disabilities in future the key health issues facing disadvantaged communities and effective approaches to these what has worked in BIG s education projects to date small grants BIG s role in building the capacity of international development organisations in the UK and overseas promotion of UK partnerships delivery model of International Communities and Tsunami programme promotion of the needs of women in BIG s funding 2.12. Funding distribution for the programmes evaluated is summarised in the table below. Appendix B provides a list of all grants included in this evaluation, along with the countries they work in. Appendix C provides information on each project s outcomes in the IC programme. Appendix D provides a summary of the IS grants. Programme and outcome areas International Communities Amount Number of grants 39,049,734 90 grants in 38 countries: 45 Africa 35 Asia 8 Central and S America 1 Middle East 1 Europe 40 (34 completed) 35 (32 complete) 36 (26 complete) 20 (10 complete) Health Human Rights Livelihoods Natural Resources Education 16 (12 complete) Strategic 26,583,254 6 grants in 14 countries (all active completing 2013/14) 4 Africa, 2 Asia Education 1 Mother and child health 2 HIV/AIDS 3 BIG s approach to international funding - key points There is no explicit theory of change underpinning the international programmes, though programme priorities and cross-cutting themes are key elements, for example the cross-cutting themes can be interpreted as pre-conditions to achieving outcomes. Millennium Development Goals provide a framework for programme outcome themes but these are very broad. Some useful research has been done already on possible themes to consider in future funding. J Needham, A Sanders and C Sexton January 2013 4

3 CONTEXT 3.1. In a paper to the UK Funding Committee of June 2012 2, BIG identifies the other most relevant actors in the context of UK funders of international development work to be the Department for International Development (DFID) and Comic Relief. The approach of both of these agencies is changing. 3.2. DFID 3 dispersed 599 million of bilateral aid through Civil Society Organisations (CSOs) globally in 2009/10, with 362 million of that going to UK based CSOs. To do this, DFID uses several mechanisms: the Girls Education Challenge Fund; Governance and Transparency Fund; the Civil Society Challenge Fund; UK Aid Match; the Global Poverty Action Fund (GPAF) and Programme Partnership Arrangements (PPAs). Of the above, only UK Aid Match and GPAF are still open to new applicants. DFID has recently consulted on initiatives to double its impact on numbers of people securing access to water, sanitation and hygiene facilities in countries that are very behind on this MDG, by the 2015 deadline. 3.3. In March 2011 DFID announced its priorities for the next four years, including the 27 countries in which it will concentrate resources and impact. 4 DFID s key funding mechanism to CSOs is the Programme Partnership Arrangement (PPA), through which 360m has been awarded to 41 NGOs for the period 2011-2014. These are strategic, multi-year, unrestricted grants (i.e. core funding) to CSOs who share a common vision and ethos with DFID, are able to deliver against agreed outcomes, and who represent value for money. Outcomes are in line with DFID priorities as articulated in DFID s Business Plan 2011 2015, their Results Framework and also, to some extent DFID s Structural Reform Plan of 2010. 3.4. DFID s other key funding mechanism for CSOs is the Global Poverty Alleviation fund (GPAF). This is project oriented i.e. restricted, and targets poverty reduction linked to addressing the MDGs, particularly the off-track MDGs. The Fund is delivered through 2 windows : the Impact window, with 36m delivered annually, up to a maximum grant of 4m, to fund large scale initiatives delivering results at a greater scale, but with lower risk; and the Community Partnership Fund, primarily supporting smaller organisations (i.e. annual income less than 1m) to reduce poverty at the community level. The current Fund runs until 2013, but is likely to be extended, probably to 2015. Applicants are encouraged to align project outcomes and indicators with DFID Country Strategies and Results Framework, making use of DFID standard indicators across a range of thematic sectors. 3.5. It is as yet unclear what will replace the PPAs when the current round expires in 2015. Current PPA holders, working through Bond 5, are putting a case to DFID re the value of strategic, unrestricted funding, and lobbying for it to continue in some form. An external evaluation of the PPA mechanism is being carried out by the recently established Independent Commission for Aid Impact (ICAI) and is due to report in March 2013, so it is unlikely that any clarity will emerge until after that date. 2 Report to UK Funding Committee presenting the Terms of Reference for the International Programme Review, Bryan,S and Manion, C 11 June 2012 3 Bryant,L and Pharoah, C. Global Grantmaking The Nuffield Foundation January 2012 4 http://www.dfid.gov.uk/aidreviews 5 The membership body for UK CSOs working in international development J Needham, A Sanders and C Sexton January 2013 5

BIG s own research 6 points to the possibility that GPAF may now become the focus of DFID s funding to CSOs, with a format similar to BIG s International Communities programme. This may be combined with a broadening of GPAF in terms of both outcomes and the size of organisations eligible to apply. Elsewhere in DFID there has been a move towards funding larger grants to consortia, with a focus on Maternal and Child Health, Basic Service Delivery and empowering women and girls. These consortia are more likely to be multi-country, thematic project funding, with an emphasis on large-scale beneficiary reach, rather than on transformational societal change through advocacy, for example, where change is more difficult to measure. It is also likely that the DFID emphasis on value for money will continue. Therefore, post 2014, most DFID funding may be restricted, not core funding, and made to larger agencies or consortia. 3.6. On a more general level, DFID strategy post 2015 may be informed by David Cameron s so-called golden thread of development, a concept which has featured in his speeches on development since 2005. Cameron has said: I am convinced that we need to focus more than ever on the building blocks that take countries from poverty to prosperity. It s about establishing the building blocks of democracy, the independence of the judiciary and the rule of law, with the majority prepared to defend the rights of the minority, the freedom of the media, a proper place for the army in society and the development of effective state institutions, political parties and wider civil society. 7 DFID has recently (2011) turned its attention to examining how the huge potential of private sector investment can be harnessed for international development, referring to it as the engine of development 8. 3.7. DFID, Comic Relief and BIG all use contribution to progress on off-track MDGs as a framework for their grantmaking, to varying extents. The Prime Minister co-chairs the High Level Panel established to consider what mechanism may be put in place after the MDGs expire in 2015 and, according to commentators such as Lawrence Haddad of the Institute for Development Studies: The Golden Thread may well become a very important organising principle for the London G8 Summit in June and for the High Level Panel on the post 2015 agenda. 9 3.8. Global Grant Making, a report for the Nuffield Foundation 10 also quotes a figure of total UK foundation international funding to UK and global civil society of 292m, just under 50% of that of DFID. The authors acknowledge that figures are a best estimate based on published information. In this analysis, BIG sits 4 th in a table of the 90 UK based international development funding foundations ( 14.4 m in 2009-10) after the Wellcome Trust ( 59m), Comic Relief ( 41m) and the Children s Investment Fund Foundation UK ( 22m). Comic Relief is due to launch a new international strategy in March 2013. It is likely that an increasing proportion of their 40 million annual international budget will be more proactively spent i.e. that they will seek out leading organisations and other grant makers to identify projects to fund and make more direct awards to African organisations. They are also likely to spend a greater proportion of funding on social investment through loans, loan 6 Ibid no 1 7 David Cameron's address to the United Nations General Assembly. Wednesday 26 September 2012 8 Ibid no 3 p46 9 http://www.developmenthorizons.com/2012/10/david-camerons-golden-thread-how-to.html - Institute of Development Studies blog by Lawrence Haddad 12/10/12 10 Ibid no 3 J Needham, A Sanders and C Sexton January 2013 6

guarantees and equity purchases rather than grants. Further information on Comic Relief is in Section 11. 3.9. Global Grant-making 11 shows that the funding priorities (by geographic region, theme and beneficiary) of the 90 listed foundations are similar to UK expenditure on bilateral international support. Africa receives most support (37% or 89.9m), followed by Asia (23% or 55.9m), then the Americas (13% or 31.6m). Within this, East Africa emerges as the area of greatest funding interest (22%) and then South Asia (17%). Spread by beneficiary group is: children & youth (37%); general disadvantage and poverty (18%); people with health/disability or special needs 18%; women and families 18%; elderly people and indigenous / displaced people both at 4%. The report notes a trend, observed since 2007, showing a significant increase in funders supporting disadvantaged people. It is thought this reflects funders concern about the scale of disadvantage in developing countries. Also noted is an increase in the number of foundations spending more than 50k a year on international development. Reasons for this include that many foundations have widened their criteria for inclusion; the establishment of new trusts and programmes and being able to build on earlier research and foundations increasingly acknowledging both their role in global citizenship and the transformational power which relatively low-level funding can have in a developing country context. 3.10. Global Grant-making also highlighted the following emerging trends in foundations that fund international development: A growing involvement that reflects foundations different motivations and interests A spending focus on similar areas to government and other official international development funding such as investing in health, education and sustainable development, particularly in Africa Responsiveness to emerging needs in international development by establishing new programmes and strategically reviewing and re-focusing existing programmes Maintaining a highly independent and diverse pattern of funding, tackling areas which become neglected or marginalised Increasingly tackling international development at both strategic and grassroots levels and working with multiple stakeholders and partners where this is the best way of achieving impact Targeting their relatively limited resources at the level of civil society where they can have most impact Providing sustained support for existing strengths in developing countries as well as sustainable forms of support for new ones 3.11. The Scottish Government recently announced 12 a number of changes to the operation of its International Development Fund, which focuses on South Asia and Malawi. The next round of South Asia funding will focus on Bangladesh, Pakistan and the Indian states of Bihar, Madhya Pradesh and Orissa and will be aligned with national and regional development strategies in the relevant countries and the achievement of the MDGs. Secondly, they announced their intention to launch a small grants funding round. The thematic and geographic focus are yet to be decided but consultation will take place over the coming months. 11 Ibid no 3 12 http://www.terravivagrants.org/home/view-grant-makers/government-organizations/scottish-internationaldevelopment-fund J Needham, A Sanders and C Sexton January 2013 7

3.12. BIG is currently investigating compliance with the International Aid Transparency Initiative (IATI), which aims to make information about aid spending easier to access, use and understand. It is a voluntary, multi-stakeholder initiative that has brought together donor countries, developing country governments, non-governmental organisations and experts in aid information. Together they supported the aspirations of the IATI Accra Statement of 2008 in improving the transparency of aid. The major international donors are signatories, including DFID. Impact context BIG DFID 3.13. As described in the previous section, BIG s approach to impact on its international programmes is to set broad outcomes for each programme. Applicants must demonstrate how their proposals will address these outcomes and reflect the crosscutting themes. Project milestones are required to demonstrate the activities that will support the achievement of the outcomes. There is no framework of indicators to which projects are required to adhere. Two other approaches are summarised below to provide a comparison. 3.14. DFID s Results Framework 13 is a tool to monitor and manage progress and report publicly on delivery. It sets out the development outcomes DFID is seeking to contribute to, the actual results to be delivered, and the metrics used to measure organisational effectiveness and efficiency. This is the first results framework to set out actual development outputs that DFID will deliver against and for which it can be held accountable. Previous DFID results and performance frameworks relied on monitoring and reporting progress against global development outcomes, in particular progress towards MDGs. DFID views these as less useful for measuring the specific contribution of DFID or other donors at country level. 3.15. The framework identifies a core subset of results that DFID will seek to influence over the next four years and a range of indicators to provide data on DFID s operational and organisational efficiency. It was informed by best practice in other donor institutions in particular the Asian and African Development Banks and the International Development Association. 3.16. The framework is organised into four levels that capture each main stage through which input are transformed into developing country results. Level 1 consists of indicators that represent development outcomes to which DFID is seeking to contribute in partner countries and which cannot be attributed to DFID alone. Level 2 contains indicators measuring outputs and intermediate outcomes that can be directly linked to DFID interventions. This can be best illustrated by taking one of the MDG indicators, such as education, down to level 2: Level 1 MDG 2: Achieve universal primary Net enrolment ratio in primary education Proportion of pupils starting grade 1 who reach last grade of primary 13 http://www.dfid.gov.uk/documents/publications1/dfid-external-results.pdf J Needham, A Sanders and C Sexton January 2013 8

education Level 2 DFID bilateral indicators Level 2 DFID multilateral indicators Literacy rate of 15-24 year olds Proportion of children that can read with sufficient fluency for comprehension in early grades Number of children supported by DFID in primary education (per annum) Number of children supported by DFID in lower secondary education Number of children completing primary education supported by DFID Number of teachers trained Number of teachers recruited Number of education ministry officials trained and coached in strategic planning and management 3.17. Level 3 contains indicators for monitoring DFID s operational effectiveness, leading to better delivery of results and greater value for money. The department will seek to report annually against the performance areas of: Portfolio quality a measure of the extent to which DFID s interventions are on track to delivery their expected outputs and outcomes Pipeline delivery data on DFID s pipeline of programmes either approved or under design to help assess whether sufficient plans are in place to achieve results commitments and spend its budget effectively Monitoring and evaluation data on the extent to which DFID is actively reviewing its programmes and learning lessons for the future Structural reform data to assess how well DFID is delivering against its corporate objectives and areas of coalition priority. 3.18. Level 4 indicators aim to monitor improvements in the efficiency of DFID s internal corporate processes to help inform capacity to provide more effective frontline delivery. The Improve It Framework 3.19. The Bond Effectiveness Team is leading a process of identifying common assessment methods for the sector. The Improve It Framework 14 will enable NGOs to assess, manage and report their effectiveness confidently and consistently across eight thematic areas and five ways of working. The first version of the online framework was due to be launched in autumn 2012. The framework will support organisations in identifying what they want to change, the evidence they will collect, how they will collect data and how to ensure the quality of date and analysis. It is based on indicators and tools currently being used by UK-based NGOs. 3.20. Taking education as a sample illustration again, there are 6 domains of change : Barriers to education are removed and all children access quality education where they learn useful knowledge and skills Governments and other mandated bodies ensure all children access quality education 14 http://www.bond.org.uk/pages/improveit.html J Needham, A Sanders and C Sexton January 2013 9

Schools are well managed, safe and well resourced All children receive their full allocation of high quality teaching Children participate in shaping their education and school life Communities actively support the education of all children These are supported by 16 outcomes for which a set of indicators and tools are suggested. BIG s current distinct role 3.21. BIG staff meet on a regular basis with Bond, DFID and Comic Relief currently the Bond Effectiveness Framework is high on the agenda. There has been a suggestion that BIG and Comic Relief might trial it. 3.22. As part of a review of its international funding prior to commissioning this evaluation, BIG carried out a piece of work to look at BIG s position within the international funding sector 15 and the key findings relevant to BIG s position have been extracted below. 3.23. The paper found that all those interviewed expressed significant support for BIG s International Communities programme; it was seen as crucial, valuable and needed and interviewees felt that it was symbolically significant that Britain has a role in global society. BIG s work and staff are respected in the sector and form part of relevant groups including the Advisory Board of DFID s GPAF and the Association of Charitable Foundations (ACF). 3.24. Funders felt BIG added variety to grant holders and to the funding landscape, also reflected by grant holders who valued BIG s broad remit. Most respondents had limited awareness of BIG s strategic focus, but agreed they were sensible and reflected the needs of the sector, though funders suggested these could be narrowed further to measure impact more effectively. Some respondents queried the wide-ranging list of countries eligible for funding, as most funders focus on specific countries or used the UN s index for the 50 least developed countries. None of the respondents were aware of the impact to date of BIG s IC Programme, although all funders stated that measuring impact in this area is a challenge, particularly aggregating data. Context - key points BIG sits 4 th in a table of the 90 UK based international development funding foundations. The funding landscape, most notably priorities beyond the MDGs, are under review and it will be critical to keep up to date with the direction of travel when considering how BIG can offer best value from its funding and complement other funding streams. There are useful comparator models available from DFID and Bond to support development of impact frameworks. BIG is engaged with the principal stakeholders in the sector, but there is limited awareness of its strategic focus, though there is significant support for the 15 Sappal, B. August 2012. BIG s Position Within the International Funding Sector. J Needham, A Sanders and C Sexton January 2013 10

International Communities programme. J Needham, A Sanders and C Sexton January 2013 11

4 AN OVERVIEW OF IMPACT OF BIG s INTERNATIONAL GRANTS 4.1. Our approach to assessing impact against programme objectives has been to group projects thematically by BIG s outcome areas, i.e. health, human rights, livelihoods, natural resources and education. This supported a review of projects from the IC and IS programmes alongside each other. 4.2. Tsunami projects were allocated to the relevant outcome area. There was only one project that did not lend itself to this approach, as its focus was on supporting communities to respond to and prepare for future emergencies. 4.3. It should be noted that projects have several outcomes and these often span themes, so many will feature in more than one theme. This is also due to many target groups being identified as having multiple needs (for example street children may have health and education needs and families may have livelihood needs). A thematic approach such as this does not support looking holistically at the needs and outcomes for the target group, but this is a function of the programme design. 4.4. The information available on IC grants was as follows: Merlin 16 reports completed by BIG staff summarising end of grant information, including whether outcomes were met a selection of end of grant reports provided by grantholders (21 readily available electronically) a selection of independent evaluations (20 readily available electronically) 4.5. For IS grants the information available was as follows for all six grants: Merlin reports (as above) independent mid-term evaluations end of year reports latest quarterly reports 4.6. For almost all completed IC grants, BIG staff had assessed whether outcomes had been met and recorded this. The only exceptions are where the grant is fully paid but the Merlin report has not yet been completed in this case this was assessed by considering the end of grant forms or external evaluations. It should be noted however that it seems that in a small number of cases BIG staff perhaps in the absence of any other information from the grant holder have had to make an assumption that if all milestones had been completed the outcome had been achieved. An extract from a Merlin report is provided to illustrate this point: Lead International - Building Sustainable Livelihoods on the Tsunami-affected Nicobar Islands Outcome 3 900 families are reporting increased awareness of health and sanitation issues and the benefits of health supplements: all milestones under this outcome have been achieved, therefore this outcome is considered to be fully achieved. Outcome 4 A minimum of 150 people from the target communities are reporting increased levels of participation in local decision-making processes with great and more frequent attendance of relevant meetings and increased levels of trust and confidence in decision makers: all milestones under this outcome have been achieved therefore this outcome is considered to be fully achieved. 16 Reports extracted from BIG s Merlin database J Needham, A Sanders and C Sexton January 2013 12

Assessing change 4.7. The researchers aimed to assess the change at an individual level by extracting information on the numbers of beneficiaries related to a particular outcome and collating these. Change at a community/organisational level was identified from the reports and summarised, as was any strategic level change in policy and practice. All Merlin reports were reviewed to extract this information and in many cases end of grant reports and independent evaluations were also used to clarify figures. It should be noted that in a small number of cases no quantitative information was available but BIG staff had recorded that the outcome has been fully met in these cases the figure proposed in the original outcome was used. In other cases changes were presented in other forms, for example income rises by 50%, with no figures on the number of individuals benefiting, so no data was available for collation. It should also be noted that the quality of end of project reports and external evaluations was variable, with some not necessarily clearly reporting on progress towards outcomes. 4.8. While accepting that this methodology makes a number of assumptions and is not completely robust, it does allow a quantitative overview of each theme. The results for each sector are summarised in the following sections. Identifying sub themes 4.9. Where possible, projects were grouped into subthemes and a further level of analysis carried out. The results of these are given in the following sections. Exemplar projects 4.10. For each theme a number of projects were reviewed in detail, drawing on the documentation above. These projects were selected to present examples of change at an individual, community and strategic level, as well as highlight where there may be potential for learning or replication. A longer selection of these is also provided in Appendix E. 4.11. The table below presents an overview across all themes (note that each grant has several outcomes which can cover different themes). It should also be noted that some outcome areas such as health or human rights include projects that are designed for breadth - such as raising awareness for high numbers of people - as well as projects that are aiming for depth such as those targeting one specific condition or target group and that this is reflected in the higher beneficiary estimates for those outcome areas. Programme and outcome areas Amount Number of completed grants Estimated number of beneficiaries International 32,973,985 67 grants - 54 met outcomes 1.7 million Communities Health 34 grants with health outcomes 1.3 million - 26 met outcomes Human Rights 32 grants with human rights 28 met outcomes 477,000 J Needham, A Sanders and C Sexton January 2013 13

Livelihoods 26 grants with livelihoods outcomes 24 met outcomes Natural 10 grants with natural Resources resources outcomes 10 met outcomes Education 12 grants with education outcomes-9 met outcomes Strategic 26,583,254 6 grants in 14 countries (all active completing 2013/14) Education 1 47,500 30,000 28,500 Mother and child 2 health HIV/AIDS 3 J Needham, A Sanders and C Sexton January 2013 14

5 IMPACT ON HEALTH 5.1. The overall impact achieved under the health outcomes is summarised here: Outcomes: International Communities - Improved health for the most disadvantaged people, specifically: Improved measures to prevent HIV/AIDS and mitigate its impact Improved mother and child health Improved reproductive health Improved measures to prevent and control locally significant diseases Improved mental health International Strategic: Improved measures to prevent HIV/AIDS and mitigate its impact Improved mother and child health 40 International Communities grants were awarded in 25 countries (i.e. almost half of all grants or 44%) contributing to these outcomes 5 International Strategic grants were awarded in 13 countries contributing to these outcomes. Disadvantage: About 1 in 4 IC projects target those living with HIV/AIDS. Other disadvantaged groups targeted by IC grants include disabled people, pastoralists and migrant workers. Several projects are based in some of the poorest countries in Africa (Sierra Leone, Ethiopia, Zimbabwe and Zambia) while other target communities affected by the Tsunami. Of the 40 IC grants, 34 grants are complete and 6 are in progress. Of those complete, 26 grants are identified / classified as having achieved their outcomes, 7 grants are identified as having partly achieved their outcomes and 1 is classified as not having met its outcomes. Impact Individuals: From the IC grants, there are an estimated 1.3 million individuals 17 who have had improvements to their health, although this benefit can range from increased awareness through to lives saved. The IS grants to date show: 38% decrease in neonatal deaths in the project populations in Bangladesh and India (UCL) An increase in the number of women accessing maternal health services and ante natal care in Uganda, Malawi and Ethiopia (Interact) over 198,000 people within mobile populations in India, Nepal and Bangladesh supported to reduce their vulnerability to HIV/AIDS (Care) Over 45,000 people have improved access to support services on HIV/AIDs in Southern Africa (Skillshare) Institutions: There is some evidence of impact from IC grants at this level, with a few examples of strengthening of health teams and increased knowledge and skill of health workers. Some examples of impact from IS grants are: Stronger links between local clinics and health services in Bangladesh (UCL) 17 Grants have a number of outcomes and so can span more than one theme J Needham, A Sanders and C Sexton January 2013 15

Almost 3,000 health workers trained in maternal and child health in Uganda, Malawi and Ethiopia Over 600 peer educators and outreach workers trained to support mobile populations in India, Nepal and Bangladesh Strategic policy and practice There is limited evidence of this from IC grants, however all IS portfolios can demonstrate evidence of influencing health policy and practice at regional, country and district level UCL has produced significant learning on improving maternal and child health in low income countries supported by robust evidence, with potential for influencing policy and practice at international level. Themes improved measures to prevent HIV / AIDS and mitigate its impact (14 IC and 3 IS grants contributing) improved maternal and child health / reproductive health / women s health (11 IC grants and 2 IS contributing) improved awareness, prevention and control of disease (9 IC grants contributing) improved health for people with disabilities (4 IC grants contributing) improved mental health (3 IC grants contributing) improved health for children and young people (3 IC grants contributing) Types of impact some illustrative examples International Strategic 5.2. A summary of projects within each portfolio and partners is provided in Appendix D As the strategic grants are still in progress, final conclusions on the extent to which outcomes have been met are still in development. An external mid-term review has been conducted for each grant. A review of evaluations to date and other information suggests that these grants are achieving impact in the following areas: Maternal and child health 5.3. Some examples of impact to date from UCL s portfolio include: A 38% decrease in neonatal deaths in the project populations A large increase in the coverage of women s groups for example in Bangladesh, the number has increased by a factor of five and an increase in attendance by 28%. Increased coverage and attendance at women s groups are understood as key reasons for the impact on neonatal mortality. Significant improvements in behaviour contributing to reduced mortality, such as earlier breastfeeding and improved basic hygiene practices. Women report feeling more supported and better informed about recognising potential health problems and also about accessing care. Stronger links with government health services and women are clearer about their entitlements. Strategic influence at local, country and international level, for example: J Needham, A Sanders and C Sexton January 2013 16

o In Bangladesh, there is increasing linkage between local clinics and health services and the project has contributed to the national guidelines on neo natal death and the maternal healthcare strategy o A project dissemination workshop was held for key organisations in the field of maternal and child health in London in November 2012 and a dissemination meeting was held in Bangladesh in January 2013 to discuss findings from the project which was attended by representatives from WHO, UNFPA and SCF. o The Good Practice Guides being produced by Women and Children First are attracting interest at international level, including from organisations working in sectors other than maternal and child health, such as water and sanitation. o There has been considerable media coverage and programme findings have been cited in various reports of leading development organisations, which have been disseminated widely. o A submission has been made to the UK Parliament s International Development Committee, and learning from the portfolio has contributed to reports for World Bank and the UN. 5.4. Transferrable learning from UCL s portfolio is significant and is being widely disseminated amongst relevant audiences. UCL s portfolio has had a particularly rigorous and robust process evaluation in place from the beginning. The study design is complex and requires technical expertise and funds but has enabled robust quantifiable evidence about what is attributable, how it works and what it costs. Randomised control trials and rigorous data systems are in place. A data safety monitoring board scrutinises evidence on a regular basis. There is potential for learning for other large-scale projects from the approach adopted by this portfolio. 5.5. Some examples of impact to date from Interact Worldwide s portfolio include: In some cases, the number of women utilising services relating to maternal health has doubled Cumulatively, 2,914 health workers and 1,000 community volunteers have been trained Ante natal care visits have increased significantly compared to baseline in all countries Useful learning from applying and adapting approaches and models across the three countries (Ethiopia, Malawi and Uganda) e.g. motorbike ambulances; maternity huts; support groups; linking community health volunteers with health centres; income generation. Strategic influence, for example: o There has been considerable interest in Ethiopia by government and donors in the motorbike ambulance model o A new government initiative in Ethiopia of using women educators is integrating with the project s women s groups o In Malawi, based on work at district level, service level agreements are being developed with government. Portfolios addressing outcomes in HIV / AIDS 5.6. Some examples of impact to date from Care International s portfolio include: J Needham, A Sanders and C Sexton January 2013 17