Development Priorities Window Call for expressions of interest: impact evaluations in environment, governance, infrastructure and public finance sectors Issue date: 2 December 2015 Deadline for questions: 18:00 Indian Standard Time, 4 January 2016 Application submission deadline: 18:00 Indian Standard Time, 18 January 2016 1. Introduction The International Initiative for Impact Evaluation (3ie) requests research organisations or consortia to submit expressions of interest (EOI) for undertaking one or more impact evaluations under 3ie s new Development Priorities Window (DPW). This window is aimed at promoting rigorous and policy-relevant impact evaluations in a variety of development sectors where high-quality evidence is scarce. In this first call (DPW1), 3ie invites applications for support to produce rigorous and policy-relevant impact evaluations of programmes or policies in any of these four sectors: environment, governance, infrastructure or public finance to achieve relevant sustainable development goals (SDGs). 1 Proposed impact evaluations must be of interventions undertaken in a low- or middle-income country as classified by the World Bank. 3ie aims to make approximately 18 awards of up to a total value of US$7.5 million through this window. 2. Scope of impact evaluation The selected DPW1 sectors are intrinsically linked to the SDGs, which are expected to be used by countries to set and manage their development priorities over the next 15 years. Table 1 maps DPW1 target sectors to relevant SDGs. 1 These sectors are specific only to DPW1. Subsequent DPWs will focus on other sectors.
Table 1: Target sectors for DPW1 and their relevance to the SDGs Sector Sustainable Development Goals Environment Goal 6: Ensure availability and sustainable management of water and sanitation for all. Goal 7: Ensure access to affordable, reliable, sustainable and modern energy for all. Goal 13: Take urgent action to combat climate change and its impacts. Goal 14: Conserve and sustainably use the oceans, seas and marine resources for sustainable development. Goal 15: Protect, restore and promote sustainable use of terrestrial ecosystems; sustainably manage forests, combat desertification, and halt and reverse land degradation and halt biodiversity loss. Governance Infrastructure Goal 16: Promote peaceful and inclusive societies for sustainable development, provide access to justice for all, and build effective, accountable and inclusive institutions at all levels. Goal 9: Build resilient infrastructure, promote inclusive and sustainable industrialisation and foster innovation. Public finance Goal 1: End poverty in all of its forms everywhere. (Target 1.3: Implement nationally appropriate social protection systems and measures for all.) Goal 17: Strengthen the means of implementation and revitalise the global partnership for sustainable development. (Target 17.1: Strengthen domestic resource mobilisation, including through international support to developing countries, to improve domestic capacity for tax and other revenue collection.) Below we present some important evidence gaps and references for each of the four sectors. It should be noted that the questions listed here and references are only illustrations of questions that may be addressed through proposed impact evaluations under DPW1 and are neither indicative of any preference nor represent an exhaustive list of possible questions for this window. Page 2 of 10
2.1 Environment In the environment sector, much of the focus has been on monitoring: collecting data on indicators of status and trend, such as pollution levels, habitat areas and management effectiveness scores. 2 For environment policy and programmes, the promise of experimental or quasi-experimental designs lies in their ability to complement non-experimental evaluations and intuition. Here are some of the questions that may be addressed by interventions in environment: How may scarce water resources be used more efficiently? What is the appropriate level of subsidies and taxes for reducing greenhouse gas emissions? What are the most effective ways to mitigate and adapt to adverse effects of climate change? How can firms and citizens be incentivised to adopt energy-efficient technologies in order to lower energy consumption and thereby reduce greenhouse gas emissions? What is the appropriate level of subsidies for promoting a sustainable market for renewable energy usage? What marketing methods are most effective in increasing adoption and use of renewable energy technologies? What is the magnitude of impact and trade-offs of renewable energy sources on employment and livelihoods? What is the impact of population awareness and behaviour change campaigns on adoption of energy-efficient technologies and/or on combating climate change? How can we limit human damage to marine ecosystems and what can be done to restore damaged ecosystems? What methods may help sustainably manage and restore healthy ecosystems such as forests, soils and wetlands? 2.2 Governance Governments spend billions of dollars each year on basic services and development programmes to improve the lives and well-being of vulnerable and marginalised populations. But the effectiveness of public spending requires good governance practices. Despite the crucial importance of good governance for development, many questions persist about effective governance and how it can be improved. Here are some illustrative questions 3 that may be addressed. a) Improvement in participation in political and policy processes Does decentralisation increase citizen participation, improve policy outcomes and increase electoral accountability? How? What types of election procedures and information aid in selecting leaders for improved policymaking and how? What characteristics do leaders need to have to ensure that governance has a positive impact on growth? How does greater democratisation lead to greater growth? What measures reduce barriers to political participation by women? What are the ways to improve citizen satisfaction and how does it affect policy? What are the measures that reduce elite capture of electoral processes? How does ethnic diversity affect policymaking? What kind and amount of information improves responsiveness of political actors, political parties and voters? b) Reducing leakages in public programmes What is the magnitude of leakages in public programmes? What are its key determinants? What is the impact of corruption on firm performance, government expenditures and provision of 2 Ferraro (2009) 3 These are primarily summarised from Olken and Pande (2011) Page 3 of 10
public goods and services? What compensation structures and non-monetary incentives for civil servants are effective in reducing corruption? What measures are likely to improve the capacity of civil servants to deliver public programmes? What kind of government monitoring mechanisms and community monitoring programmes are likely to reduce corruption and improve compliance? How should bureaucracy be structured so that leakages are reduced? How may technology reduce corruption? Which transparency measures reduce corruption and how? Under what conditions does transparency lead to greater accountability? 2.3 Infrastructure The number of impact evaluations of infrastructure interventions is increasing. However they are challenging because of difficulties in finding a counterfactual. 4 For this grant window, infrastructure as a sector includes roads, highways, ports, railways, power, irrigation, and information and communications technology. Here are some of the questions that impact evaluations in this sector may answer: How may access and quality of various types of infrastructure be improved? What is the effect of improving access on poverty? What incentive structures help to improve the maintenance of different types of infrastructure? What is the willingness to pay for building, maintaining and rehabilitating different types of infrastructure? What is the short- and/or long-term impact of building, repairing and maintaining various types of infrastructure on outcomes, such as traffic, transit time, transport costs, accidents, local growth, income level, household access rates to various social services, academic results, food availability, disease, mortality, job creation, firm growth, gender equality and labour mobility? 2.4 Public finance Public finance decisions on revenue and expenditure are critical for effective governance. 5 However it is challenging to assess the effectiveness of different public finance options. Recent improvements in impact evaluation techniques allow for increasingly reliable answers to questions in this sector. Additionally, a growing number of collaborations between public administrations and academics has facilitated the application of randomised evaluations and other quasi-experimental methods to questions in public finance, but many more need to be answered. 6 Here are some of the questions that can be addressed in the public expenditure area: What reforms in public finance management enhance the effective delivery of public goods? How can and how far should the private sector be involved in financing and providing services to poor and vulnerable populations? How can fiscal policy (expenditures, subsidies and taxes) be used to address inequality? What are the relative merits of various instruments such as cash versus in-kind, conditional cash versus unconditional cash transfers, transfers versus subsidies? What is the net effect of transfers under social protection schemes? 4 Estache (2010); McKenzie (2011) 5 We recognise that there is a significant overlap between governance and public finance and some of the questions listed under governance can be seen as falling under public finance and vice versa. We leave it to the research team to categorise their proposed impact evaluation to the sector(s) they think it fits best. 6 Pomeranz (2015) Page 4 of 10
Here are some illustrative evidence gaps on revenue raising 7 : What communication and auditing strategies are most effective for tax compliance among individuals or firms? What is the behavioural response of taxpayers to the tax structure? How does the variation in public goods provision affect tax compliance? How do tax compliance behaviours spread through social networks? What are the long-term impacts of tax compliance interventions, where the various actors are expected to undertake this repeatedly? What is the relationship between attitudes measured through surveys or in the laboratory and actual tax behaviour? How can impact evaluations integrate findings from behavioural science to improve attributes, such as timing, framing, complexity, tone or visual presentation, in non-deterrence approaches to tax compliance? 8 3. Characteristics of successful proposals Proposed impact evaluations must be of environment, governance, infrastructure, or public finance intervention(s) conducted in low- or middle-income countries. We seek impact evaluation proposals that clearly demonstrate the following attributes: Fill an important policy and research gap: Proposals should clearly justify why the proposed impact evaluation is important and how it potentially will be used to inform policies, strategies and/or decisions in a developing country context. If an EOI is selected, the full proposal should also include a short review of previous work in the area and indicate how the proposed impact evaluation will contribute to the discipline. Have a strong theory of change: All proposals should contain a description of a possible causal pathway between the intervention and the outcomes of interest. It should clearly lay out the assumptions behind each link and describe how the problem will be addressed. By incorporating mixed methods of data collection and analysis, we expect the impact evaluation study to answer the what, how, why, how much (magnitude of impact and cost) and for whom questions. We also expect them to assess unintended consequences. Be informed by formative work: Proposals will also be expected to indicate how past formative work has either informed the assumptions in the causal links in the theory of change or have failed to do so and what the proposed study will help to inform. Previous formative work, either undertaken by applicants or others, should be referenced to provide contextual information about the programme. It will be important for demonstrating the strength of the proposal. Use mixed methods: The full proposal should clearly lay out how an experimental or quasi-experimental design and a mix of quantitative and robust qualitative approaches and data sources will be employed and how analysis will be integrated to deliver a strong and rigorous impact evaluation. 7 Summarised from Hallsworth (2014) 8 The non-deterrence approach claims that the taxpaying decision does not rest solely on the financial decisions, but rather is influenced by factors such as social norms, perceptions of fairness, tax morale and the provision of public goods (Hallsworth 2014). Page 5 of 10
Demonstrate strong technical quality: Applicants will need to ensure that the technical quality of the proposal is high. If selected for phase two (see section 4), applicants will need to present detailed methods and timelines and demonstrate the ability and feasibility of using experimental or quasi-experimental methods to help measure attributable change in targeted outcomes and impact variables. Submitted full proposals must meet 3ie s definition of impact evaluation. They should have sound identification methods and should be adequately powered to measure outcomes. They should measure impact using counterfactuals that are constructed using experimental or quasi-experimental methods. They should be able to account for common threats to causal validity including programme placement bias, confounding, sample selection bias, performance biases, including spillovers, and contamination, as well as biases in outcomes data collection and reporting. Experimental designs involve random assignment of the intervention at individual or cluster levels. Quasi-experimental approaches employ statistical techniques to address the selection bias when the intervention is not randomly assigned. This includes instrumental variables estimation, discontinuity assignment, difference-in-difference and statistical matching approaches. Use of complementary approaches and methods: 3ie encourages complementing the experimental and quasi-experimental approaches with other approaches that are simulation-based to understand general equilibrium and secondary effects. 3ie also requires evaluation studies to be gender sensitive when the study question requires gender analysis for full understanding of results and be able to analyse relevant equityrelated effects. Full impact evaluations should also incorporate cost calculations for programmes to the extent possible to enable cost-related indicators to be computed. All impact evaluations must plan for process evaluation and good monitoring to ensure and track implementation fidelity. Please see 3ie s principles for impact evaluation. Have national researchers: All proposed research teams in the full impact evaluation proposal must have one or more nationals from the country in which the programme is being evaluated, preferably with one being a named principal investigator (PI), and demonstrate substantive roles for all of them throughout the study. All applicable ethical approvals must be granted before the collection of the intervention and endline data. Final study reports must be submitted to 3ie which will undergo peer review. Thereafter which grantees will have three months to produce the revised final report, which must include all analyses and results and be submitted in the 3ie report template and follow any content requirements. 4. Submission, review procedure and criteria All submissions must be made using 3ie s online Grant Management System. Submissions are requested in two phases: 4.1 Phase one: submission of an expression of interest A sample version of the EOI application form is available for reference on the 3ie website. Please note, that EOI will only be accepted through 3ie s online grant management system. Page 6 of 10
3ie has published a list of frequently asked questions (FAQ) on the 3ie website. All additional queries on the call for EOIs should be submitted to dpw1@3ieimpact.org with DPW1 query in the subject line by 18:00 Indian Standard Time on 4 January 2016. Response to the queries will be provided on the on 13 January 2016 in a Q&A document on the DPW page on the website. The deadline for submitting EOIs is 18:00 Indian Standard Time on 18 January 2016. All EOIs will be reviewed by 3ie. All applicants will be provided comments on their EOI applications. 4.2 Phase two: submission of the full proposal If an EOI is selected for the second phase, the full proposal must be submitted by 18:00 Indian Standard Time on 12 May 2016. Please note that an invitation to submit a full proposal is not a guarantee of an award. The provisional template for the full proposal is available here. All full proposals are reviewed by at least one internal 3ie reviewer and two independent external reviewers. Subsequently, a selection panel will work with 3ie to select the impact evaluations that will be provisionally awarded a grant. Final approval of awards is made by the 3ie Board of Commissioners. 3ie reserves the right not to make any award. 4.1.2 Review criteria Phase one: expression of interest Policy relevance and research gap addressed by the proposed impact evaluation Identification methods and use of rigorous design and mixed methods 25 Sector and impact evaluation experience of the team 25 Prior and proposed work with developing country researchers 20 3ie membership or associate membership of the research or implementing agency Weight (%) 25 Phase two: full proposal Weight (%) Qualifications of the research team 20 Capacity building of developing country researchers 10 Internal validity of the proposed method 30 External validity of results 10 Stakeholder engagement and potential for uptake of findings 20 Cost 5 3ie membership or associate membership of the research or 5 implementing agency 5 Page 7 of 10
5. Eligibility 1. EOIs must evaluate programmes or policies in at least one of the four DPW1 sectors. Multicountry applications are eligible, so long as all countries are listed as low- or middle-income countries. 2. Only legally registered organisations and consortia of registered organisations, not individuals, may apply. 3. Each proposal must be submitted by a single organisation that may include other organisations or team members as sub-grantees or sub-contractors (subject to 3ie s direct and indirect cost policies). 4. For-profit organisations are eligible to apply, but are restricted to the same indirect cost limits as non-profit organisations and may not charge a fee. 5. The applicant organisation must be able to sign the 3ie grant agreement. 6. Organisations may submit more than one EOI and may be included on more than one EOI. They should, however, have the capacity to implement any and all grants awarded to them under the window. 7. Researchers may be included in multiple applications, but must indicate in writing that they are available and able to carry out fully all of the stated time commitments concurrently in the case of multiple awards. 7. Evaluation teams should include nationals of the country in which the study is being implemented. National researchers must be resident in that country and working for an organisation that is registered in that country. They must be engaged in substantive tasks for the study during design, analysis, writing, stakeholder engagement and research communication for evidence uptake and use. 6. Timeline Expression of interests are due by 18:00 Indian Standard Time on 18 January 2016. If selected, full proposals are due by 18:00 Indian Standard Time on 12 May 2016. 3ie plans to announce provisional awards for grants on 29 July 2016. 3ie will host a two-day post-award workshop on 21-22 September 2016 for awardees to discuss revised proposals and 3ie requirements. At least one PI from the project team who fully understands the impact evaluation design and one member from the implementing agency must be available to attend the postaward workshop. The workshop will be held in New Delhi, India. Selected teams would be responsible for making their own travel arrangements for the workshop. 3ie will reimburse the cost of economy class return air or train fare for one PI from their home country and one representative of the implementing agency. 3ie will also make arrangements for accommodation and cover meals during the workshop. Any other expense during the workshop will be covered according to the 3ie cost policy, which will be shared with the selected team before the workshop. Grant signing is contingent on attendance at the post-award workshop and 3ie approval of requested revisions to the full proposal. The grant signing process will take three to six weeks after submission of the revised full application, depending on the degree of revisions requested to the proposed research designs. The summary timeline is given in the following Table 2. Page 8 of 10
Table 2: Provisional summary timeline Key activity or deliverable Dates (by 18:00 Indian Standard Time) Deadline for submitting queries on the call for EOIs (Q&A) 4 Jan 2016 Q&A document with responses to queries posted on the 3ie website 13 Jan 2016 Phase 1 deadline for submission of applications to EOI 18 Jan 2016 Decision about selection of EOIs announced via email 1 Mar 2016 Phase 2 deadline for submitting full proposal by selected EOI applicants from phase one 12 May 2016 Provisional impact evaluation awards announced via email 29 Jul 2016 Post-award workshop in New Delhi 21-22 Sep 2016 Revised full proposal submission due 7 Oct 2016 Expected date by which grant agreement is signed 9 Nov 2016 Final report due 31 Oct 2019* Revised final report due after incorporating reviewer comments 31 Jan 2020* *NB: To be discussed with 3ie. More information about this call can be found on the DPW webpage. Please direct any questions related to this call to dpw1@3ieimpact.org with DPW1 query in the subject line by 18:00 Indian Standard Time 4 January 2016. By 13 January 2016, a single document with all questions and answers will be made publicly available on the DPW page on the 3ie website. This call for EOIs does not constitute a guarantee of an award. References Estache, A, 2010. A survey of impact evaluations of infrastructure projects, programs and policies. European Centre for Advanced Research in Economics (ECARES) Working Paper 5 (2010) Ferraro, PJ, 2009. Counterfactual thinking and impact evaluation in environmental policy. New Directions for Evaluation 2009.122 (2009): 75-84. Hallsworth, M, 2014. The use of field experiments to increase tax compliance. Oxford Review of Economic Policy 30.4 (2014): 658-679. Mckenzie, D, 2011. Strategies for evaluating the impact of big infrastructure projects: How can we tell if one big thing works? World Bank blog <last accessed on 15 October 2015 http://blogs.worldbank.org/impactevaluations/strategies-for-evaluating-the-impact-of-biginfrastructure-projects-how-can-we-tell-if-one-big-thing> Page 9 of 10
Olken, BA and Pande, R, 2011.Governance Review Paper. J-PAL Governance Initiative Pomeranz, D, 2015. Impact evaluation methods in public finance. Harvard Business School, Working Paper 16-049 Page 10 of 10