Alpbach Technology Forum, The Efficiency of RTI Investments, 26 August 2011 EU RESEARCH : VALUE FOR MONEY? Wolfgang Burtscher DG Research and Innovation European Commission
Structure PART I. About the Framework Programme PART II. Evaluating EU R&I funding PART III. Illustration of evaluation evidence PART IV. Future strengthening of evaluation system
Part I About the Framework Programme
EU research: Characteristics of the Framework Programme (1) Most important multinational research programme world wide Project funding only True common put no national quota Competitive process with annual calls and peer review proposal evaluation Main focus on collaborative projects with partners from several countries Open to academia and industry alike Open to the world, not restricted to Europe
FP7 Structure Cooperation Collaborative research Multi-partner projects, top-down agenda Ideas Frontier Research Individual projects, bottom-up agenda Capacities Research Capacity Sharing of infrastructures People Marie Curie Actions Mobility programmes for post-docs, Training courses,
FP7 Budget ( million)
FP7 Facts and Figures (2007 to 2010) 59.000 proposals received, 312.000 applicants 12.000 proposals retained, 69.000 participants Out of these: Universities 30% Industry 25% Research Organisations 23% 16,6% of all participations are SMEs
Part II Evaluating EU R&I funding
Evaluating R&I investments is difficult Time lags some results have an impact in the longer term Multiple kinds of impacts R&I have an impact in several dimensions, which might not be easily identified: Scientific impacts Technological impacts and impacts on innovation Economic impacts Social impacts Environmental impacts Structural impacts on ERA Impacts on EU policies European Added Value Impacts on SMEs Attribution R&I is often funded from different sources Valuation it is difficult to provide a monetary value of impacts to make them comparable
New context for evaluating R&I in Europe - Economic crisis is threatening expenditures on research & innovation, which need to demonstrate high impacts and economic returns - Next Multiannual Financial Framework will be focusing on instruments with proven European Added Value and value for money
Over 20 years of efforts in evaluating Framework Programme FP 5 FP 6 FP 7 Annual monitoring 5 year assessment FP6 ex post FP7 Ex ante Impact Assessment CSF Ex ante Impact Assessment FP7 Interim Evaluation Thematic level evaluations National Impact studies 1998 2000 2002 2004 2006 2008
Framework Programme Evaluation System Consisting on ex-post evaluation and ex-ante impact assessment Increasingly geared to capture value for money Traditional evaluation methods: Expert panels, Interviews, Surveys of programme participants New methodologies: Linking of different data sources to programme participation (bibliometrics, networks analysis, innovation impacts) Ex-ante modelling of impacts
FP7 Interim Evaluation Carried out in 2010 Group of 10 external experts Covered FP as a whole Final report online http://ec.europa.eu/research/evaluations/ Political follow-up Commission Communication Council Conclusions Report by the European Parliament
FP7 Interim Evaluation: Key Message FP7 is on course and is clearly making a significant contribution to European science and the development of the European Research Area There are acknowledged difficulties in some aspects of its implementation, but it is important to applaud what is good about it
Evidence used for the Impact assessment of Horizon 2020 Robust quantitative and qualitative evidence : ex-post and interim evaluations (FP6, FP7, CIP, Marie Curie, ERC, ) foresight and forward looking studies statistical data (FP, CIP, Community Innovation Survey, ) analyses of science, technology and innovation indicators; (EC, ESTAT, OECD..) econometric modelling exercises; (NEMESIS, DEMETER ) reviews of academic literature on market and systemic failures and the impact of research and innovation, and of public funding for research and innovation; sectoral competitiveness studies; expert panels and expert hearings; etc. On-line surveys among FP and CIP beneficiaries.
Part III Illustration of evaluation evidence
Outputs and reach Framework Programme research funding supported thousands of projects and participants: - involving different types of top (A-team) research actors : - business enterprises, - research centres, - higher education institutions - NGOs - from a large number of countries and regions: - EU 27 Member States and their regions, - associated and candidate countries - 108 third countries including BRICs
FP Evaluation Studies: Top 50 Participants Top 50 Participants account for 25% of FP7 funding Strong indication that FP7 is not a closed shop Rank Organisation Name Countr y Participations EU Contribution in Mio 1 CENTRE NATIONAL DE LA RECHERCHE SCIENTIFIQUE FR 501 231,0 2 FRAUNHOFER-GESELLSCHAFT ZUR FOERDERUNG DER ANGEWANDTEN FORSCHUNG E.V DE 331 153,1 3 COMMISSARIAT A L' ENERGIE ATOMIQUE FR 234 118,7 4 MAX PLANCK GESELLSCHAFT ZUR FOERDERUNG DER WISSENSCHAFTEN E.V. DE 238 115,4 5 THE CHANCELLOR, MASTERS AND SCHOLARS OF THE UNIVERSITY OF CAMBRIDGE UK 215 97,8 6 ECOLE POLYTECHNIQUE FEDERALE DE LAUSANNE CH 165 97,4 7 THE CHANCELLOR, MASTERS AND SCHOLARS OF THE UNIVERSITY OF OXFORD UK 176 96,7 8 FONDATION EUROPEENNE DE LA SCIENCE SUP 9 93,5 9 EIDGENOESSISCHE TECHNISCHE HOCHSCHULE ZUERICH CH 170 91,7 10 IMPERIAL COLLEGE OF SCIENCE, TECHNOLOGY AND MEDICINE UK 175 86,9
Projects Additionality % of FP6 participants who did/would abandon the project without FP funding in comparison with national programmes participants Source: FP data based on 20 studies of additionality of EU support; national programme data based on studies for Member State programmes in Austria, Belgium (x2), Finland and Norway.
Leveraging private funding The multiplier effect of the FP7 Risk-Sharing Finance Facility, is expected to be over 30 between the EU contribution and the additional leveraged investment in RDI.
Impact on R&I capabilities Different FP ex-post impact studies demonstrate that EU funding, i.e.: increased FP participants ability to network, administrating and managing international projects (Study on FP6 behavioural additionality) increased skills and research capabilities of participants key research staff (Study of the impact of FP6 on new Member States) A UK evaluation of FP6 and FP7 found that the FP has a big impact on the nature and extent of UK researchers' international relationships and networks, as well as on their knowledge base and scientific capabilities.
Structuring effects: durable changes in the european RTDI landscape Scientific excellence is promoted via EU-wide, competitive ERC grants Creating research infrastructures of pan-european importance (ESFRI roadmap) Developing joint industrial strategic research agendas (ETPs and JTIs) Bringing together fragmented national research funding to tackle important societal challenges (ERA-NETs) Creating persistant R&I networks through the FP collaborative research Increasing the attractiveness of Europe as a place to carry out research (ERC, MCA) Influencing the design of Member State research policies, especially in the new Member States
Scientific impact - FP projects produce thousands of scientific publications - FP6 lead scientists have better scientific impact than scientists without FP participation (Study on FP6 network effects) - According to a German evaluation of FP6 scientific personnel participating in FP6 stated that a substantial part of their publications was due to their participation in the FP.
Impact on technology, innovation and competitiveness Firms motivation for participation: knowledge + networking opportunities, less direct commercialisation Nevertheless, FP produces large numbers of patents and innovations Framework Programme participants are more likely to apply for a patent :
Impact on growth & jobs On the basis of the NEMESIS econometric model, the long-term FP7 macro-economic impact was estimated at an extra 0.96 percent of GDP, an extra 1.57 percent of exports, and a reduction by 0.88 percent of imports The long-term employment impact of FP7 was estimated at 900,000 jobs, of which 300,000 in the field of research.
Policy impact Past Framework Programmes have an impact on the decision-making process, i.e.: At the international level, EU research related to climate change contributed to the International Panel on Climate Change Water and soil projects played a large role in the formulation and implementation of the Water Framework Directive. According to an Irish evaluation of FP6 each project counted, on average, 0.4 new or significantly improved regulation or policy.
Part IV Future strengthening of evaluation system
Monitoring & Evaluation system in Horizon 2020 Learning from the past, the new Framework Programme will have better monitoring and evaluation tools : - Comprehensive : evaluation coverage of all CSF action lines, with a detailed timetable for evaluation. updated and revised annually, taking into account new developments - Well-timed : A comprehensive interim evaluation in 2017 (3 years into the programme), and a full-scale ex-post Evaluation in 2023 (2 years after the end of the programme) - Harmonised : common templates, methodologies and indicators - Strong focus on throughput, output and impact - Supported by an appropriate data archive, experts, dedicated research activity, and increased cooperation with Member States and Associated States - Valorised through appropriate dissemination and reporting: transparency of the evaluation process is a key element of an overall strategy for full accountability.