Indicators and Results of Analysis 1 Set of Indicators 1.1 First set of indicators 1. Efficiency Criteria Indicators / evaluation parameters Unit/scale Source Composite indicators Complexity of the establishment of work programmes Efficiency of the management/evaluati Number of staff involved in the elaboration of each work programme Time of elaboration of work programme Cost of the elaboration of each work programme Lifetime of work programme (main release) Number of sub programmes Efforts spent in the evaluation of proposals Full-time equivalent Months Months (delay between each work programme) Number of sub programmes Full time equivalent EC (database) Number of staff involved in the elaboration of each work programme / time of elaboration of work programme Time of elaboration of work programme / lifetime of work programme (main release) Cost of the proposal evaluation experts / number of proposal
on of work programmes (economies of scale in terms of EC management tools costs and resources) Efficiency of the evaluation of work programmes Synergies and avoidance of overlaps between different programmes Volume of proposal received Number of staff involved in the management of each work programme Number of projects running Cost of the evaluation experts and tools used by the EC Type of beneficiaries targeted by the programme Geographical coverage Number of proposals Full time equivalent Average number of projects Average breakdown percentage between participants type (SMEs/large industries/research/other) in budget allocated to the different participants R&D capabilities multiple choice: o Low o Medium o high Industrial sectors (NACE 2) Multiple choice: o Regional o National o EU27 o beyond EC (database) EC (database) EC (database) received Number of staff involved in the management of work programme / lifetime of work programme Cost of the evaluation experts / number of projects running Matrix of the gaps and overlaps between the different programmes Type of innovation funded Multi-choice: o Basic research (>10 y)
Coordination efforts between relevant stakeholders and departments Comparison with other regional and national innovation policy landscapes in large and regionalised countries (France, Germany, UK, Sweden, Finland, Italy, Spain, and the US) Type of funding Characteristics of project funded Outputs of the coordination meetings between the different DGs and between DGs and stakeholders Overall complexity of innovation policy landscapes in these countries Volumes of national programmes in these countries Agencies / Authorities coordinating o Applied research (5-10 y) o Development (3-5 y) o Industrialisation (<3 y) o Support (no RTD) o Mobility o Training o Other Multi-choice: o Grant o Co-funding o Loan Duration in months Cost in Consortium in number of partners EC contribution in Average annual number of meetings inter DGs Average annual number of meetings between DGs and external stakeholders Number of different innovation programmes in these countries Budget in Number of bodies involved in coordination/management of these Programmes Regional authorities (questionnaire) Regional authorities (questionnaire) Regional authorities (questionnaire)
Simplification effects through uniform terminology, rules, information tools, etc. Number of targets knowing the different programmes they are eligible for Number of targets participating in different programmes % from a sample % from a sample Innovators (questionnaire) 2. Effectiveness Criteria Indicators / evaluation parameters Unit/scale Source Composite indicators Programme impact on people with respect to its initial Training of skilled resources Reinforcement of Europe s image Involvement of people objectives Creation/support of entrepreneurs Programme impact on technology with respect to its initial Development/creation of business Customer Vision Cooperation objectives Preparation Multi-choice: low-average-high Multi-choice: low-average-high Innovators (questionnaire) Innovators (questionnaire) Programme Innovation Usability Multi-choice: low-average-high Innovators
impact on market with respect to its initial objectives Intellectual property Infrastructures Knowledge (questionnaire) Programme impact on finance with respect to its initial objectives Instrument Multi-choice: low-average-high Innovators (questionnaire)
1.2 Second set of indicators 1. Transversal 1.2.1.1 Beneficiary characteristics Indicator Detail Unit Measure Type Multi-choice (C.f. innova mapping) Number of employees Turnover Average turnover growth per year on the last three years Creation date Number Euros Percentage Year 1.2.1.2 Proposal preparation Indicator Detail Unit Measure Efforts spent in the preparation of a proposal Men.months Cost of the preparation of a proposal Number of proposals written per year Men.months and other expenses (travels, sub contracting ) Euros Number Number of proposals accepted per year Number Success rate Number of proposal win / written Percentage Efficiency
Efficiency of proposal preparation Yearly efforts in the preparation / success rate Men.months per winning proposal Efficiency Efficiency of proposal preparation (2) Yearly costs of the preparation of a proposal / success rate Euros per winning proposal Efficiency Time to contract Delay between call deadline and contract signature date Months Efficiency 1.2.1.3 Project implementation Indicator Detail Unit Measure Average project duration Months Efforts spent in the management of European projects Men.months Other costs of the management of European projects Any cost but man power Euros Efforts spent in the activities of European projects All project activities except management Men.months Other costs of the activities of European projects Any cost but man power All project activities except management Euros Efficiency of project management Efforts spent in the management / (management efforts + other efforts) Percentage Efficiency Efficiency of project management (2) Management costs / (management costs + other costs) Percentage Efficiency 1.2.1.4 Post project activities
Indicator Detail Unit Measure Number of patents generated thanks to the project Included the ones generated during the project Number Efforts spent in post project nonfunded activities Costs of post project nonfunded activities Evaluation, audits Men.months Efficiency Evaluation, audits Euros Efficiency Burden of post project activities Efforts spent in post project activities / (management efforts + other efforts) Percentage Efficiency 2. Financial tools Indicator Detail Unit Measure Type of funding received Average annual income from European projects Multiple choice Loans Guarantee Venture capital Grants Euros Complementary funding received from other sources Leverage effect of European projects Average annual cost for product/service development and launching on the Funding obtained directly from other sources and facilitated by the award of the European project Complementary funding / income from project Euros Percentage Euros
market Project budget significance Average budget allowed per project / turnover Percentage Effectiveness Project budget significance (2) Average budget allowed per project / average annual cost for product/service development and launching on the market Percentage Effectiveness 3. Services Indicator Detail Unit Measure Type of services received Number of solicitations Number of useful solicitations Multiple choice Technology transfer Partnering IPR support & licensing e-business support On-line tools Number Number Usefulness of provided services Number of solicitations / number of useful solicitations Percentage Effectiveness 4. Interaction Indicator Detail Unit Measure Number of contacts established thanks to projects Number of useful contacts established thanks Number Number
to projects Usefulness of contacts Number of contacts / number of useful contacts established thanks to projects Percentage Effectiveness Human resources exchanged Both hosted and sent Men.year 5. Demand Indicator Detail Unit Measure Volume of public contracts Euros Public procurement Share of turnover coming from public bodies Percentage Efficiency Export area Multi-choice Regional National EU 27 International Regulation of the Single choice Yes / no market
Results of Analysis 2 Analysis of programme s public information 2.1 Proportion of SMEs participating in the programmes SMEs are recognised as one of the primary target of European programmes supporting innovation. Major programs have a target of 15% minimum of participating SMEs and while reports still underline that SMEs are deterred by complexity in procedures and delays in contracts 1, the target of 15% is close to be reached. However, the situation is mixed when looking at sub-levels and the SMEs participation is mostly achieved through SMEs dedicated calls (Figure 1) which may appear as a good way to achieve SME participation. Nevertheless, the situation is more complex. Without yet deepening the evaluation of impact generated at SMEs level, one can mention cases where intermediaries exist between the European fund and the SMEs. Well known examples of such a situation is the JEREMIE program which operates as a tool-box to create financial instruments for the benefit of SMEs using ERDF funds held and managed by Fund Holders on behalf of national and regional authorities 2 or the Enterprise Europe Network 3 financed through the CIP. FP7 CIP Eureka-Eurostars 1 Interim Evaluation of the Seventh Framework Programme, report of the expert group, November 2010. 2 http://www.eif.org/what_we_do/jeremie/faq/what%20is%20jeremie.htm 3 http://www.enterprise-europe-network.ec.europa.eu/index_en.htm
Figure 1: SMEs participation (Sources: 4th Progress report on SMEs participation in the 7th R&D Framework Programme, December 2009; CIP ICT-PSP Interim Evaluation Panel Report, May 2009; Interim Evaluation of the Intelligent Energy-Europe II Programme within the Competitiveness and Innovation Framework Programme, April 2009; OSEO Congrés Curie 19 juin 2009) 2.2 Type of participants Looking at the profiles of all participants shows that: FP7 mostly attract interest from private organisations, research organisations and higher or secondary organisations (Figure 2) CIP attract a large number of governmental structures in both it ICT-PSP and CIP-IEE parts. Private (commercial and non for profit) making most of the remaining population (Figure 3) Eureka participants are to a large extent SMEs (45%) whereas the other half of participants is made of large industries, research institutes and universities in comparable levels (Figure 4). Public authorities and development agencies were the most active group of beneficiaries across all 3 LIFE+ components (2007-2008) Research organisation 26% Higher or secondary education 38% Public body 5% Private for profit 28% Other 3% Figure 2: FP7 participants profile (source E-Corda database extract February 2011) (a) ICT PSP Figure 3: CIP participants profile (Source: CIP Implementation report, 2009) (b) IEE
SMEs; 45% Large companies; 18% Research institutes; 19% Large companies; 6% Research institutes; 9% University; 10% National administrations ; 3% University; 16% SMEs; 75% (a) Eureka (b) Eureka-Eurostars Figure 4: Eureka participants (700 projects, 2600 participants - July 2008). (Source OSEO Congrés Curie 19 juin 2009) Figure 5: Life+ participants profiles in projects funded in 2007 and 2008 (source Mid-Term Evaluation of the implementation of the LIFE+ Regulation, march 2010).
2.3 Size of projects Number of participants: 5000 4500 4000 3500 3000 2500 2000 1500 1000 500 0 1 3 5 7 9 1113151719212325272931333537394144464951555964 Figure 6: Number of projects vs number of participants in FP7 (Source: E-CORDA database extract February 2011) The overall average number of participants in FP7 is 5.5. Only looking at the collaborative projects (STREP and IP) this average is 11.6 partners/project. This is higher than the average seen in other programs. As an example, in LIFE+ program, the majority of consortia comprise 1 to 5 partners. LIFE+ call Number of partners 0 1-5 6-10 >10 2007 29 92 16 4 2008 31 141 17 2 Total 60 233 33 6 Table 1: Breakdown of funded projects based on the number of partners showed that the majority of 2007 and 2008 LIFE+ projects involved between 1 and 5 partners (0 means 1single contract holder) (source Mid-Term Evaluation of the implementation of the LIFE+ Regulation, march 2010). Projects duration
3000 2500 Number of projects 2000 1500 1000 500 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 57 63 72 84 Duration (Month) Figure 7: Duration of projects in FP7 (Source: E-CORDA database extract February 2011). For accounting reasons, most of FP7 projects have a round number of years duration, with most of the project in the range 2 to 4 years. Proposal preparation: success rate 14000 12000 10000 8000 6000 4000 2000 0 40% 35% 30% 25% 20% 15% 10% 5% 0% Nb proposals received Nb proposals selected Success rate Figure 8: success rate ( Sources: Interim Evaluation of the Seventh Framework Programme - Report of the Expert Group, November 2010; Interim Evaluation of the Competitiveness and Innovation Framework Programme (2007 2013), March 2010) If the total number of proposals submitted under the FP7 and the CIP are totally different (more than 12 000 under the FP7 and only 650 under the CIP in 2007) the proposal success rates are rather similar
(around 20 %). Information to be provided is roughly the same in both CIP and FP7 proposals (it should be noted that the introduction Participant Identification Code (PIC) in FP7 simplified the administrative procedures for participants being involved in several proposals within the FP7) so cost incurred by proposer is roughly the same under FP7 and CIP. In comparison, the first Eurostars call (deadline 08/02/2008) has a success rate of 43% (Table 2). Submission Stage 1st Call (Deadline : 08/02/2008) Applications Received 215 Total Budget 300 M Number of Applicants 667 Evaluation Stage Applications Eligible 189 Applications Above Threshold 133 Funding Stage Projects Funded* 92 Total Budget 132 M Public Funding ** 53 M Members States Funding 40 M EC Funding 13 M Table 2: application results for Eurostars first call (source eureka-eurostars) Nature and Biodiversity Environment Policy and Governance Information and Communication Number of proposals 2007 call 2008 call Number of Average Number of Number of selected success selected proposals projects rate (%) projects Average success rate (%) 264 57 21.59 227 80 35.24 325 72 22.15 288 99 34.38 118 11 9.32 98 17 17.35
Total 707 140 19.80 613 196 31.97 Table 3: average success rate across 3 Life+ components in 2007 and 2008 (source Mid-Term Evaluation of the implementation of the LIFE+ Regulation, march 2010). Time to contract: The time to contract if defined by the duration between the call deadline and the date at which the contract is signed. Short time to contract are essential, especially in fast moving markets, as a valuable project rationale may become obsolete at project start. Figure 9: Time (days) to grant in days for FP7 grant agreements signed in 2007-2009 (Source: third FP7 monitoring report (European Commission), 2009) The average global time to contract in FP7 is around 350 days, i.e. one year. In comparison, Eureka- Eurostars claim to have a time to contract of about 3 months.
3 Analysis of web survey results The following analysis is based on the results of the web survey conducted on a sample of 132 beneficiaries of EU programmes. Respondents profiles The websurvey has been strongly disseminated within the networks of organisations and agencies supporting the access by innovators to EU funding instruments. As a direct consequence, most of the respondents have been SMEs (Figure 10) providing consultancy and support services (Figure 11). Figure 10: Web-survey respondents type of organization
Figure 11: Web-survey respondents main activity Involvement in EU projects Most of the respondents have a large experience in EU funded innovation projects as most of them have been involved in more than 5 such projects over the past 5 years (Figure 12). To a large extent, participation took place within the FP7, which is one of the largest EU programme. CIP, LLP, ERDF are then mentioned at equivalent levels. It is interesting to note that some overlap is said to exist between these instruments as highlighted in the life+ interim evaluation (Figure 14)
Figure 12: involvement in EU funded innovation project. Figure 13: used EU funding programmes.
Figure 14: ranking of programmes which are felt to most overlap with LIFE+ (where 1 is most overlap ) (source Mid-Term Evaluation of the implementation of the LIFE+ Regulation, march 2010 survey results). Figure 15: participation in national projects. While 80% of the respondent indicated having participated in a EU funded program, only 64% mentioned a participation in a national program. A reason for such a difference may by explained by the profile of the respondents ( ) and the lesser representation of technological SMEs, usually mentioned to prefer national funding. Time to contract
Figure 16: observed time to contract. As observed through the indicators analysis, the average time to contract estimated by most of participants is between 6 and 12 months, and is considered as a too long time frame especially for SMEs that have shorter business time scales. Success rate The average success rate estimated by the participants is around 45%, which is higher in comparison of the average success rate observed in the analysis of the existing indicators (around 20%) but which encloses more programmes (including LLP, ERDF, EBF...). Benefits/inconvenients from participation in EU programmes Figure 17: outputs vs expectations Generally speaking, the satisfaction level of the participants in terms of outputs is good. Most of the comments received related to the issue of cash flow in the project and request a shortened time to grant.
Figure 18: important benefits from innovation projects. The networking opportunities are still the first mentioned benefit obtained when participating in innovation projects. In second rank, with comparable level, appear the financial benefits, the capacity building through skills development and the development of new technologies, products or processes. Figure 19: barriers faced by participants But project administration overhead is still pointed out as the most undesirable effect of the participation in European projects (Figure 19). In relation with the time to grant mentioned earlier, the timescale to realise the benefit still appear to be too long. Finally, the low success rate of proposals negatively position the efforts made for consortium building and proposal preparation. Vision of European programmes by the beneficiaries
Figure 20: evolution of EU programmes Despite the underlined barriers related to administrative issues and time to grant, half of the participants recognise that things are getting better from a harmonisation of procedures point of view (Figure 20). But most of the other half think that issues remain, indictaing that there is still room for efficiency improvement. Figure 21: effectiveness of EU programmes. Finally, most of the participants think that EU innovation support is effective (15%) or fairly effective (46%) in achieving the European 2020 objectives of a smart sustainable growth. 20% of the respondents perceive it as being ineffective. Nevertheless, the 2020 objectives are much more recent than the evaluated programmes which were not fully targeting this 2020 agenda. 4 Conclusions The analysis of existing indicators and answers obtained from the websurvey highlighted the variety of existing schemes and their difficult evaluation on a comparable basis. Nevertheless, some known trends have been confirmed. The need to reduce the time to contract and even more important the time to grant appear as a key issue of most of the European programmes. The administrative burden are still highlighted but a majority recognise that efforts are being made and should be pursued to reduce thus overhead, this increase programmes efficiency.
Harmonisation of terminologies and standardisation of procedures among programmes appear as being the next necessary steps to increase programmes efficiency. From an effectiveness point of view, the programmes are recognised as being fairly effective and would gain in being more tightly aligned with EU2020 objectives.