Evaluation of EFSA s Science Grants and Procurement Schemes 1

PROJECT REPORT OF EFSA Evaluation of EFSA s Science Grants and Procurement Schemes 1 European Food Safety Authority (EFSA), Parma, Italy 2 ABSTRACT In 2008, EFSA was asked by its Advisory Forum to assess cooperation with Member States under the grants and procurement schemes. In 2009, EFSA conducted an evaluation of its Article 36 grants and procurement schemes from 2007 to 2009 to assess the efficiency and effectiveness of the process and procedures, and the impact of the work carried-out under Article36 and procurement to EFSA and the partner organisations. Two separate questionnaires were distributed to EFSA units involved in a grant or procurement and organisations who were awarded a grant or procurement project between 2007 to 2009. In total 192 questionnaires were distributed; 92 to EFSA units and 100 to organisations. In addition available project specific data was analysed to calculate indicators, timing descriptors, activity and resource allocation descriptors, and application and participation descriptors. In total 120 completed questionnaires were received; 66 (72% response rate) from EFSA units and 54 (54% response rate) from organisations. The results show that procurement projects have mainly been used to support the evaluation of authorisation dossiers, whereas grant projects have mainly been used for cooperation projects in the general area of risk assessment. The average budget per project was higher for grants (156,500 ) compared to procurement (60,768 ) but the average budget per partner (between 55,893 and 53,777 ) is similar for both types of projects as a result of the higher number of organisations forming consortia and therefore sharing the funding for grants (n=3.0) than for procurement (n=1.1). A higher percentage of respondents of the grant scheme (65%), compared to the procurement scheme (48%) felt that some administrative practices could be improved. The time spent on applying to a call was considerable higher for grants (87 hrs) compared with procurement (30 hrs). Overall, the procurement procedure was viewed by organisations as simpler to complete and had less administrative burden. Cooperation and project implementation was rated positively by both EFSA staff and organisations. In conclusion, the evaluation suggests that the schemes are effective in achieving their objectives; supporting EFSA`s scientific work as well as benefiting organisations through fostering networking and further collaboration opportunities. This assessment also confirms the complementary role of the two schemes. 1 EFSA Project No. P-2009-00011 2 Correspondence: Cooperation.Article36@efsa.europa.eu Suggested citation: European Food Safety Authority; Evaluation of EFSA s Grants and Science Procurement Schemes. [47 pp.]. Available online: www.efsa.europa.eu European Food Safety Authority, 2010 1-47

SUMMARY Background Scientific cooperation between EFSA and the Member States is a key priority for EFSA to help support the development of risk assessments within EFSA s remit. In this context, EFSA has implemented the grants and procurement schemes to carry out scientific cooperation projects with Member State organisations. The overarching objective of these schemes is to support the scientific work of EFSA, i.e. the Scientific Committee, the Scientific Panels, and the Scientific Cooperation and Assistance Directorate Units. The legal basis for the use of the grants scheme is found in Article 36 of EFSA s Founding Regulation 3 and Commission Regulation (EC) No 2230/2004 4. In line with these regulations, a list of competent organisations capable of assisting EFSA in meeting its aims and objectives was adopted by EFSA s Management Board in December 2006. This list is updated regularly with new organisations designated by Member States. For both schemes, the detailed administrative procedures stem from the EFSA Financial Regulation and its Implementing Rules, which in turn are based on the EU s Framework Financial Regulation and Implementing Rules 5. In view of their significance to EFSA s scientific work and their important role in the strategy for cooperation and networking with Member States 6, EFSA decided to carry out an evaluation of the different scientific cooperation schemes with individual Member State organisations. The objective of this evaluation was to assess the efficiency and effectiveness of the grants and procurement schemes, and if possible to provide recommendations for any improvements to the schemes. The evaluation was based on a survey of the main actors using the schemes, the partner organisations and EFSA scientific units and on available statistics internally in EFSA. The data were compiled through October-November 2009 and the analysis was carried out in November-December 2009. 3 Regulation (EC) No 178/2002 of the European Parliament and of the Council of 28 January 2002 laying down the general principles and requirements of food law, establishing the European Food Safety Authority and laying down procedures in matters of food safety, (OJ L 31, 1.2.2002, p. 1 24). 4 Commission Regulation (EC) No 2230/2004 of 23 December 2004 laying down detailed rules for the implementation of European Parliament and Council Regulation (EC) No 178/2002 with regard to the network of organisations operating in the fields within the European Food Safety Authority s mission, (OJ L 379, 24.12.2004, p. 64 67). 5 Council Regulation (EC, EURATOM) No 1605/2002 of 25 June 2002 on the Financial Regulation applicable to the general budget of the European Communities, (OJ L 248, 16.9.2002, p.1-48); and as amended by Commission Regulation (EC, EURATOM) No 652/2008 of 9 July 2008 (OJ L 181, 10.07.2008, p.23). 6 EFSA (European Food Safety Authority), 2006. Strategy for Cooperation and Networking between the EU Member States and EFSA. <http://www.efsa.europa.eu/cs/blobserver/resource_efsa/about/core/mb_strategy_28thmeet_en_6a.pdf> 2-47

Annual Work-Programmes The budget allocated for grant or procurement activities in the different scientific areas is set out in EFSA s Annual Work Programmes. These plans take into consideration the research or scientific needs of its Scientific Committee and Scientific Panels, and are prepared in consultation with Member States via the EFSA Advisory Forum in order to identify common priorities and avoid the duplication of activities. Compared to the planning and execution of the 2007 and 2008 Annual Work Programmes, the adoption of the 2009 Annual Work Programmes was brought forward by the EFSA Management Board in order to ease its timely execution. The choice on whether to address a research need via a grant or procurement scheme is based on the provisions set in the Financial Regulation and its Implementing rules and on EFSA s experience with these schemes. The procurement procedures provide a flexible tool to address a variety of EFSA s targeted needs. Between 2007 and 2009 procurement projects have mainly been used to support the evaluation of authorisation dossiers, which are of a very targeted nature, whereas grant projects have mainly been used for cooperation projects in the general area of risk assessment. Both grant and procurement projects have equally been used to address research needs for data collection. Projects on horizontal issues are mainly Focal Point agreements with Member States. In general, the procurement procedure is viewed by some organisations as simpler to follow and less bureaucratic. EFSA staff indicate that the main limiting factor in the use of the grants scheme is the lack of specific expertise available on the Article 36 List. Participation of Member State organisations in grants and procurement activities Since 2007, many Member State organisations have applied for and been awarded a grant or procurement project. From 2007 to 2009, 45 different organisations have been awarded a grant project; 62 have been awarded procurement projects and 90 7 have been awarded either a grant or a procurement. The applicants cover a broad range of scientific fields and geographical areas over Europe. Interestingly, most procurement projects (around 70%) are carried out by organisations that are already on the Article 36 list for grants. The application rate (number of applications per call) is lower for grants compared to procurement projects, (1.5 vs 2.8 (all calls included) or 1.5 vs 3.6 (only open calls). The lower rate of applicants for grants could be due to a number of reasons such as the lack of appeal to the organisations (e.g. cost-effectiveness relative to the budget proposed and the nature of the tasks to be performed), the non-availability of very specific expertise, the difficulty to find partner organisations within the timing of the call, or any combination of the above. This low application rate has a higher risk factor of leading to unsuccessful calls. Between 2007-2009, there were seven (14%) unsuccessful 8 grant calls (of which four were in 2009, after the enlargement of the Article 36 list with new nominations) and four (2.6%) unsuccessful procurement calls (of which 3 were in 2009) (see Appendix D). 7 Note that due to the same organisations taking part on both grants and procurement, the figures do not add up. 8 Either no applications were received or proposals were not of sufficient quality when evaluated against the exclusion, selection and award criteria. 3-47

Project implementation and cooperation Both EFSA and the organisations awarded a grant or procurement contract rate positively the cooperation between each other and the process of project implementation. Specifically, organisations find the terms of reference for both grants and procurement projects clear and EFSA staff are content with the results and project deliverables. While some EFSA units indicated that projects were delayed by an average of 2 months, they also declared that this did not result in a subsequent delay in the respective EFSA scientific output. The use of financial and human resources From 2007 to 2009, EFSA have spent approximately 16 million in funds for grant and procurement activities; approximately 6.7 million on grants projects and 9.3 million on procurement projects (including agreements). Taking the contribution of Member State organisations into account (which is at least 20% of the total budget), the total budget allocated to grant projects further increases by a factor of at least 1.25 (i.e. reaching 8.4 million ). The total amount of EFSA funds spent or planned for cooperation projects with organisations in Member States to support EFSA in its scientific tasks has considerably increased (nearly threefold) between 2007 and 2010. The increase of the available funds has been most pronounced for the Support for the evaluation of authorisation dossiers from around 0.5 million in 2007 to nearly 3 million planned for 2010, representing more than one third of the total funds devoted in 2009 and 2010 to grants and procurement contracts for scientific cooperation. Funds for cooperation projects for risk assessments activities and data collection have also significantly increased and each represent between 20 and 30 % of the budgets in 2009 and 2010. The average budget per project is higher for grants (156,500 ) compared to procurement (60,768 ) but the average budget per partner (between 55,893 and 53,777 ) is similar for both types of projects as a result of the higher number of organisations forming consortia and thus sharing the funding for grants (n=3.0) than for procurement (n=1.1). The time allocated by EFSA staff on the different stages (i.e. planning, preparation, evaluation, implementation) of a grant or procurement process is similar, with a higher percentage of time spent by scientific staff (73%) than administrative staff (27%). However, the time spent by applicants in applying to a call (the preparation stage) is considerably higher for grants (87hrs.) compared to procurement projects (30hrs.), with a considerably higher share of administrative staff time in grants (34%) compared to procurement (17%). This shows the differences in the amount of administrative work involved. Comments received by organisations highlighted the high administrative burden in the grants procedure (mainly attributed to the detailed financial reporting and consortia formation) which some felt was disproportionate to the financial value of the project and therefore not cost effective. Satisfaction with the administrative procedures The majority of organisations (76%) and EFSA staff (68%) who responded to the survey believe that there are no un-clarities in the administrative procedure that negatively impact on the grant or procurement scheme. However, a higher percentage of respondents of the grant scheme (65%), compared to the procurement scheme (48%) felt that some administrative practices could be improved. The main suggestions for improvements were, i) better templates for documentation, ii) clearer guidance on administrative procedures and iii) less bureaucracy and more flexibility e.g. regarding the volume of documentation to submit, deliverable dates, and particularly for grants, on the financial checks. Some organisations and EFSA staff believe that there is a need for training in administrative procedures of grants and procurement schemes. 4-47

Impact to EFSA and organisations The assessment confirms that the schemes are effective in achieving their objectives, namely the importance of these activities to EFSA`s scientific work as well as their positive impact on the organisations that conducted this work for EFSA. This was demonstrated by the positive feedback received from both EFSA staff and organisations on the contribution of grant and procurement projects to EFSA scientific outputs, as well as to other uses, such as the production of publications, and on fostering networking and further collaborations. This assessment also confirms the complementary role of the two schemes, due to their suitability for specific application areas and the differences in the benefits they confer. Based on the different legal provisions governing the two tools, procurement projects are of a more targeted nature, more flexible, and mainly serve to directly support EFSA s specific activities and needs. On the other hand, grant projects are well suited to bringing together the expertise available in the Member State competent organisations to meet EFSA s needs but also those of the Member States, and produce outputs that are amenable for multiple uses. Very importantly, grant projects were found to have a strong networking effect among the Member State organisations, in line with one of the main objectives of Article 36. Recommendations The following actions are proposed: To better address EFSA s scientific needs and the Member State organisations capacity to participate in grants and procurement projects, EFSA to: consider launching larger projects with a longer duration, particularly for grants; improve medium-term planning by exploring the use of more multi-annual service contracts and framework contracts. To improve the application process for both grants and procurement schemes, EFSA to take the following measures: pre-notify potential applicants in a timely manner by publishing more details on grants and procurement calls planned for launch; allow sufficient time for applicants to respond to a call (increase time period from launch of call to deadline of applications and spread calls evenly throughout the year); consider automating the application procedure e.g. via the development of an online application tool and to re-use administrative documentation (if applicants have applied previously). To improve the knowledge and appeal of the two schemes, EFSA to take the following measures: continue improving guidance for Member State organisations and EFSA staff; continue providing training activities to Member State organisations and EFSA staff; continue simplifying the administrative procedures whilst respecting the legal provisions (particularly for grants), by duly considering the comments and suggestions received by Member State organisations and EFSA staff through the survey. 5-47

To further improve the networking with and between the Article 36 organisations, EFSA and Article 36 organisations to actively use the recently developed IT tools (Article 36 Organisations database and Article 36 Extranet workspace). To ensure a broad scientific and geographical coverage in the Article 36 list, EFSA and Member States (with support of Focal Points) to consider further expanding the Article 36 list. To monitor further progress, EFSA to re-assess the schemes in 2-3 years. 6-47

TABLE OF CONTENTS Abstract... 1 Summary... 2 Mandate... 8 Objectives... 8 Evaluation... 9 1. Introduction... 9 2. Materials and Methods... 9 2.1. Project-specific data collected by EFSA... 10 2.2. Data collected from a survey to evaluate the grants and procurement schemes... 11 3. Results... 12 3.1. Implementation of the schemes... 12 3.1.1. Annual Work-Programmes... 12 3.1.2. Participation of Member State organisations... 16 3.1.3. Project implementation... 19 3.2. The use of resources... 21 3.2.1. Financial resources... 21 3.2.2. Human resources... 24 3.3. The administrative process... 26 3.3.1. Satisfaction and potential improvements in the administrative process... 26 3.4. The impact of science grants and procurement... 28 3.4.1. The impacts on EFSA... 28 3.4.2. The impacts on organisations participating in EFSA`s grants and procurement schemes 30 Discussion and Conclusions... 32 Appendices:... 36 A. APPENDIX: Questionnaire distributed to EFSA staff... 37 B. APPENDIX: Questionnaire distributed to organisations... 42 C. APPENDIX: Differences between grants and procurement... 46 D. APPENDIX: List of unsuccessful calls1 (grants and procurement) in the period 2007-2009... 47 7-47

MANDATE The Article 36 network is one of the main tools of cooperation between EFSA and Member States. An annual report on the activities under Article 36 was prepared at the end of 2008 which foresaw that EFSA should assess the efficiency and effectiveness of the grants and science procurement schemes by the end of 2009. Moreover, the EFSA Advisory Forum has expressed the importance of this evaluation at its 27th and 31st meetings in Oslo on 12 June 2008 and in Bucharest on 22 April 2009, respectively. In view of the importance of scientific cooperation activities with Member State organisations in supporting the scientific work of EFSA s Scientific Committee and Panels, EFSA embarked on an evaluation of the grants and procurement 9 schemes under EFSA Project No. P-2009-00011. OBJECTIVES The objectives of this assessment are: To assess the operational efficiency and effectiveness of the Article36 and procurement schemes. To assess the impact of the work carried-out under Article36 and procurement to EFSA and the partner organisations. This report provides a synthesis of the evaluation of the grants and procurement schemes. 9 Procurement for scientific needs only. 8-47

EVALUATION 1. Introduction Scientific cooperation between EFSA and the Member States is a key EFSA priority in order to support the scientific work of EFSA, i.e. risk assessments and other scientific activities in all fields within EFSA s remit. In this context, EFSA has implemented the grants and procurement schemes to carry out scientific cooperation projects with Member State organisations. The overarching objective of these schemes is to support the scientific work of EFSA. Cooperation with designated Member State competent organisations is described in Article 36 of EFSA s founding Regulation 10 and in Regulation (EC) 2230/2004 11, which states a list of competent organisations capable of assisting EFSA in its tasks. This list was adopted by EFSA s Management Board in December 2006 and has since been regularly updated. According to the provisions of Article 5 of Regulation (EC) 2230/2004, financial support is provided in the form of grants. Scientific cooperation with organisations is also carried out via public procurement procedures (contracts), as well as through the establishment of other specific agreements, such as Focal Point agreements. The detailed administrative procedures followed in EFSA s grants and procurement schemes stem from European legislation, namely the EFSA Financial Regulation and its Implementing Rules, which in turn are based on the EU s Framework Financial Regulation and Implementing Rules 12. 2. Materials and Methods As the grant scheme (under Article 36) was first implemented in 2007 (following the approval of the list of competent organisations by the Management Board in December 2006), this assessment considered projects initiated between 2007 and 2009. Two difference sources of data were used for this assessment: Available data on factual information on grant and procurement schemes. Data collated from a questionnaire on the evaluation of the grants and procurement schemes from EFSA units and the Member State organisation staff involved in a grant or procurement contract, who provided their feedback on the various stages of the two schemes. 10 Regulation (EC) No 178/2002 of the European Parliament and of the Council of 28 January 2002 laying down the general principles and requirements of food law, establishing the European Food Safety Authority and laying down procedures in matters of food safety, (OJ L 31, 1.2.2002, p. 1 24). 11 Commission Regulation (EC) No 2230/2004 of 23 December 2004 laying down detailed rules for the implementation of European Parliament and Council Regulation (EC) No 178/2002 with regard to the network of organisations operating in the fields within the European Food Safety Authority s mission, (OJ L 379, 24.12.2004, p. 64 67). 12 Council Regulation (EC, EURATOM) No 1605/2002 of 25 June 2002 on the Financial Regulation applicable to the general budget of the European Communities, (OJ L 248, 16.9.2002, p.1-48); and as amended by Commission Regulation (EC, EURATOM) No 652/2008 of 9 July 2008 (OJ L 181, 10.07.2008, p.23). 9-47

2.1. Project-specific data collected by EFSA EFSA collects detailed project-specific data encompassing the complete project life cycle (i.e. from preparation of the calls to project finalisation). These data were used to calculate indicators providing an overview of the grants and procurement activities during the period under study. The dataset that was used for this evaluation does not cover all grants and contracts initiated in the period 2007-2009, as the detailed databases were developed in 2009, therefore a limited set of data is only available for the years 2007 and 2008. Furthermore many of the projects initiated in 2009 were still in the evaluation phase when this report was drafted. The following descriptors were used: Timing descriptors, for the period 2008-2009 (data not available for 2007): launching distribution (grants and procurement), per year; signature distribution (grants and procurement), per year. Activity and resource allocation descriptors, for the period 2007-2009: number and budget (actual for 2007, 2008; estimated for 2009) of projects by EFSA priority area (grants and procurement), per year and in total; number and budget (actual for 2007, 2008; estimated for 2009) of procurement projects by type, i.e. i) open calls (service contracts, multi-annual and renewable service contracts and framework contracts), ii) negotiated procedures (< 60,000 ) and iii) agreements (Focal Point agreements, Food consumption data provision and processing agreements, Service Level Agreements with the EC s Joint Research Centre) for the period 2007-2009. Application and participation descriptors, for the period 2007-2009: number of successful / unsuccessful applications per country (grants and procurement); number of successful / unsuccessful applicants (different organisations counted once) per country (grants and procurement); number of successful / unsuccessful applications per applicant organisation (grants and procurement); number of consortia formed per project (grants and procurement); number of applications received per call (grants vs procurement); number and shares of procurement projects carried out by Article 36 organisations. 10-47

2.2. Data collected from a survey to evaluate the grants and procurement schemes Two questionnaires (see Appendix A and B) were designed and distributed to all EFSA scientific units and Member State organisations who participated in a grant or procurement project between 2007 and 2009. The aim of these questionnaires was to collect data on the following areas: human resources allocated to grants and procurement projects; recommendations to improve processes; quality of collaboration and project delivery; the impact on EFSA and the collaborating organisations (outputs and networking); the choice of the scheme (grants or procurement). Ongoing projects were included in this survey due to the limited number of finalized projects. These were only able to respond to certain questions based on the current stage of the project. The questionnaire was distributed to a named contact person identified from EFSA s database on grants and procurements. Each contact person was requested to complete a separate questionnaire for each grant or procurement project, however some respondents compiled their answers for multiple projects in one questionnaire. In these few cases, the same replies were attributed to all relevant projects, and the time estimation was equally distributed. All questions provided a choice of Yes/No/Not Applicable or a rating (ranging from excellent to poor), as well as the opportunity to provide additional comments. The comments were then analysed and aggregated in categories where feasible to ease the analysis and description of the results. Blank responses were excluded in the analysis. This document provides a summary of the survey results and comments received. 11-47

3. Results 3.1. Implementation of the schemes For the purpose of this report the implementation of the schemes is broken down into three parts: Annual Work-Programmes participation of Member State organisations project implementation. 3.1.1. Annual Work-Programmes Grants and procurement projects are based on an Annual Work Programme and, as a rule, are implemented through calls relating to specific tasks. The planning of the Annual Work Programme typically takes place the year before its implementation and follows a cycle in which i) the scientific and budgetary needs for grants and procurement are identified by the EFSA scientific committee, scientific panels and units (preliminary plan), ii) the EFSA Advisory Forum and Scientific Committee are consulted on the preliminary Work-Programme to identify priorities and avoid duplication (intermediate plan), and iii) the final Work-Programme is proposed for adoption by the EFSA Management Board. The Annual Work-Programmes are implemented following their adoption by the EFSA Management Board, which involves the launch of calls, the evaluation of the received proposals, and the award of grants or contracts to the successful applicants within the respective calendar year. Figures 1 and 2 show the number of calls launched and projects signed, respectively, for grants and procurements under the 2008-2009 work-programmes 13. The figures illustrate that the calls were not evenly launched throughout the calendar year. Few calls were launched in the first part of the year, while the majority of calls were launched either in the summer period or close to the end of the year. Most projects were also awarded (committed) close to the end of the financial year, potentially leading to subsequent risks related to budget execution. The above demonstrates that the timing of launching and awarding of projects has not been optimised yet, which may have repercussions for the interested organisations and the EFSA units involved in the management of the calls. The Annual Work Programme for 2010 was adopted by EFSA s Management Board in October 2009 which is earlier than in previous years. This will allow EFSA to prepare and launch calls earlier (e.g. already in 2009 for projects under the 2010 Work-Programme) and evenly throughout the year. 13 Data for 2007 are not available. 12-47

Figure 1: Distribution of launching grants and procurements for the 2008 and 2009 Annual Work-Programmes (2007 data not available). The dotted line represents the date of adoption of the Annual Work-Programmes by the EFSA Management Board. Grants launched before the adoption of the Work-Programme for each respective year, correspond to grants that were launched as part of the previous year s Work-Programme, but which were signed as part of the subsequent year s Work-Programme. Figure 2: Distribution of signing grants and procurements for the 2008 and 2009 Annual Work-Programmes (2007 data not available). 3.1.1.1. Choice of scheme An important aspect of the planning phase concerns the decision on the type of scheme that should be used for addressing the identified scientific needs. Appendix C lists the main differences between the grants and procurement schemes. The following practical issues are among the main factors for EFSA science units to consider when choosing between the two schemes: the character of the scientific need (e.g. urgency, the necessity to respect specific deadlines, duration and recurrence); the nature of the project and the underlying tasks (e.g. need for networking, tasks described under Regulation (EC) No 2230/2004, expertise required and its availability on the market and on the Article 36 list); 13-47

the ownership of the outcome; the anticipated interest of organisations on the Article 36 list to co-fund specific projects. As illustrated in Figure 3, several procurement procedures (e.g. service or framework contracts following open calls for tender, negotiated procedures or agreements) have been used in the period 2007-2009 by EFSA depending on the nature of its needs (e.g. long term continuous need, urgent need, unique or rare expertise), making procurement a valuable, flexible tool for scientific cooperation with organisations. Figure 3 also shows that: the majority (57%) of procurement projects follow open calls for tender, therefore taking advantage of all available expertise in the market; (32%) specific agreements with partner organisations in case of direct designation by Member States (e.g. Focal Points and Food Consumption Data agreements) or of a monopoly in the type of service to be provided; (11%) by negotiated procedures, in case of targeted projects with a lower budget, benefiting therefore from lighter administrative procedures. Among the open calls for tender, an important share (61%) of the procurement activities addresses EFSA s continuing needs through multi-annual service contracts and framework contracts. Figure 4 demonstrates that between 2007 and 2009 procurement projects have been mainly used to support the evaluation of authorisation dossiers (e.g. preparatory work providing summary datasheets for food additives or guidance documents for the risk assessment of pesticides), which are indeed of a very targeted nature. Grants have mainly been used for cooperation projects in the area of risk assessment. Regarding data collection, both grants and procurement have been used equally. Focal Point agreements with Member States are an activity resulting from the Strategy on Co-operation and Networking, endorsed by the Management Board in December 2006. After a test phase starting in 2007 and following the review of the Strategy on Cooperation and Networking, these contracts have been renewed in 2008 with increased funding. Figure 5 shows that about a quarter of the survey respondents (organisations and EFSA units) involved in the grant scheme preferred carrying out the project under a procurement scheme. The organisations indicated that the procurement process was simpler to follow, less bureaucratic, while EFSA staff highlighted the limited specific expertise available on the Article 36 list as the main reason. 14-47

Figure 3: Breakdown of procurement activities (budget in and % of total, number of projects) by type for the period 2007-2009. For framework contracts, the number refers to the specific contracts or order forms (in parentheses the number of framework contracts). Figure 4: Total amounts (in ) committed (2007 and 2008) and estimated (2009) in the four activity areas for procurements/agreements and grants between 2007 and 2009. Numbers on the chart refer to the number of projects. 15-47

Figure 5: Survey results from EFSA units and organisations for the question Would you have preferred to carry out the project under the other scheme (procurement versus grant scheme)? 3.1.2. Participation of Member State organisations Figures 6 and 7 provide a breakdown of the number of successful and unsuccessful applications per Member State for each scheme. The figures indicate that in general, the same countries are more active (total and successful applications) in both schemes. However, some countries are more active in one type of scheme (e.g. France, Spain and Austria are active in grants whereas Denmark is more active in procurement). Figures 6 to 8 show the active participation of many Member State organisations from 2007 to 2009; 45 different organisations in grant projects, 62 in procurement projects and 90 when both grants and procurement are considered. Figure 8 indicates that the majority of organisations have participated in only one EFSA grant or procurement project, while a small number have been involved in several projects. The organisations cover a broad range of scientific and geographical areas. Grants are limited to the organisations on the Article 36 list, but Figure 9 illustrates that most of the procurement projects are also awarded to public organisations and universities (around 80%), who are listed on the Article 36 list (around 70%). While access to a larger pool of expertise may be an important differentiating factor favouring procurement, other differences (see section 3.1.1.1, Choice of scheme) in the nature of the two schemes are probably also important in the choice of the two schemes. 16-47

Figure 6: Applications for grants by country in the period 2007-2009. Figure 7: Applications for procurement by country in the period 2007-2009. Figure 8: Participation of organisations in grant and procurement projects in the period 2007 2009 (2009 data as available by 16 Nov 2009). 17-47

Figure 9: Breakdown of organisations awarded procurement projects in the period 2007-2009 (2009 data as available by 16 Nov 2009). Figures 6 and 7 show that a higher number of applications are received for procurement schemes, which coincides with the larger number of launched projects as well as the higher application rate (applications received per call) in procurement compared to grants. The average number of applications per successful call is 1.5 for grants versus 2.8 (all calls included) and 3.6 (only open calls) for procurement projects. This could partly be because the applicants to grants calls are restricted to organisations on the Article 36 list. Between 2007-2009, there were seven (14%) unsuccessful 14 grant calls (of which four were in 2009, after the enlargement of the Article 36 list with new nominations) and four (1.8%) unsuccessful procurement calls (of which three were in 2009) (see Appendix D). The vast majority of Article 36 organisations (90%) believed they were adequately informed when a call is launched. Some suggest that an email notification system would be beneficial. It is important to highlight that EFSA already has an email notification system in place, comments received by organisations highlight that more experts may need to be included on the notification list. Many EFSA respondents feel that the current Article 36 list is not comprehensive and would benefit from strengthening some scientific areas such as feed additives. 14 Either no applications were received or proposals were not of sufficient quality when evaluated against the exclusion, selection and award criteria. 18-47

3.1.3. Project implementation Survey results show (Figure 10) that the majority of organisations believe that the terms of reference for both grant and procurement calls were sufficiently clear to be able to draft a proposal and ensure delivery of a quality output. The main comments received for those who answered no concerned better guidance on the report format. It should be noted, however, that the majority of these projects were carried out before the recent implementation of a template for EFSA external scientific and technical reports. Figure 10: Survey results from organisations for the question Do you believe that the terms of reference were sufficiently clear as a guidance to draft a proposal to ensure the delivery of an output of high quality? Similarly, EFSA units rate positively the quality of the final reports (Figure 11) as well as the pertinence of the project results to answer the original expectations and needs (Figure 12). While there were some delays (average delay of 2 months) in the delivery of reports (more for grants compared to procurement; Figure 13), EFSA respondents indicated that these delays did not result in a subsequent delay in EFSA outputs (Figure 14). Most delays in grant deliverables were attributed to delays in the start of the project, high workload involved, and the large number of partners involved. For procurement projects the most common cited causes of delay were data availability, amendments to contracts and revisions to the draft report. Overall, the cooperation between EFSA and organisations was rated positively by both sides (Figure 15). 19-47

Figure 11: Survey results for the question directed to EFSA units only How do you consider the quality of the final report (meeting the objectives, presentation, etc)? Figure 12: Survey results for the question directed to EFSA units only At what level did the project results answer your needs/original expectations? 20-47

Figure 13: Survey results from EFSA units for the question Was there a delay in receiving the final acceptable output? Figure 14: Survey results from EFSA units for the question Did the delay result in a delay in a subsequent EFSA output (e.g. opinion)? Figure 15: Survey results from EFSA units and organisations for the question How would you rate your cooperation with the beneficiary/contractor? 3.2. The use of resources 3.2.1. Financial resources Figure 16 shows that from 2007 to 2009, EFSA will have paid approximately 16 million for grant and procurement activities; approximately 6.7 million on grants and 9.3 million on procurement (including agreements). Figure 16 also shows that the total amount of EFSA funds spent or planned for cooperation projects with institutions and organisations in Member States to support EFSA in its scientific tasks has considerably increased between 2007 and 2009 for both schemes. 21-47

Figure 16: Total amounts committed, estimated or planned (in ) for procurements and grants between 2007 and 2010. For 2007 and 2008, the amounts represent the funds effectively committed, for 2009 the current (November 2009) estimate, and for 2010 the proposed amount. Numbers on the chart refer to the number of projects. In total, from 2007 to 2009, EFSA will have paid approximately: 4 million on supporting the evaluation of authorisation dossiers; 4 million on general risk assessment activities, 6 million on data collection; 2 million on horizontal activities and scientific cooperation (Figure 4). The increase of the available funds has been most pronounced for the Support for the evaluation of authorisation dossiers from around 0.5 million in 2007 to nearly 3 million planned for 2010, representing more than one third of the total available funds in 2009 and 2010 (Figure 17). Funds for cooperation projects for risk assessments activities and data collection have also significantly increased and each represent between 20 and 30 % of the total budgets in 2009 and 2010. 22-47

Figure 17: Total amounts (in ) committed (2007 and 2008), estimated (2009) or planned (2010) in the 4 activity areas for procurements and grants between 2007 and 2010. Numbers on the chart refer to the number of projects. Figure 18 shows that the average budget per project is higher (2.6 times) for grants compared to procurement 15, which concurs with the smaller and more targeted nature of procurement projects. However, the average budget per partner (grants = 55,893, procurement = 53,777 ) is similar for both types of projects as a result of the higher number of organisations forming consortia and thus sharing the funding in grants. Moreover, it is important to note that for grant projects a financial contribution is provided by the awarded organisations, a minimum of 20% of the total project costs. Taking this contribution into account, the total budget allocated to grant projects further increases by a factor of at least 1.25 to a total of 8.4 million. 15 In the case of framework contracts, the specific contracts are counted as separate projects. 23-47

Figure 18: Average EFSA expenditure per grant or procurement project for the period 2007-2009. 3.2.2. Human resources Results from the survey (Figure 19) indicate that the time spent by EFSA staff on the different stages of a project is similar for grants and procurement, with more time spent by scientific staff compared to administrative staff. As expected, a larger share (47%) of the work is spent in the implementation stage of the process, which concerns project management by the unit once the project is signed. The time spent by organisations in the preparation stage (i.e applying to a call) is considerably higher (three times) for grants compared to procurement (Figure 20). As for EFSA staff, more time is spent by scientific officers than administrative staff; however organisations indicate a higher percentage of time spent by administrative staff for grants compared to procurement (34% vs 17%, respectively). This reflects the differences in the amount of administrative work involved. Comments received by organisations highlight the time-consuming process of preparing to answer a call, particularly for grants. Many felt that participating in such projects was not cost effective with the time spent being disproportionate to the financial value of the project. The time spent in the implementation phase by organisations staff was not part of the questionnaire as it would vary according to the project specifications. Organisations commented that due to the more complex nature of grant projects (consortia and financial reporting), projects of a short duration should be avoided. It should be noted that the figures on the time spent were rough estimations and therefore should be treated with caution. There was a large variation in the responses received to this question. As a result of the large variation and the limited number of available data, no further breakdown or analysis by type of procurement procedure was possible (e.g. open procedure, negotiated procedure, etc). However, it is known that negotiated procedures and agreements are less time consuming in the preparatory and evaluation stages for EFSA and the Member State organisations. 24-47

Figure 19: Survey results for the question What is the total estimated time (in hours) of EFSA staff spent in each stage of a grant or procurement procedure? Figure 20: Survey results for the question What is the total estimated time (in hours) of partner organisations staff spent in the preparatory stage of a grant or procurement procedure? 25-47

3.3. The administrative process The detailed administrative procedures used in EFSA s grants and procurement schemes stem from relevant legislation, namely the EFSA Financial Regulation and its Implementing Rules, which in turn are based on the EU s Framework Financial Regulation and Implementing Rules 16. 3.3.1. Satisfaction and potential improvements in the administrative process Both EFSA units and organisations were asked if any improvements were needed to the administrative procedure of the two schemes. Figure 21 shows that more respondents believe that administrative practices could be improved for grants (65%) compared to procurement (48%); about half of the procurement respondents compared to two-thirds of the grants respondents. The main suggestions for improvements were: improved templates for all documentation; clearer guidance on the administrative procedure - such as the types of documentation to submit; less administrative burden and more flexibility e.g. regarding the volume of documentation to submit, the strict deadlines for deliverables, and particularly for grants on the financial controls on expenditures. Many organisations highlighted the significant amount of paperwork that is currently related to the submission of an application, and suggested that an on-line submission process and the re-use of documentation (such as the Legal Entity form) could help make the process simpler and less time consuming. However, the majority of the respondents believe that there are no unclarities that negatively impact on the whole scheme (Figure 22). Figure 23 shows that about half of EFSA respondents believe that there is a need for training on the schemes, compared to only a quarter of organisations participating in grants and procurement. Some EFSA staff commented that training on administrative process and procedures would be particularly useful for new staff to understand best practice and to receive updates on changes. Organisations commented that training for organisation staff at the start of a project would be useful. EFSA currently recommends administrative staff from organisations to attend the first project meeting (kick-off meeting) in order to receive explanations and instructions on the administrative procedures. 16 Council Regulation (EC, EURATOM) No 1605/2002 of 25 June 2002 on the Financial Regulation applicable to the general budget of the European Communities, (OJ L 248, 16.9.2002, p.1-48); and as amended by Commission Regulation (EC, EURATOM) No 652/2008 of 9 July 2008 (OJ L 181, 10.07.2008, p.23). 26-47

Figure 21: Survey results from EFSA units and organisations for the question Are there any aspects of the administrative practices which could be improved? Figure 22: Survey results from EFSA units and organisations for the question Are there any un-clarities in the administrative procedure that negatively impact on the whole scheme? Which are these? 27-47

Figure 23: Survey results from EFSA units and organisations for the question Do you think there is a need for training on the administrative procedure? 3.4. The impact of science grants and procurement 3.4.1. The impacts on EFSA The overarching objective of the grants and procurement schemes is to support the scientific work of EFSA. Survey results illustrate that both grant and procurement outputs are viewed as having successfully contributed towards informing EFSA s scientific outputs (Figure 24). The results also suggest that procurement outputs may have been more effective (Figure 25). Figure 24: Survey results from EFSA units for the question Have the results of the project been useful in helping to inform EFSA outputs (e.g. opinions, statements or guidance documents)? 28-47

Figure 25: Survey results from EFSA units for the question How would you rate the contribution of the project deliverables to the relevant EFSA output? Figures 26 and 27, and relevant comments received, show that grant outputs have been particularly useful in contributing to additional activities, such as producing publications, building capacity (the development of databases) and in general adding valuable knowledge for future work. Figure 26: Survey results from EFSA units for the question Have the results been useful in another way to EFSA, e.g. building capacity supporting other projects/activities? Figure 27: Survey results from EFSA units for the question Are you aware of any further use of the project deliverables (e.g. reports, databases, etc)? 29-47

3.4.2. The impacts on organisations participating in EFSA`s grants and procurement schemes Figure 28 shows that besides the final deliverable, there has been further use of the project outcome. About half of the survey respondents indicated that the project outputs have been used e.g. in subsequent publications in scientific journals 17, in teaching, in building capacity, and in providing a leveraging effect to further research. A number of reports have also been published this year on EFSA s website, raising the scientific visibility of the participating organisations as well as EFSA. There was no difference between grants and procurement in the further use of projects deliverables. Figure 28: Survey results from organisations for the question Besides the final deliverable (report, database etc.) have you or your partners made any use of the project outcome? Grants appear to have been particularly effective in fostering the networking among the Member State organisations (90% of the responses were positive for grants compared to 33% for procurement) (Figure 29). The most common example of further networking cited by organisations is collaboration in the preparation of research proposals and participation in other research projects (e.g. under the EU Framework Contracts for research, FP6 and FP7). This effect may also have been influenced by the fact that the formation of consortia was a requirement in the terms of reference for a number of grant projects. Figure 30 shows that most grant projects (58%) have been carried out by consortia (overall average of 3 partners per project), compared to the 5% for procurement (overall average of 1.1 partners per project). 17 In general, the provisions for the publication of grants and procurement outputs are stipulated in the calls for proposals/tenders as well as the grant agreements and contracts. Depending on the specific provisions, a written authorisation by EFSA may be required before the further publicity of results, which can only take place, as a rule, after the report has been published on EFSA`s website. 30-47

Figure 29: Survey results from organisations for the question In case you have been working with partners in a consortium, have you started any other collaborative projects with any of those partners? Figure 30: Share of projects carried out by consortia versus single organisations in grant and procurement projects for the period 2007-2009, including summary data (2009 data as available by 16 Nov 2009). 31-47