FINAL EVALUATION REPORT

Transcription

1 EVALUATION OF THE SMALL AND MEDIUM-SIZED ENTERPRISES (SMEs) ACCIDENT PREVENTION FUNDING SCHEME EUROPEAN AGENCY FOR SAFETY AND HEALTH AT WORK FINAL EVALUATION REPORT IDOM Ingeniería y Consultoría S.A. BIO 8958

2 Foreword IDOM INGENIERÍA Y CONSULTORÍA S.A., Bilbao, Spain, carried out evaluation. The company gratefully thanks all those people who answered its questionnaires and interviews for their useful feedback and kind collaboration. The Evaluation Team Bilbao, 2003 BIO 8958

3 TABLE OF CONTENTS EXECUTIVE SUMMARY 1 0- THE STRUCTURE OF THE EVALUATION REPORT OBJECTIVES AND METHODOLOGY OBJECTIVES OF THE EVALUATION METHODOLOGY FOR THE EVALUATION OF THE FUNDING SCHEME METHODOLOGY STRUCTURE EVALUATION PHASES QUESTIONNAIRES AND SURVEY SAMPLES IMPLEMENTATION OF THE SURVEY IMPLEMENTATION OF INTERVIEWS THE SME ACCIDENT PREVENTION FUNDING SCHEME: PROGRAMMING, MANAGEMENT AND MONITORING ISSUES DESCRIPTION OF THE FUNDING SCHEME AIMS AND SCOPE OF THE FUNDING SCHEME PHASES OF THE SME ACCIDENT PREVENTION FUNDING SCHEME _ PROGRAMMING OF THE FUNDING SCHEME CONTENTS OF THE AGENCY S CALL FOR PROJECT PROPOSALS THE CALL, APPLICATION AND TECHNICAL SPECIFICATION FORMS PROCESSING APPLICATIONS BY THE AGENCY SELECTION PROCEDURE AND ADVISORY BODIES MANAGEMENT AND MONITORING OF THE FUNDING SCHEME AIM OF THE CHAPTER AGENCY S PERFORMANCE WHEN HANDLING AND MONITORING THE PROJECTS EVALUATION OF THE ACHIEVEMENTS PROMOTION OF THE FUNDING SCHEME BY THE AGENCY PROMOTION OF THE AIMS AND OBJECTIVES OF THE FUNDING SCHEME AND THE CALL INFORMATION CONCERNING CO-FUNDED PROJECTS DISTRIBUTION OF PROJECT RESULTS ANALYSIS OF THE COHERENCE OF THE SME ACCIDENT PREVENTION FUNDING SCHEME CONTEXT FOR ANALYSING COHERENCE INTERNAL COHERENCE OF THE FUNDING SCHEME COHERENCE OF THE FUNDING SCHEME REGARDING OTHER COMMUNITY POLICIES RESULTS AND IMPACTS OF THE FUNDING SCHEME SELECTED PROJECTS: LEVEL OF ORGANISATION AND CATEGORIES THE OUTCOME OF THE PROJECTS 61 BIO 8958

4 RELEVANCE OF THE FUNDING SCHEME IN RELATION TO OSH NEEDS AND PRIORITIES PERFORMANCE OF PROJECT HOLDERS. EFFECTIVENESS AND EFFICIENCY OF THE PROJECTS OUTCOME OF THE PROJECTS IN RELATION TO THE CALL PRIORITIES PROJECT SUSTAINABILITY IMPACT OF THE FUNDING SCHEME IMPACT OF THE FUNDING SCHEME IN HEIGHTENING AWARENESS OF OCCUPATIONAL ACCIDENTS RISKS AT EU LEVEL TRANSFERABILITY OF PROJECTS EW 2001 AND THE TOPIC ON ACCIDENT PREVENTION OVERALL ASSESSMENT RECOMMENDATIONS FOR FUTURE SME FUNDING SCHEMES PRIORITY RECOMMENDATIONS FOR THE ENTIRE FUNDING SCHEME SPECIFIC RECOMMENDATIONS FOR THE FUNDING SCHEME RECOMMENDATIONS FOR FUTURE EVALUATIONS 95 ANNEX ANNEX1: METHODOLOGY PROCESS EMPLOYED FOR THE COLLECTION AND ANALYSIS OF INFORMATION 97 ANNEX 2: QUESTIONNAIRE FOR PROJECT HOLDERS 98 ANNEX 3: QUESTIONNAIRE FOR NATIONAL FOCAL POINTS 103 ANNEX 4: DATA FROM THE SURVEY (PROJECT HOLDERS) 106 ANNEX 5: DATA FROM THE SURVEY (NATIONAL FOCAL POINTS) 114 BIO 8958

5 Executive Summary 1. Introduction The SME Accident Prevention Funding Scheme is a Grant Scheme focusing on reducing accidents in Small and Medium-sized Enterprises (SMEs) throughout the European Union. It was launched by the European Agency for Safety and Health at Work (hereafter referred to as the Agency ) in April 2001 and has co-funded a total of 51 projects at national and transnational levels. The European Parliament allocated a 4.5 Million budget to the Agency. The Agency is a tripartite European Union organisation uniting representatives from three key decision-making groups in each of the European Union s Member States namely, governments, employers and workers organisations. The different bodies comprising the Agency network are: The Administrative Board: that sets the Agency s goals and strategy, The Bureau: that oversees the Agency s operational performance, The Agency: the Director of the Agency is the official representative of the Agency and is responsible for the day-to-day running of the same, National Focal Points: these are the Agency s principal safety and health information network in each Member State. National Focal Points are nominated by each government as the Agency s official representative in that country and are normally the competent national authority for Safety and Health at Work. The role of National Focal Points is to provide information and feedback to the Agency concerning initiatives and products and they are consulted about all information activities related to the national level. The general objective of this appraisal is to establish whether the objectives of the SME Accident Prevention Funding Scheme have been met in terms of impact, effectiveness, efficiency and sustainability, in addition to making recommendations and assisting the Agency in the formulation of future Funding Schemes. In order to attain these objectives, the Evaluation Team has developed and implemented an evaluation method based, on one hand, on the analysis of secondary sources such as project documentation (Application Forms and Final Activity Reports) and, on the other, analysis of information obtained from a survey and interviews conducted for this purpose. This, the First Evaluation of the Funding Scheme, covers the management and accomplishment of the objectives of the Scheme, as well as the performance of the Agency and the project holders during the Scheme s different phases, in addition to including recommendations for improving future Schemes. 1

6 2. The Scope The Evaluation of the Funding Scheme was launched in November 2002 and implemented over a period of 4 months. Evaluation concluded with this Final Evaluation Report. In accordance with the specifications contemplated in the Restricted Call for Tender (OSHA/TF/2002/T2) regarding External Evaluation of the Funding Scheme, evaluation has focused upon the following items: 1. Quality and suitability of the Call for Project Proposals. 2. Agency performance regarding dissemination of the Funding Scheme. 3. The processing of applications by the Agency in order to evaluate the extent to which services provided by the Agency ensured quality at the outset and supported implementation. 4. The selection procedure regarding evaluation of the composition and effectiveness of the bodies evaluating the applications and the quality of selected applications. 5. Evaluation of the implementation of selected projects. 6. Results and impact of the Scheme in terms of outcome, sustainability and transferability. 3. Methodology The methodology used for this evaluation exercise was a combination of desk- (compilation and analysis of information) and fieldwork (questionnaires and interviews). Two main sources of information were used for the evaluation: Information provided by the Agency; namely Application Forms, Final Activity reports of all co-funded projects, the Funding Scheme reference documents (Call for Project Proposals, Application Form, Technical Specifications) and other related documents. Information generated during the evaluation. This qualitative information was gathered using a survey conducted among 51 project holders and 15 National Focal Points, all of whom were sent questionnaires, in addition to interviews held with project holders, National Focal Points, the Agency and other related parties. Other documents and publications such as newsletters, reports, etc. The information was gathered and analysed according to the following stages: 1. Information was gathered from the Agency. 2. Identification and preliminary analysis of essential aspects of the Funding Scheme. 3. Survey: Elaboration of Questionnaires for Project Holders and National Focal Points. 4. Personal interviews. 2

7 5. Information generated during the evaluation process was analysed. 6. The Final Evaluation Report was drafted, including issues identified in the Terms of Reference. It should be mentioned that there was a high degree of response to questionnaires (100% National Focal Points and 90% project holders) and collaboration among the parties interviewed. This fact provided the Evaluation Team with important information that further improved the results of the evaluation. 4. Organisation of the Evaluation Report The Evaluation Report has been set out in Chapters and drawn up in accordance with the following issues: Chapter 1 Objectives of the Funding Scheme and Evaluation Methodology. Chapter 2 Programming, Management and Monitoring of the Funding Scheme. Chapter 3 Coherence of the Funding Scheme. Chapter 4 Results and Impact of the Funding Scheme. Chapter 5: Recommendations for future Funding Schemes. This Executive Summary has been structured in accordance with the Evaluation Report and each of its Chapters includes both the main conclusions and recommendations elaborated by the Evaluation Team. 5. Chapter 1: Objectives of the Funding Scheme and Evaluation Methodology In 2001, the European Agency for Safety and Health at Work launched the Accident Prevention Funding Scheme for Small and Medium-sized Enterprises (SMEs). The Scheme supported projects aimed at reducing work-related accident risks throughout European SMEs. The overall objective of the Funding Scheme was to identify, communicate and support activities and projects with effective added value that could motivate, develop, support and sustain the organisation of OSH/OSH management in SMEs. It was imperative to achieve this goal by providing effective Good Practices that would meet the relevant legislative requirements established by European Directives, national legislation and guidelines. Three project categories, focusing on priority hazards and high-risk sectors at both national and transnational levels, were financed: Training Information and Communication Provision of Good Practices The Evaluation Process was divided into four phases: 3

8 Phase 1- Structuring. The objective of this phase was to determine the most appropriate evaluation method and gather relevant information. It was carried out from the 21 st November to the 5 th December 2002 and concluded with a Progress Report being submitted on the 5 th December. Phase 2- Data collection and preliminary analysis. This phase focused on data collection related to the identification of those target groups that would respond to the questionnaires, the selection of project holders for interviews and the elaboration of questionnaires and their despatch to all project holders and National Focal Points. The timeframe for this phase was from the 5 th December to the 26 th December This phase concluded with the Inception Report being submitted on the 19 th December. Phase 3- Implementation of fieldwork. Interviews, reception of questionnaires and complementary activities were carried out. This phase was implemented from the 26 th December 2002 to the 30 th January 2003 and concluded with the Second Progress Report being submitted on the 30 th January. Personal interviews during Phase 3 were held with: 1. Thirteen project holders. 2. The Agency (Director, Task Force Campaigns and Programmes, Head of the Working Environment Unit and Head of the Information and Communication Unit). 3. The Agency s Administrative Board Members. 4. National Focal Points in Greece, Denmark, Germany and Belgium. 5. The European Jury. 6. Other interested parties (European Commission, European Parliament and Social Partners at EU level). Phase 4- In-depth evaluation and judgement. This phase was implemented from February to the end of March It focused on an in-depth analysis of all information gathered during Phases 2 and 3 and served to produce this Final Evaluation Report. 6. Chapter 2: Programming, Management and Monitoring of the Funding Scheme This chapter describes the Funding Scheme and assesses its implementation in accordance with three basic, inter-related steps: Programming of the Funding Scheme: this is the planning of the Agency s Call for Project Proposals and the Selection Procedure. Management and Monitoring of the Funding Scheme: the Agency s performance while handling and monitoring projects, in addition to evaluating the Scheme s achievements. Promotion of the Funding Scheme: assessment of the promotion of the Scheme at different levels. 4

9 6.1. Agency s Call for Project Proposals The Call for Project Proposals was based on a decision of the European Parliament. It was published in the Official Journal of the European Communities on the 21 st April 2001 (2001/S ). The Call for Project Proposals, Application Form, Technical Specifications and Checklist were drafted in eleven languages in order to allow project holders to submit their applications. The majority (88%) of project holders stated that these documents were drawn up in a clear and comprehensible manner and that they contained sufficient information for potential applicants. In spite of this, some room for improvement was identified. Avoiding repetitive questions would shorten the Application Form. In addition to which, relevant information concerning Budget or Administrative Requirements could be more precise. These modifications should ease the task of Project Holders when filling out their applications. Evaluation also identified the need to include reference indicators in the Application Form in order to allow projects to be monitored and evaluated. Potential applicants are required to quantify these indicators upon submitting their Application Forms. It would be advisable to provide potential applicants with an indication of how their applications are to be assessed. Introducing scores to measure the quality of the technical and financial proposal of the project holder could do this. The Agency promoted the Scheme in an efficient manner. The Call for Project Proposals, Application Form and Technical Specifications were published on the Agency s website, together with the OJEC publication. Promotional activities were held by the Agency within its own network and presented to other interested parties. Most potential applicants knew about the Funding Scheme through the Agency s website or network. However, in order to increase Scheme visibility, it would be advisable to reinforce promotional campaigns. The vast majority (96%) of project holders considered that the deadline for submitting Applications was appropriate. From the date of publication of the Call, national applicants had 10 weeks in which to submit their applications. Transnational applicants had 12 weeks The Agency s Performance The Agency answered enquires from potential applicants from the date the Call was published till the deadline for submission of applications. Two-thirds of the project holders expressed their opinion that the Agency provided project applicants with excellent service. In addition to which, 41% of all project holders assessed support provided by the Agency as excellent and a further 41% assessed it as being good. 5

10 The Agency registered and filed all applications. In the year 2001, the Agency received 456 applications, most of which (90%) had been properly completed according to the specifications of the Call and received by the Agency within the specified deadline. Applications checked against these eligibility criteria were assessed in accordance with selection criteria regarding specific priorities. The final selection of co-funded projects was carried out by the Agency in October The selection procedure was impartial, transparent and fair. The Agency established a schedule and appointed groups of experts to assess eligible applications. As a result of the interviews carried out, it can be concluded that the timeframe for receiving, processing and selecting applications was extremely short. National and transnational project selection groups were made up of specialists in the field of Safety and Health at Work, in addition to a number of social partners Selection procedure of national and transnational projects Evaluation procedure took approximately three months and particular attention was paid regarding the compliance of applications with selection criteria. Interviews held led to the conclusion that having more time for the selection of applications would have produced a more positive impact. National Focal Points assessed national applications in co-operation with social partners and a European Jury comprising government representatives, social partners, a delegate from the European Commission and two independent experts evaluated transnational applications. Of 410 eligible applications, only 51 passed the selection process. This illustrates that only 12% of all applications submitted received co-funding from the Agency. In view of the limited amount of human resources allocated to the European Agency Task Force Campaigns and Programmes (responsible for management the Funding Scheme) and many National Focal Points, we suggest that more time and resources be allocated to the evaluation process. Should the Scheme continue to function on an annual basis, then it might be necessary to reduce the number of applications to be assessed. This could be achieved by introducing stricter eligibility criteria. Priority was given to the transparency and fairness of the evaluation process at both national and transnational levels. In order to attain these conditions, the Agency provided national and transnational evaluation teams with common rules and precise guidelines for assessing applications in order to avoid judge and jury problems and conflicts of interest. Posterior analysis and interviews carried out reveal that the selection procedure was impartial, transparent and just. Nevertheless, an alternative option that would avoid any problem in the application assessment procedure would be to subcontract this work to external specialists. This external evaluation would be conducted under the surveillance and monitoring of the Agency. 6

11 6.4. Selected Projects The Funding Scheme co-funded 35 national and 16 transnational applications with a total budget of 4.5 Million EUR spent on an annual basis. The grant received by each project varied between and National projects had smaller budgets than transnational ones. This allocation enabled 51 projects to be co-funded. Nevertheless, this budget is very small in comparison with other allocations of Community funds and the impact is not, therefore, easily detected in the SME sector. Evaluation reflected the need to increase the SME Accident Prevention Funding Scheme visibility. This could be achieved as follows: By financing a lower number of larger-size projects (broad partnerships) in order to ensure that the message can reach and be effectively used by others. By increasing project budgets and promoting the creation of integrated projects (projects that include more than one category: Training, Information and Communication and Good Practices). By making the Funding Scheme an "Integrated Programme". This would permit the creation of a specific Scheme for SMEs to be developed regarding issues related to accident prevention in the workplace and would prevent the Funding Scheme from being isolated to a greater, or lesser, degree from other activities occurring at Community level. Selected Project Holders received written confirmation from the Agency. However, this letter did not include any indication or evaluation regarding those strong or weak points of their applications. Project holders interviewed expressed their interest in receiving the Agency s feedback in this particular respect in order to permit them to improve the quality of their applications in future Calls for Project Proposals. Depending on the amount of the conceded grant, Project Holders were contractually required to present Interim and Final Activity Reports. This allowed the Agency to better monitor projects. The Survey and interviews carried out showed that project holders had no problem in understanding what the Interim and Final Reports should contain. Time was the main impediment regarding project implementation under the Funding Scheme. Approximately half of the project holders stated that the deadline for submitting the Final Activity report was unsuitable. Additionally, project holders emphasised the difficulties encountered in submitting their financial reports. Since the Agency works on an annual basis, it might be advisable to follow the following steps in order to gain time: Provide project holders with clear information concerning the need to comply with these tight deadlines. Ensure that all contracts are signed rapidly. Ensure intermediate monitoring of all projects, via telephone or Interim Reports. This assistance from the Agency could help project holders to resolve their problems whilst the project was still in progress. Once the project has been completed, allow for one or two additional weeks for the submission of financial reports. 7

12 The three project categories presented were appropriate for contributing to a reduction in accidents in SMEs. Likewise, with regard to co-funded projects, it was observed that these categories corresponded more closely to the needs of intermediate bodies (trade unions, associations, business organisations, etc.) than to those of SMEs, who might prefer direct funding for fixed assets and safety and health plans in the workplace Promotional Activities of the Funding Scheme Agency Funding Scheme promotional activities were carried out at the following levels: Dissemination of the objectives of both the Funding Scheme and the Call for Project Proposals. A public launching of the Scheme accompanied the publication of the Call in the OJEC, the Agency s website dedicated a specific section to the Funding Scheme, the Agency network participated in dissemination activities, etc. At this stage, the promotional campaign of the Scheme was sufficient, though it would be advisable to extend publicity campaigns in collaboration with National Focal Points in order to increase diffusion of the Call. Dissemination of selected projects: the Agency prepared a report containing a brief summary of selected projects and also published this information on its website. Those project holders consulted considered this to be satisfactory. Dissemination of results of co-funded projects. The Funding Scheme is still at a promotional stage. At the time this Evaluation Report is still being drawn up, a document containing the results of the Scheme is being confected by the Agency. Though it is not possible for us to give an assessment of promotional activities at this stage, it is important to underline the need to make greater efforts to disseminate projects that have received funding, including their results. In this sense, the Evaluation Team recommends an initial, kick-off meeting being set up before launching the Scheme in order to make project holders familiar with the results of previous Schemes. 7. Chapter 3: Coherence of the Funding Scheme Funding Scheme coherence was analysed at two levels: Internal coherence: evaluating the extent to which Funding Scheme activities contributed to achieving the objectives pursued by the Agency. Coherence with other Community policies: evaluating the degree of complementarity between activities contemplated by the Funding Scheme regarding other Community Policies that might also include activities related to Safety and Health in the Workplace. The results of the evaluation permit us to draw the conclusion that the Funding Scheme s degree of coherence with Agency objectives is high. The results of co-funded activities contributed to meet the Agency s objectives to a great extent. Regarding its coherence with other Community Policies and Community Institutions, research and analysis carried out conclude that the SME Funding Scheme has its own, peculiar characteristics and objectives that are clearly different to other instruments 8

13 developed by Community Institutions. Accordingly, it should be pointed out that SME Funding Scheme does not overlap other Community programmes and activities, but rather complements them. Likewise, it might be advisable to ensure co-ordination between various community institutions concerning matters dealing with Health and Safety. This would avoid overlapping of similar activities. 8. Chapter 4: Results and Impact of the Funding Scheme The qualitative information gathered from the Survey and interviews was taken into account in order to draw up this Chapter which analyses the outcome of the co-funded projects, their sustainability, the impact of the Scheme and the overall evaluation Outcome of the Projects The Funding Scheme produced an impact at both national and EU levels. The results of the projects led to an increase in the prevention of accident hazards and fostered the development and identification of Good Practices. Those project holders consulted appraised the Funding Scheme most favourably. National Focal Points consulted indicated that the Scheme contributed very positively to: The development and identification of Good Practices The development of risk evaluation and prevention practices The development of sustainable and European added value in Occupational Safety and Health Activities in different countries. Projects were implemented according to the agreements signed to that purpose, with no significant deviations from the agreed budget. Most project holders (62%) reportedly accomplished their project objectives. Performance level achieved by project holders was very good and, when interviewed, project holders declared their satisfaction in having taken part in the Funding Scheme. As those indicators identified by the Funding Scheme s Evaluation Team illustrate, cofunded projects proved extremely effective. The Funding Scheme produced more than 250 publications, almost 500 Information and Communication Activities and more than 500,000 SMEs who benefited indirectly from the Funding Scheme. These indicators were quantified with the information obtained from questionnaires, Application Forms and Final Activity Reports from the 51 co-funded projects. Project holders were highly motivated and showed interest in completing their projects, something that was a key issue for the success of this first Funding Scheme. Co-funded projects also showed a high level of efficiency. Established objectives were achieved with no significant deviations from the budget originally set for the projects. The projects presented the following positive results: 9

14 An effective partnership. The average number of partners per project was three (five for transnational projects and two for national projects). Project holders appraised the partnership work carried out as being very good. Project holders particularly stated that transnational partnerships gave them the opportunity to learn about occupational safety and health situations in other countries and sectors. An effective participation of the social partners and workforce. Project holders stated that the social partners and workforce provided constant support throughout the project implementation process. The workforce was the clear recipient of training activities, seminars and information. Project holders consulted expressed their satisfaction regarding both the participation of the workforce and their social partners in the projects. Projects targeted SMEs and micro-businesses. This was especially clear in transnational projects, 88% of which concentrated on SMEs, as compared to 71% in the case of national projects. Projects targeted priority and high-risk sectors (construction, metalworking, fisheries, etc.). The lack of time available to implement projects reduced the effectiveness of projects. This aspect had a more damaging effect on smaller organisations that had little experience in implementing projects and programmes of this nature Project Sustainability Project sustainability was one of the selection process criteria to which projects had to comply. Accordingly, all projects developed under the Funding Scheme were sustainable. However, an independent assessment of project applications and Final Activity Reports showed that some project holders misunderstood the terms sustainability and dissemination, thinking that projects were sustainable when they were properly disseminated. A project is sustainable when its results last in the middle- or long-term. The definition of sustainability should be clarified in the Call for Project Proposals. It would be then advisable to insist in the Call that project sustainability should focus on: Sustainability of the results of projects, once funding has ceased. Project results that do not harm the environment. Sustainability was a key criterion for evaluation and a precise, clear definition of the same in the future would avoid confusion among project holders and guarantee potential applicants making an effort to submit sustainable projects. In view of the brief period of time that has elapsed since the projects were completed, it is difficult for us to assess the impact of the Funding Scheme. In spite of this, we believe that the Scheme had a favourable impact on heightening the awareness of occupational accident risks at national and EU levels. As a result, most project holders (56%) pointed out the excellent contribution of the Scheme concerning SME needs regarding occupational safety and health. In the opinion of project holders, the Scheme allowed for work to start in a field where there was an important need for prevention of accidents in 10

15 SMEs. Direct participation of SMEs and micro-businesses also gave an insight into the true situation small companies faced. The Funding Scheme heightened the awareness of accident prevention, fostered training and produced information material allowing project holders to continue working on accident prevention in the future Transferability of project results The transferability of project results was a key issue concerning guaranteeing the success and impact of the Funding Scheme. Transferability of project outcomes was a selection criterion applied in the selection process, a condition that automatically lead to the conclusion that all results from co-funded projects were transferable. Main project results such as publications, CD-ROMs, websites and training materials were, in general, easy to transfer at low cost. All project holders stated that their project results were easily transferable to other SMEs. The evaluation of Project Applications and Final Activity Reports confirmed this fact. Consulted National Focal Points were of the same opinion inasmuch as the dissemination of project results provided excellent opportunities for exchanging experiences and practical solutions. National Focal Points insisted on the need to disseminate all projects at a national level, not only those corresponding to their own country, but also those of other countries as well. In this respect, the Evaluation Process includes a number of recommendations to ensure transferability: An initial kick-off or start-up meeting with selected project holders. This would allow project holders to exchange ideas and experiences and could well be the first step towards encouraging and reinforcing the transfer of results. The Agency will edit a publication containing all project results for dissemination purposes. Inviting project holders to participate in promotional events organised by the Agency. Dissemination of documents in several languages Impact of the Funding Scheme Due to the fact that it heightened awareness of Occupational Safety and Health at EU level, the impact of the Funding Scheme was considered satisfactory. The SME Scheme was successful in improving working conditions and preventing accidents in the workplace, especially regarding those projects oriented towards Information and Communication Activities. Of all activities included in the Funding Scheme, those activities in the Information and Communication Project category reached the largest target public. The Agency intended to spend its budget equally between national and transnational projects. Most of the co-funded projects, however, were national (69%). Transnational 11

16 projects required larger budgets in order to carry out their activities and more time for coordination and dissemination purposes. The majority of co-funded projects fell within the Information and Communication category. The target public of the projects in this category is the largest of the three. Most Information and Communication activities concentrated upon improving access to occupational safety and health information and encouraging workforce participation in Occupational Safety and Health Practices. The survey indicated that 80% and 78% of project holders respectively were of this opinion. Analysis of the evaluation results and assessment of Applications and Final Activity Reports allows us draw the conclusion that this category was the most efficient project category, especially concerning transnational projects. The Provision of Good Practices was the second largest category. An analysis of the qualitative data, Project Applications and Final Activity Reports illustrated that project holders had difficulties in understanding the meaning of Good Practices. It is therefore recommended to include a more precise definition of this project category in the Call for Project Proposals. Training was the project category that received the least funding. The target public of this category - mainly workforce - was also quite restricted. As a result of national regulations, language differences and other factors, these projects had limited impact in terms of dissemination and transference of final results at EU level. The strategy and objectives of the Funding Scheme did indeed contribute to a reduction in the risk of accidents and the objectives it pursued coincided with the needs of project holders to a great degree. Hence, the Scheme proved most useful in preventing accidents in the workplace. 12

17 9. Chapter 5: Recommendations for Future Funding Schemes Priority Recommendations 1. To restructure the SME Accident Prevention Funding Scheme into a Pluri-annual Programme in order to avoid time constraints. 2. To heighten the visibility of the SME Accident Prevention Funding Scheme in relation to other Community Programmes. 3. To establish a co-ordination mechanism to assure the complementarity between different Community institutions in occupational safety and health matters, thereby avoiding overlaps and activities with ones having similar contents. 4. To assure transparency and impartiality in the evaluation process. Subcontracting the evaluation of applications to external specialists will prevent problems arising in the evaluation procedure. 5. Should the Funding Scheme continue to operate on an annual basis, it would be advisable to slim down the entire procedure. The Application Form should be simplified as much as possible. This would reduce the work involved for potential project applicants as well as for those taking part in the task of project evaluation. 6. The Funding Scheme should facilitate the participation of Eastern European countries, conceding greater preference to those projects including partners from Eastern countries. This will contribute towards reducing the differences in Occupational Safety and Health concerns between EU and Eastern countries. 7. To dedicate more staff to management the SME Accident Prevention Funding Scheme. The Campaigns and Programmes Task Force managed to put this Scheme into operation with a team made up, basically, of only four people. The team worked in a high workload environment under a great deal of pressure in order to comply with the Evaluation Scheme s objectives and tight deadlines. In view of the Scheme s timetable, allocating additional staff to handle the Funding Scheme is recommended. 8. Documents disseminated across Europe should be published in the highest number of languages possible. In order to do this, the possibility of collaborating with National Focal Points in the translation of these documents should be considered. 9. Barely six months have elapsed since completion of the Funding Scheme projects, something that has made assessing the impact of these projects a difficult task. Simple monitoring of projects from the Funding Scheme is recommended. 13

18 Specific Recommendations 1. The need for more time to complete the projects. Project holders had difficulties implementing their projects within the deadlines, which reduced project effectiveness. 2. To stress the sustainability of the projects, thereby avoiding confusion between sustainability and dissemination of project results. 3. To improve the transferability of the results of projects. A start-up, initial meeting with project holders might well be a first step towards reinforcing the transfer of results, together with the need to edit a publication contemplating the results of the Funding Scheme in the highest number of languages possible. 4. To encourage the participation of SMEs, associations and consortiums in order to increase the number of SME project holders. 5. The Agency should organise an initial, kick-off meeting with project holders in order to clarify evaluation scheme procedures and to allow project holders to establish contact with both the Agency and other project holders. 6. The need to support potential applicants in creating partnerships. The Agency should create and disseminate a database containing details of those applicants who submitted applications to the Funding Scheme. 7. To improve the definition of each project category. The Funding Scheme co-funded three different types of activities: Training, Information and Communication Activities and the Provision of Good Practices. These categories caused a certain degree of confusion among project holders, some of whom did not really know in which category they should participate. 8. Much stricter eligibility criteria should be applied in order to ensure the technical and administrative quality of the projects submitted. This would stop the Agency s receiving applications having little likelihood of being accepted. 9. To improve the clarity of the Call for Project Proposals and the Application Form. To include reference indicators in the Application Form in order to allow for projects to be monitored and evaluated. 10. To extend the Call for Project Proposals publicity campaigns in order to improve the diffusion of the message. 11. To establish a minimum number of promotion activities by all National Focal Points in order to counter-balance the dissemination and publicity of the Call for Project Proposals. 12. To use the Agency s website as a means for channelling all the information produced as a result of the Funding Scheme. The SME Accident Prevention Funding Scheme should have a heightened, more visible presence on the Agency's website. 14

19 0- The Structure of the Evaluation report The Evaluation Report contains an Executive Summary and five Chapters. Chapter 1 introduces the objectives of the evaluation, evaluation questions, methodology used and the different evaluation phases. In Chapter 2, the report proceeds with a description and detailed analysis of programming, management, monitoring and promotion issues related to the Funding Scheme. Chapter 3 examines the coherence of the Funding Scheme. This analysis is carried out within the framework of internal coherence with other community policies. Since the Funding Scheme was launched by the European Agency on Safety and Health at Work, this analysis should consider the context of other related policies and programmes of the European Commission. Chapter 4 deals with that information collected either through surveys or by means of personal interviews with beneficiaries of the Funding Scheme, the Agency, the National Focal Points and other parties involved. Likewise, an independent assessment was made of the project documents provided by the Agency (Application Forms and Final Activity Reports) together with other relevant documents. After obtaining a clear idea of the Funding Scheme and the context in which it is operating, the next step of the evaluation process focused on the analysis of results from the survey and interviews held with parties involved in the Scheme. The Evaluation Report includes a set of recommendations to improve the Funding Scheme in the future. 15

20 1- Objectives and Methodology 1.1- OBJECTIVES OF THE EVALUATION The purpose of this Evaluation Report is to analyse the SME Accident Prevention Funding Scheme , the Scheme set up to tackle work-related accidents in Small and Medium-sized Enterprises (SMEs) throughout the European Union. The European Agency of Safety and Health at Work 2 launched the Funding Scheme in April The Agency is a tripartite European Union organisation bringing together representatives from three key decision-making groups namely, governments, employers and worker s organisations - in each of the European Union s Member States. The different bodies making up the Agency network are: The Administrative Board: establishes the Agency's goals and strategy. The Bureau: oversees the Agency's operational performance. The Agency: The Director of the Agency is the official representative of the Agency and is responsible for day-to-day Agency operations. The National Focal Points: these are the Agency s principal Safety and Health Information networks in each Member State. Each government designates National Focal Points as the Agency's official representative in that country and these are normally considered the competent national authority for Safety and Health at Work. The role of the Focal Points is to provide information and feedback to Agency initiatives and products and they are consulted on all information activities related to national levels. This report is the First Evaluation of the Funding Scheme. To a great extent, the results stemming from this evaluation will provide useful information for improving future Funding Schemes. The specifications contemplated in the Restricted Call for Tender (OSHA/TF/2002/T2) stipulate that external evaluation should be made of the Funding Scheme. This evaluation has focused upon the following items: Agency s Call for Project Proposals Formulation of the Call for Project Proposals, Application Form and Technical Specifications. Promotion of the Funding Scheme by the Agency. Deadline for submitting applications (from the publication of the Call to the closing date for the submission of applications) Processing applications by the Agency Quality of the service rendered by the Agency to potential applicants. Suitability of the timetable for receiving, processing and selecting applications. Selection procedure (national and transnational projects) and advisory bodies Was the selection procedure reasonable, transparent and fair? 1 Hereinafter called the Funding Scheme 2 Hereinafter called the Agency 16

21 Suitability of timetable for project selection. Suitability of national and international groups of experts. Involvement of social partners. Quality of the information provided by the Agency for project selection. Quality of selected projects in relation to the objectives of the Funding Scheme. Implementation of selected projects Performance of project holders. Agency performance when accompanying monitoring projects. Clarity of the Interim and Final Activity Reports. Quality of the information provided by the Agency concerning project selection. Results and Impact Outcome of projects. Sustainability of projects. Impact of projects. Transferability of project results. Overall Assessment of different project categories 1.2- METHODOLOGY FOR THE EVALUATION OF THE FUNDING SCHEME Before describing the methodology used in the Evaluation, a few comments must be made concerning the difference between programme evaluation and monitoring. These activities are inter-related, but their objectives are quite different. Monitoring refers to regular analysis of the progress being made with the programme, using a follow-up of all programme-related activities. By and large, monitoring implies gathering a great deal of information and adopting clear criteria in the selection and organisation of this relevant information. Information produced by the monitoring process should be used as input in the evaluation process. Evaluation assesses the implementation of a programme, not only in terms of its impact, but also from the viewpoint of those planning, management and monitoring procedures adopted. It should be clearly stated that, since monitoring does not, on occasion, produce information in an easily accessible way, evaluators are mistakenly asked to supply this. However, the role of the evaluator is neither to produce the information supplied or produced by the monitoring process mechanism, nor to organise this in a proper manner when it is found scattered over numerous files and documents. As the Commission itself states, 3 the expectations placed on evaluation should not exceed what it can really contribute. Evaluation should not be given roles it is not designed to play, such as auditing and monitoring. 3 MEANS Collection, Volume 1. Evaluating socio-economic programmes. Evaluation design and management. European Commission. 17

22 Methodology Structure From a methodology viewpoint, evaluation of the Funding Scheme was based on gathering and analysing two different sources of information: Information provided by the Agency in relation to the Funding Scheme: Application Forms and Final Activity Reports of the 51 co-funded projects The SME 2001 Call for Project Proposals The SME 2001 Application Form Technical Specifications and Checklists The SME 2001 Decision and Notification of the SME 2001 Decision to National Focal Points Financial Planning for the Selected SME 2001 Projects Documents (Matrix Document for the Bureau, Feb-March 2001, SME 2001 Key Issues, Common Guidelines for National Focal Points) Notes (to the Bureau, Feb-March 2001, to National Focal Points and Board Members to inform of the SME 2001 Call, Overall Note for the Assessment Meeting, 17 th 18 th October 2001) Minutes (SME/EW Group Meeting 29 th 30 th March 2001, SME European Jury Meeting 13 th 14 th September 2001 and Jury Recommendations, Assessment Meeting, 17 th 18 th October 2001). Information generated during the evaluation The evaluation has made use of qualitative information collected using surveys (questionnaires sent to project holders and National Focal Points), complementary activities (telephone calls) and interviews (personal and telephone interviews with project holders, interested parties, National Focal Points and the Agency). The following groups were targeted: Beneficiaries of the 2001 Funding Scheme (mainly project holders) Agency staff Agency Administrative Board Members European Jury and SME/EW Group National Focal Points Other Interested parties: European Commission, European Parliament, European Trade Union Confederation (ETUC/TUTB), Union of Industrial and Employers Confederations of Europe (UNICE), Federation des Entrepreneurs Generaux de la Construction, ECOSOC (Economic and Social Committee of the European Union) 18

23 The methodology used for collecting and analysing the information was based on deskwork (compilation and analysis of information) and fieldwork. These processes are set out below (see chart in Annex 1): Gathering of information from the Agency Identification and preliminary analysis of essential aspects of the Funding Scheme Elaboration of Questionnaires for Project Holders and National Focal Points: Survey Personal interviews (Agency Members, Project Holders, National Focal Points, Interested Parties) Summary of Information generated during the evaluation process Evaluation phases Funding Scheme evaluation was structured as per the 4 phases described in the following table: 19

24 Table1: Evaluation Phases (from the 21 st November 2002 to the 31 st March 2003) Main Objectives Timescale Meetings Reports Phase 1: 21 st November Structuring 5 th December Phase 2: Data Collection and Preliminary Analysis Phase 3: Fieldwork Phase 4: In-Depth Evaluation and Assessment Structure of general evaluation method Collection of information provided by Agency Questionnaire target groups Preliminary selection of project holders and other interested party interviews Templates for questionnaires (in English, French, Spanish, Italian and Portuguese) Despatch of questionnaires to 51 project holders and 15 National Focal Points (20 th December) Reception of questionnaires Reinforcement activities Interviews Final reception of questionnaires End of Interviews Draft of the Final Evaluation Report Final Evaluation Report 2 weeks 5 th 26 th December 3 weeks 26 th December 30 th January 5 weeks 30 th January 31 st March 8 weeks Kick-off meeting 21 st November Progress meeting 11 th December Progress meeting 31 st January Progress meeting: 21 st February Meeting on Draft Evaluation Report: 20 th March First Progress Report: 5 th December This Progress Report covered progress made from the kick-off meeting. The objective of the report was to present, on one hand, the planning for implementing the foreseen evaluation tasks (work plan, time schedule) and, on the other, the methodology used in carrying out the evaluation. Inception Report: 19 th December The Inception Report was based on the First Progress Report. The objective of the Report was to present the conclusions of the Planning Phase. The Report included an agreed work-plan and evaluation framework (evaluation focus, methodology used, target groups for interviews and indicators). Templates for questionnaires were included in the Annex. Second Progress Report: 30 th January This report described the results and activities carried out during Phase 3: the activities implemented regarding the questionnaires and interviews. Checklists for interviews were included in the Annex. Findings: 19 th February Main results obtained from the interpretation of the survey process were presented on this report. Third Progress report: 28 th February The Third Progress Report described the results and activities carried out during Phase 4 (from the end of Phase 3 to the end of February). Draft Evaluation Report: 13 th March Final Evaluation Report. 31 st March 20