Mid-term evaluation of Erasmus+ and the predecessor programmes: Lifelong Learning Programme and Youth in Action

Transcription

1 Ref. Ares(2017) /07/2017 Mid-term evaluatio of Erasmus+ ad the predecessor programmes: Lifelog Learig Programme ad Youth i Actio FINAL REPORT Warsaw, 30 Jue 2017

2 Mid-term evaluatio of Erasmus+ ad the predecessor programmes: LLP ad YiA Cliet: Miistry of Natioal Educatio al. J. Ch. Szucha Warszawa Study coducted by: Evalu Sp. z o.o. (cosortium leader) ul. Dzika 19/23 lok Warszawa Phoe: Fax: WYG PSDB Sp. z o. o. (cosortium parter) ul. Bitwy Warszawskiej 1920 r Warszawa Phoe: Fax: Edited by Paweł Peszko Report authors: Jarosław Chojecki Paweł Huras Aleksadra Kamińska Magda Matysiak Paweł Peszko Wojciech Pieiążek Cezary Przybył Jerzy Radziwiłł Irea Wolińska Małgorzata Zub 2 Page

3 Mid-term evaluatio of Erasmus+ ad the predecessor programmes: LLP ad YiA Cotets EXECUTIVE SUMMARY INTRODUCTION Subject ad objectives of the study Geeral research cocept DISCUSSION OF STUDY METHODOLOGY STUDY RESULTS Assessmet of effectiveess of measures take to achieve Programme objectives Assessmet of Programme efficiecy Assessmet of Programme relevace Assessmet of iteral ad exteral Programme coherece Assessmet of Europea added value of the Programme CONCLUSIONS AND RECOMMENDATIONS APPENDICES Page

4 Mid-term evaluatio of Erasmus+ ad the predecessor programmes: LLP ad YiA EXECUTIVE SUMMARY This report was created as part of the mid-term evaluatio of the Erasmus+ programme ad the predecessor programmes: Lifelog Learig Programme ad Youth i Actio, coducted i 2017 by the cosortium of Evalu sp. z o.o. ad WYG PSDB sp. z o.o., ad commissioed by the Miistry of Natioal Educatio. The evaluatio covers decetralised actios beig implemeted i Polad. The followig activities have bee performed as part of this evaluatio: desk research, diagostic ad creative workshop, survey of beeficiaries, usuccessful applicats, ad participats, case studies of participatig orgaisatios ad idividuals, idividual i-depth iterviews with represetatives of the Natioal Authorities, the Natioal Agecy ad the Idepedet Audit Body, experts who evaluate applicatios, ad represetatives of istitutios that did ot apply for co-fiacig. The followig key fidigs have bee formulated as a result of the evaluatio: 1. Actios covered by the evaluatio cotribute to the performace of specific objectives ad the majority of geeral objectives of the Erasmus+ programme. The strogest effects are produced by the iteratioalizatio of al istitutios ad traiig istitutios, givig youg people opportuities for itercultural commuicatio, icreasig the level of participats laguage competeces, self-reliace, ad self-cofidece. However, the evaluatio did ot idetify mechaisms that would lead us to believe that decetralised iitiatives sigificatly cotribute to a reductio i the umber of early drop-outs from, to a icrease i the share of populatio with higher, or to a icrease i ad equalizatio of opportuities of all youg people o the labour market (see sectio 3.1 for further details). 2. The followig iitiatives were particularly effective: a. with respect to the developmet of key competeces mobility of VET learers ad school staff b. with respect to the support for labour market participats studet traieeships ad mobility of vocatioal studets c. with respect to stregtheig the potetial of orgaisatios mobility i the HE sector, ad strategic parterships (sectio 3.1). 3. The Programme objectives are up-to-date from the viewpoit of stakeholders, i.e. project providers, participats, academic circles ad implemetig istitutios staff. Noetheless, there are eeds for a additioal directio for itervetios (sectio 3.3). 4. There is a variatio betwee regios i the level of activity of applicats ad Programme participatio. NA udertakes adequate activities couteractig this variatio (sectio 3.1). 5. Itegratio of sectoral programmes i a sigle programme facilitated their uificatio, harmoisatio, ad the existece of syergies betwee projects, as well as reduced the admiistrative costs, ad simplified ad streamlied the maagemet of the Programme. The curret structure of the Programme is adequate (sectio 3.1). 6. The budget size was adequate for the Programme s objectives thaks to the ivolvemet of additioal ad sigificat fuds from the Europea Social Fud. Provisio of aother source of fudig is recommeded i case the ESF co-fiacig ceases (sectio 3.1). 7. The simplified grat system eabled the reductio of the admiistrative burde for the Natioal Agecy ad programme beeficiaries ad participats. Efforts aimig to further limit or elimiate the burde should be cotiued (sectio 3.2). 8. The system of cooperatio ad divisio of resposibilities betwee Polish istitutios participatig i the executio of Erasmus+ is efficiet ad fuctios well. The level of huma ad fiacial resources that is available for the implemetatio of the Programme i Polad is adequate. Further arragemets ad possible improvemets are required for certai areas cocerig cooperatio betwee the Commissio ad atioal istitutios (sectio 3.2). 4 Page

5 Mid-term evaluatio of Erasmus+ ad the predecessor programmes: LLP ad YiA 9. NA s actios aimed at target groups, icludig iformatioal ad promotioal campaigs, are effective. Moitorig visits, coducted to support beeficiaries i project executio, also elicit a positive evaluatio. Proposals for certai modificatios of coducted activities are give i sectio Erasmus+ is very well kow i the HE ad VET sectors, ad well kow i the SE sector. I the AE ad Youth sectors it is kow to a certai part of the potetial beeficiaries ad oly some of them have ecessary resources to apply for fudig (sectio 3.3). 11. A rage of factors hamperig the Programme s implemetatio has bee idetified; however, they do ot costitute severe burdes threateig the achievemet of the Programme s objectives (sectio 3.1). 12. As a result of the study, a array of recommedatios have bee formulated (preseted i detail i sectio 4), cocerig: a) reviewig applicatio evaluatio criteria to icrease the umber of ew applicats ad decrease the prevalece of experieced beeficiaries; b) reistatig the fudig of preparatory visits; c) algorithm for calculatig costs of itellectual output of idividuals participatig i the projects; d) lump-sum travel fudig; e) requiremets for the implemetatio ad dissemiatio of itellectual output, ad icreased fudig for this activity, ad eforcemet of the requiremets; f) further actios reiforcig the use of the approach based o the learig outcomes at al facilities participatig i the Programme; g) providig potetial participats with a mobility assessmet expressed by curret participats; h) support for disadvataged target groups; i) extedig support forms with short mobilities; j) promotio i the commuicatios policy of positive attitudes towards foreigers i Polad ad the possibility of coductig strategic parterships for combatig discrimiatio; k) iformatioal activities related to implemetig core curriculum through projects; l) remueratio for implemetig tachers; m) promotio of liguistic support. 5 Page

6 Mid-term evaluatio of Erasmus+ ad the predecessor programmes: LLP ad YiA 1 INTRODUCTION 1.1 Subject ad objectives of the study 1 The mai objective of the study was to assess the implemetatio of the Erasmus+ Programme i Polad, takig ito accout the previous Lifelog Learig ad Youth i Actio programmes. The evaluatio referred to the followig detailed areas: effectiveess of measures take to achieve Programme objectives; Programme efficiecy; Programme relevace; exteral ad iteral Programme coherece; Europea added value of the Programme. The study used the followig evaluatio criteria: 1. Effectiveess uderstood as a assessmet of the degree to which the objectives of the Erasmus+ programme have bee achieved; 2. Efficiecy uderstood as a assessmet of the ratio betwee expeditures (fiacial ad huma resources) to results ad outcomes achieved i the Erasmus+ programme; 3. Relevace uderstood as a assessmet of whether ad to what degree the adopted Erasmus+ objectives match idetified issues i the area covered by the Programme. 1.2 Geeral research cocept This evaluatio was a mid-term evaluatio, i.e. oe coducted while the Programme was ogoig. The purpose of the coducted study was to estimate the degree to which the assumed objectives have bee achieved i light of previous assumptios ad to idetify the relevace of such assumptios i relatio to curret situatio i the itervetio area. The results of mid-term evaluatio are used potetially to modify programme documets, ad primarily to optimize the itervetios. Iformatio obtaied thereby ca also be used whe desigig itervetios i the ext programmig period. Because of these factors, this evaluatio will be used to verify the chage theory adopted i the programmig. We took the Theory Based Evaluatio (TBE) approach recogized as the most adequate i these circumstaces. 2 DISCUSSION OF STUDY METHODOLOGY Desk research Aalysis of secodary data ecompassed documets relevat to the Erasmus+ programme, as well as documets relevat to the Lifelog Learig ( ) ad Youth i Actio ( ) programmes. Other aalysed items icluded promotioal ad iformatioal material o the Programmes made available, amog others, by o-govermetal orgaisatios ad other beeficiaries, as well as other results of dissemiatio activities. We also used public statistical data (from the CSO ad Eurostat amog others) o higher, school ad target groups of the Programmes, as well as participat data collected withi the Programme. For the purposes of aalysig exteral cosistecy, we also aalysed data ad documets cocerig regioal exchage programmes, scholarships ad traieeships. The aalysis also icluded blogs ad Iteret forum posts related to Programme mobilities. The documets aalysed to aswer particular research questios are listed i Appedix 3. 1 Chapter compiled o the basis of the Terms of Referece. 6 Page

7 Mid-term evaluatio of Erasmus+ ad the predecessor programmes: LLP ad YiA Aalysis of secodary data from participat reports The Research team was grated access to electroic databases cotaiig uit data from idividual participat reports. Addresses were provided together with aswers foud i participat reports for those participats who coseted to havig their addresses disclosed for the purposes of evaluatio. The aswers were subjected to quatitative aalysis. Questioaire iterviews Questioaire iterviews were coducted with the followig groups: beeficiaries (2300 filled-out questioaires) ad usuccessful applicats (472); idividual participats (3736). Case studies (icludig biographical studies) Two types of case studies were coducted: a moographic case study of the beeficiary istitutio (6 studies), a biographical case study of a project participat (9 studies). Idividual i-depth iterviews (IDI) The iterviews were coducted with represetatives of atioal istitutios ivolved i implemetig the Erasmus+ Programme. 1. Natioal Agecy Foudatio for the Developmet of the Educatio System (FRSE) 8 iterviews 2. Natioal Authority: The Miistry of Natioal Educatio Departmet of Structural Fuds 1 iterview; The Miistry of Sciece ad Higher Educatio Departmet of Iovatio ad Developmet 1 iterview. 3. Idepedet Audit Body a sectio auditig Erasmus+ i the Miistry of Fiace, Departmet of Public Fuds Audit 1 iterview. Additioally, idividual iterviews with experts assessig the applicatios i each sector were coducted 15 iterviews i total. Aother group of respodets i idividual iterviews will be represetatives of istitutios from the, traiig ad youth areas which so far have ot applied for co-fiacig from the Erasmus+ programme or previous programmes covered by this study 16 iterviews coducted. Diagostic ad creative workshop 5 workshops were coducted, oe for each sector. Each workshop was atteded by 9-10 participats represetig various stakeholder groups (beeficiaries, project participats, employers, sector experts). More detailed iformatio about the methodology is give i Appedices 1 (Methodology report), 2 (umber of quatitative iterviews) ad 3 (methodology used for each research questio). 3 STUDY RESULTS 3.1 Assessmet of effectiveess of measures take to achieve Programme objectives 1.1. To what extet have Erasmus+ ad its predecessor programmes cotributed to the realisatio of the Erasmus+ specific objectives i Polad? Are there ay differeces across fields? Please provide, where relevat, your assessmet for each of the specific objectives ad provide evidece ad examples where possible. 7 Page

8 Mid-term evaluatio of Erasmus+ ad the predecessor programmes: LLP ad YiA Specific objective: to improve the level of key competeces ad skills, with particular regard to their relevace for the labour market ad their cotributio to a cohesive society, i particular through icreased opportuities for learig mobility ad through stregtheed cooperatio betwee the world of ad traiig, ad the world of work; The Erasmus+ programme eables achievig this objective i all fields maily through the developmet of laguage competeces (which are addressed also i aother specific objective) ad soft competeces (such as ability to pla oe s learig ad work, readiess to face challeges, opeess to other cultures). The latter are liked to the commoly observed effect of improved self-cofidece ad self-reliace of the participats for whom a idividual, log travel abroad costitutes a importat experiece at the oset of adulthood. This type of psychological chages also affects the way a perso fuctios i his/her professioal ad social life. A related outcome of travels, also cofirmed by may sources (beeficiaries 2, idividual participats, NA staff), is gettig acquaited with multiligual ad multicultural social eviromets, which i these times of globalisatio provides a valuable experiece makig it easier for a perso to take up professioal or scietific career i such eviromets. Furthermore, two i three studets udergoig traieeships ad VET learers have declared a icrease i vocatioal skills. Give the high mobility i the HE sector, the impact of the Erasmus+ Programme o the achievemet of this objective should be assessed as substatial. Especially beeficial, however, as regards the improvemet of competeces ad self-cofidece, is mobility of VET learers. The reaso for this may be greater iitial deficits ad less iteratioal experieces. Specific objective: to foster quality improvemets, iovatio excellece ad iteratioalisatio at the level of ad traiig istitutios, i particular through ehaced trasatioal cooperatio betwee ad traiig providers, ad other stakeholders; Data collected durig iterviews show that this objective was achieved. The study results have provided a solid foudatio for the thesis accordig to which the Erasmus+ programme cotributes to the iteratioalisatio of the activities udertake by ad traiig istitutios. I respect of may actios, i particular Key Actio 2 of the Erasmus+ programme, the Grudtvig ad Leoardo da Vici parterships, ad mobility i the HE sector (both LLP ad Erasmus+), most beeficiaries declared that participatio i the Programme facilitated the broadeig of iteratioal cotacts betwee orgaisatios ad the exchage of experiece. Accordig to the represetative of the NA, the requiremets applicable to strategic parterships eforce geuie cooperatio betwee istitutios from differet coutries, rather tha a mere distributio of tasks. A lower, although still substatial, proportio of beeficiaries have observed the effect of ehacemets of the way the orgaisatio operates (the largest of which, of more tha 50%, occurred i mobility i the HE sector). At least 40% of staff from each Erasmus+ sector expressed a opiio that their participatio traslated ito various aspects of operatio of their paret istitutios, such as teachig methods, maagemet, iteratioal cooperatio, etc. Observig foreig practices ecourages to itroduce them i domestic istitutios. Such reports, provig that the Programme has achieved its specific objective, were recorded also i case studies ad previous research i the SE sector coducted by NA 3. Specific objective: to promote the emergece ad raise awareess of a Europea lifelog learig area desiged to complemet policy reforms at atioal level ad to support the moderisatio of ad traiig systems, i particular through ehaced policy cooperatio, better use of EU trasparecy ad recogitio tools ad the dissemiatio of good practices; The decetralised actios of Erasmus+ ad its predecessor programmes, beig the subject of this evaluatio, pertaied to this specific objective oly margially, although, from the perspective of the NA the Programme supported the implemetatio of domestic reforms, particularly i the SE (e.g. itegratig ew techologies) ad VET (e.g., collaboratio with employers) sectors. There was also a exchage of good practices betwee 2 Throughout this report, beeficiaries are uderstood as participatig orgaisatios, whose represetatives were iterviewed. 3 Pachocki M. (2016). Mobile teachers chage their schools. Warszawa: Fudacja Rozwoju Systemu Edukacji. 8 Page

9 Mid-term evaluatio of Erasmus+ ad the predecessor programmes: LLP ad YiA schools. Stakeholders participatig i workshops poited out problems with the recogitio of qualificatios obtaied i Erasmus+ projects (i case of higher studets also with the recogitio completed classes) i their home coutries, ad metioed a iadequate itegratio of Erasmus+ mobility with EU recogitio tools. Nevertheless, FRSE s activities promotig such a itegratio (e.g. the activities of the Polish ECVET Expert Team, iformatioal brochures) should be commeded. Specific objective: to ehace the iteratioal dimesio of ad traiig, i particular through the cooperatio betwee Programme ad Parter-Coutry istitutios i the field of VET ad i higher, by icreasig the attractiveess of Europea higher istitutios ad supportig the EU's exteral actio, icludig its developmet objectives, through the promotio of mobility ad cooperatio betwee Programme ad Parter-Coutry higher istitutios ad targeted capacity buildig i Parter Coutries; Iterviewees from NA oted that the Polish higher istitutios, especially from the Lubuskie ad Podkarpackie regios, used the fudig obtaied uder the Erasmus+ programme to attract studets from the Easter Partership coutries, i particular Ukraie. This shows the effort made i order to achieve the specific objective. The impact of this effect is limited due to the fairly low budget allocated to mobility with the participatio of the Parter Coutries. Specific objective: to improve the teachig ad learig of laguages ad to promote the Uio s broad liguistic diversity ad itercultural awareess. The impact of mobility projects o liguistic competeces has bee cofirmed by almost every participat. This impact is of a threefold ature: first of all, it etails the improvemet of laguage skills because the laguage is used o a daily basis; secodly, it helps to overcome metal barriers associated with speakig a foreig laguage (i.e. it boosts self-cofidece); thirdly, it raises awareess about the laguage as a useful medium which eables commuicatio with people of other atioalities. Furthermore, the stakeholders, icludig the participats themselves, uiversally report eterig ito actual cotact with people livig abroad ad with their cultures, which is a experiece ot to be foud durig tourist travels. The participats ofte cofirm that they maitai the cotacts they made durig travels, which is of cosiderable importace to overcome the sese of alieatio from other Europea cultures. Furthermore, oe should remember that, accordig to aecdotal evidece but also earlier studies 4, acquaitaces resultig from the participatio i mobility ofte tur ito log-lastig relatioships, also marriages, which form a sort of bridge across the respective atioal cultures. Improved liguistic ad pedagogical competeces of laguage teachers costitute a additioal chael through which the Programme exerts its impact. Therefore, the Programme's impact with regard to the implemetatio of the adopted specific objective should be regarded as particularly strog. Specific objective: to improve the level of key competeces ad skills of youg people, icludig those with fewer opportuities, as well as to promote the participatio i democratic life i Europe ad the labour market, active citizeship, itercultural dialogue, social iclusio ad solidarity, i particular through icreased learig mobility opportuities for youg people, those active i youth work or youth orgaisatios ad youth leaders, ad through stregtheed liks betwee the youth field ad the labour market; Most of the beeficiaries ad participats i the Europea Volutary Service, the mobility of youth workers ad youth exchages uder the Erasmus+ programme have cofirmed the acquisitio of key competeces (except for commuicatio i the mother togue), such as commuicatio i foreig laguages, learig how to lear, sese of iitiative ad etrepreeurship ad the stimulatio of social ad civic egagemet. EVS also helps to activate local commuities 5. This costitutes a argumet i favour of ackowledgig the Programme's impact o the implemetatio of the specific objective i questio, although its itesity would be hard to assess. 4 The Erasmus Impact Study (2014). Luxembourg: Publicatios Office of the Europea Uio. 5 Jeżowski M., (2016). Wpływ projektów Wolotariatu Europejskiego a społeczości lokale. Warszawa: Fudacja Rozwoju Systemu Edukacji. 9 Page

10 Mid-term evaluatio of Erasmus+ ad the predecessor programmes: LLP ad YiA Specific objective: to foster quality improvemets i youth work, i particular through the ehaced cooperatio betwee orgaisatios i the youth field ad/or other stakeholders; 46% of beeficiaries of the Youth sector foud the Erasmus+ programme to be sigificatly cotributig to the objective of improvig youth-orieted activities, ad 37% declared the same with regard to the icreased iovativeess of the activities. While fairly large, these proportios are lower tha i other objectives to which the questios pertaied. O this basis, the, the impact which the Programme exerts o the achievemet of the specific objective ca be regarded as moderate. Specific objective: to complemet policy reforms at local, regioal ad atioal level ad to support the developmet of kowledge ad evidece-based youth policy as well as the recogitio of o-formal ad iformal learig, i particular through the ehaced policy cooperatio, better use of Uio trasparecy ad recogitio tools ad the dissemiatio of good practices; The oly actio desiged to achieve this type of effects as part of decetralised actios is the KA347 Support for reforms i actio. Its impact o the achievemet of the objective is limited due to the small scale of the activities (49 cotracts ad participats i years , compared to more tha 2 thousad regioal ad local self-govermets ad more tha 6 millio youg people). Specific objective: to ehace the iteratioal dimesio of youth activities ad ehace the capacity of youth workers ad orgaisatios i their support for youg people i complemetarity with the Europea Uio's exteral actio, i particular through the promotio of mobility ad cooperatio betwee stakeholders from Programme ad Parter Coutries ad iteratioal orgaisatios. Oe ca observe a widespread iterest (esp. amog youth workers) i mobility projects with the participatio of easter Europea coutries ad the Caucasus. The support for the Erasmus+ programme therefore beefits from the itesificatio of iteratioal activities of this type. This effect is limited by the threshold uder which ot more tha 25% of the Key Actio 1 budget ca be allocated to the cooperatio with Parter Coutries. A lesser ivolvemet o the part of the Parter Coutries was observed i Key Actios 2 ad 3, where they participated i a umber of projects To what extet has the progress o the realisatio of the specific objectives cotributed to the realisatio of the Erasmus+ geeral objectives i Polad? The aswer will be broke dow ito idividual Erasmus+ geeral objectives. Achievemet of the Europe 2020 Strategy objectives Erasmus+ decetralised actios do ot cotribute to the implemetatio of the Europe 2020 strategy objective for i Polad, i.e. to reduce the proportio of people who abado early ad icrease the proportio of people with higher. This stems from the fact that o mechaism has bee idetified, which would lead to the executio of the Programme ifluecig such idicators. It is worth otig that i Polad the scale of the problem of premature droppig out of the system is very small (sice its etry ito the EU the value of the correspodig idicator has ever exceeded 6%), with both idicators reachig a target set at EU level ad approachig the atioal target value 6. Achievemet of the objectives of the strategic framework for Europea cooperatio i ad traiig (Educatio ad Traiig 2020) Empirical data collected i this study shows that the Erasmus+ programme fosters the achievemet of strategic objective No 1, i.e. the implemetatio of the lifelog learig idea. This is ot due to the activatio of people so far passive i terms of, as they do ot usually participate i projects, 7 but to allowig iterested parties to participate i traiig that is ot available to them outside the Programme (e.g. due to the fact that 6 Eurostat data: 7 Cf. Viešosios politikos ir vadybos istitutas / Public Policy ad Maagemet Istitute (2011). Iterim Evaluatio of the Lifelog Learig Programme ( ). Fial report. 18 February Europea Commissio. 10 Page

11 Mid-term evaluatio of Erasmus+ ad the predecessor programmes: LLP ad YiA i Polad such iche traiig or traiig related to the latest treds ad techologies are ot coducted case studies provide such examples. However, the scale of this effect is small. The executio of the remaiig strategic objectives is facilitated by the Programme's effects i the form of improved quality ad developig competecies. I some aspects (e.g. developmet of laguage competecies) the Programme's cotributio ca be cosidered substatial. Additioal coclusios o strategic objective 3 are cotaied i the followig paragraphs. Achievemet of the mai objectives of the reewed framework for Europea cooperatio i the youth field ( ) The implemetatio of the Erasmus+ specific objective i the youth field stems explicitly from the implemetatio of the reewed framework's mai objective aimig at "promotig civic participatio, social iclusio ad solidarity amog all youg people" these objectives are almost idetical, ad also with strategic objective No 3 of the strategic framework for Europea cooperatio i ad traiig. Support i the Youth field, however, is ot sufficietly widespread to cosider the impact o the mai objective as strog. A sigificat idirect role i buildig geeratioal solidarity o a Europea scale could be played by mobility projects i other fields, which give youg people the opportuity to iteract with peers from other coutries, but cofirmatio of this hypothesis is outside the scope of this study. Erasmus+ greatly icreases the chaces of youg people i the field of, but to a small extet cotributes to providig them with greater ad equal opportuities o the labour market, for example because the decisive factors for these opportuities deped primarily o the demad side of the labour market. However, o the basis of cocluded iterviews ad case studies it is possible to coclude that experieces acquired by the participats make it easier for them to work abroad or i multiatioal corporatios. Promotio of Europea values accordig to Art. 2 of the Treaty o Europea Uio The Erasmus+ programme supports the values of huma digity, freedom, democracy ad respect for huma rights, above all by breakig dow itercultural barriers ad fosterig tolerace through foreig exchages, iteratioalisig istitutios' activities ad life experieces of people, as well as promotig great liguistic variety i the Europea Uio ad itercultural awareess. I additio, Erasmus+ (especially i the VET ad HE fields) provides the participats with skills useful o the labour market, which are likely to alleviate their susceptibility to job isecurity, which promotes radicalisatio To what extet did Erasmus+ activities im pact the developmet of policy i the area of, traiig, youth ad sport i Polad? Which activities were the most effective? Were ay differeces betwee idividual areas observed? The evaluatio provided examples of a impact of Erasmus+ o the developmet of policy, but the scope of this ifluece is limited. Due to scat evidece o this impact, it is ot possible to make a compariso betwee activities or idividual areas. NA declares that Erasmus+ activities support al reforms itroduced i Polad (especially i the SE sector, e.g. developig ew curricula ad itegratig ew techologies i teachig, ad the VET sector, e.g. fosterig school-busiess collaboratio). It is difficult, however, to ame specific impacts of the Programme o the developmet of public policies. Activities focused directly o developig policies, i.e. KA 3, are decetralised ad coducted i the Youth sector oly. The stakeholders participatig i workshops oticed that participatig i KA 3 positively affects the ivolvemet of youg people i commuity life ad local politics. Effects of this kid are also declared by a substatial group of KA 1 participats. O the other had, stakeholders mostly failed to otice ay impact o political decisio makers ad explaied this by the sporadic ad ot log-term ature of coducted activities (for example oe-time debates betwee youg 8 Cf. de Polavieja, J. G. (1999). How do labour market experieces affect political attitudes?: aalysig the political effects of labour market dualisatio i Spai. Istituto Jua March de Estudios e Ivestigacioes; Emmeegger, P., Marx, P., & Schraff, D. (2015). Labour market disadvatage, political orietatios ad votig: how adverse labour market experieces traslate ito electoral behaviour. Socio-Ecoomic Review, 13(2), The geeral objectives of the "Sustaiable developmet of parter coutries i the field of higher " ad the "developmet of the Europea dimesio i sport, i particular commo sport" exceed the scope of this evaluatio. 11 Page

12 Mid-term evaluatio of Erasmus+ ad the predecessor programmes: LLP ad YiA people ad decisio makers). Oe ca, however, poit to may examples i which a youth policy project resulted i actual ad lastig outcomes, for example chagig school bus schedules or establishmet of a couty-level youth coucil. I this case, the key success factor was the willigess of local authorities to hold a ope dialogue with youg people. 1.4 What specific approaches (such as co-fiacig, promotio or others) have you take i order to try to ehace the effects of Erasmus+ i your coutry? To what extet have these approaches bee effective? Ca ay particular poits for improvemet be idetified? Approaches take by NA should be assessed as effective. The activities udertake (iformig, promotig, simplifyig) cotribute to the ehacemet of the effects of Erasmus+ i Polad, ad they iclude: Iformatioal ad promotioal activities focused o potetial applicats, whose purpose was to popularise iformatio cocerig the possibility of obtaiig co-fiacig i the Programme ad adjustig publicly available materials (leaflets, brochures, presetatios) to the specific ature of particular target groups (school studets, uiversity studets, teachers, academic staff, NGO represetatives, youg people, adults). The followig chaels have bee used: Iteret (websites, for example as regards providig aswers to frequetly asked questios, social media, ewsletter, glossary), radio ad TV programmes, films, direct meetigs, cofereces, workshops, fairs, traiig sessios (such as the Natiowide Iformatio Day at the Warsaw Natioal Stadium or the Europea Youth Week that resulted i collaboratio with media parters), as well as publicatios drafted by the Erasmus+ Programme Natioal Agecy, a project implemeted by FRSE whose purpose is to reach commuities that did ot implemet projects i Erasmus ad provide them with detailed iformatio, i.e. the Erasmus+ Bus. Project recipiets are commuicated a importat message about Programme simplicity, for example as regards settlemets ad ucomplicated applicatio procedures (i cotrast with projects implemeted uder ESF sposorship), which is essetial due to the multitude of actios ad sectors supported by Erasmus. The coducted iformatioal ad promotioal activities should be cosidered successful. I the quatitative survey coducted amog beeficiaries ad usuccessful applicats, the vast majority i all sectors agreed with the statemet that it was easy to access iformatio o who ca be a beeficiary ad that iformatio o what ca programme fuds be used for was trasparet. I tur, iterviews with represetatives of istitutios that did ot apply for co-fiacig show that a vast majority was aware of Programme opportuities but decided ot to follow up, maily because of implemetig other udertakigs. At the same time, however, over oe half of usuccessful applicats (56%) ad almost oe half of beeficiaries (49%) agreed that ways of otifyig about the Programme should be improved so that everyoe who could participate would kow he or she might be able to. At the same time the iterviewees did ot specify what chages i iformatioal activities they expect from NA. It is, therefore, recommedable to focus promotioal activities o iactive orgaisatios. Ivitatios to participate i iteratioal meetigs (semiars) o which the specific character of Erasmus activities is explaied to participats are set first to etities foud o rejected applicatio lists. Such meetigs allow them to meet potetial parters ad joi aother implemeted project. The maer of coductig moitorig to focus the attetio of the beeficiary. Oe of the ispectio objectives, apart from stadard verificatio of whether the task was implemeted correctly, is focusig the beeficiary s attetio o implemetig optimal solutios. Ivolvig KED OP fuds i implemetig the Programme allowed to expad the scope of Erasmus+ impact. Thaks to this approach, recogized as iovative by the Europea Commissio, the scale of implemeted projects is much larger tha the expected Erasmus+ budget. At the same time, the coducted study demostrated that the list of approaches stregtheig Erasmus+ impact should be exteded with: 12 Page

13 Mid-term evaluatio of Erasmus+ ad the predecessor programmes: LLP ad YiA wider support o project desig stage for applicats with o experiece beeficiaries cited the isufficiet umber of Erasmus+ NA iitiatives i this respect, icreasig the accessibility of traiig sessios ad cofereces by orgaizig them outside Warsaw iterviewees poited that evets hosted i the capital predomiate, makig access to them difficult for some, icreasig the umber of traiig sessios ad workshops coducted i the form of webiars, more efficiet traslatio of documets ecessary durig the applicatio desig process accordig to some beeficiaries, certai Eglish laguage provisios might be iterpreted icorrectly Do you cosider that certai actios of the Programme are more effective tha others? Are there differeces across fields? What are the determiig factors for makig these actios of the Programme more effective? The effectiveess of actios i evaluated programmes, measured by the percetage of beeficiaries or participats who cofirmed the presece of a particular effect, differs. Certai activities ca be show to be particularly effective i a give respect. As regards the improvemet of quality, iovatio excellece ad iteratioalizatio of al, traiig ad youth istitutios, apparet effectiveess is show particularly by mobility projects i the HE sector, both cocerig exchages betwee programme coutries (actio KA103) ad with parter coutries (actio KA107), as well as coutries from previous programmig period (actio ERA02). Represetatives of a vast majority (over two thirds) of orgaisatios participatig i projects of this kid declared that they cotributed to developig cotacts with similar istitutios from other coutries ad a exchage of experieces cocerig solutios used there. More tha 37% of KA103 beeficiaries ad 61% of KA107 beeficiaries stated that participatio i the Programme led to a icreased efficiecy i the fuctioig of their istitutios. Iterestigly, these declaratios occurred oticeably less frequetly i mobility projects from other sectors. O the other had, istitutioal effectiveess was almost as frequetly apparet i parter projects, especially i Leoardo da Vici (actio LE04), Grudtvig ad actio KA201 of the Erasmus+ programme (Strategic partership i school cooperatio betwee regios ad collaboratio betwee istitutios active i the area of ). As regards the developmet of key competeces, this effect is admitted by about 90% VET learers, less frequetly by studets doig traieeships, ad least frequetly (though still i the majority of cases) by Europea volutary workers ad studets leavig to study abroad. The exceptios iclude certai social ad civic competeces (for istace, uderstadig such cocepts as democracy, equality or huma rights, tolerace towards other people ad behaviours) whose developmet through mobility is less frequetly oticed by studets doig traieeships. The key factor icreasig effectiveess, thus uderstood, is the level of competeces extat prior to mobility participatio. By way of example, VET learers ad studets leavig for traieeships almost ever deied that the trip gave them a opportuity to improve their commuicatio skills i a foreig laguage, yet the latter were six times as ready (16.1% compared to 2.7%) to declare they did ot beefit from this because their laguage skills had already bee fluet whe they departed. Studets who took up their first or subsequet job before the evaluatio was made, mostly (75% i actio KA107 ad 55% i actio KA103) declared that they were iflueced by the Programme. The percetage was slightly higher for studets who did traieeships (61%) compared to those that did ot (52%). Amog o-staff VET ad Youth sector participats, the percetages were 38% ad 44% respectively. Developig professioal skills was more frequetly declared by vocatioal studets or graduates (67%) ad studets doig traieeships (66%) tha EVS participats (33%) ad studets i study programmes (32%). As far as the developmet staff competeces related to workig with school studets or youg people are cocered, the highest effectiveess was oted i the school sector ad the lowest i the higher 13 Page

14 Mid-term evaluatio of Erasmus+ ad the predecessor programmes: LLP ad YiA sector. The type of mobility was probably of importace here i the HE sector the purpose of most mobilities was teachig, which was of margial importace i the other sectors domiated by participatio i a course of traiig. Oe caveat should be appeded to the above results, as available data oly related to whether a participat oticed ay chage ad ot to its size. Assessig effectiveess thus allows us to compare how may participats were affected by a observed effect but ot its itesity To what extet has the itegratio of several programmes ito Erasmus+ made the Programme more effective i your coutry? The itegratio of programmes harmoized coducted itervetios for participatig orgaisatios, helped them to combie activities from differet Key Actios, ad also facilitated ad streamlied Programme maagemet (which was stregtheed by the fact there is oly oe NA i Polad). Potetially, this may traslate to icreasig its effectiveess, although specific examples of this ifluece may be hard to fid. Issues that are cotroversial amog shareholders cocer ot so much itegratio itself as the solutios brought alog cocurretly with it (for example cost settlemet usig lump sums) ad amig covetios. Opiios were voiced that icludig all sectors uder the Erasmus+ umbrella is ot without its disadvatages, as some beeficiaries still use old sector programme ames (such as Grudtvig) ad the Erasmus+ ame is commoly associated mostly with higher, hiderig commuicatio targeted at other sectors. However, it should be stated that givig a sigle recogizable ame to the etire Programme makes it more trasparet. For iformatioal ad promotioal purposes, the use of compoud ames featurig old appellatios was allowed (e.g. Erasmus+: Leoardo da Vici). Do you see scope for chages to the structure of Erasmus+ or its successor programme that could icrease effectiveess? The study results suggest that, due to the aforemetioed beefits form itegratio, maitaiig a sigle programme structure divided ito sectors is recommeded to maitai the specific character of these sectors. Implemetig itersectoral iitiatives that produce syergy betwee idividual programme activities remais a challege 10. Such iitiatives are beig udertake, but their small scale 11 shows that the itersectoral barriers persist. A remedy to these barriers, however, lies i project selectio ad, perhaps, fiacial allocatio rather tha i chagig the structure of Erasmus Is the size of the budget appropriate ad proportioate to what Erasmus+ is set out to achieve? The scale of itervetios i Erasmus+ matches set objectives thaks to usig fuds from the Europea Social Fuds to implemet additioal ad large projects. These fuds make it possible to implemet good quality projects from reserve lists ad to provide additioal support for disadvataged groups (people with disabilities, people ecoomically disadvataged). If the fiacig from cease i the future, the appearig fiacial gap should be covered from other fudig sources. Apart from this, expasio of the Programme budget should be cosidered to icrease the fudig rate for employees of orgaisatios ivolved i iteratioal collaboratio i strategic parterships. It would provide extra motivatio to eter ito parterships ad egage i collaboratio, especially sice the disparity betwee wages eared by represetatives of differet coutries is cosidered extremely high ad prejudicial to employees of Polish orgaisatios. 10 Cf. Viešosios politikos ir vadybos istitutas / Public Policy ad Maagemet Istitute (2011). Iterim Evaluatio of the Lifelog Learig Programme ( ). Fial report. 18 February Europea Commissio. 11 Accordig to aalysed databases, the value of itersectoral parterships amouted to less tha 5% of the total value of all strategic parterships. NA coducts umerous itersectoral activities, but the umber of their participats is egligible whe compared to the whole populatio of programme participats. 14 Page

15 Mid-term evaluatio of Erasmus+ ad the predecessor programmes: LLP ad YiA Is the distributio of fuds across the Programme s fields ad actios appropriate i relatio to their level of effectiveess ad utility? The objectives assumed i Erasmus+ ad the high effectiveess of activities might speak for icreased focus o mobility of VET learers, AE staff ad school staff at the cost of the curretly most itesely supported HE sector (also effectively supportig studets) What challeges ad difficulties ca you ecouter while implemetig the various actios of Erasmus+? What chages would eed to be itroduced i Erasmus+ or its successor programme to remedy these? The barriers listed below are mostly related to the Programme implemetatio process ad its maagemet by the Erasmus+ NA. The followig challeges ad difficulties have bee idetified: complex Programme structure; varied awareess of beig eligible for support amog potetial beeficiaries from each sector stregtheig activities popularisig the vocatioal sector would be reasoable; ivolvig KED OP fuds fiacig some projects from ESF fuds obliges beeficiaries (FRSE ad idirectly project providers) to meet certai obligatios assiged to KED OP beeficiaries; limited ability of Erasmus+ NA to export aggregated data from EC systems; Erasmus+ NA admiistrators have o rights to perform some operatios i IT systems; isufficiet umber of EC traiig sessios devoted to operatig IT systems; lack of detailed EC guidelies, for example i the risk aalysis area; less tha itese collaboratio betwee atioal agecies; problems ad ambiguities with defiitios, for example cocerig the otio of 'low qualificatios'; complex Programme documetatio affectig especially applicats from the adult sector; favorig experieced beeficiaries over ew istitutios i the assessmet process; o collaboratio betwee etities i the local eviromet, for example schools ad supervisig authorities. Quatity surveys with usuccessful applicats ad beeficiaries of the Erasmus+ programme ad previous programmes have demostrated that the challeges ad barriers discussed above have had limited impact o the Programme maagemet process. Oly less tha thirty percet of usuccessful applicats were of the opiio that project maagemet should be improved to stregthe the Programme s impact. This opiio was shared by oe half of beeficiaries. The followig should be cosidered i order to streamlie the Erasmus+ implemetatio process: desigig clear checklists of documets which beeficiaries ad participats should draft at each stage of project implemetatio, for example a checklist of documets which a mobility participat should produce to the sedig orgaisatio (for all sectors), coductig more traiig sessios/workshops for iterested applicats, icludig those with o experiece beeficiaries poited that the rules of applyig for support i Erasmus+ are markedly differet tha rules i effect i previous programmes ad that the procedures are more difficult for them ow; it was also oticed that etities which did ot implemet project i this format may be uable to provide the required documetatio, coductig traiig sessios for project coordiators before sigig a project implemetatio agreemet i Erasmus+, draftig detailed istructios for accoutats employed i schools ad supervisig authorities o how to settle costs sustaied durig projects, harmoisig the activities of all Erasmus+ Natioal Agecies i Europe, 15 Page

16 Mid-term evaluatio of Erasmus+ ad the predecessor programmes: LLP ad YiA eergise cooperatio betwee Erasmus+ beeficiaries i each sector, for example by creatig a platform for kowledge exchage betwee beeficiaries To what extet are the approaches ad tools that are used for dissemiatig ad exploitig the results of Erasmus+ ad its predecessor programmes i your coutry effective? Was Programme implemetatio suppor ted by a coordiated commuicatio policy based o diagosed iformatio eeds ad habits of groups targeted by the commuicatio? The approach assumed, ad tools used as part of this approach, should be deemed effective i all areas covered by iformatioal ad promotioal campaigs ad dissemiatio activities. Sice the evaluatio did ot iclude a detailed study of effectiveess of idividual actios, we use the desk-based assessmet of coherece of the strategy of the Promotio ad Commuicatio Departmet of FRSE, ad its qualitative assessmets expressed durig IDIs with represetatives of istitutios ivolved i the Programme implemetatio, telephoe iterviews with etities that did ot apply for co-fiacig, ad as part of diagostic ad creative workshops. The commuicatios policy that accompaies the implemetatio of the Programme is see as coordiated ad based o a previous assessmet of target group eeds. This policy is implemeted o the basis of three processes: dissemiatio, evet orgaisatio, ad exteral commuicatio. These processes are maaged by the Promotio ad Commuicatio Departmet of FRSE. Actios of the Departmet are targeted maily at those who could further relay the message to potetial participats: orgaisatios ad istitutios active i a sector, schools ad uiversities, teachers ad academic staff. The basis for Departmet strategy is commuicatio that relies o two chaels cosidered essetial: direct commuicatio (meetigs, traiig sessios, persoal cosultatios) ad the Iteret (mostly social media). Other chaels, such as TV, radio, press ad leaflets play a auxiliary role, complemetig the message set to recipiets usig the two mai chaels. Results of quality surveys (iterviews, workshops ad case studies) demostrate that such choice of chaels esures that all iterested parties are effectively reached. A challege for commuicatio activities i Erasmus+ is rebradig from previous ames (Grudtvig, Comeius, Leoardo) to Erasmus+. This requires reachig target groups accustomed to previous support structures ad ames. Aother major challege at the NA level is reachig potetial beeficiaries which did to participated i previous programmes. That those resposible for promotio ad distributio focus most of their attetio o these groups must be viewed as reasoable. What areas show room for potetial adjustmet? A importat issue (barrier to commuicatio) is the size ad level of complexity of the Programme Guide. Assessmets of beeficiaries participatig i diagostic ad creative workshops idicate that it is see as cofusig. This issue is ackowledged by the represetatives of the Promotio ad Commuicatio Departmet. Oe of the critical goals of promotio aimed at beeficiaries is to offer readers a isight ito its cotets usig the tool kit defied i the commuicatios strategy. This is satisfactorily doe by the Departmet by meas of traiigs, direct commuicatio ad prited materials (such as leaflets). Actios should be cotiued to preset the Programme structure i a more trasparet ad friedly way, both i the Programme Guide ad o a dedicated websites, guidig a potetial participat, applicat or beeficiary through relevat iformatio, followig a path that is customised with regard to his sector, profile ad iterests. The quality ad effectiveess of activities udertake by beeficiaries, ad thereby their ifluece o geeral Programme image, is highly varied. Some beeficiaries, usually orgaisatios ad istitutios with cosiderable experiece i implemetig iteratioal projects, act i a highly professioal way, reach potetial participats ad dissemiate project results. While this effect may ot apply to schools ad uiversities, it is more apparet i the youth ad AE sectors. 16 Page

17 Mid-term evaluatio of Erasmus+ ad the predecessor programmes: LLP ad YiA Did commuicatio activities coducted withi the Erasmus+ programme cotribu te to its effective implemetatio? To summarise, commuicatio activities implemeted by the NA cotributed to effective implemetatio of the Programme. For activities coducted by beeficiaries o project level, this was largely depedet o the project iitiator s experiece: etities with richer experiece ad resources i that respect were more likely to effectively implemet their activities Is the Erasmus+ programme participatio scale geographically diversified? Is this a issue give the implemetatio of the equal access ad social iclusio priciple? If ay differeces exist (for example betwee provices or urba/rural areas), what might be their cause? The scale of participatio i Erasmus+ is geographically diversified. This should be uderstood as differeces betwee provices i the umber ad value of submitted applicatios ad approved projects, as well as umbers of project participats (i absolute terms). Geographical diversity is specific for idividual sectors but a holistic view allows recogizig that the majority of projects ad project participats i the Erasmus+ Programme hailed from the Mazowieckie, Śląskie, Małopolskie, Wielkopolskie, Dolośląskie ad Pomorskie provices. The project iitiator structure is domiated by orgaisatios based i provicial capitals (Warszawa, Wrocław, Katowice, Kraków, Pozań, Gdańsk, Łódź, Lubli) which serve as cetres of growth o a atiowide scale, developig iovativeess, parterships betwee sciece ad busiess, ad iteratioal cooperatio. The geographical diversity of Erasmus+ participatio is the result of diverse potetial demostrated by each provice. This potetial ca be defied with respect to idividual sectors but also o a holistic level (defied i relative terms). For example, i the higher sector, comparig the umber of approved Erasmus+ projects to the umber of uiversities with the Erasmus Charter for Higher Educatio, it must be stated that provicial cocetratio is abset (Florece s cocetratio ratio is 0,0487 o a scale from 0 to 1, proof that projects are highly dispersed ad that this occurrece is ot limited to particular locatios). I other sectors, the potetial should be defied with referece to demographic structure or the level of social ad ecoomic developmet. The potetial ca also be defied o a summary level usig sythetic idices, for example related to developmet of social capital i each provice. Regioal diversity was also apparet i previous years (e.g. Erasmus, Youth i Actio) i which, as i Erasmus+, lower activity was show by less socially ad ecoomically developed provices (such as i easter Polad) or those with lower level of social capital (Lubuskie, Opolskie). It should be emphasized, however, that oticeable chages are occurrig i this respect, for example the icreased activity of orgaisatios ad participats from the Lubelskie provice, which ca be iterpreted as a result of more itese implemetatio of iformatioal ad promotioal activities by the Erasmus+ Natioal Agecy. The idetified geographical diversity of Erasmus+ participatio is a occurrece to be moitored ad potetially supported with corrective actios (if relative diversity betwee regios is high or blak spots (areas ot active i applyig i Erasmus+) are idetified). This is ot, however, a problem from the viewpoit of the equal access ad social iclusio priciple as it is drive by processes of a exteral ature (the demographic, social, cultural ad ecoomic potetial of idividual regios) that evolve i the log term ad are beyod potetial impact of the Programme. I the Programme, a sufficiet umber of activatio ad commuicatio activities to ecourage participatio has bee udertake. Itesifyig these activities will ot lead to limitig geographical diversity i the Programme as they will ot be able to break dow barriers of a structural ature Did the maer of Programme implemetatio esure equal access to the Programme, regardless of ay exclusio risk factors, icludig for people with disadvataged backgrouds ad people with disabilities? I particular, was material situatio a obstacle to participatig i Actio 1 mobility? I case of log -term mobilities, did persos iterested i mobility have a opportuity to obtai ecessary meas to provide for their eeds ad could they participate i the Programme? Was disability a obstacle to participatig i the Programme? I particular, did project providers esure a opportuity to participate to 17 Page