Interim Evaluation of Erasmus Mundus

Transcription

1 Interim Evaluation of Erasmus Mundus Final Report June 2007 Centre for Strategy & Evaluation Services LLP

2 Interim Evaluation of Erasmus Mundus Contents SECTION PAGE GLOSSARY I. EXECUTIVE SUMMARY i. 1. INTRODUCTION Introduction 1.2 Evaluation aims and scope 1.3 Methodological approach 1.4 Report structure ERASMUS MUNDUS PROGRAMME Legal basis of the Erasmus Mundus programme 2.2 Key objectives 2.3 Policy context and legal issues 2.4 Erasmus Mundus Actions, eligibility and geographic scope 2.5 The Windows and the External Co-operation Window 2.6 EM and the awarding of joint, double or multiple degrees 2.7 Linkages with other EU programmes in the field of higher education 2.8 Participation in Erasmus Mundus 2.9 Financing aspects of Erasmus Mundus 2.10 Management of EM 2.11 Application and selection process 2.12 Quality assurance, monitoring and evaluation 2.13 Promotion and marketing RESEARCH ISSUES Programme logic and objectives 3.2 Programme design 3.3 Programme management 3.4 Programme outcomes (including progress towards objectives) CONCLUSIONS AND RECOMMENDATIONS Overall findings 4.2 Programme logic and objectives 4.3 Programme design 4.4 Programme management 4.5 Programme outcomes (including progress towards objectives)

3 Interim Evaluation of Erasmus Mundus Section Contents 4.6 Sustainability 4.7 Community Value Added 4.8 The dissemination of results APPENDICES PAGE A. INTERVIEW LIST 103 B. PROGRAMME IMPLEMENTATION - OVERVIEW OF CALLS FOR PROPOSALS 105 C. PARTICIPATION IN ERASMUS MUNDUS BY YEAR 106 D. ORGANIGRAM - EUROPEAN COMMISSION AND EXECUTIVE AGENCY 127 E. SURVEY RESULTS (BOUND SEPARATELY) F. CASE STUDIES (BOUND SEPARATELY) 2

4 Interim Evaluation of Erasmus Mundus Glossary Glossary Below is a short glossary setting out some of the commonly used acronyms in the interim evaluation report. Where appropriate, a short explanation is also provided: EM - Erasmus Mundus DG EAC - Directorate General for Education and Culture The DG s Erasmus Mundus Unit is responsible for the overall strategic management of the programme EACEA - Education, Audiovisual and Culture Executive Agency (the Agency ) The Agency is responsible for the day to day management and administration of the programme NS - National Structures The National Structures are responsible for promoting awareness about, and participation in Erasmus Mundus, as well as for providing ad hoc ongoing support to higher education institutions involved in Erasmus Mundus Masters Courses HE - Higher Education HEIs - Higher education institutions MS - Member States VET - Vocational Education and Training Note: The opinions expressed in this evaluation are those of the authors and do not necessarily reflect the views of the European Commission.

5 Interim Evaluation of Erasmus Mundus - Executive Summary I. EXECUTIVE SUMMARY 1. About the Erasmus Mundus programme Erasmus Mundus is a co-operation and mobility programme in the field of higher education intended to promote the European Union as a centre of excellence in learning around the world. It aims to support the development of top-quality European Masters Courses and to enhance the visibility and attractiveness of European higher education in third countries. The programme has, as its strategic aims, to improve the quality of higher education in Europe and to promote intercultural understanding through co-operation with third countries. The specific aims of the programme are to: promote quality and excellence in European higher education; encourage the incoming mobility of third-country graduate students and scholars; foster structured co-operation with third-country higher education institutions; and improve the profile, visibility and accessibility of European higher education in the world. The programme consists of four main Actions, namely: Action 1 - Erasmus Mundus Masters Courses, comprising integrated courses at masters level offered by at least three universities in three different European countries; Action 2 - Erasmus Mundus scholarships for students and scholars from third countries; Action 3 - Partnerships with higher education institutions in third countries, comprising scholarships for students and scholars from EU countries for mobility towards third countries; Action 4 - Projects to enhance the worldwide attractiveness of European higher education. The Erasmus Mundus programme was allocated a budget of 230m euros for the period Supplementary financing of 57.3m euros was made available in the years through the Asian Windows as well as 8.8m euros in the year 2007 through the ACP Window and the Western Balkans Window. These are financial envelopes to fund additional scholarships for students from specific countries which have been allocated through the EU s external relations budget. Consequently, a total budget of 296.1m euros is available for the programming period Evaluation Design 2.1 Aim and Scope Article 12 of the Decision 2317/2003/EC of 5 December 2003 establishing the Erasmus Mundus Programme, requires an interim evaluation of the programme to be carried out. The evaluation assignment was carried out by the Centre for Strategy and Evaluation Services (CSES) on behalf of the Directorate-General for Education and Culture over an 8-month period. The scope of the interim evaluation was the period , during which a number of calls for proposals have taken place to implement the programme. While Erasmus Mundus Masters Courses, scholarships and attractiveness projects (Actions 1, 2 and 4) commenced in the academic year , Partnerships (Action 3) began one year later in The aims of the interim evaluation were to: address the relevance and utility, efficiency, effectiveness, sustainability and Community Value Added of the programme; and provide the Commission with recommendations as to how the programme s intervention logic, objectives, design, implementation arrangements, results and impact can be further improved. i

6 Interim Evaluation of Erasmus Mundus - Executive Summary 2.2 Methodological approach The methodological approach adopted for the assignment included desk research to review relevant literature; seven separate surveys with key programme stakeholders (as well as rejected applicants) including HEIs participating in EM Masters Courses, EM Partnerships and attractiveness projects (Actions 1, 3 and 4), and participating third-country and EU students and scholars (Actions 2 and 3); and interviews with key stakeholders, including the Commission, the Education, Audiovisual & Culture Executive Agency, EM National Structures and participants in the four Actions of the programme. Case study visits with seven EM Masters Courses were also carried out - which provided feedback on the operation of courses and the views of students on the ground. 3. Evaluation Results Overall, the EM programme appears to have made a very positive start. The programme has generated genuine enthusiasm amongst students and HEIs alike, suggesting high relevance to identified needs. The programme was perceived by HEIs as adding value in a number of ways, for example by promoting the development of joint, double and multiple degree awards between HEIs in different countries, in line with the aims of the Bologna process of strengthening the European dimension in education and promoting increased mobility. EM has also begun to make a contribution to the promotion of academic excellence in European higher education, in particular by encouraging European HEIs to foster cooperation and joint working with other HEIs regarded as world-class in particular subject disciplines. Likewise, from the perspective of students, a wide range of benefits were identified, including the personal development benefits that arise from exposure to new cultures and languages and the academic benefits of studying on a Masters Course which demonstrates academic excellence. Participation in EM was also viewed as bringing benefits for students in terms of their future career development. However, given that the programme only commenced in 2004, its impact on the employment prospects of potential students will need to be assessed through longitudinal studies of EM graduate destinations (an initial tracking study will shortly be launched in this regard). The financial envelope for the programme allowed the funding of a number of high-quality courses and scholarships that was in line with initial expectations. However, demand for both courses and scholarships increased throughout the period under review and was of a sufficiently high level to justify additional funds in the future. The relatively easy financial absorption of those scholarships made available through the Windows suggests that a bigger financial envelope could have been absorbed without difficulty and with a significant increase in impact for the programme. The terms of reference for the evaluation required the presenting of evaluation results, firstly, according to the traditional key evaluation issues (relevance, effectiveness, efficiency, etc.), and, secondly, according to four headings: programme intervention logic and objectives; programme design; programme management and implementation; and programme outcomes. Additionally, aspects relating to finance which cut across these headings are explored. The evaluation results are presented in the main report according to the European Commission s suggested dual structure. ii

7 Interim Evaluation of Erasmus Mundus - Executive Summary 3.1 Evaluation Issues Below we set out key conclusions from the assessment of the evaluation issues Relevance The policy context has not changed radically since the programme proposal was drawn up for the programming period. The intervention logic remains relevant both from a policy perspective (strengthening co-operation with third countries in the field of higher education, fostering intercultural dialogue, promoting the development of integrated courses leading to the award of joint, double or multiple degrees at European level in line with the aims of the Bologna process) and in meeting the identified needs of programme beneficiaries (HEIs and students and scholars from both Europe and third countries). There are strong linkages between the objectives of the EM programme and the Lisbon Strategy, in that the programme seeks to attract the brightest minds from around the world to study in the European Union. There are also links between the programme and the Education and Training 2010 agenda, which links in with the Bologna process, in particular the need to open up education and training systems to the wider world as part of the European Community s response to the challenges and opportunities presented by globalisation. Europe must compete in an environment where the number of international students is growing rapidly and where competition for these students is intensifying Effectiveness and impact EM has made some progress in beginning to develop more structured co-operation in higher education between the EU and third countries, although the low participation rates in Partnerships (Action 3) suggest that more still needs to be done in this area. In terms of improving the accessibility of European higher education, in particular by enabling highly qualified graduates and scholars from all over the world to study and/or teach in the EU, the scholarship scheme (Action 2) has greatly facilitated access to European higher education for high calibre students from third countries. The attractiveness of European HE has also been promoted, through both Action 4 projects which seek to promote European HE as a whole as well as through the participation of international students on specific EM Masters Courses. With regard to the promotion of intercultural understanding, EM has been successful in encouraging cultural exchange of experiences. This process has been assisted by the fact that there was a very good distribution of nationalities among third-country scholarship holders. Moreover, students from third countries identified cultural and linguistic benefits as being more important than the benefit to their future career prospects. However, students from third countries have to date benefited more from the programme than EU students due to the more limited financing scope for the mobility of EU students. The experience of students - academically, culturally and linguistically - is much richer in EM courses where there is a good balance between European (including host country) and international students, as opposed to courses that are dominated by non-eu students. EM has also had a positive impact on those EU countries where there was previously no legal framework for the accreditation of joint, double or multiple degrees involving partners in different EU countries. Even where joint degree accreditation problems remain, EM Masters Courses have implemented effective double or multiple degree structures which have permitted genuine integration and co-operative study to take place. Joint degrees also help make the EU labour market more transparent and accessible to European students. iii

8 Interim Evaluation of Erasmus Mundus - Executive Summary Certain European countries were underrepresented in terms of the number of projects in which their institutions participated, an imbalance that the Commission will need to consider ways of addressing, mainly through promotion and awareness raising since excellence will need to remain the key criterion for the selection of courses. With regard to subject breadth, EM Masters Courses cover practically all academic disciplines, although there is a clear preponderance of engineering and natural science courses Efficiency and cost-effectiveness The Commission and the Executive Agency were regarded by the National Structures and HEIs running EM Masters Courses as having played an effective role in overseeing the programme s implementation to date. Communication and information flows in particular were regarded as having been regular and satisfactory. Responses to queries from HEIs have normally been dealt with promptly and useful feedback and comments have been provided to consortia on their reports. The high levels of satisfaction in respect of programme management by students and HEIs suggest that the programme has been managed efficiently and effectively by the Commission and the EACEA. The use and the level of unit costs and lump sums applied for the implementation of the programme were largely considered cost-effective, helping the programme to reach its aims at the lowest possible costs. However, the evaluation suggests that universities participating in EM Masters Courses and European students are under-funded in the current programme. Moreover, many HEIs did not regard the 15,000 euros per annum per consortium grant allocation as adequate to administer a high quality transnational integrated Masters Course (see financing aspects) Utility, added value and sustainability Many EM Masters Courses supported through the programme were pre-existing courses that have been modified to fit into the EM framework. The programme appears to have been successful in encouraging HEIs to modify their existing high-quality courses through cooperation and joint curriculum development with HEIs in other European countries offering the same subject discipline and able to meet the requisite academic excellence criteria at European/international level. The research suggests that most of these courses could not have continued in their present form without funding from the EM programme. While this suggests low levels of sustainability, looked at from another perspective it suggests that European money is being used to promote activities that would not otherwise be able to take place on the same basis. Another issue is whether students and scholars (particularly those from third countries) would have participated in the programme without financial support. In this context, 95% of thirdcountry students stated that they could not have participated in the programme without the scholarship. This suggests high levels of additionality European added value One of the core elements of the EM programme is the requirement to include a strong 1 EM Masters Courses receive a yearly lump sum of 15,000 euros. EU students receive a scholarship of 3,100 euros for a study period of three months at a third-country HEI. iv

9 Interim Evaluation of Erasmus Mundus - Executive Summary transnational dimension to Masters Courses, with mobility in a minimum of two different EU countries being an integral component. Given the transnational nature inherent in all courses, EM rates highly in respect of Community Value Added. The close coherence between EM and the Bologna process also suggests Community Value Added, in particular the role of the programme in achieving various aims connected with the further development of the concept of a European Higher Education Area, for example through the promotion of the European dimension in education, mobility, joint degrees and European level co-operation on quality assurance. 3.2 Key programme aspects - Findings and recommendations Below we set out key conclusions from the assessment of the four main headings of the Terms of Reference referring to the key programme aspects. Where appropriate, the main recommendations are also highlighted (detailed conclusions and recommendations are set out in section 4 of the main report) Programme intervention logic and objectives A number of evaluation questions were defined in the terms of reference in respect of the assessment of EM s programme intervention logic. These included the extent to which the needs analysis for the programme has proven to be correct and the extent to which the programme s objectives were relevant in addressing identified needs. Another key evaluation consideration was the degree to which there were synergies or overlap as compared with other EU and national programmes in the field of higher education. Key findings are now summarised: 1. The intervention logic remains highly relevant to the identified needs of programme beneficiaries, namely higher education institutions, students and scholars from Europe and from third countries, with survey responses indicating a close alignment between EM Masters Courses and the academic objectives of HEIs. 2. The programme intervention logic demonstrates strong internal coherence with clear linkages between the two general strategic objectives, the four specific objectives and the four Actions supported. Each of EM s four operational objectives is explicitly linked to a separate Action. 3. The intervention logic of the EM programme demonstrates strong external coherence with the wider European policy framework, in particular with the Lisbon Strategy, the Bologna process (several of whose Action Lines are directly relevant to EM) and the Education and Training 2010 Agenda, which emphasises the need to open up education and training systems to the wider world as part of the Community s response to the challenges and opportunities presented by globalisation. 4. The research suggests that the programme closely reflected the needs of beneficiaries in the case of each of the four Actions. Taking Action 1, EM Masters Courses were found to be closely aligned with the academic objectives of European HEIs, in particular the desire to strengthen the European and international dimensions of their academic offering. The provision of scholarships awards for Action 2 students also meets the significant latent demand to study in Europe among international students. 5. The making available of scholarships for up to 3 scholars (more in cases where the amount spent on scholars remains within the budgetary limit of the amount foreseen) v

10 Interim Evaluation of Erasmus Mundus - Executive Summary from the EU and third countries also meets identified needs and constitutes a valuable part of the programme. Scholars from third countries interviewed appreciated the opportunity to teach and carry out research in an EU country and highlighted the learning opportunities of comparing different teaching styles and methodologies between HEIs in third countries and those in the EU. 6. The effectiveness of the grants for scholars could perhaps be further improved by providing clearer guidance to consortia in relation to the role that visiting scholars should play within the context of EM courses - drawing on good practices where scholars have been fully integrated into the delivery of Masters course curricula. 7. The research identified similarities - as well as differences - between the EM programme and other Community financed programmes in the field of higher education (such as the Atlantis and EU-Canada Co-operation Programmes, the Tempus programme and the ALFA and Alßan co-operation programmes with Latin America). There is an ongoing need to ensure that overlap and duplication is avoided in so far as possible. 8. Looking ahead, there would at some future point be an argument in favour of combining at least some of the various co-operation programmes in the field of higher education into a single programme to improve the coherence of the policy framework in this area. Recommendation 1: The Atlantis and the EU-Canada Cooperation Programme (and likewise pilot projects with Japan, Australia and New Zealand) could in future be combined with Erasmus Mundus so as to improve the coherence of the Commission s approach to strengthening co-operation in the field of higher education between the EU and third countries Programme design In terms of programme design, a number of key evaluation issues were explored, notably the overall effectiveness of the programme s design and the extent to which the structure continues to be appropriate including the division of activities into four separate Actions. A key consideration was the extent to which there are linkages and synergies between the different EM Actions. Other evaluation issues included whether the design is conducive to attracting EU students to participate in EM and whether the programme should remain focused on the Masters level or be extended to PhD level. Key findings are now summarised: 9. The interim evaluation considered whether the existing programming structure was the optimal approach. In this regard, some stakeholders believe it may be more effective in the next programming period to unify Actions 1, 2 and 3 into a single Action in order to simplify and improve the coherence of programming (given the close inter-linkages between these Actions). However, the evidence was not conclusive in this regard. Recommendation 2: Erasmus Mundus Masters Courses should be encouraged not only to develop co-operation with higher education institutions in third countries but to enable these institutions to become full partners in the course itself. One means of achieving this aim would be to combine Actions 1 and 3 (currently Masters Courses and Partnerships with third country institutions respectively) together in a single Action. 10. Concerns were expressed through the interview programme whether the current programme design was conducive to encouraging the participation of students from within the European Union to take EM Masters Courses. While some financing is targeted at European students (in particular for outwards mobility through Action 3 Partnerships), scholarships have not been made available for intra-eu mobility. vi

11 Interim Evaluation of Erasmus Mundus - Executive Summary The lack of participation by EU students in EM was attributed to various factors, including the comparatively high cost of tuition fees for EM Masters Courses in comparison with national Masters courses; the lack of a scholarship for EU students to facilitate intra-eu mobility and marketing and branding issues with a perception amongst some students that the EM programme is for international students. Recommendation 3: Scholarships should be awarded to EU students to participate in the Erasmus Mundus programme on a competitive basis. There is a need to ensure that EU students participate on a more equal footing with their counterparts from third countries. Another evaluation question was whether the EM programme should remain focused on Masters Courses or be extended to the first and third degree cycles, i.e. to undergraduate and PhD levels. 11. There was little support amongst key programme stakeholders for extending EM to undergraduate level given the difficulties of operating an excellence programme at bachelor level. 12. There were, however, mixed views with regard to the possible extension of the EM programme to PhD level. While some NS and HEIs were strongly in favour of this on the basis that it would represent a natural progression for the EM programme, others expressed reservations. Concerns included the financial implications of supporting PhD studies of between three and five years duration, as well as the difficulties in monitoring the academic excellence of PhD courses, linked to the nature of PhDs themselves. In many EU countries, there is a strong tradition of PhDs primarily involving independent research, rather than a taught study programme. The extent to which PhDs could be genuinely integrated between different HEIs in different Member States was a concern. There appeared to be strong support amongst students, however, as well as amongst many NS and HEIs interviewed for extending the programme to the PhD level. 13. There are already examples of transnational PhD courses within the EU, notably the French Co-tutelle model, a national initiative which entails PhD students being supervised jointly by academics at French universities and universities in partner countries (examples to date have included the UK, Australia, etc.). The PhD student carries out research in a HEI in two countries and upon completion receives a joint, double or badged PhD from these institutions. Recommendation 4: The Erasmus Mundus programme should - finance permitting - be extended to the PhD level both at the level of courses and in respect of scholarships. Issues around quality assurance in respect of PhD programmes will need to be carefully thought through. Particular care should be taken to avoid duplication with Marie Curie research scholarships. 14. There was strong support amongst students participating in EM courses for an internship/career placement dimension to be incorporated as an integral part of courses. 15. However, there would be practical barriers in requiring internships to be incorporated into all Masters Courses. These would include the relatively short duration of some Masters courses and logistical difficulties to fit in a work placement given that students will move between different HEIs in different EU countries (some move between institutions as often as once every 3 months). vii

12 Interim Evaluation of Erasmus Mundus - Executive Summary Recommendation 5: Consideration should be given to providing EM students with placement opportunities on a more systematic basis. This would contribute significantly to raising the profile of the programme among employers. Undertaking a placement should not, however, be made compulsory since this would not necessarily be appropriate in the case of all Erasmus Mundus Masters Courses Programme management and implementation Key evaluation issues examined in relation to programme management and implementation included the extent to which implementation mechanisms were viewed as having been satisfactory in ensuring desired outcomes both at the programme level and by individual Action. Other evaluation considerations included the extent to which selection mechanisms for selecting EM Masters Courses (Action 1) helped ensure academic excellence and quality and whether selection mechanisms applied by EM Masters consortia for selecting students (Action 2) helped ensure the excellence of selected students. 16. The Commission s Erasmus Mundus Unit was regarded by the National Structures and HEIs as having played an effective role in overseeing the programme s implementation to date. Communication and information flows in particular were regarded as having been regular and satisfactory, with staff viewed as knowledgeable and helpful about the programme and about many specific EM Masters Courses. 17. The EACEA was also perceived as having done a good job in administering Actions 1, 2 and 3 since it assumed responsibility for these on 1 January 2006 (Action 4 remains under the auspices of DG EAC until June 2007). Responses to queries from HEIs have been dealt with promptly and regular feedback and comments have been provided to EM consortia both in respect of monitoring reports (interim and final reports on the implementation of EM Masters Courses) and at annual course meetings. 18. Another important component of EM s programme implementation structure is the EUwide network of National Structures set up by the European Commission. NS were regarded as having played a valuable and effective role. They are an important mechanism at the national level for providing information and advice to applicants, potential applicants as well as HEIs, students and scholars already in the programme 19. The decision to finance NS directly by the EU Member States reflected a desire to avoid putting in place a full EU agency structure at national level to implement EM, which would have been administratively burdensome, less flexible and costly. 20. Looking ahead to the next programming period, however, given that the EM budget (and hence both the number of EM Masters Courses supported and scholarships awarded) is likely to increase, NS could have a considerably increased workload. The varying resource allocation to NS between different Member States could become problematic, 21. There have been difficulties in obtaining adequate financing for the NS in some EU countries, particularly the New Member States. There is a need to promote reasonably balanced participation in EM Masters Courses between different countries. This will be difficult, as some NS have considerably more resources than others at their disposal to help promote participation in and awareness about EM. Recommendation 6: The National Structures should continue to be financed largely by the Member States in order to avoid a full EU Agency structure at national level (which does not appear to enjoy support amongst National Structures). viii

13 Interim Evaluation of Erasmus Mundus - Executive Summary Nevertheless, some EU financing should be made available to help co-finance promotional activities. The Commission should therefore make provision for restricted calls for proposals on a thematic basis to enable the NS to finance specific initiatives they wish to support, such as marketing and promotional activities, etc. 22. In terms of the quality assurance of EM Masters Courses, the current approach places a strong emphasis on assessing quality during the selection phase through a rigorous assessment of prospective courses carried out by the academic experts responsible for assisting the Executive Agency in selecting Masters Courses. 23. The selectors examine the quality of applications for Action 1 Masters Courses not only from the perspective of course content but also seek to ensure that internal QA mechanisms have been put in place (in particular, regular self-evaluation procedures) by EM Masters Consortia to ensure quality is continuously assessed during the 5 years of operation of courses. 24. Other means of assessing quality include: monitoring visits by the Executive Agency - some of which are also attended by the NS - and scrutiny by the Executive Agency of progress and final reports submitted by the consortia. Additionally, the Executive Agency organises annual co-ordination meetings which bring together HEIs involved in EM Masters consortia. 25. There is a need, however, to consider how self-evaluation might be combined with some form of external quality assurance to ensure that course quality is maintained throughout the five years duration of EM Masters courses. Over such a time period, factors may arise which cannot be known at selection stage - such as the extent to which there will be staff continuity on courses, course content needing to be revised and updated, etc. 26. Good practice suggests that quality assurance procedures in respect of Masters Courses should be centred first and foremost around self-assessment but with some external evaluation to help validate course quality, where appropriate. Recommendation 7: There is a need for the European Commission to turn its attention to quality assurance once Erasmus Mundus-branded Masters Courses are actually up and running - although self-evaluation should remain the fundamental starting point for ensuring continuous course quality. A representative sample of courses should be subject to external quality assessment. This could be carried out by external quality assurance bodies with previous experience in assessing both the quality of academic content and the integration of courses. 27. In terms of the appropriateness of selection mechanisms, these were felt to be broadly appropriate. There was wide agreement that Actions 1, 3 and 4 should continue to be managed on a centralised basis, given the transnational nature of activities supported. 28. In relation to Action 1, the use of two external academic experts as part of the appraisal process for the selection of EM Masters Courses appears to have been successful in leading to the selection of high quality courses demonstrating academic excellence. 29. Likewise, the selection of third country students and scholars (Action 2) on a decentralised basis, with HEIs involved in EM Masters consortia having responsibility for selecting excellent students for scholarship awards was also felt to be highly appropriate - HEIs themselves are evidently best placed to select students. ix

14 Interim Evaluation of Erasmus Mundus - Executive Summary 30. The comitology principle (a requirement to consult the EM Programme Committee as well as the European Parliament in respect of all four Actions) has resulted in delays, but has had a particularly adverse impact in delaying the awarding of Action 2 scholarships. Recommendation 8: The comitology principle for Erasmus Mundus scholarships (Action 2) should be discontinued in order to speed up the finalisation of the selection process. This could accelerate the scholarship award decision by as much as 6-8 weeks Programme funding The interim evaluation also examined financing aspects, including the extent to which the financing allocation by Action was the most efficient means of achieving desired outcomes. Issues under examination included whether the financial allocation for EM Masters consortia of 15,000 euros per year for Actions 1 and 3 was appropriate and whether the level of scholarship set at the programme outset for Action 2 students (21,000 euros per year up to a maximum duration of 2 years) was appropriate or should be raised or lowered. 31. The majority of HEIs participating in the programme viewed financing levels to cover the cost of administering Action 1 Masters Courses - 15,000 euros per EM Consortium per year - as insufficient and failing to reflect the true cost of administering a fully integrated high quality transnational Masters. 32. However, it should be emphasised that many consortia recoup some administrative costs through the common tuition fee. Moreover, the financial allocation of 15,000 euros per annum was set to help avoid the need for full EU reporting by each Masters consortium, which could prove administratively burdensome. 33. Views on the adequacy of financing varied between different EU countries, depending in part on the financing situation for higher education at national level. In EU countries where tuition fees are the norm, the low level of financing per consortium was perceived as being particularly problematic. Recommendation 9: Consideration should be given to increasing the finance allocated to EM Masters Consortia to better reflect the actual cost of administering an integrated Masters Course on a cross-border basis. 34. Action 2 scholarships for students from third countries were set at an internationally competitive level with rival scholarship schemes (21,000 euros per year for a maximum of 2 years) - notably the Fulbright scholarship in the US. Since the scholarship was set, however, foreign currency movements have occurred (notably the decline in the US dollar) which mean that the scholarship now appears comparatively generous. 35. The scholarship level was regarded as either highly adequate or adequate by almost all survey respondents. Some students pointed out, however, that because some HEIs and EU countries have higher tuition fees than others, while at first sight the scholarship appears generous, this depends in practice very much on the choice of academic institution and of country of study made by students from third countries. Recommendation 10: The scholarship level for third-country students should not be reduced from the current level of 21,000 euros per year. However, the Commission should continue to monitor scholarship levels of other scholarship schemes, such as the Fulbright, Chevening and DAAD (German Academic Exchange) scholarships. x

15 Interim Evaluation of Erasmus Mundus - Executive Summary 36. A universal scholarship amount should continue to be given with no differentiation either on the basis of where a given student decides to study or on the grounds of their country of origin. This is the only equitable approach and other approaches do not seem workable. Common tuition fees determined by individual Masters consortia should be retained in respect of EM Masters Courses Programme outcomes Outcomes to the end of 2006 were as follows: 80 EM Masters Courses (Action 1); 2325 scholarships for incoming third-country students (Action 2); 19 Partnerships (Action 3); 23 attractiveness projects (Action 4). Participation rates in the programme so far have been broadly in line with expectations, with the exception of Partnerships (Action 3), where participation was lower than anticipated. 37. Progress against objectives was summarised in section of this summary (effectiveness and impact). The EM programme can be said to have begun to make a very positive contribution towards achieving key objectives - such as promoting the European and international dimensions in education, promoting academic excellence, fostering intercultural understanding and supporting structured co-operation in higher education between European HEIs and those in third countries. In overall conclusion, the Erasmus Mundus programme should be continued with increased financing to reflect the need to attract European as well as international students and to facilitate the expansion of the programme to include the third cycle. The various recommendations in our report should be considered by the Commission so as to further improve the programme s efficiency and effectiveness in the next period. xi

16 Interim Evaluation of Erasmus Mundus - Introduction Section 1 1. INTRODUCTION 1.1 Introduction This document contains the final report in respect of the Interim evaluation of the Erasmus Mundus programme. The work was carried out by the Centre for Strategy and Evaluation Services (CSES) on behalf of the European Commission s Directorate-General for Education and Culture (hereafter DG EAC). The evaluation assignment has been carried out over an 8-month period. Erasmus Mundus was established by Decision No 2317/2003/EC of 5 December 2003 and will operate from inclusive. Erasmus Mundus is a co-operation and mobility programme in the field of higher education intended to promote the European Union as a centre of excellence in learning around the world. It aims to support the development of top-quality European Masters courses and to enhance the visibility and attractiveness of European higher education in third countries. EUfunded scholarships for third-country nationals who participate in these masters courses are provided, as well as some grants for EU nationals participating in EM and studying in third countries. The interim evaluation is particularly timely since it will closely inform the preparation of the new Erasmus Mundus programme post Evaluation aims and scope The evaluation of EU-financed programmes serves two basic functions: Accountability - looking backwards, evaluation helps explain to stakeholders what has been achieved through the implementation of a programme Improving the quality and effectiveness of future programmes - looking forwards, evaluation helps ensure that lessons are learnt and used to improve the efficiency and effectiveness of existing and future programmes The specific aims of the interim evaluation of the programme were to: Meet the legal requirement under Article 12 of the Decision establishing the EM programme to carry out an interim evaluation of the programme Address the key evaluation issues set out in the terms of reference, in particular the relevance and utility, efficiency, effectiveness, sustainability and Community Value Added of the programme Provide the Commission with recommendations as to how the programme s intervention logic, objectives, design, implementation arrangements and results and impacts can be further improved The scope of the interim evaluation was the period , during which several calls for proposals have taken place under each Action. While Actions 1, 2 and 4 commenced in the academic year 2004/05, Action 3 began one year later in 2005/06. 1

17 Interim Evaluation of Erasmus Mundus - Introduction Section Methodological approach The methodological approach adopted for carrying out the evaluation assignment is summarised diagrammatically below: Resume of Work Plan - Interim evaluation of Erasmus Mundus Programme Phase 1 Preliminary Tasks Phase 2 Fieldwork/survey work Phase 3 Analysis and Reporting Set up meeting Literature review Finalise evaluation methodology Finalise workplan and timetable Prepare inception report Desk research Survey work of institutions participating in Actions 1, 3 and 4 Survey work with third-country and EU students and scholars Interview programme with key stakeholders in Erasmus Mundus Good practice case studies of EM Masters Courses Preparation of interim report Analysis and reporting Preparation of draft final report Steering Group workshop outlining evaluation findings and policy recommendations Inception Report Interim Report Final Report September/October 2006 February 2007 April (draft)/june 2007 (final) As part of the evaluation work, seven surveys were carried out in parallel from January-March An overview of the survey responses across the different types of participants in the programme - HEIs students and scholars - is provided below: Survey Questionnaire Response % Action 1 co-ordinators/partners Action 1 non-selected* Action 2 students Action 2 scholars 84 NA Action 3 partner institutions Action 3 EU students 125 NA Action 3 EU scholars 27 NA Total 990 Note: the response rate in respect of Action 2 scholars, European students and scholars is not known since contact details were not available directly for these. HEIs were asked to send these out directly. * 133 non-selected applicants were sent a survey. Only contact details for 2007 applicants were available. 2

18 Interim Evaluation of Erasmus Mundus - Introduction Section 1 Almost 1,000 survey responses were received across the various surveys. The response level to the different surveys has in most cases exceeded the targets set out in CSES proposal document. In particular, there was an encouraging response from Action 2 students benefiting from an EM scholarship, with over 600 respondents. This is testament to the high satisfaction ratings of the EM programme amongst students. The response rate from HEIs participating in Action 1 EM Masters consortia was also encouraging, with 36.8% of co-ordinators and partner institutions having responded. In addition, almost a quarter of rejected applicants who were approached responded to the survey. An interview programme was also carried out to investigate key evaluation issues in greater depth. The main programme stakeholders in EM were consulted through this research. In particular, interviews were undertaken with: Commission officials and staff from the Executive Agency responsible for programme administration since 1 January 2006 A representative sample of the EM National Structures A representative sample of EM Masters Courses (Action 1) A representative sample of EM Partnerships (Action 3) A representative sample of EM projects to enhance the attractiveness and visibility of European higher education (Action 4) Third-country students (Action 2) and their European counterparts (Action 3) benefiting from an EM scholarship Other organisations active in the higher education field with an interest in the EM programme A full interview list is provided in Appendix A. 1.4 Report structure The report is structured as follows: An executive summary is provided at the beginning of the report Section 1 provides an introduction to the interim evaluation work Section 2 outlines key elements of the EM programme Section 3 considers key evaluation issues grouped around the headings: programme logic and objectives, programme design, programme management and programme outcomes (including progress towards objectives) Section 4 provides a summary of key evaluation findings and outlines recommendations at interim evaluation stage The appendices provide an interview list and various supporting information including a statistical annex 3

19 Interim Evaluation of Erasmus Mundus - Erasmus Mundus Programme Section 2 2. ERASMUS MUNDUS PROGRAMME In section 2, key features of the EM programme are outlined. This includes an assessment of the legal basis and programme objectives, the policy context, activities supported through EM s four Actions and the programme s geographic scope. Participation rates to date, funding and programme management and implementation arrangements are then considered. Last but not least, promotion and marketing activities are examined. It should be noted that this section addresses factual aspects of the programme - key evaluation issues are explored in Section Legal basis of the Erasmus Mundus programme The legal basis for the Erasmus Mundus programme stems from Article 149 of the EC Treaty, which makes provision for Community intervention to develop the European dimension in education, foster cooperation with third countries and to contribute to the development of quality in education. 2.2 Key objectives The Erasmus Mundus programme seeks to promote the EU as a centre of excellence in learning around the world in the field of higher education through the strengthening of co-operation with third countries and through intra-eu and outwards mobility. The programme has two overarching strategic objectives, namely to: Improve the quality of higher education in Europe Promote intercultural understanding through co-operation with third countries There are then a set of four operational objectives, which are to: Promote the European quality offer in higher education Encourage incoming mobility of third-country graduate students and scholars Foster structured co-operation with third-country higher education institutions Improve the profile, visibility and accessibility of European higher education in the world 2.3 Policy context and legal issues Policy context The emphasis in the EM programme on the promotion of excellence and quality in European education, and on the development of co-operation between higher education institutions in the EU and in third countries, should be seen in the wider 4

20 Interim Evaluation of Erasmus Mundus - Erasmus Mundus Programme Section 2 context of various EU policy developments including the inter-governmental Bologna process which begun with the Bologna Declaration in June This recognised that Europe s higher education sector should acquire a degree of attractiveness in the wider world equal to Europe s major cultural and scientific achievements. Bologna also emphasises the importance of strengthening the European dimension in higher education. The EM programme is of direct relevance to a number of the Bologna Action Lines, including Action 1 (adoption of a system of easily readable and comparable degrees), Action 4 (promotion of mobility), Action 5 (promotion of European co-operation in quality assurance), Action 6 (promotion of the European dimension in higher education) and Action 9 (promotion of the attractiveness of the European Higher Education Area). The EM programme contributes directly to all these aims, with the latter aim being directly linked to Action 4 of EM. The Lisbon Strategy (March 2000, and relaunched in 2005) provides an overarching strategic framework for transforming Europe into the world s most competitive, knowledge-driven economy by An underlying driver of Lisbon is the need for Europe to face up to the multifaceted challenges of globalisation. As part of the Community s response to globalisation and the need to make Europe more competitive, policy makers have recognised the importance of extending transnational co-operation and mobility in the field of higher education beyond the EU (and Europe s immediate neighbours) to third countries. The EM programme s awarding of joint, double or multiple degrees/diplomas from more than one country is very much in keeping with efforts in the education field to make higher education more responsive to the challenges and opportunities of globalisation as part of the education sector s contribution to Lisbon. Masters students from third countries participating in such courses may derive a number of benefits from participating in the programme, including the opportunity to study in a different academic environment, exposure to different EU academic traditions and teaching styles, cultures and languages, and improved employment prospects on return to their country. As part of their commitment to achieving the Lisbon aims, European Ministers in charge of higher education meeting in Prague (19 May 2001) emphasised the importance of enhancing the attractiveness of European higher education to students from Europe and other parts of the world. This led to the 2001 Communication on strengthening EU-third country co-operation in higher education (COM/2001/0385). The Communication noted that a further effort is required at EC level, to encourage institutions systematically to integrate new co-operation with third countries into a wider partnership framework. The Commission has subsequently published other relevant policy documents on European higher education of relevance to EM, including a Communication in 2005, Mobilising the brainpower of Europe: enabling universities to make their full contribution to the Lisbon Strategy (COM/2005/0152), which emphasised the need to continue the reform process already underway in higher education as critical from the Lisbon perspective. 5