Evaluation and Rating: facts & figures

www.nrf.ac.za Evaluation and Rating: facts & figures

30 Years of Evaluation & Rating CONTENTS This year marks the 30th anniversary of evaluations and rating and the information that this issue provides offers considerable insight into the progress that South Africa has made in overcoming its historical challenges in the fields of scientific research. A triennial publication, the normal publication date for this volume would have been 2013. This year, however, was unique due to a rating policy change implemented in 2008 which extended the validity period to six years. As a result, in 2013 no invitations for a reevaluation were issued which would have made comparisons with other years problematic. It was, therefore, decided to include 2013/14 data where applicable, but the publication will focus on the areas where trends could be tracked with comparable data. This publication of the NRF Reviews and Evaluation Directorate contains: Information and statistics on NRF evaluation and rating of individual researchers (NRF rating); Guidelines on how researchers can apply for a rating; Answers to frequently asked questions (FAQs) about the NRF Evaluation and Rating System; and Report back on progress made with recommendations of the Higher Education South Africa (HESA) Review of the NRF Evaluation and Rating system conducted in 2007/8. About the NRF Rating System... 2 List of Acronyms... 3 Foreword by the NRF CEO... 4 Foreword by the Deputy CEO (RISA)... 5 Foreword by the Executive Director: KME... 6 Evaluations and Rating Trends... 7 The A to Z of Rating... 18 The Key Role of the Peer Reviewers... 21 Striving for a Fair and Credible Process... 22 Report Back on Progress Made with the Recommendations of the HESA Review... 23 Appendix A... 26 Appendix B... 43 Postal address: National Research Foundation PO Box 2600 Pretoria, 0001 South Africa Physical address: NRF Building Meiring Naudé Road Brummeria Pretoria Website: www.nrf.ac.za/rating Email: factsandfigures@nrf.ac.za Telephone: +27 12 481 4075 Published September 2014 A B

About the NRF Rating System List of Acronyms The National Research Foundation (NRF) is South Africa s national agency for promoting and supporting research across all fields of the humanities, social and natural sciences, engineering and technology. Its head office is in Pretoria, with other business units and national research facilities across South Africa. The vision of the NRF is to transform society via the promotion of world-class research. The mission of the organisation is to contribute to the knowledge economy in South Africa by attaining at least 2% of global R&D output by 2015. The NRF s Research and Innovation Support and Advancement (RISA) business unit is responsible for investing in knowledge, people and infrastructure and the rating system is a key driver in building a globally competitive science system in South Africa. It is a valuable tool for benchmarking the quality of the country s researchers against the best in the world. NRF ratings are allocated based on a researcher s recent research outputs and impact as perceived by local and international peer reviewers and the system encourages researchers to publish high quality outputs in high impact journals/outlets. The evaluation and rating system is a mechanism to nurture scholarship and grow the country s research capacity. Its origins were based on the need to create a mechanism to ensure that the training of the next generation of researchers (i.e. postgraduate students) was entrusted to capable academics. It was also intended to be a mechanism for the recognition of Peer review is an important part of all scientific disciplines. In the mathematical sciences it is very important that peers are chosen in areas related to the candidate s field of research. Reviewers provide valuable input into an area of science that has struggled with being isolated from the mainstream of research for a long period of time. The feedback provided by reviewers and the review process can prove to be an invaluable tool in the development of a long term vision for an applicant s research. Prof E. Momoniat Pr.Sci.Nat., Professor and Head of School: Computational and Applied Mathematics University of the Witwatersrand the achievements of researchers and, by way of funding, to assist them to maintain levels of research excellence. The rating of individual researchers is based primarily on the quality and impact of their research outputs over the past eight years, taking into consideration the evaluation made by local and international peers. It identifies researchers who count as leaders in their fields of expertise and gives recognition to those who constantly produce high quality research outputs. Several South African universities use the outcomes of the NRF evaluation and rating process to position themselves as research-intensive institutions while others provide incentives for their staff members to acquire and maintain a rating, giving special recognition to top-rated researchers. Since its inception in 1999, the NRF has placed increasing emphasis on mechanisms to address deficiencies in the National System of Innovation (NSI) and to stimulate the development of a research culture at Higher Education Institutions (HEIs). This contributed to the gradual delinking of funding from rating over time. The rating system came to be seen mostly as an honorific system, i.e. recognition of excellence and, due to its credibility, was used widely in the country s NSI for performance management purposes as the criteria used by both systems overlapped considerably. An unintended consequence of this was that the rating became a proxy for the kind of recognition which is usually given through national academies in other systems. The above deviation from the rating system was noted in the 2005 NRF Institutional Review and, as a result, an in-depth, external review of the NRF system for the evaluation and rating of individual researchers was commissioned by Higher Education South Africa (HESA) in 2007. The review found that the evaluation and rating system maintained a high level of credibility in the higher education system and was widely used. The Review Steering Committee recommended that the system should be retained with the following enhancements: 1. Rating should be directly linked to funding of individual researchers. 2. The current rating category names and descriptors need to be reviewed as there seems to be a perception that the current categories are too complex and their designations (e.g. A, B, C, etc) demeaning to categories lower than A. 3. The NRF should take responsibility for demystifying the rating process to enhance the transparency and credibility of the system in the research community. 4. The NRF should continue to improve and refine the rating system to address criticisms raised by the research community in the course of reviews. 5. The NRF should align the rating system to international best practice in monitoring and evaluation and ensure that quality remains the cornerstone of the rating system. AISA ARC CAPRISA CEO CGS CIR COHE CPUT CSIR CUT DoE DoEA DoEA:O&C DUT EEC FTE GCRO HartRAO HEI HEMIS HESA HSRC IT ithemba LABS KMC KZNM MSA MUT NBDA NECSA NHLS NICD NMMU NRF NSET NSI NWU Africa Institute of South Africa Agricultural Research Council Centre for AIDS Programme of Research in South Africa Chief Executive Officer Council for Geoscience Considerable International Recognition Council On Higher Education Cape Peninsula University of Technology Council for Scientific and Industrial Research Central University of Technology (Free State) Department of Education Department of Environmental Affairs Department of Environmental Affairs Branch: Oceans & Coasts Durban University of Technology Executive Evaluation Committee Full Time Equivalent Gauteng City-Region Observatory Hartebeesthoek Radio Astronomy Observatory Higher Education Institution Higher Education Management Information System Higher Education South Africa Human Sciences Research Council Information Technology ithemba Laboratory for Accelerator Based Sciences Knowledge Management Corporate KwaZulu-Natal Museum Monash South Africa Mangosuthu University of Technology National Biotechnology Development Agency Nuclear Energy Corporation of South Africa National Health Laboratory Service National Institute for Communicable Diseases Nelson Mandela Metropolitan University National Research Foundation Natural Sciences and Engineering National System of Innovation North-West University NZG R&D RISA RU SA SAAMBR SAAO SACSA SAEON SAIAB SAMRC SANBI SANSA SARB SASRI SAWS SKA SSH&L SU TUT UCT UFH UFS UJ UKZN UL UNISA UP UV UW UWC UZ VUT WH WoS WSU National Zoological Gardens of South Africa Research & Development Research and Innovation Support and Advancement Rhodes University South Africa Oceanographic Research Institute South African Astronomical Observatory St Augustine College of South Africa South African Environmental Observation Network South African Institute for Aquatic Biodiversity South African Medical Research Council South African National Biodiversity Institute South African National Space Agency South African Reserve Bank South African Sugarcane Research Institute South African Weather Service Square Kilometre Array Social Sciences, Humanities and Law Stellenbosch University Tshwane University of Technology University of Cape Town University of Fort Hare University of the Free State University of Johannesburg University of KwaZulu-Natal University of Limpopo University of South Africa University of Pretoria University of Venda University of the Witwatersrand University of the Western Cape University of Zululand Vaal University of Technology Westville Hospital Web of Science Walter Sisulu University 2 3

Foreword by the NRF CEO Building South Africa s Knowledge Economy With the current number of NRF rated researchers at 2 959, we have exceeded our target set for NRF Vision 2015. At the same time we have seen considerable growth in the numbers of scholars from the social sciences, humanities and law participating in the programme now accounting for 33% of all rated researchers. This is exceptionally good news as it means that the rating system is achieving a degree of recognition as a credible benchmarking tool to measure the impact and standing of our research community. This bodes well for South Africa s fledgling knowledge economy. As our researchers attain international recognition, their world-class work will provide sustainable benefits to society, enabling the NRF to deliver on its mandate. There are, however, still demographic challenges which need to be overcome if we are to boast of having achieved a transformed science work force. We still need to produce more black and female rated researchers. Black scientists constitute 22%, and females 30% of rated researchers. Annual growth in these key areas currently stands at only 1%. This slow growth can be partially attributed to the demographic composition of staff at Higher Education Institutions (HEIs) and NRF-recognised Research Institutions. We are going to be doing more to address this challenge with some specific interventions already in place, and we are likely to see an increase in these critical areas in coming years as these interventions gain traction. The rating system itself has undergone some changes designed to make it more efficient, effective and fair. The appeals process has been strengthened and some category descriptions have been refined. In addition, we have instituted the use of multiple panel assessments to address rating applications from researchers working in inter-disciplinary and multidisciplinary areas. As for the ratings themselves, a survey was conducted at the beginning of 2013 to determine whether the rating categories were adequately defined, differentiated and understood by members of the research community. The majority opted for retaining the present categories although their significance and value were sometimes misunderstood. Despite the overall approval of the status quo, however, we will continuously monitor the entire rating system and the perceptions among its stakeholders in an effort to ensure its ongoing effectiveness. In speaking to some of the many researchers who have undergone the rating process, I have found that they overwhelmingly believe that their rating has helped to enhance their careers and standing in the scientific community. It has assisted them in obtaining funding for their research, and enabled them to track their achievements over the course of their careers. At the same time, however, many feel that serving as a member of a rating assessment panel is almost as important as obtaining a rating itself in so far as it provides a deeper insight into the rating process and how one is actually reviewed by peers. I would like to take this opportunity to not only thank those researchers who have subjected themselves to peer assessment, but also the thousands of reviewers and committee members who have, over the years, contributed their time and expertise to improving the rating system. It is their efforts that helped to build the system into what it is today. Dr Albert van Jaarsveld Chief Executive Officer: National Research Foundation Foreword by the Deputy CEO The Evaluation and Rating System is an Evolutionary Process With the numbers of NRF-rated researchers growing exponentially, I am proud to report that currently there are 2 959 NRF-rated researchers. That is an increase of 815 since the last issue of Evaluation and Rating: Facts & Figures in 2010. What is interesting to note is that one third of the current rated researchers hail from the social sciences, humanities and law - a significant increase that bodes well for the recognition of individuals in these fields. I am also pleased to note that the NRF has provided R100 million in incentive funding for research-related purposes over the last year, enabling our country s scientists to maintain their research careers. However, there are still challenges within our research landscape, the most important of which is the need to develop more black and female rated researchers. The shortage of black female rated researchers is of particular concern and we are addressing this through NRF programmes such as Thuthuka, which provides nine years of funding for this important group three years for a doctoral degree, three years for post-doctoral training and three years to establish themselves as researchers. In addition, a new instrument has been created to fund those researchers in the Y- and P-rated categories to enable them to establish themselves as highly recognised researchers in their fields. The NRF conducts surveys and workshops on a regular basis in order to ascertain the effectiveness of its rating system and to implement changes where necessary. Feedback in these areas has indicated satisfaction with the system and its alignment with the recommendations of the recently conducted NRF Review. We have also confirmed that our Reviews and Evaluation business unit, recently refocused as a result of the NRF s Research and Innovation Support and Advancement (RISA) Renewal process and which is responsible for the evaluations of bursaries and research proposals, review of programmes and, of course, the evaluation of ratings, is aligned with the recommendations of the NRF Review and the rating system. We are also currently developing guidelines for reviews and evaluations in line with the European Science Research Foundation (ESRF). The Executive Evaluation Committee (EEC) deliberates on cases where there is no consensus as well as ratifies those who have been nominated for A and P categories. Another important change that the system has undergone is the formation of the guidelines and criteria for appeals. South Africa is only one of a few countries in the world to maintain a fully operational rating system. In a developing knowledge economy such as ours, we see it as a critical tool for recognising and affirming the excellence of research and researchers. What makes our particular rating system effective is the fact that we do not count the number of publications that researchers have produced but rather weigh the outputs through an array of established metrics. The system is structured so that excellence remains the cornerstone of the decision making process. The panels that adjudicate the ratings and assess peer reports consist of experts in their particular fields. It is not the NRF that rates researchers but their peers who do. It is a platform for our researchers to determine their standing in the eyes of their fellow researchers. Dr Gansen Pillay Deputy CEO and Managing Director: Research and Innovation, Support and Advancement (RISA) 4 5

Foreword by the Executive Director: KMC Change is the Only Constant The fundamental nature of the evaluation and rating system is one of change. If evolution has taught us anything, it is that nothing can remain unchanged for too long and the NRF has recognised the need to take the process forward in keeping with the evolutionary landscape of South Africa s research environment. Where issues of race and demography of panel members arise, we seek to address them equitably. Panel membership runs on a four-year cycle and we strive to transform the composition of panel members in terms of race and gender as much as possible. When nominating new panel members, current Specialist Committee members are requested to nominate possible replacements to avoid creating an expertise gap within the committees. However, challenges still exist as many researchers decline their nominations and many struggle to divide their time between their own research work at their institutions and the various panels they are asked to serve on. The NRF is cognisant of the challenges and transformation is continuously taking place. EVALUATION AND RATING TRENDS Rated researchers in the Higher Education Institutions (HEIs) landscape The NRF tracks the number of rated researchers in different fields and at different institutions over time. These trends highlight subject areas and environments where South Africa has ample research leadership and expertise, as well as those where capacity needs to be strengthened. Table 1: Distribution of researchers with a valid NRF rating in 2009/10 compared to 2013/14 Number of rated researchers 2009/10 2013/14 SA Gov/Research Insts. 47 42 SA Industry 22 7 SA Museums 17 22 SA National Facilities 33 46 SA HEIs 1 885 2 660 SA Science Councils 52 110 SA Other * (Self-employed, retired, NGOs, Private HEIs) 14 19 International Other 74 53 Total 2 144 2 959 Table 2: NRF Ratings: Participation rates from HEIs The annual Executive Evaluation Committee/Conveners and Assessors Workshop is a forum for raising issues that affect the rating and evaluation system. Concerns raised during the year are discussed and decisions are 120 100 80 60 40 20 0 NRF Rating Panels: Transformation 2007 2010 2014 White Black NRF Rating panels: Transformation by Gender made by panel members themselves with the EEC making recommendations for the incorporation of those decisions into policy. This ensures that panel members become ambassadors for the rating and evaluation system within their own institutions. Observer status, an intervention introduced in the last cycle, allows research office staff and nominated researchers who are concerned about the rating process to attend panel meetings as observers to see and experience the process in action. This enhances their understanding of the process, particularly that it is completely peer-driven. In terms of h-index importance in the rating process, it is agreed that it be used as a supplementary source of information for the validation of opinions expressed in peer reviewer reports. It is important to the integrity and fairness of the system that peer-review take precedence over h-index information or other metrics. No. of rated researchers FTE (Instruction/ research professors) Percentage Headcount Percentage Headcount with doctoral qualification Percentage 2009/10 2 285 21 421 10.67% 16 230 14.08% 5 694 40.13% 2010/11 2 471 21 969 11.25% 16 684 14.81% 5 956 41.49% 2011/12 2 635 23 274 11.32% 16 934 15.56% 6 345 41.53% Higher Education Management Information System (HEMIS) data: *Table 3.1: Utilisation of FTE personpower resources according to programme/ sub-programme (national) **Table 3.4 Headcount of instruction/research professionals with permanent appointments (national) Table 2 lists the ten universities in South Africa with the highest percentage of rated researchers on their staff. For more detail on the number of rated researchers at all South African HEIs, see Tables 14, 15 and 16 in Appendix A. How many South African researchers have an NRF rating? 120 100 80 60 40 20 0 2007 2010 2014 Male Female Reviewer fatigue is another challenge. As South Africa maintains such as small pool of reviewers who understand the rating and evaluations process, those participating in the process are being over-utilised by the various panels. The NRF is currently developing mechanisms to ensure that reviewers are not extended beyond their limits. Dr Daisy Selematsela Executive Director: Knowledge Management Corporate Of the nearly 17 000 staff members in academic and related positions at, 11% have a valid rating from the NRF (see Table 2). The total number of rated researchers, at all institutions locally and abroad, increased by 38% from 2 144 in 2009, to 2 959 in 2013. 6 7

Table 3: The top 10 universities in South Africa in 2009/10 and 2012/13 (latest HEMIS figures available) in terms of the percentage of their research/instruction staff with a valid NRF rating Institution Rated researchers 2009/10 Headcount: Instruction/ research professionals* 2009/2010 Percentage rated (headcount) FTE personpower resources*** Percentage Rated (FTE) University of Cape Town 334 965 34.61% 1 184 28.21% Stellenbosch University 287 873 32.88% 1 111 25.83% University of Pretoria 280 1 635 17.13% 1 937 14.46% University of KwaZulu-Natal 193 1 452 13.29% 1 533 12.59% University of the Western Cape 76 524 14.50% 641 11.86% University of the Witwatersrand 217 979 22.17% 1 873 11.59% North-West University 119 996 11.95% 1 251 9.51% Nelson Mandela Metropolitan University 56 569 9.84% 671 8.35% University of the Free State 92 779 11.81% 1 129 8.15% University of South Africa 98 1 391 7.05% 1 598 6.13% University of Johannesburg 86 856 10.05% 2 020 4.26% Institution Rated researchers 2012/13 Headcount: Instruction/ research professionals* 2012/13 Percentage rated (headcount) FTE personpower resources*** Percentage Rated (FTE) University of Cape Town 412 1055 38.86% 1 523 26.92% Stellenbosch University 313 939 33.44% 1 206 26.04% University of Pretoria 334 1 281 26.00% 1 973 16.88% University of the Western Cape 94 536 17.16% 656 14.02% University of KwaZulu-Natal 209 1 470 14.22% 1 664 12.56% University of the Witwatersrand 243 1 044 23.18% 2 064 11.72% North-West University 143 1 194 12.06% 1 371 10.50% University of the Free State 105 846 12.41% 1 097 9.57% Nelson Mandela Metropolitan University 61 579 10.54% 733 8.32% University of South Africa 132 1 515 8.78% 1 937 6.87% University of Johannesburg 114 871 13.09% 1 984 5.75% *Higher Education Management Information System (HEMIS) Report, Department of Education (DoE), (Report 3.4 Headcount of instruction/research professionals with permanent appointments according to highest most relevant qualification and rank ***Higher Education Management Information System (HEMIS) Report, Department of Education (DoE), Report 3.1 Utilisation of FTE personpower resources according to programme/sub-programme Transformation trends (race, gender and discipline) Table 4 Changes in the profile of rated researchers (2007-2013*) Total NSET % SSH&L % White % Black % Male % Female % 2007 1 699 1 197 70% 502 30% 1 440 85% 259 15% 1 275 75% 424 25% 2008 1 924 1 360 71% 564 29% 1 609 84% 316 16% 1 424 74% 500 26% 2009 2 144 1 481 69% 663 31% 1 775 83% 369 17% 1 566 73% 578 27% 2010 2 285 1 545 68% 740 32% 1 852 81% 433 19% 1 651 72% 634 28% 2011 2 471 1 644 67% 827 33% 1 964 79% 507 21% 1 754 71% 717 29% 2012 2 635 1 758 67% 877 33% 2 067 78% 568 22% 1 856 70% 779 30% 2013 * 2 959 1 957 66% 1002 34% 2 291 77% 668 23% 2 070 70% 889 30% * As at 31 March 2014 Figure 1: Instruction/research professionals* 18 000 16 000 14 000 12 000 10 000 8 000 6 000 4 000 2 000 0 2009 2010 2011 18 000 16 000 14 000 12 000 10 000 8 000 6 000 4 000 2 000 0 White Coloured Indian Black HEMIS data * Table 3.3: Headcount of personnel with permanent appointments according personnel category, race and gender The number of rated black and female researchers and their share of the total pool of rated researchers are increasing steadily, as illustrated by Figures 4 and 5. Since 2002, when the social sciences, humanities and law were first included, the number of rated black researchers has more than tripled. Black 2009 2010 2011 Male Female researchers now comprise 17% of the total pool of rated researchers compared to 8% in 2002. The number of rated female researchers more than doubled from 2002 to 2009. By 2009 nearly one third of all rated researchers at South African education institutions were women. At one time the preserve of the natural sciences, the NRF rating system has become increasingly embraced across the spectrum of academic disciplines. Its strengths are that it is founded upon peer evaluation, so it self-adjusts for different disciplinary approaches to scholarship, and it is based on scholarly quality as the principle consideration. Dr Peter Clayton Deputy Vice-Chancellor: Research and Development, Rhodes University 8 9

Table 5: Rating applications submitted 2008-2013 Sources and outcomes 2008 2009 2010 2011 2012 2013 2007 cycle Total 2008 cycle Total 2009 cycle Total 2010 cycle Total 2011 cycle Total 2012 cycle Total A 26 70 18 78 17 85 7 85 21 89 27 93 P 6 14 2 16 2 17 3 16 4 17 3 14 B 134 391 108 441 98 473 88 502 112 533 168 569 C 298 943 271 1 066 283 1 206 254 1 310 346 1 434 409 1 553 Y 55 211 72 253 76 295 72 324 91 364 72 382 L 14 55 13 60 12 61 0 50 0 34 0 24 Rating unsuccessful (RU*) Applications withdrawn 49 52 64 60 92 79 39 20 39 36 29 28 TOTAL 621 1 684 556 1 914 591 2 137 520 2 287 695 2 471 786 2 635 *RU (as %) 8% 9% 11% 12% 13% 10% Year-on-Year increase Total 14% 12% 7% 8% 7% Type # of % # of % # of % # of % # of % # of % New 245 40% 262 47% 340 58% 288 55% 367 53% 338 43% Re-submissions 373 60% 294 53% 251 42% 232 45% 328 47% 448 57% Total 618 100% 556 100% 591 100% 520 100% 695 100% 786 100% NSET 351 57% 400 72% 402 68% 335 64% 483 69% 480 61% SSH&L 267 43% 156 28% 189 32% 185 36% 212 31% 306 39% Total 618 100% 556 100% 591 100% 520 100% 695 100% 786 100% Male 468 76% 396 71% 401 68% 359 69% 483 69% 576 73% Female 150 24% 160 29% 190 32% 161 31% 212 31% 210 27% Total 618 556 100% 591 100% 520 100% 695 100% 786 100% Black 59 10% 60 11% 67 11% 94 18% 133 19% 131 17% Coloured 21 3% 18 3% 27 5% 27 5% 27 4% 18 2% Indian 33 5% 33 6% 42 7% 44 8% 47 7% 41 5% White 505 82% 445 80% 455 77% 355 68% 488 70% 596 76% Total 618 100% 556 100% 591 100% 520 100% 695 100% 786 100% Table 6: The number of rated researchers at South African universities in 2006, 2009 and 2012 Institution (listed alphabetically) 2006 2009 2012 NSET SSH&L NSET SSH&L NSET SSH&L Cape Peninsula University of Technology 6 4 8 7 16 7 Central University of Technology, Free State 5 1 5 1 4 3 Durban University of Technology 11 1 10 1 10 1 Mangosuthu University of Technology 0 0 0 0 2 0 Monash South Africa 0 0 0 2 1 2 Nelson Mandela Metropolitan University 33 17 37 20 44 17 North-West University 56 29 72 45 83 60 Rhodes University 39 12 43 9 45 20 St Augustine College of South Africa 0 0 0 1 0 1 Stellenbosch University 151 80 185 84 210 103 Tshwane University of Technology 16 4 30 4 31 9 University of Cape Town 196 75 223 93 283 129 University of Fort Hare 5 1 8 4 14 5 University of Johannesburg 41 28 44 32 63 51 University of KwaZulu-Natal 102 40 124 56 134 75 University of Limpopo 8 1 5 1 7 1 University of Pretoria 154 60 178 79 219 115 University of South Africa 18 63 21 73 38 94 University of the Free State 49 20 55 27 67 38 University of the Western Cape 35 32 42 33 49 45 University of the Witwatersrand 135 39 146 50 177 66 University of Venda 1 0 4 0 14 1 University of Zululand 9 5 9 5 5 6 Vaal University of Technology 1 1 2 1 7 0 Walter Sisulu University 2 0 4 1 6 1 NSET: Natural Sciences and Engineering SSH&L: Social Sciences, Humanities and Law Where do NRF-rated researchers work? About 90% of researchers with a currently valid rating from the NRF are attached to South African universities, while close to 6% are employed in research positions at local science councils, museums or national research facilities. The rest are in private sector institutions in SA or have left the country to work abroad (see Table 1). 10 11

Table 7: The spread of NRF ratings per institution in 2013 Government departments/research institutions (SA) South African Industry SA HEIs A B C L P Y Total COHE 1 1 DoEA 1 1 DoEA:O&C 1 1 GCRO 1 1 NBDA 1 1 NHLS 2 4 2 8 NICD 2 4 1 7 NECSA 5 2 7 SAAMBR 2 2 SANSA 2 2 4 SASRI 6 6 SAWS 1 1 SARB 1 1 SKA 1 1 Creation Breeding Innovations 1 1 Eskom 1 1 Exxaro Coal (Pty) Ltd 1 1 Fruitgro Science 1 1 Panthera 1 1 Randgold Resources Ltd 1 1 Roche SA 1 1 CPUT 1 16 1 8 26 CUT 5 1 6 CAPRISA 1 1 DUT 1 11 12 MUT 1 1 2 MSA 3 3 NMMU 2 5 56 5 68 NWU 3 17 119 2 2 28 171 RU 2 16 42 13 73 SACSA 1 1 SU 13 93 176 1 3 58 344 TUT 2 36 1 4 43 UCT 33 150 183 2 6 69 443 UFH 1 16 5 22 UJ 6 28 74 1 26 135 UKZN 6 39 151 2 34 232 UL 8 3 11 UP 11 70 231 3 2 55 372 UNISA 2 14 106 1 19 142 UFS 12 93 1 11 117 UWC 1 25 67 1 11 105 UW 15 80 138 2 2 53 290 UV 1 10 6 17 UZ 14 1 15 VUT 5 2 7 WSU 4 1 2 7 WH 1 1 A B C L P Y Total Albany Museum 2 2 Bayworld Centre for Research & Education 1 2 3 DITSONG Museums of South Africa 1 1 2 SA Museums East London Museum 1 1 Iziko Museums of South Africa 3 1 4 KwaZulu-Natal Museum 1 4 5 National Museum 4 1 5 HartRAO 2 2 ithemba LABS 5 8 4 17 SA National Facilities NZG 1 1 2 SAAO 1 5 8 3 17 SAEON 2 2 SAIAB 1 3 2 6 AISA 1 1 ARC 4 21 7 32 CGS 2 2 SA Science Councils CSIR 10 16 1 12 40 HSRC 1 3 9 1 14 Mintek 1 1 2 SAMRC 1 1 6 3 11 SANBI 2 4 2 8 Self-employed/retired Self-employed/retired 2 10 1 13 Other (abroad) Other (abroad) 1 11 26 1 14 53 Grand total 99 615 1 724 22 16 483 2 959 The participation rate of researchers from the Social Sciences, Humanities and Law (SSH&L), as well as black and female researchers, has grown at approximately the same rate (on average 1% per annum) as their appointments in the HEIs. It should, however, be noted that this information is not limited to those who are in possession of a doctorate which accounts for only 42% of the total number. 12 13

Table 8: The spread of rated researchers in the various rating categories per subject areas (grouped according to NRF Specialist Committees) for 2013 Table 9: The top performing higher education institutions in South Africa during 2013 in terms of the number of rated researchers per NRF Specialist Committee (per subject areas) NRF Specialist Committees - Social Sciences, Humanities & Law A B C L P Y Total Social Sciences, Humanities & Law Natural Sciences and Engineering Anthropology, Development Studies, Geography, Sociology and Social Work 2 32 65 2 18 119 Communication, Media Studies, Library and Information Sciences 6 22 2 6 36 Economics, Management, Administration and Accounting 1 14 96 2 26 139 Education 1 15 107 5 6 134 Historical Studies 2 16 35 4 57 Law 4 39 73 1 18 135 Literary Studies, Languages and Linguistics 4 24 93 1 13 135 Performing and Creative Arts, and Design 8 31 5 44 Political Sciences and Philosophy 2 16 35 2 4 59 Psychology 2 9 50 6 67 Religious Studies and Theology 4 12 56 5 77 Total for the social sciences, humanities and law 22 191 663 10 5 111 1002 NRF Specialist Committees - Natural Sciences and Engineering A B C L P Y Total Animal and Veterinary Sciences 12 49 146 1 1 40 249 Biochemistry, Molecular & Cell Biology 1 21 44 21 87 Chemistry 4 30 75 5 35 149 Earth Sciences 3 47 93 3 24 170 Engineering 8 40 125 56 229 Health Sciences 22 76 213 4 1 63 379 Information Technology 10 55 1 20 86 Mathematical Sciences 7 42 76 1 27 153 Microbiology and Plant Pathology 4 15 59 1 18 97 Physics 9 60 90 1 4 37 201 Plant Sciences 7 34 85 31 157 Total for the natural sciences and engineering 77 424 1061 12 11 372 1957 Total for all disciplines 99 615 1 724 22 16 483 2 959 Panel Anthropology, Development Studies, Geography, Sociology and Social Work Communication, Media Studies, Library and Information Sciences Economics, Management, Administration and Accounting Education Historical Studies Law Literary Studies, Languages and Linguistics Performing and Creative Arts, and Design Political Sciences and Philosophy Number Institution of rated researchers 1. UCT 21 2. UKZN 17 3. UW 15 1. UP 7 2. UCT 6 3. UKZN 4 3. UNISA 4 1. UP 24 2. SU 21 2. UCT 21 3. UW 14 1. UP 20 2. SU 15 2. UCT 15 3. UKZN 14 1. UCT 15 2. UNISA 6 2. UW 6 3. UJ 5 1. UNISA 32 2. UCT 19 3. UP 16 1. NWU 17 2. SU 14 2. UP 14 2. UW 14 3. UCT 13 1. UCT 11 2. SU 6 2. UP 6 2. UW 6 3. UNISA 4 1. SU 13 2. UP 7 3. UJ 6 1. SU 7 Total number of rated researchers 119 36 139 134 57 135 135 44 59 Panel Animal and Veterinary Sciences Biochemistry, Molecular & Cell Biology Chemistry Earth Sciences Engineering Health Sciences Information Technology Mathematical Sciences Microbiology and Plant Pathology Physics Plant Sciences Institution Number of rated researchers 1. UP 66 2. UCT 21 3. SU 20 1. UCT 21 2. SU 13 3. UP 12 1. SU 18 2. UCT 15 2. UW 15 3. UKZN 14 1. UW 34 2. UCT 30 3. UJ 13 1. UCT 43 2. SU 37 3. UP 32 1. UCT 83 2. SU 49 3. UW 45 1. UCT 17 2. UP 15 3. NMMU 9 3. UNISA 9 1. UCT 27 2. UW 24 3. UP 20 1. UP 22 2. SU 18 3. UFS 15 1. UCT 27 2. UW 21 3. UKZN 18 1. SU 25 2. UKZN 19 3. UP 17 Total number of rated researchers 249 87 149 170 229 379 86 153 97 201 157 1. UCT 7 1. UP 7 Psychology 2. NWU 6 67 2. UW 6 3. UKZN 5 3. UNISA 5 Religious Studies and Theology 1. UP 16 2. SU 15 3. UNISA 11 77 14 15

Table 10: The age profile of rated researchers employed at and museums in 2013 Age A B C L* P Y Total number of rated researchers 20-29 0 0 0 0 1 23 24 30-39 0 32 187 1 15 370 605 40-49 15 185 634 11 0 26 871 50-59 34 199 530 7 0 0 770 60 - > 45 145 228 0 0 0 418 Grand Total 94 561 1 579 19 16 419 2 688 *The L Committee was discontinued on 01 January 2010. Researchers with a valid L-rating will retain their ratings until the rating lapses. A rating is valid for six years. Figure 2: Percentage of rated researchers by age in 2012 800 700 600 500 400 300 200 100 0 20-29 30 39 40 49 50 59 Over 60 Figure 4: Age distribution of the combined number of A-, B- and C-rated researchers in 2003, 2006, 2009 and 2012 at and museums 100% 80% 60% 40% 20% 0% 2003 2006 2009 2012 Figure 5: Rated researchers in 2014: age and gender comparison 1000 800 600 400 200 0 Up to 30 31-40 41-50 51-60 61-70 over 70 Figure 3: Comparison of the percentage of A-, B- and C-rated researchers in 2003, 2006, 2009 and 2012, according to their age, at and museums 30-39 40-49 50-59 60+ Female Male 100% 80% 60% 40% 20% 0% < 40 40-49 50-59 60 < 40 40-49 50-59 60 < 40 40-49 50-59 60 < 40 40-49 50-59 60 2003 2006 2009 2012 A B C The average annual increase in the number of applications is approximately 10% as is the number of unsuccessful applications as a proportion of the total number of applications submitted (after screening). The proportion of re-evaluations vs new applications remains at around 50% 16 17

THE A TO Z OF NRF EVALUATION AND RATING The evaluation and rating of individual researchers In the evaluation and rating of individual researchers, the NRF approaches external peer reviewers to evaluate the impact and standing of South African researchers against global norms. These peer reviewers base their opinions on the quality and impact of each applicant s research outputs and achievements in the last eight years. In addition to an extensive network of reviewers, the following committees/persons play a key role in the rating process: 22 Specialist Committees, comprising over 100 academic researchers, who each select a Convener from their own ranks to coordinate their annual activities; Five externally appointed Chairpersons and five Assessors; The Executive Evaluation Committee (EEC), comprising five external Chairpersons; one Convener representing the NSET panels and one Convener representing the SSH&L panels; and three senior NRF executives; The Appeals Committee, comprising five external members (including the chairperson) and the NRF CEO. Selection and role of Assessors and Chairpersons Assessors are selected from the ranks of highly respected researchers who have served on Specialist Committees, normally as Conveners, in the past and are, therefore, familiar with the process and who have earned a reputation during their tenure on these Specialist Committees for their wisdom and objective judgements. They have to ensure that the same standards are maintained across the different fields of research; that the rating process is fair and independent and that the same criteria are applied consistently by all of the Specialist Committees A Chairperson is a senior researcher of repute who is thoroughly familiar with the NRF rating system. Their role is to facilitate decision making and, if necessary, to act as a second Assessor. To be able to fulfil this role properly, both Chairpersons and Assessors need to participate in a number of panels (up to five). They may not sit on a panel that deals with research in which they, themselves, perform research. Selection and role of Specialist Committee members The members of Specialist Committees are respected members of the South African research community who, themselves, are rated and represent a specific field of research within their discipline. They are appointed for a period of four years on a rotation basis. Existing committee members propose the names of potential committee members with similar research expertise to replace them once their term has expired. Committee members screen the evaluation and rating applications for appropriateness of the primary panel and premature submissions. They are also responsible for the selection of an appropriate mix of reviewers for each applicant. Their main goal is to ensure that a sufficient number of reports of acceptable quality, with a balance in the number of reports submitted by reviewers nominated by the applicant and those nominated by the Specialist Committee, is available at the panel meeting. They assess the usability of reports and translate the reviewers comments into a rating category at the panel meeting to determine the placement of the applicant in a rating category. It is important to note that members of Specialist Committees do not serve as peer reviewers for the applicant. They only interpret the reports submitted by peer reviewers. Figure 6. Steps in the evaluation and rating of individual researchers Submission of applications NRF Submission System Screening by Designated Authorities (DAs) at institutions, submission to NRF with proposed rating Screening of application and selection of peer reviewers by members of SCs Reviews and Evaluation (RE) staff approach reviewers and coordinate (with Convener of SC) the process to obtain [six] reviewer reports RE staff send meeting docs to members of Assessment Panels SC including Convener Chairperson Assessor Referred back via RE as premature Referred to another SC The role of Assessors in the evaluation and rating process Assessors must ensure that the same standards and criteria are applied consistently throughout the process. they study the applicant s documentation and reviewers reports to reach an independent opinion on an appropriate rating category for each applicant. Independent preparations How does the NRF define research? For purposes of the NRF, research is original investigation undertaken to gain knowledge and/or enhance understanding. Research specifically includes: The creation and development of the intellectual infrastructure of subjects and disciplines (e.g. through dictionaries, scholarly editions, catalogues and contributions to major research databases). the invention or generation of ideas, images, performances and artefacts where these manifestly embody new or substantially developed insights. building on existing knowledge to produce new or substantially improved materials, devices, products, policies or processes. It specifically excludes: routine testing and analysis of materials, components, instruments and processes, as distinct from the development of new analytical techniques. the development of teaching materials and teaching practices that do not embody substantial original enquiry. Joint meeting of Assessment Panel Allocation of a rating communicated by RE to institution Outcome letter with feedback (compiled by SC/feedback writer) sent by RE staff Appeals process No consensus Nominations: A & P More reports Virtual meeting Executive Evaluation Committee 18 19

THE KEY ROLE OF THE PEER REVIEWERS Definition of rating categories The definitions of the rating categories are given below. Descriptions of sub-categories in the A, B, C and Y categories have also been indicated. It must be borne in mind that the peer evaluation process is intricate and not mechanistic. Ultimately, the judgement of the members of the Assessment Table 11: Detailed definitions of NRF rating categories and sub-categories Category Definition Sub-category Description A B C P Y Researchers who are unequivocally recognised by their peers as leading international scholars in their field for the high quality and impact of their recent research outputs. Researchers who enjoy considerable international recognition by their peers for the high quality and impact of their recent research outputs. Established researchers with a sustained recent record of productivity in the field who are recognised by their peers as having: Produced a body of quality work, the core of which has coherence and attests to ongoing engagement with the field Demonstrated the ability to conceptualise problems and apply research methods to investigating them. Young researchers (normally younger than 35 years of age), who have held the doctorate or equivalent qualification for less than five years at the time of application and who, on the basis of exceptional potential demonstrated in their published doctoral work and/or their research outputs in their early post-doctoral careers are considered likely to become future international leaders in their field. Young researchers (40 years or younger), who have held the doctorate or equivalent qualification for less than five years at the time of application, and who are recognised as having the potential to establish themselves as researchers within a fiveyear period after evaluation, based on their performance and the production of quality research outputs during their doctoral studies and/or early post-doctoral careers.. A1 A2 B1 B2 B3 C1 C2 C3 Y1 Y2 Committees and their wisdom which has some intangible components must be relied upon. Hence the interpretation of words such as broad field, narrow area, considerable, etc. form an important part of the Assessment Committees task in their role of assessment of reviewers reports. A researcher in this group is recognised by all reviewers as a leading scholar in his/her field internationally for the high quality and wide impact (i.e. beyond a narrow field of specialisation) of his/her recent research outputs. A researcher in this group is recognised by the overwhelming majority of reviewers as a leading scholar in his/her field internationally for the high quality and impact (either wide or confined) of his/her recent research outputs. All reviewers are firmly convinced that the applicant enjoys considerable international recognition for the high quality and impact of his/her recent research outputs, with some of them indicating that he/she is a leading international scholar in the field. All or the overwhelming majority of reviewers are firmly convinced that the applicant enjoys considerable international recognition for the high quality and impact of his/her recent research outputs. Most of the reviewers are convinced that the applicant enjoys considerable international recognition for the high quality and impact of his/her recent research outputs. Some reviewers as already enjoying considerable international recognition; OR The overriding majority of reviewers as being a scholar who has attained a sound/solid international standing in their field, but not yet considerable international recognition; OR *The overriding majority of reviewers as being a scholar whose work focuses mainly on local and/or regional issues and who as a scholar at a nationally leading level has substantially advanced knowledge and understanding in the field by contributing to new thinking, a new direction and/or a new paradigm.* All of the reviewers are firmly convinced that the applicant is an established researcher as described. The applicant may, but need not, enjoy some international recognition for the quality and impact of his/her recent research outputs. Most of the reviewers concur that the applicant is an established researcher (as described). Researchers in this group are recognised by all or the overwhelming majority of reviewers as having demonstrated the potential of becoming future international leaders in their field on the basis of exceptional research performance and output from their doctoral and/or early post-doctoral research careers. A young researcher (within 5 years from PhD) who is recognised by all reviewers as having the potential (demonstrated by quality research products) to establish him/herself as a researcher with some of them indicating that he/she has the potential to become a future leader in his/her field. OR A young researcher (within 5 years from PhD) who is recognised by all or the overwhelming majority of reviewers as having the potential to establish him/herself as a researcher of considerable international standing on the basis of the quality and impact of his/her recent research outputs. A researcher in this group is recognised by all or the overwhelming majority of reviewers as having the potential to establish himself/herself as a researcher (demonstrated by recent research products). * This definition is restricted to those researchers whose area of research prevents (or precludes) them from meeting the requirements of either definition 1 or definition 2 [Last update: 29 July 2014] Thousands of peer reviewers from South Africa and around the world participate in the NRF evaluation and rating system as reviewers. They provide their expertise and time free of charge and contribute to scholarly endeavours. The NRF depends on its Specialist Committees to constantly identify new individuals to avoid overburdening existing reviewers with repeated review requests. The NRF received 571 applications for evaluation and rating in 2009/10 and approached 7 188 reviewers to assess these applications. This means that, on average, more than 12 reviewers were approached for each application received. More than half (55%) of these reviewers responded and eventually 3 600 reviewers participated in the process. Below are the comparative figures for 2012/13 which shows a similar trend but indicate the greater willingness of reviewers outside the country to submit a report. Figure 7: 2012/13 Rating Reviewers - Who are they? 5 000 4 000 3 000 2 000 1 000 0 Nominated by applicant South Africa Submitted a report Outside of South Africa Choosing reviewers When a researcher applies for evaluation, he/she also submits a list of six potential reviewers. Applicants are requested to indicate their relationship with each reviewer and to give reasons for each nomination in order to provide the Specialist Committee with additional information for prioritisation which influences the order in which they are approached. The NRF guidelines recommend that up to half of the reviewers approached should be selected from the applicant s list, while the Specialist Committees nominate the remainder. However, in reality, this balance might not be achieved as, in some instances, the reviewers suggested by the applicant are not available, while others may decline the review request. Specialist Committee members are encouraged to consult colleagues (locally or abroad) and have access to electronic publication and citation databases such as Web of Science (WoS) and Scopus in order to guide reviewer selection. In some cases reviewers who have declined to review suggest names of other potential alternative reviewers. What if I am working in more than one field? Members of Specialist Committees in these cases consult with other Specialist Committees that would be able to suggest suitable reviewers in the fields that form part of their key research area. Reviewers (normally more than six, especially if the fields are very divergent) are chosen to ensure that the scope and impact of all the work is adequately covered (the publication record and narrative fields in the application should provide guidance in this regard). The final rating outcome is determined by the field in which the applicant is the strongest. Specialist Committees are, however, reminded that the key rating criterion of coherence is not overlooked in the process. Why is it important to select the correct primary panel and how do I do it? The members of the Specialist Committee each represent a specific field which is a key research area of the panel and are therefore familiar with the context in which the research is done. This is important when identifying reviewers and interpreting their comments. The document Key Research Areas and Types of Research Outputs contains the following details for each Specialist Committee and should always be consulted as a first step when applying for a rating: Key research areas covered; Research outputs taken into account in the rating process and their relative importance; and Boundaries and overlaps between Specialist Committees allowing applicants whose fields of research straddle more than one committee to ascertain where their application will best be handled. All reports are not equal Members of Assessment Panels play an interpretative role when they assess reviewers reports. Given the specialist committee members knowledge of the field, they are able to assess the veracity of the comments made by reviewers. For example, if a reviewer states that the applicant is publishing in top journals, yet the journals are in the bottom tier of those in the field, then the validity of the reviewer s report must be called into question. Reviewers who provide a thorough analysis and critique of recent research outputs and substantiate comments/judgements on the quality and impact of the research outputs of the last eight years as well as the international/national standing of the applicant, are given more weight in the decision making process. Reports from reviewers who have indicated that they have not read any of the research outputs of the last eight years; provided unsubstantiated testimonials; or write a report that is obviously biased or hostile, are not used. Number of reports required for a credible decision Although the aim is to procure six reviewer reports, four reports of acceptable quality expressing a consistent opinion is the NRF benchmark. 20 21

STRIVING FOR A FAIR AND CREDIBLE PROCESS REPORT BACK ON PROGRESS MADE WITH THE RECOMMENDATIONS OF THE HESA REVIEW The credibility and fairness of the NRF evaluation and rating system are strengthened by: The collective effort and wisdom of the Specialist Committees in identifying appropriate reviewers The availability of a sufficient number of quality reports on which to base the rating decision. Assessors and Chairpersons who moderate the process. The availability of good policy/guideline documents on all aspects of the rating process (e.g. selection of reviewers; key research areas and types of research outputs for each Specialist Committee; the meaning of considerable international recognition (CIR) and appropriate indicators thereof for each Specialist Committee). The following checks and balances are built into the system to further ensure fairness: Both reviewers and members of Specialist Committees have to declare any conflicts of interest beforehand and will not be able to access the documents unless they have done so; Information on the process is freely available. Names of Specialist Committee members as well as all rating policy and process guidelines are posted on the website; At least six reviewers (although usually up to 12) are approached for each applicant. If the Assessment Panel members are of the opinion that the reports do not provide enough credible information to make a clear-cut decision, they will request reports from additional reviewers until they are satisfied; The relevant Specialist Committee, Assessor and Chairperson study the reviewers reports independently. They then meet to compare their recommended rating for each applicant in order to reach consensus. The use of a panel member facility (IT tool) that requires all Specialist Committee members to submit their assessment of reports independently before the official panel meeting means that no room for group think or dominance of one panel member at the meeting exists; Across all disciplines, all nominations for A and P ratings are referred to the Executive Evaluation Committee for a decision. Table 12: Number of appeals against the outcome of evaluation and rating Should the Assessment Panels not reach consensus, the case is referred to the Executive Evaluation Committee (This occurs in less than 2% of cases). Appealing against an evaluation result Applicants may appeal against the outcome of an evaluation, with or without the support of their institution, but all appeals must be processed through the relevant authority at the applicant s institution. Although no restrictions are placed on the grounds upon which an applicant may appeal an evaluation decision, the appeal documentation must indicate the extent to which the particular evaluation outcome was seriously incongruent with the institution s assessment of the applicant or applicant s own assessment and must substantiate why the institutional assessment is to be preferred over that of the assessment panel. The Appeals Committee will assess both substantive and/or procedural issues brought up in the letter of appeal. It will also determine whether evidence of materiality (i.e. is any identified irregularity sufficiently substantial enough to constitute a material deviation from a required norm to warrant overruling of the previous decision) exists. The decision of the Appeals Committee is final and binding. The revised feedback model now provides a reviewer profile Comprised of the following information which can indicate potential procedural flaws: Number of reviewers approached Number of reports received Number of reports used Number of reports from reviewers based in South Africa Number of reports from reviewers based outside of South Africa Number of reports from reviewers nominated by the applicant Number of reports from reviewers selected by Specialist Committee The percentage of appeals against a rating decision has remained consistent at between 4% and 7% of total evaluations processed over the past seven years, as shown in Table 12 below. 2007 2008 2009 2010 2011 2012 Number of applications for evaluation 621 556 591 520 695 786 Total number of appeals 27 25 27 26 23 29 Total number of appeals successful 5 3 6 3 9 6 Percentage of total applications appealed 4% 4% 5% 5% 3% 4% % Successful 19% 12% 22% 12% 39% 21% An in-depth, external review of the NRF system for the evaluation and rating of individual researchers was commissioned in 2007 in response to concerns raised in the 2004/5 NRF Institutional Review. The Review Steering Committee made the following recommendations/observations: 1. Rating and funding Stakeholders perceived a level of ambivalence to the value and use of the rating system and how it was to be integrated within its other programmes Figure 8: Grants awarded by institution according to race and gender from the NRF. A major criticism of the rating system was the fact that it was no longer directly linked to NRF funding and that a rating was often not even among the eligibility criteria for funding when decisions were made for financial support for self-initiated research done by individuals. It was recommended that the NRF should relink rating and funding. NRF s Response/Commitment: Incentive funding is now fully rolled out. Millions of Rand 0 5 10 15 20 25 ARC AM CPUT CUT CGS CSIR DUT HartRAO HSRC ithemba LABS IMCT KZNM MUT Mintek NHLS NICD NMISA NM NZG NMMU NWU NECRA SAAMBR PEM RU SAMRC SAAO SAEON SAIAB SANBI SANSA SASRI SAWS SU TUT UCT UFH UJ UKZN UL UP UNISA UFS UWC UW UV UZ VUT WSU Black Grant-holders (Female) Black Grant-holders (Male) White Grant-holders (Female) White Grant-holders (Male) 22 23

2. Sharpened focus on excellence The review showed that rating is often used as a benchmarking tool by institutions due to the credibility of the system. Furthermore, there was a continuous call from the HEIs to broaden the criteria of assessment (e.g. research administration, teaching loads, innovation outputs, etc). The rating system, however, does not reveal anything about the rating of teams; an individual s ability to manage a team; or how to conduct or stimulate multi-purpose research. It was recommended that the NRF should not divert from the rating system s intended purpose - namely the recognition and reward of researchers for excellence. The rating system should remain an assessment of the quality of research performance only. NRF s Response/Commitment: The NRF recognises that institutional ranking and benchmarking are becoming standard practice around the world and considers ratings as a contribution to this practice in the NSI. The NRF s rating system would continuously strive to align the system with internationally recognised best practice and ensure fitness for purpose of the rating system. 3. A Kaizen approach This refers to the Japanese philosophy of continuous improvement through stakeholder engagement and is aligned to the credibility of the rating system. The review found that the process of evaluation and rating was considered to be mature, but complex and time consuming. NRF s Response/Commitment: The NRF expressed its commitment to the continued improvement and refinement of the system and its willingness to strive to streamline the process without compromising its quality. The NRF noted that it is widely accepted that the peer review process is not flawless but that it is globally still the most widely accepted system and the NRF s rating system therefore undertook to: Continue to use peer review as the foundation of the rating system. A definition of a peer has been introduced, the selection criteria for reviewers by members of the Specialist Committees had been reviewed and refined; the minimum number and quality requirements of reports needed for a credible decision had been formalised etc. Enhance the credibility of the system and decrease the possibility of a dominant member of the Specialist Committee influencing decisions. Independent assessment of each report by every member of the Specialist Committee have to be submitted electronically before the panel meeting. The consolidated sheet of all assessments is then used to focus attention on contentious assessments which the members of the Specialist Committee have to motivate. Ensure well prepared panel members. Due to the immediate availability of the reports on an online panel member facility, members of Specialist Committees have adequate time to prepare for the meeting. They also need to submit their assessments independently prior to the meeting. Incorporate quality assurance as a cornerstone of the system (The guidelines for assessment of the usability/quality of reports had been reviewed and refined). Facilitate/investigate the incorporation of additional information sources such as citation profiles/indices and other benchmarking measures to enhance the quality of the peer review process. The h-index information [Scopus and WoS (ISI)] is currently provided to members of the Specialist Committees but it is only used as a supplementary source of information to validate the opinions expressed in peer reviewer reports. Members of Specialist Committees have discretion on how to use it in the context of their discipline. The use of other indices (e.g. Google Scholar profiles) and bibliometric tools are, however, continuously examined by the Knowledge Management Corporate directorate of the NRF for potential added value to the rating process. Refine the process and the nature of constructive feedback to applicants as the review reports contained several references to problems stakeholders raised in terms of the feedback that applicants received as part of their outcome letters. The problems were mainly related to the content of, and the time when, the feedback became available. The feedback policy guidelines had been revised to include a reviewer profile in addition to narrative fields that: o Summarise the reason a particular rating category had been decided upon. o Provide developmental feedback from the reviewers. Applicants levels of satisfaction with the feedback are annually monitored by a survey and trends in terms of suggestions are fed into the rating policy structures to improve the feedback policy model. 4. Demystifying the rating process The reviewers found that misperceptions on how the process operates and how decisions are made were rife. NRF s Response/Commitment: In consultation with its stakeholders, the NRF has committed itself to address the criticisms/misperceptions regarding various aspects of the rating system, e.g. rating categories, simplification of processes, transparency, procedural aspects, etc. Examples of system changes to address this include: providing all rating information/documents on the NRF website; interactive workshops at all HEIs facilitated by senior members of the rating panels; the attendance of Panel Meetings by observers from Research Offices; the NRF rating categories survey (2012/13); changes to the descriptors of the rating categories (mainly the C category); the inclusion of a reviewer profile as part of the feedback; the revision of feedback policy guidelines (2011/12) as well as the appeals procedure (2012/13). 5. Buy-in from the Social Sciences, Humanities and Law researchers The acceptance levels of the rating system among the natural scientists are higher than among the social scientists. NRF s Response/Commitment: The NRF noted this finding and expressed its commitment to engage with these stakeholders to make the system accessible and credible for them. Examples of system changes to address this include constant refinement of guideline documents e.g. Key areas of research and types of research output and The meaning of considerable international recognition providing information specific to each Specialist Committee. 6. Multidisciplinary research and rating The perception was that the system poses challenges for the evaluation of generalists and those working in multidisciplinary and new interdisciplinary research areas. NRF s Response/Commitment: The NRF committed to continue exploring international best practices and measures to address this challenge. They particularly noted the role that the selection of appropriate peers for submitting reviews on researchers working in multidisciplinary areas could play. Examples of system changes to address this include in 2012 a policy decision was implemented that ensured intensive consultation with applicants when recommendations in terms of changes to their primary panel selection are made by the Specialist Committees; a proposal was accepted at the EEC/ Conveners/Assessors workshop in March 2014 to make it possible for applicants to indicate that their research specialisations are spanning more than one discipline/field to alert members of the Specialist Committees to the need for consultation when identifying reviewers. 7. Social costs: The review found that the rating system had a high social cost in terms of putting pressure on academics time. This was caused mainly by the fact that the system benchmark required six usable reports per applicant. The high decline rate in certain disciplines often resulted in the need to approach a large number of reviewers and, of those who were willing, some were already overburdened. NRF s Response/Commitment: An intervention that was proposed to alleviate this high social cost was to expand the role of the Convener to: Play a more pro-active coordinating role in obtaining good quality reports from peer reviewers; Perform quality control on the feedback that forms part of the outcome letters; and Investigate the role of bibliometrics (e.g. the h-index) in alleviating the challenge. Examples/interventions: The h-index proved to be a useful adjunct/validity tool to the reviewers reports in cases where the rating and the discipline benchmarked h-index were misaligned. It was worthwhile investigating reasons for this misalignment in those cases, e.g. poor quality/biased reports; unique publication track record, etc. It was crucial to prevent the overloading of reviewers to ensure good quality reports and NRF IT systems should be designed to prevent this. The availability of status reports and minutes through the panel member facility would be important tools to facilitate the process. A good common understanding of what constituted a peer and a good report would be very important. Who is a peer and who should ideally be nominated as a reviewer? A peer is a researcher or a person with a research background who has the requisite knowledge and experience as well as the ability to exercise objective fair judgment of the applicant and to provide an appropriate assessment of the applicant s research and research standing. The emphasis should be on the person s experience and ability to provide an appropriate assessment. 24 25

Appendix A Table 13: Percentage rated researchers at South African universities in 2008 compared to 2012 *As at 22 July 2013 **DoE 2012 Rated researchers* 2008 2012 Instruction/Research professionals** % Rated Rated researchers* Instruction/Research professionals** UCT 291 937 31,1% 412 1 055 39,1% SU 247 867 28,5% 313 939 33,3% UP 241 1 638 14,7% 334 1 281 26,1% UW 177 979 18,1% 243 1 044 23,3% UKZN 155 1 476 10,5% 209 1 470 14,2% NWU 102 927 11% 143 1 194 12% UNISA 78 1 313 6% 132 1 515 8,7% UFS 78 756 10,3% 105 846 12,4% UJ 71 853 8,3% 114 871 13,1% UWC 57 518 11,0% 94 536 17,5% NMMU 56 531 10,5% 61 579 10,5% RU 40 307 13,0% 65 319 20.4% TUT 29 793 3,7% 40 836 4,8% CPUT 14 696 2,0% 23 763 3% UZ 14 238 5,9% 11 270 4,1% DUT 13 578 2,2% 11 592 1,9% UFH 9 336 2,7% 19 291 6,5 UL 8 753 1,1% 8 806 1% CUT 5 221 2,3% 7 269 2,6% WSU 5 537 0,9% 7 607 1,2% VUT 3 280 1,1% 7 334 2,1% UV 1 283 0,4% 15 325 4,6% MUSA 3 MUT 2 193 1,0% SACSA 1 % Rated How long does it take to process an evaluation application? During 2012 it took an average of 8,25 months to process an application from the closing date (28 February) to the time the e-mail with the evaluation outcome was sent to the relevant research office. Social Sciences, Humanities and Law Table 14: The number of rated researchers in the social sciences, humanities and law per NRF Specialist Committee, per institution and per rating category during 2013 (as at 20 February 2014) Anthropology, Development Studies, Geography, Sociology and Social Work Type of institution Institution A B C P Y Total Government: departments/research institutions (South Africa) Gauteng City-Region Observatory (GCRO) 1 1 South African museums DITSONG Museums of South Africa 1 1 KwaZulu-Natal Museum 1 1 Durban University of Technology 1 1 Nelson Mandela Metropolitan University 1 1 North-West University 3 2 5 Rhodes University 1 1 2 4 Stellenbosch University 2 8 10 University of Cape Town 1 12 3 2 3 21 University of Johannesburg 1 5 6 University of KwaZulu-Natal 4 11 2 17 University of Pretoria 1 5 4 10 University of South Africa 6 1 7 University of the Free State 5 5 University of the Western Cape 2 8 1 11 University of the Witwatersrand 1 6 6 2 15 South African science councils Human Sciences Research Council 1 2 3 Grand Total 2 32 65 2 18 119 Communication, Media Studies, Library and Information Sciences Type of institution Institution B C L Y Total Cape Peninsula University of Technology 2 2 Monash South Africa 1 1 North-West University 1 1 Rhodes University 1 1 2 Stellenbosch University 1 1 Tshwane University of Technology 1 1 University of Cape Town 1 2 1 2 6 University of Fort Hare 1 1 University of KwaZulu-Natal 2 2 4 University of Pretoria 1 6 7 University of South Africa 4 4 University of the Western Cape 1 1 University of the Witwatersrand 1 1 University of Zululand 1 1 South African science councils Council for Scientific and Industrial Research 1 1 Other (institutions abroad, self-employed, retired) 1 1 2 Grand Total 6 22 2 6 36 26 27

Economics, Management, Administration and Accounting Type of institution Institution A B C L Y Total Cape Peninsula University of Technology 2 1 3 Nelson Mandela Metropolitan University 5 5 North-West University 7 1 4 12 Stellenbosch University 3 13 5 21 Tshwane University of Technology 6 6 University of Cape Town 1 3 12 1 4 21 University of Johannesburg 4 2 6 University of KwaZulu-Natal 2 8 10 University of Pretoria 4 15 5 24 University of South Africa 8 8 University of the Free State 3 1 4 University of the Witwatersrand 2 10 2 14 Council for Scientific and Industrial Research 1 1 South African science councils Human Sciences Research Council 2 2 Other (institutions abroad, self-employed, retired) 1 1 Other (South African - self-employed/retired) 1 1 Grand Total 1 14 96 2 26 139 Education Type of institution Institution A B C L Y Total Cape Peninsula University of Technology 3 1 4 Central University of Technology, Free State 2 2 Nelson Mandela Metropolitan University 3 3 North-West University 11 1 1 13 Rhodes University 4 4 Stellenbosch University 2 11 1 1 15 Tshwane University of Technology 1 1 University of Cape Town 3 12 15 University of Fort Hare 1 1 University of Johannesburg 2 6 8 University of KwaZulu-Natal 1 12 1 14 University of Pretoria 1 17 1 1 20 University of South Africa 1 8 9 University of the Free State 6 6 University of the Western Cape 4 3 1 1 9 University of the Witwatersrand 1 1 7 9 Other (institutions abroad, self-employed, retired) 1 1 Grand Total 1 15 107 5 6 134 Historical Studies Type of institution Institution A B C Y Total Monash South Africa 1 1 North-West University 3 3 Rhodes University 2 2 Stellenbosch University 2 2 4 University of Cape Town 5 9 1 15 University of Johannesburg 1 3 1 5 University of KwaZulu-Natal 4 4 University of Pretoria 1 1 2 4 University of South Africa 2 4 6 University of the Free State 3 3 University of the Western Cape 1 2 3 University of the Witwatersrand 1 3 1 1 6 Other (institutions abroad, self-employed, retired) 1 1 Grand Total 2 16 35 4 57 Law Type of institution Institution A B C P Y Total Government: departments/research institutions (South Africa) South African Reserve Bank 1 1 Nelson Mandela Metropolitan University 2 2 North-West University 2 6 1 9 Rhodes University 1 1 Stellenbosch University 2 6 1 1 10 University of Cape Town 8 7 4 19 University of Fort Hare 1 1 University of Johannesburg 3 4 2 9 University of KwaZulu-Natal 1 4 5 University of Pretoria 2 7 5 2 16 University of South Africa 2 25 5 32 University of the Free State 2 2 4 University of the Western Cape 4 6 1 11 University of the Witwatersrand 2 6 2 10 University of Zululand 1 1 Walter Sisulu University 1 1 Other (institutions abroad, self-employed, retired) 1 1 2 Other (South African - self-employed/retired) 1 1 Grand Total 4 39 73 1 18 135 28 29

Literary Studies, Languages and Linguistics Type of institution Institution A B C L Y Total Nelson Mandela Metropolitan University 3 3 North-West University 2 14 1 17 Rhodes University 1 3 4 Stellenbosch University 4 7 3 14 University of Cape Town 2 5 6 13 University of Fort Hare 1 1 University of Johannesburg 2 6 8 South African higher education institution University of KwaZulu-Natal 1 10 1 12 University of Pretoria 3 11 14 University of South Africa 1 1 9 1 12 University of the Free State 5 1 6 University of the Western Cape 2 8 10 University of the Witwatersrand 4 6 4 14 University of Venda 2 2 University of Zululand 2 2 Other (institutions abroad, self-employed, retired) 1 1 2 Other (South African - self-employed/retired) 1 1 Grand Total 4 24 93 1 13 135 Performing and Creative Arts, and Design Type of institution Institution B C Y Total North-West University 1 1 Rhodes University 2 2 Stellenbosch University 1 4 1 6 Tshwane University of Technology 1 1 University of Cape Town 3 8 11 University of Johannesburg 1 2 3 University of KwaZulu-Natal 1 1 University of Pretoria 6 6 University of South Africa 4 4 University of the Free State 2 2 University of the Witwatersrand 2 2 2 6 University of Zululand 1 1 Grand Total 8 31 5 44 Political Sciences and Philosophy Type of institution Institution A B C P Y Total Central University of Technology, Free State 1 1 Nelson Mandela Metropolitan University 2 1 3 North-West University 1 1 Rhodes University 2 2 Stellenbosch University 4 8 1 13 University of Cape Town 1 2 2 5 University of Fort Hare 1 1 University of Johannesburg 1 3 1 1 6 University of KwaZulu-Natal 2 1 3 University of Pretoria 6 1 7 University of South Africa 2 2 University of the Free State 3 3 University of the Western Cape 1 1 2 University of the Witwatersrand 3 2 5 University of Zululand 2 2 South African science councils Africa Institute of South Africa (AISA) 1 1 Other (institutions abroad, self-employed, retired) 2 2 Grand Total 2 16 35 2 4 59 Psychology Type of institution Institution A B C Y Total Nelson Mandela Metropolitan University 1 2 3 North-West University 6 6 Rhodes University 1 3 4 Stellenbosch University 2 5 7 University of Cape Town 2 2 2 1 7 University of Johannesburg 3 1 4 University of KwaZulu-Natal 1 4 5 University of Limpopo 1 1 University of Pretoria 6 1 7 University of South Africa 1 4 5 University of the Free State 3 1 4 University of the Western Cape 2 1 3 University of the Witwatersrand 5 1 6 University of Zululand 1 1 South African science councils Human Sciences Research Council 1 1 2 South African Medical Research Council (MRC) 1 1 Other (institutions abroad, self-employed, retired) 1 1 Grand Total 2 9 50 6 67 30 31

Religious Studies and Theology Type of institution Institution A B C Y Total Nelson Mandela Metropolitan University 1 1 North-West University 1 3 2 6 St Augustine College of South Africa 1 1 Stellenbosch University 1 3 11 15 University of Cape Town 2 1 2 1 6 University of Fort Hare 1 1 University of Johannesburg 1 1 University of KwaZulu-Natal 2 5 7 University of Pretoria 1 1 13 1 16 University of South Africa 1 9 1 11 University of the Free State 1 8 9 University of the Western Cape 1 1 2 Other (institutions abroad, self-employed, retired) World Council of Churches 1 1 Grand Total 4 12 56 5 77 It is a privilege for me to be rated as an NRF scientist. This is an admirable system which promotes South African researchers and encourages them to compete on a global scale. I would urge all young researchers to look beyond the boundaries keep publishing and think big! Prof William Bishai Howard Hughes Medical Institute University of KwaZulu-Natal Natural Sciences and Engineering Table 15: The number of rated researchers in the natural sciences, engineering and technology per NRF Specialist Committee, per institution and per rating category during 2013 (as at 20 February 2014) Animal and Veterinary Sciences Type of institution Institution A B C L P Y Total Government: departments/research institutions (South Africa) Department of Environmental Affairs Branch: Oceans & Coasts 1 1 National Institute for Communicable Diseases (NICD) 1 1 Oceanographic Research Institute (SAAMBR) 2 2 South African Sugarcane Research Institute 2 2 Industry (South Africa) Panthera 1 1 South African museums South African National Facilities South African science councils Other (institutions abroad, self-employed, retired) Albany Museum 2 2 Bayworld Centre for Research & Education 1 1 2 DITSONG Museums of South Africa 1 1 East London Museum 1 1 Iziko Museums of South Africa 2 2 KwaZulu-Natal Museum 1 2 3 National Museum 3 3 National Zoological Gardens of South Africa (NZG) 1 1 2 South African Institute for Aquatic Biodiversity (SAIAB) 1 3 2 6 Central University of Technology, Free State 1 1 Durban University of Technology 1 1 Nelson Mandela Metropolitan University 1 4 1 6 North-West University 1 4 1 6 Rhodes University 1 4 5 1 11 Stellenbosch University 2 3 10 1 4 20 University of Cape Town 3 10 7 1 21 University of Fort Hare 1 1 1 3 University of Johannesburg 1 5 1 7 University of KwaZulu-Natal 1 1 10 2 14 University of Limpopo 3 3 University of Pretoria 1 16 34 15 66 University of South Africa 1 1 2 University of the Free State 8 2 10 University of the Western Cape 1 3 4 University of the Witwatersrand 2 1 8 2 13 University of Venda 1 1 1 3 University of Zululand 1 1 Agricultural Research Council (South Africa) 1 13 2 16 Council for Scientific and Industrial Research 1 1 2 South African National Biodiversity Institute 3 3 1 1 3 1 6 Other (South African - self-employed/retired) 1 1 Grand Total 12 49 146 1 1 40 249 32 33

Biochemistry, Molecular and Cell Biology Type of institution Institution A B C Y Total South African science councils Cape Peninsula University of Technology 1 1 1 3 Nelson Mandela Metropolitan University 2 2 North-West University 1 1 2 Rhodes University 2 1 3 Stellenbosch University 1 4 6 2 13 University of Cape Town 7 8 6 21 University of Johannesburg 1 1 University of KwaZulu-Natal 3 1 4 University of Pretoria 4 6 2 12 University of South Africa 1 1 University of the Free State 3 3 University of the Western Cape 5 5 University of the Witwatersrand 1 4 4 9 University of Zululand 1 1 Agricultural Research Council (South Africa) 1 1 2 4 South African Medical Research Council (MRC) 1 1 Other (institutions abroad, self-employed, retired) 1 1 2 Grand Total 1 21 44 21 87 Getting an NRF rating was an important milestone in my career. Before my NRF rating I was just another researcher with a PhD. Being a small, black female researcher it is very easy for some people in academia not to take me seriously but as soon as I mention my B2 rating many of them sit up and pay attention because it communicates to them the quality and standing of researcher I am. I do not have to argue with people about my standing as a scholar because my NRF rating does it for me. It may sound not important but in this business and in this country people will make judgements about you on the basis of what you look like. I know that I do not look like an academic and so my rating says I do not have to look like an academic to be a good scholar and I like that. Chemistry Type of institution Institution A B C L Y Total Government: departments/research institutions (South Africa) Nuclear Energy Corporation of South Africa (NECSA) 2 1 3 Cape Peninsula University of Technology 1 1 2 Durban University of Technology 2 2 Nelson Mandela Metropolitan University 4 1 5 North-West University 6 1 7 Rhodes University 1 4 1 6 Stellenbosch University 2 5 7 4 18 Tshwane University of Technology 1 3 1 5 University of Cape Town 9 3 3 15 University of Fort Hare 1 1 University of Johannesburg 7 5 12 University of KwaZulu-Natal 1 1 9 3 14 University of Pretoria 3 2 2 7 University of South Africa 1 3 1 5 University of the Free State 5 1 2 8 University of the Western Cape 2 4 1 7 University of the Witwatersrand 4 7 4 15 University of Zululand 1 1 Vaal University of Technology 1 1 Walter Sisulu University 1 1 2 Council for Scientific and Industrial Research 2 1 1 2 6 South African science councils Mintek 1 1 South African Medical Research Council (MRC) 1 1 Other (institutions abroad, self-employed, retired) 1 1 1 3 Other (South African - self-employed/retired) 2 2 Grand Total 4 30 75 5 35 149 Prof Mamokgethi Phakeng, PhD (UW) MASSAf Vice Principal: Research & Innovation, University of South Africa 34 35

Earth Sciences Type of institution Institution A B C P Y Total Government: departments/research institutions (South Africa) South African museums South African national facilities South African science councils Department of Environmental Affairs 1 1 Nuclear Energy Corporation of South Africa (NECSA) 1 1 South African Weather Service 1 1 Bayworld Centre for Research & Education 1 1 Iziko Museums of South Africa 1 1 2 KwaZulu-Natal Museum 1 1 National Museum 1 1 2 Hartebeesthoek Radio Astronomy Observatory 1 1 ithemba Laboratory for Accelerator Based Sciences 1 1 South African Environmental Observation Network (SAEON) 2 2 Mangosuthu University of Technology 1 1 Nelson Mandela Metropolitan University 1 1 2 4 North-West University 1 6 1 8 Rhodes University 1 6 7 Stellenbosch University 5 3 8 University of Cape Town 14 10 2 4 30 University of Fort Hare 3 1 4 University of Johannesburg 1 3 8 1 13 University of KwaZulu-Natal 2 5 2 9 University of Pretoria 2 8 1 11 University of the Free State 6 6 University of the Western Cape 1 1 University of the Witwatersrand 1 14 15 1 3 34 University of Venda 1 1 Agricultural Research Council (South Africa) 1 1 Council for Geoscience 2 2 Council for Scientific and Industrial Research 2 7 5 14 Other (institutions abroad, self-employed, retired) 1 1 2 Other (South African - self-employed/retired) 1 1 Grand Total 3 47 93 3 24 170 Engineering Type of institution Institution A B C Y Total Government: departments/research institutions (South Africa) National Biotechnology Development Agency 1 1 Eskom 1 1 Industry (South Africa) Exxaro Coal (Pty) Ltd 1 1 Randgold Resources Ltd 1 1 South African national facilities ithemba Laboratory for Accelerator Based Sciences 1 1 Cape Peninsula University of Technology 2 4 6 Durban University of Technology 2 2 Nelson Mandela Metropolitan University 6 6 North-West University 9 3 12 Stellenbosch University 1 9 17 10 37 Tshwane University of Technology 11 2 13 University of Cape Town 2 10 19 12 43 University of Johannesburg 1 1 4 5 11 University of KwaZulu-Natal 2 10 1 13 University of Pretoria 2 8 18 4 32 University of South Africa 1 1 1 5 8 University of the Witwatersrand 1 5 14 3 23 Vaal University of Technology 3 2 5 Agricultural Research Council (South Africa) 1 1 South African science councils Council for Scientific and Industrial Research 1 2 3 Mintek 1 1 Other (institutions abroad, self-employed, retired) 2 3 2 7 Other (South African - self-employed/retired) 1 1 Grand Total 8 40 125 56 229 Initially I had doubts about the impartiality of the ratings system. Having been an observer to a panel meeting, however, and having heard the discussion on reviewer reports, I am persuaded both that the reviewer system is the fairest way of evaluating the worth of rating applicants and that the panel review process is so thorough as to minimize bias in the assignment of ratings. Mr Michael Cosser Head: Capacity Enhancement, Human Sciences Research Council 36 37

Health Sciences Type of institution Institution A B C L P Y Total Government: departments/research institutions (South Africa) National Health Laboratory Service (NHLS) (including National Institute for Communicable Diseases (NICD)) 2 4 2 8 National Institute for Communicable Diseases (NICD) 2 3 1 6 Industry (South Africa) Roche SA 1 1 South African national facilities ithemba Laboratory for Accelerator Based Sciences 1 1 South African science councils Cape Peninsula University of Technology 3 3 Central University of Technology, Free State 1 1 Centre for AIDS Programme of Research in South Africa (CAPRISA) 1 1 Durban University of Technology 1 1 Monash South Africa 1 1 Nelson Mandela Metropolitan University 2 2 North-West University 1 2 25 6 34 Rhodes University 1 1 2 Stellenbosch University 2 15 24 8 49 Tshwane University of Technology 2 2 University of Cape Town 9 28 37 9 83 University of Fort Hare 1 1 University of Johannesburg 1 1 4 3 9 University of KwaZulu-Natal 1 5 23 1 9 39 University of Limpopo 2 2 4 University of Pretoria 1 1 22 1 3 28 University of South Africa 1 1 University of the Free State 1 7 1 9 University of the Western Cape 2 11 2 15 University of the Witwatersrand 5 13 18 2 7 45 University of Venda 5 1 6 Vaal University of Technology 1 1 Walter Sisulu University 3 3 Westville Hospital 1 1 Council for Scientific and Industrial Research 1 1 2 Human Sciences Research Council 1 1 3 1 6 South African Medical Research Council (MRC) 1 3 3 7 Other (institutions abroad, self-employed, retired) 1 3 1 5 Other (South African - self-employed/retired) 1 1 2 Grand Total 22 76 213 4 1 63 379 Information Technology Type of institution Institution B C L Y Total Nelson Mandela Metropolitan University 1 6 2 9 North-West University 2 1 3 Rhodes University 3 2 5 Stellenbosch University 2 5 7 Tshwane University of Technology 1 1 University of Cape Town 2 11 4 17 University of Fort Hare 1 1 2 University of Johannesburg 1 2 1 4 University of KwaZulu-Natal 1 1 2 University of Pretoria 2 11 1 1 15 University of South Africa 8 1 9 University of the Western Cape 1 1 University of the Witwatersrand 2 3 5 South African science councils Council for Scientific and Industrial Research 2 1 1 4 Other (institutions abroad, self-employed, retired) 1 1 2 Grand Total 10 55 1 20 86 Mathematical Sciences Type of institution Institution A B C P Y Total Cape Peninsula University of Technology 1 1 Nelson Mandela Metropolitan University 3 3 North-West University 4 3 2 9 Rhodes University 2 2 Stellenbosch University 1 2 9 1 3 16 Tshwane University of Technology 3 3 University of Cape Town 4 10 9 4 27 University of Fort Hare 2 2 University of Johannesburg 1 2 4 2 9 University of KwaZulu-Natal 4 5 2 11 University of Pretoria 5 12 3 20 University of South Africa 2 5 1 8 University of the Free State 2 1 3 University of the Western Cape 1 3 4 University of the Witwatersrand 1 7 10 6 24 University of Venda 1 1 2 University of Zululand 1 1 Walter Sisulu University 1 1 Council of Scientific and Industrial Research 1 1 South African science councils Human Sciences Research Council 1 1 South African National Biodiversity Institute 1 1 Other (institutions abroad, self-employed, retired) 1 3 4 Grand Total 7 42 76 1 27 153 38 39

Microbiology and Plant Pathology Type of institution Institution A B C P Y Total Central University of Technology, Free State 1 1 Durban University of Technology 5 5 Nelson Mandela Metropolitan University 1 1 North-West University 1 1 2 Stellenbosch University 7 7 4 18 Tshwane University of Technology 4 4 University of Cape Town 1 2 1 4 University of Fort Hare 1 1 University of Johannesburg 1 1 2 University of KwaZulu-Natal 5 2 7 University of Pretoria 3 3 9 1 6 22 University of South Africa 1 1 University of the Free State 3 11 1 15 University of the Witwatersrand 2 4 6 University of Venda 1 1 Agricultural Research Council (South Africa) 4 1 5 South African science councils South African Medical Research Council (MRC) 1 1 Other (institutions abroad, self-employed, retired) 1 1 Grand Total 4 15 59 1 18 97 Physics Type of institution Institution A B C L P Y Total Government: departments/research institutions (South Africa) South African national facilities Council on Higher Education 1 1 Nuclear Energy Corporation of South Africa (NECSA) 2 1 3 South African National Space Agency (SANSA) 2 2 4 Square Kilometre Array 1 1 Hartebeesthoek Radio Astronomy Observatory 1 1 ithemba Laboratory for Accelerator Based Sciences 5 5 3 13 South African Astronomical Observatory (SAAO) 1 5 8 3 17 Cape Peninsula University of Technology 1 1 Nelson Mandela Metropolitan University 7 7 North-West University 2 3 1 1 7 Rhodes University 2 1 3 Stellenbosch University 8 7 2 17 University of Cape Town 3 9 6 2 7 27 University of Fort Hare 1 1 University of Johannesburg 4 3 1 8 University of KwaZulu-Natal 3 10 5 18 University of Limpopo 1 1 University of Pretoria 3 6 1 1 11 University of South Africa 1 4 2 7 University of the Free State 1 5 2 8 University of the Western Cape 1 5 5 1 12 University of the Witwatersrand 2 7 8 1 3 21 University of Zululand 1 1 South African science councils Council for Scientific and Industrial Research 1 1 2 Other (institutions abroad, self-employed, retired) 1 3 3 7 Other (South African - self-employed/retired) 1 1 2 Grand Total 9 60 90 1 4 37 201 My rating confirmed that my work on gender and race in organisations is making a significant contribution to management and organisation studies. For an academic, getting feedback from one s scholarly community is critical. Prof Stella M Nkomo Deputy Dean for Research and Post-Graduate Studies, Faculty of Economic and Management Sciences, University of Pretoria 40 41