Options and limitations in measuring the impact of research grants evidence from Denmark and Norway

Transcription

1 Research Evaluation 24 (2015) pp Advance Access published on 26 May 2015 doi: /reseval/rvv012 Options and limitations in measuring the impact of research grants evidence from Denmark and Norway Liv Langfeldt 1, *, Carter Walter Bloch 2 and Gunnar Sivertsen 1 1 Nordic Institute for Studies in Innovation, Research and Education (NIFU), P.O. Box 5183 Majorstuen, N-0302 Oslo, Norway and 2 Danish Centre for Studies in Research and Research Policy, Department of Political Science, Aarhus University, Bartholins Alle 7, DK-8000 Aarhus C, Denmark (Future address NIFU address from September 21, 2015: PO Box 2815 Tøyen, NO-0608 Oslo, Norway) *Corresponding author. liv.langfeldt@nifu.no Competitive grant schemes are set up with the intention of improving research performance. It may, however, be difficult to find evidence of the intervention impact of research grants for ex post evaluations of grant schemes. Based on data on applicants to Danish and Norwegian open mode grant schemes research projects as well as post doc fellowships this article applies difference in difference analysis to study to what extent research grants are likely to affect the publication and citation rates of the principle investigators (PIs). The results show higher increases in the number of publications for grant recipients than for rejected applicants, while increases in mean normalized citation rates were not significantly higher for the successful applicants. In other words, the grants seem to have increased productivity, e.g. by helping PIs to add staff to their research teams, but not to have influenced the importance of the research as measured by average citations. However, along with increases in the number of publications also came a greater increase in the number of highly cited papers for grant recipients than for rejected applicants. In sum, the analyses indicate that the measurement of grant impact is sensitive to how research performance is defined and tested using bibliometric indicators. Furthermore, the applicants complex landscape of multiple projects and grants makes it difficult to isolate the output of a single grant. Hence, using bibliometrics to measure the impact of smaller grant schemes and smaller grants may often yield inconclusive results. Keywords: research performance; competitive funding; citation impact; excellence; funding schemes. 1. Introduction R&D funding schemes aiming at enhancing conditions for research and improving research output are major instruments in public research policy. A key reason for allocating part of national research funding as project grants, rather than solely relying on block funding to the research institutions, and a raison d eˆtre for research councils and other bodies funding fundamental/academic research, is to target the most promising research projects and support the best researchers. The underlying idea is that competitive grant schemes can increase research performance and optimize funding impact. In later years, impact evaluations of such funding schemes have become more common, as funding authorities would like evidence that the grants have in fact yielded intended effects on research performance and scientific quality. 1 Evaluating impact is however generally a complex and demanding endeavour. Even when we put aside the complexities of investigating the (broader) impacts of research (Donovan 2011), and concentrate on how research grants impact first-order outputs such as publications, adequate tests of ß The Author Published by Oxford University Press. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License ( which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.com

2 Options and limitations in measuring the impact of research grants. 257 the intervention impact of grant schemes are hard to design and perform. In addition to good data on the relevant funding schemes and grant awardees, we need a matching control group, to decide on adequate indicators and time-slots for measuring impact/change, and to be able to attribute the research output to specific grants (Vedung 1997; Nedeva et al. 2012). Many studies have examined the correlation between past performance and the selection/peer review of grant proposals, and found that the applicants tend to have better track records (citation scores) than non-applicants, and that awarded applicants on average have better track records than the rejected (but not always better than the best performing rejected applicants, van Leeuwen and Moed 2012; Bornmann et al. 2010; Neufeld and van Ins 2011). However, in this article we focus on the impact of the grants on the subsequent performance of the researchers. Impacts of R&D funding schemes have proved hard to measure based on bibliometric data, as has also been demonstrated in numerous studies. Some impact on productivity (number of publications) has been found, but little impact on citation rates (Sandstro m 2009; Rigby 2011; Benavente 2012). Research projects are complex activities and often lack one-to-one relations between principle investigators (PIs), grants, research activities/projects and publications. Moreover, citations are not an unproblematic proxy of research performance. Difficulties in identifying an impact of funding schemes can thus be due to complex funding structures e.g. rejected grant applications may be funded from other sources, projects may profit from multiple grants and PIs may be involved in multiple, parallel (and interrelated) projects, thus limiting the possibilities of attributing particular outcomes to a specific grant or rejection, or to comparing outputs of funded and rejected projects. Negative results may also be due to measurement problems, such as limitations in available data, highly skewed distribution of citations, and varying time lags between grant application and publication. Some notable findings in previous studies include a study matching grants and publication records of Swedish researchers which found that productivity measures (number of papers) were related to the total sum of grants (all sources for a 6-year period), but that citation scores were essentially unaffected by research grants (Sandstro m 2009). Likewise, a study comparing outcomes for successful and unsuccessful applicants for Chilean research funding found a significant impact on number of publications, but no impact on citations (Benavente 2012). Another study comparing funded and rejected applicants for large 6-year grants to young investigators in Sweden found no impact on the number of publications, but positive effects for the funded PIs were found in terms of a higher proportion of international co-authorship, as well as securing further funding for their research groups (Melin and Danell 2006). Moreover, there are some indications that grants increase the probability of passing publication and citation thresholds, but not significantly impact the total number of citations: In a study comparing outcomes for successful and unsuccessful applicants for NIH postdoc grants (comparing those with peer review scores just above and just below the cut-off), Jacob and Lefgren (2011) found that grant receipt increased the number of publications (by about 20% in the 5-year period following the grant award), and significantly increased the probability of crossing a citation threshold (200 citations in the 10-year period after the grant), but had no significant effect on the total number of citations. They argued that this is consistent with grants improving outcomes at lower levels, but have little effect on high achievers (Jacob and Lefgren 2011: 870). Another approach to study impacts of research grants is to match input and output factors by analysing publications attributed to funding sources. Whereas such studies are more promising in terms of studying the impacts of specific grants and comparing impacts of different funding schemes, data limitations are more challenging and results are ambiguous. Notably, a study of funding body acknowledgements in published papers in two disciplines (physics and biology) found that a large proportion of publications contained multiple funding acknowledgements (Rigby 2011). Hence, evaluations based on methods assuming that papers owe their existence to a single funding source may be misconceived. Moreover, when studying citation scores by number of funding body acknowledgements, no clear relationship between number of funders and research impact was found (Rigby 2011). Still, there are indications that publications with an accredited funding source have higher impact than publications which do not accredit grants. A study by Zhao (2010) comparing the citation scores of publications (within library and information sciences) with accredited funding sources and publications without accredited funding, showed higher citation scores for papers accrediting funding. Other studies found that highly cited papers within malaria research acknowledged multiple funding sources more often than papers with lower citations scores within this field (MacLean et al. 1998), and that multiple funding sources of papers within biomedical research were correlated with publication in high-impact journals (Lewison and Dawson 1998). Another study based on lists of publications accrediting grants from the Human Frontier Science Program (HFSP), found significant citation impact compared to the world average, and also higher citation impact than other publications by the co-authors (HFSP 2009). Note that these studies did not address impact/change in researchers performance, but merely the difference between publications resulting/not resulting from (specific) funding schemes. With these mixed findings in previous studies as a point of departure, the focus in this article is data from some

3 258. L. Langfeldt et al. recent evaluations of open mode funding schemes in two Scandinavian countries, Norway and Denmark. FRIPRO is a key funding instrument of the Research Council of Norway and the only Norwegian scheme allocating project grants for independent basic research in all fields of research. It is based on open calls and national competition and has a high standing and popularity in the Norwegian research community, and also a low success rate (11 per cent of applications were funded in 2010). Fostering research of high scientific quality, promoting research recruitment and developing basic theory and methods are among the aims of the scheme. University researchers are the main target group, receiving 86 per cent of the FRIPRO funding. In the studied period (grant decisions in ), 413 research projects and PhD and postdocs fellowships were funded, average total projects size was million Euros 2 and the normal project period 3 or 4 years. The programmes of the Danish Council for Independent Research to fund research projects and postdoc fellowships are very similar to the Norwegian FRIPRO programme both in terms of aims and funding criteria. Research project grants are the most important instrument for the Danish Council, accounting for around 60% of Council funding. In the period , around 2,600 project grants were awarded to a total sum of approximately 480 million Euros. In the same period, around 880 postdoc fellowships were awarded. The Danish research project grants differ somewhat from the Norwegian in terms of size and success rate. For example, the average size of project grants in 2007 and 2008 was around 0.25 million Euros and the success rate around 23%. The success rate for project grant has however fallen significantly in more recent years and is now around the same level as in Norway. Evaluations of schemes for projects grants and postdoc fellowships from the Danish Council for Independent Research and of the similar Norwegian FRIPRO scheme failed to demonstrate any substantial impact of grants on PI s publications or citation rates, but still found some notable differences between successful and rejected applicants based on their survey replies (Faber et al. 2010; Bloch et al. 2011; Langfeldt et al. 2012). In both countries the survey data indicated important impact of the funding schemes, and bibliometric analysis that successful applicants had significantly higher publication and citation rates than the rejected applicants; however, the funding or rejection of applications did not seem to have a measurable impact on the citation or publication rates of the PIs the before after changes in publication and citation rates did not differ significantly between the funded and rejected applicants. In this article we study how and to what extent research grants are likely to affect publication and citation rates, based on further analysis of the data from these Danish and Norwegian evaluations. Using the data sets provided for the evaluations, we test a variety of impact indicators and examine the possible explanations of the discrepancy between the survey data and the bibliometric analyses. Notably, when using more refined methods, more positive impacts on PI s publication and citation rates appear than in the evaluations of these schemes. In this way we identify and discuss numerous factors which complicate the identification and measurement of impact of funding schemes. The analyses are conducted using field normalized indicators of citation impact and through comparison of performance before and after grant application. Our data are thus more suited to this type of analysis than data that are only measured after application or are not field normalized (such as the number of citations). These data allows us to calculate difference in difference measures, i.e. whether publication activity or citation impact has increased more over the grant period for successful applicants than rejected applicants. This is important given that ex ante performance is typically greater for grant recipients. Hence, a finding that grant recipients also perform better ex post does not necessarily indicate an effect of grant funding. While there are some differences for Denmark and Norway, the general result of the analysis is that we find that increases in productivity are higher for grant recipients, while increases in citation impact are not significantly greater. However, it should also be noted that these increases in productivity also include greater productivity of highly cited papers. The next section describes both the Norwegian and Danish evaluations and the bibliometric data used in the analysis. In this section we also outline the methods used in the analysis. Section 3 presents the results of the analysis, while Sections 4 and 5 include a discussion of the results and their implications, followed by concluding remarks. 2. Data and methods 2.1 The data set and methods for the study of the Norwegian FRIPRO grants In the Norwegian part of the study, the publication and citation rates of PIs who were granted and/or rejected in the above-mentioned FRIPRO scheme are analysed before and after their evaluations in Both grant reception and rejection for the same PI are possible in the period, since it is the PIs of the FRIPRO applications and their scientific articles, not their projects, that constitute the main unit of study in the bibliometric analysis. The PIs are thereby divided in two partly overlapping groups, where grant recipients have been awarded at least one grant during the period and rejected applicants have not received any FRIPRO grant during the period. Their articles are divided into two 5-year periods in order to detect possible changes after funding or rejection. The

4 Options and limitations in measuring the impact of research grants. 259 Table 1. Descriptive statistics for the Norwegian samples of PIs for research project grants and postdocs PIs research projects Postdocs Granted Rejected Granted Rejected Number Sex (% women) Application size (million Euros) a Natural sciences (%) Medical sciences (%) Technical sciences (%) Social sciences (%) Humanities (%) Sources: Bibliometric data: National Citation Report for Norway (NCR)/ Thomson Reuters. Application data: The Research council of Norway. a Sum of applied amounts. Exchange rate (from Norwegian kroner, 6 March 2015). first period includes the application year, while the last period commences the year after the start-up year. The publication and citation rates are studied with data from the National Citation Report for Norway, a bibliographic database from Thomson Reuters representing all journal articles that have been indexed in Web of Science since 1981 with at least one address indicating an institutional affiliation in Norway. In the 10-year period selected for analysis here, , the database consists of 72,263 articles. Of these, 22,030 articles (30.4%) have been matched (using author names) with PIs included in this study and selected for further analysis. The Web of Science represents the scientific production in the natural sciences and medicine more comprehensively than in other areas. Compared to complete data for scientific publications recorded at the institutional level in Norway (the Cristin database), the following shares of peer-reviewed original research publications are covered by the Web of Science (Sivertsen and Larsen 2012): Natural sciences: 80%; Health sciences: 76%; Engineering sciences: 61%; Social sciences: 20%; Humanities: 11%. Not surprisingly, a number of PIs in the last mentioned fields could not be identified with publications in the database. In addition, for those that could be identified, we found a limited number of publications even at the level of disciplines in several instances. In order to derive all the indicators for this study, PIs need to have at least one publication in Web of Science in the relevant period. This has resulted in the removal of a number of PIs in the humanities and in several subfields of the social sciences. Furthermore, we have excluded a small number of very extreme values (PIs with a mean normalized citation score (MNCS) that is more than 10 times the world average). Totals of 306 granted and 1,036 rejected PIs are included in the study. Some of these PIs may be co-authors of each others articles. Thereby, the differences between the two groups may diminish. However, we found that only 7.6% of the articles represent an overlap between those that can be attributed to the granted PIs and those that can be attributed to the rejected PIs. Table 1 shows the included PIs split by gender and research area, and the total size of the applications. 2.2 The data set and methods for the study of Danish project grants and postdoc grants The Danish bibliometric data set stems from two separate, but similar, evaluations of funding instruments of the Danish Council for Independent Research. The first concerns an evaluation of the funding of postdocs and specialized programs to support younger researchers and female researchers over the period (Faber et al. 2010). The largest group examined in this study was postdocs, where in all 2,402 had applied for a postdoc grant over and of these 802 were awarded a grant. The second evaluation examines the funding of research projects over the same period, In all, 4,077 researchers submitted an application for a research project grant over the period, with 1,602 receiving a grant. As with the Norwegian evaluation, both Danish evaluations are broad reaching, drawing on a number of approaches including a questionnaire survey, interviews, analysis of register-based career and income data, and bibliometric analysis. For both types of grants, and in particular for research projects, many have received more than one grant from the Danish Council during the period. For example, the 1,602 grant recipients have been awarded in total 2,604 research project grants over the period. For multiple grant recipients, the evaluation focuses on the first grant received in the period. Rejected applicants are included in the analyses for both evaluations, and defined as not having received any grants from the Danish Council over the period. Bibliometric analyses were conducted in a similar fashion in the two Danish evaluations. In each case a small subset of grant awardees and rejected applicants were selected due to resource constraints. Grant recipients and rejected applicants were selected according to a matching procedure, where the two groups were matched according to field, application year, gender, age, academic position, and years since PhD degree. Applicants were chosen from a 3-year period (in terms of the year of application) in order to allow for the construction of 4-year windows both before the application and afterwards (starting 2 years after application), for which data on journal publications and citations was collected. The period was chosen for postdocs and for research projects. For postdocs, the bibliometric data covers 206 applicants (104 grant awardees and 102

5 260. L. Langfeldt et al. Table 2. Descriptive statistics for the Danish matched samples of PIs for research project grants and postdocs PIs research projects Postdocs Granted Rejected Granted Rejected Number Sex (% women) Application size a a (1,000 euros) Natural sciences (%) Medical sciences (%) Technical sciences (%) Social sciences (%) Humanities (%) Sources: Bibliometric data: Scopus for PIs-research projects and Thomson Reuters Web of Science for postdocs. Application data: Danish Agency for Science, Technology and Innovation. a Data not available. All postdoc grants awarded for 2-year period. rejected applicants) evenly divided across three main fields: natural sciences, medical sciences, and engineering and technical sciences. The bibliometric sample for research projects attempted to cover all five main fields (including social sciences and humanities), but coverage in the Scopus database (in particular concerning citations) was not adequate in all cases for social sciences and humanities. Out of an initial sample of 208 observations across all five main fields, the resulting sample consists of 174 PIs, 88 grant awardees, and 86 rejected applicants. Table 2 shows basic statistics for the matched samples used in the analysis below. The data source for postdocs is Thomsen Reuters Web of Science database while the source for research projects is the Scopus database. Publication and citation data were collected for each researcher using a broad search method followed by manual validation for errors. Field-based indicators were constructed based on the full database using pre-defined subject categories Options and limitations in comparing results: Similarities and differences between the Norwegian and Danish data and analysis There are a number of challenges in attempting to estimate the effects of grants on research performance. Two of the most important are constructing a counterfactual case for comparison and isolating the outcomes of grant projects from other work by the researchers in question. There are a number of similarities in the Danish and Norwegian approaches, but also some important differences. Concerning the issue of a counterfactual, given that we are unable to observe what would have happened to PIs had they not received the grants, we need to find a control group for comparison that is as similar as possible with grantees. Rejected applicants are typically the best possible choice to form a control group, and have also been used in a number of the other studies mentioned above (e.g. Melin and Danell 2006; Jacob and Lefgren 2011; Benavente 2012; Van Leeuwen and Moed 2012). Both the Danish and Norwegian studies have used rejected applicants as a control group, but have followed different approaches. The Norwegian study utilizes a broad approach that includes all grantees and rejected applicants for the years The key advantages of this approach are both that it covers the full population of grantees as opposed to a subsample, and that the large sample improves the precision of estimates. In contrast, the Danish study relies on a small sample of grantees and rejected applicants that were chosen through a matching procedure. The aim of the matching procedure is to isolate effects of the grant by ensuring that selected grant awardees and rejected applicants are comparable in all other respects than the receipt of the grant, so that eventual differences in publication activity are not due to differences in academic position, research experience, etc. The Danish samples for both postdocs and PIs for research projects consist of pairs of grantees and rejected applicants with the best match on all criteria. Given that the bibliometric data was collected after sample selection, it was however not possible to include prior publications and citations as matching criteria. Hence, for both the Danish and Norwegian samples, there is a need to control for differences in initial conditions. In both cases, we attempt to address this through the use of Differences-in-Differences (DiD), focusing on changes over the analysis period as opposed to absolute levels. DiD enables the estimation of treatment effects while eliminating individual time-invariant effects and time effects; however, the method neither controls for unobserved temporary individual-specific components nor for differential impacts of macro-effects across the groups of comparison. The Danish study thus relies on a combination of matching and DiD approaches, providing scope for unobserved determinants of participation as long as it lies on separable individual and/or time-specific components of the error term (Blundell & Costa Dias 2002). The combination of the two methods is argued to provide more reliable results, as argued by e.g. Heckman et al. (1998), Dehejia & Wahba (1999, 2002), and Smith and Todd (2005). The second issue described above concerned the isolation of outcomes of grants. The comparison of periods before and after the grant period reflects an attempt to isolate the output of grants. However, this is a very difficult task and is at best only partially possible. We do not have information about the publications that resulted from the funded projects. PIs may hold multiple parallel grants and be involved in multiple parallel research projects, and in addition one publication may be the combined outcome of multiple grants. As we use the rejected applicants and their publications and citations as a quasi-control group, we have not tried to isolate outcomes of the specific grants,

6 Options and limitations in measuring the impact of research grants. 261 but instead examine the total publications and citations of the PIs in the relevant period. 2.4 The bibliometric performance indicators Both the Danish and Norwegian bibliometric data are divided into two periods, one before (including application year) and one after (from year after startup year) the funding decision. The differences between the two periods are analysed on the basis of the following indicators:. PIs average number of publications per year. The MNCS of PIs publications (normalized by field of research and year of publication, 1 = world average). Number of publications with MNCS: (a) above the world average, and (b) two times the world average or more.. Share of publications with MNCS: (a) above the world average, and (b) two times the world average or more. MNCS measures the average impact of publications, while the number of publications measures overall productivity and the two other indicators measure the production of top articles. In order to better discern productivity effects from impact effects, the last two measures are used to examine whether the share of highly cited articles has increased for the two groups. Since the distribution of citations is highly skewed across publications, both (a) and (b) will be above the median. Other relevant performance indicators, such as international co-authorship and authorship position (first/last author) are not included in the study, mainly for simplicity reasons and the fact that the data encompasses grants in a large variety research fields. The Norwegian evaluation included analyses of international co-authorship and found that overall it increased somewhat more for the rejected than the successful PIs, whereas there were substantial differences between research areas (Langfeldt et al. 2012: 32). Hence, the likeliness that more refined analyses should find any positive impact of grants in our data set is small. Moreover, we did not find any differences in authorship position of successful versus unsuccessful applicants in the disciplines where this position is a sign of the role of the researcher in the group. 3. Overview results This section examines publication and citation activity among applicants in Norway and Denmark and tests whether there are differences in outcomes for funded and rejected applicants. We are both interested in whether there are differences in productivity and impact for the two groups. We examine the four measures defined above, both in a period prior to application, after the grant (or a corresponding period for rejected applicants), and the difference between before and after. Given the peer review process involved in making funding decisions, there is great potential for a selection effect; that on average and independent of the grant itself, grant recipients will have higher performance both before and after grant receipt. Hence, in examining whether funding has had an effect, we should consider the difference in difference (DiD), i.e. whether changes in research performance from before to after are larger for grant recipients. Figure 1 shows box plots of the MNCS for PIs of research project applications in Norway and Denmark: citation scores before the grant period, after, and difference in difference. Corresponding box plots for postdoc applicants can be found in the appendix. 4 We can see that, as expected, citation scores are highly skewed both for grant recipients and rejected applicants. Though, before after differences appear to more closely follow a normal distribution around a mean that is slightly above zero. Both for median values and middle quartiles, grantees clearly have higher scores both before and after application. However, differences are less apparent when considering the DiD scores. Hence, it is not clear that citation scores for grantees have improved more than for rejected applicants. Table 3 shows the results of statistical tests of differences between grantees and rejected applicants for the DiD measures, while separate results for before and after measures are included in the appendix. As noted above and illustrated in Figure 1, citation scores before or after grant reception are highly skewed, while the DiD measures do not show any strong indication of being skewed and appear to better approximate a normal distribution. Taking this into account, for the DiD measures we have both conducted parametric tests (T-tests) of differences in mean values and non-parametric tests (Mann Whitney tests) that examine whether distributions are significantly different from each other. In contrast, for comparison of before or after values, which are shown in tables A1 A4, only the results of Mann Whitney tests are reported. Table 3 shows average values of the DiD measures for grantee and rejected applicants in Norway and Denmark and P-values for the two tests. As the results shown in the appendix confirm, average citation impacts (MNCS) are significantly higher among grantees both before and after. This is also the case for the number of articles with impact above or more than double the world average. Results are also qualitatively the same for Norway and Denmark. Hence, performance is on average better among grantees prior to grant application, and it is also better after the grant period. However, as Table 3 shows, average impact has not improved more for grantees than for rejected applicants, neither for Norway or Denmark. Both parametric and non-parametric tests fail to find a statistically significant difference in DiD measures of average impact for grantees compared to rejected applicants (MNCS and % above world average). We do however find some varied evidence of effects based on the other indicators.

7 262. L. Langfeldt et al. Figure 1. Citation scores for research project applications, Norway and Denmark: Box plot /quartiles distribution of field standardized citation rate (per paper, 1 = world average) before and after application, for successful and unsuccessful applicants, and the increase/reduction in citation rate from the period before to period after the application. (outliers: 4 PIs with MNCS > 10 are excluded; years with 0 publ not included).

8 Options and limitations in measuring the impact of research grants. 263 Table 3. Research project applications. Comparison of differences before after for Grant recipients and rejected applicants Granted Rejected T-test (P-value) Mann Whitney (P-value) Number of publications per year (DiD) Norway (0.158) (0.086) *** Denmark (1.141) (1.096) Mean normalized citation score (MNCS, DiD) Norway (0.068) (0.043) Denmark (0.100) (0.128) Number of articles with citations above world average (DiD) Norway (0.116) (0.042) 0.050** 0.015** Denmark (0.644) (0.481) 0.089* Number of articles with citations more than twice world average (DiD) Norway (0.070) (0.024) 0.000*** 0.012** Denmark (0.375) (0.306) 0.077* Per cent of publications above world average (DiD) Norway (2.178) (1.224) 0.079* Denmark (4.097) (3.276) Per cent of publications more than twice above world average (DiD) Norway (1.459) (0.815) Denmark (2.465) (2.288) Number observations: Norway: granted = 242; rejected = 812; Denmark: granted = 88; rejected = 86. Standard error of mean in brackets. *** P < 0.01, **P < 0.05, *P < 0.1. Before after differences for the number of publications is significantly higher for grantees in Norway based on the Mann Whitney test (at 0.002), while the P-value for the t-test is not significant (at 0.137). Differences in Denmark are not statistically significant, despite the fact that differences in means are much larger than for Norway. This could potentially suggest that the Danish results are influenced by the small sample size. Differences for the number of highly cited articles are strongly significant for Norway, with average numbers of articles with both more than and double world averages for citations significantly higher for grantees. Results are again weaker for Denmark, though along the same lines, with differences weakly significant based on t-test statistics. Results are much weaker for shares of these highly cited papers, though for Norway the percentage with citations over the world average is weakly significant based on the t-test. What is our overall interpretation of these results for research project grants? First, the MNCS is the most appropriate indicator to measure the effects not related to number of publications, and here we find no evidence of a significant difference between grantees and rejected applicants. On the other hand, mean values are larger for grantees in all cases. This at least leaves open the possibility that we might have found more significant results with larger samples. While it does not appear that there is a difference in average impact between grantees and rejected applicants, there are some indications of an effect in terms of productivity, where this increase in output also includes highly cited articles. While results are fairly similar across countries concerning PIs for research project grant applicants, there are a number of differences between Norway and Denmark regarding postdoc grants. For Norway, while average citation scores are higher among grantees both before and after, the difference is not significant in either case (see Tables A1 and A2). Furthermore, average citation scores actually fall for both grantees and rejected applicants in Norway. In contrast, the number of top publications is higher for both groups after the grant period, and DiD measures are also larger for grantees, though not significant. Results for postdoc grants are shown in Table 4. Results for Danish postdoc fellowships resemble to a greater degree the results for research projects. As with research projects, differences for average citation scores (MNCS) are positive but insignificant, while differences for top publications are significant. Results for Danish postdocs in fact show a stronger effect than for research projects, in particular Danish projects. It is not fully clear why there is such a substantial difference in results among Danish and Norwegian postdoc fellowships, though a partial factor may be the smaller number of observations for Norwegian grantees. Moreover, as will be discussed in Section 4, the scope of outputs from individual fellowships can be expected to differ from those of regular (collaborative) research projects, particularly when we measure the impact on publication and citation rates of the PI. As an additional check of the robustness of our statistical results above, we employed simple regressions, examining whether the receipt of grants impacts changes in performance after controlling for basic individual

9 264. L. Langfeldt et al. Table 4. Postdoc applicants. Comparison of differences before after for grant recipients and rejected applicants Granted Rejected T-test (P-value) Mann Whitney (P-value) Number of publications per year (DiD) Norway (0.173) (0.097) Denmark (0.675) (0.670) Mean normalized citation score (MNCS, DiD) Norway (0.178) (0.100) Denmark (0.129) (0.137) Number of articles with citations above world average (DiD) Norway (0.086) (0.043) Denmark (0.269) (0.229) 0.005*** 0.025** Number of articles with citations more than twice world average (DiD) Norway (0.064) (0.027) Denmark (0.195) (0.156) 0.027** 0.057* Per cent of publications above world average (DiD) Norway (5.626) (3.257) Denmark (2.755) (3.423) 0.017** 0.026** Per cent of publications more than twice above world average (DiD) Norway (4.763) (2.043) Denmark (2.365) (2.119) Number of observations: Norway: granted = 64; rejected = 224; Denmark: granted = 104; rejected = 102. Standard error of mean in brackets. ***P < 0.01, **P < 0.05, *P < 0.1. characteristics, such as age, gender and main field of science. The analysis was conducted for both PIs of research project applications and postdoc applicants, using three of the performance measures (in differences) as dependent variables: MNCS, publications with citations over the world average, and publications with more than double the world average. The regression results can be found in the appendix. For Norway, the regression results correspond well with the statistical tests above. A significant effect of receiving a grant is found on the number of publications with citations above world averages, but not for the average citation rate (MNCS). In contrast, for Denmark no significant impact is found on the number of publications cited above world averages, nor MNCS. This can be compared with the statistical test results above, where differences were weakly significant for both measures of top publications. Hence, when we control for other basic factors, the grant effect for Danish research projects disappears. 4. Discussion As described in Section 1, the summarized results of previous studies of the impact of research grants are inconclusive; the studies apply a variety of approaches and different impact measures/citation scores, and concern different kinds of funding schemes and contexts, and yield divergent results. Are there still some overall conclusions that can be drawn? In this section we discuss the above results for Denmark and Norway in light of previous studies. In this we address factors complicating the identification and measurement of impacts of grant schemes, and in particular the difference between productivity measures and standardized measures related to impact, such as MNCS. The results for Norway above, along with the findings from the evaluation of the FRIPRO scheme, indicated that both the FRIPRO applicants and the awardees were a highly selected group: both funded and rejected applicants were more highly cited than the world average, the funded somewhat more so than the rejected. Hence, past scientific performance seems important both for applying for FRIPRO grants and for receiving them. On the other hand, bibliometric results show only marginal changes in the field-normalized relative citation rates for the period before and after funding decisions for both funded and rejected applicants. 5 Data indicate that PI s multiple projects and funding sources were part of the explanation for this negative result. PI s research typically covers several projects and funding sources within smaller or larger networks of national and international scientific collaboration, and the evaluation was unable to trace the impact of a grant on a PI s publication or citation rate. Many applicants found alternative funding sources for their rejected FRIPRO projects, and a majority of rejected projects appeared to have been implemented. For instance, several universities had economic incentives for FRIPRO applicants and awarded highly rated, but not funded, projects (Langfeldt et al. 2012). On the other hand, obtaining FRIPRO funding was also argued to open doors for additional/further funding, and a substantial proportion of those obtaining FRIPRO grants reported that the FRIPRO funding enabled them to successfully compete for funding from other external sources. The applicant survey in the evaluation also indicated substantial added value of FRIPRO funding concerning international

10 Options and limitations in measuring the impact of research grants. 265 cooperation and research results: Those obtaining FRIPRO funds much more often reported that their long-term international cooperation had been enhanced as a result of the project. Moreover, in comparing successful applicants with applicants who implemented the project with other resources, the former were more likely to report unexpected results of importance to the research field, or that the project had explored new research areas of significant importance for their future research. Still these results from the survey could not be substantiated by bibliometric analyses comparing the citation rates of the funded and rejected applicants (Langfeldt et al. 2012, cf. section 2.4 above concerning international cooperation). Survey and interview results in the Danish evaluations found similar, broader effects of receiving a grant. These included positive impacts on opportunities for career advancement, collaboration with top researchers within one s field, research management competences, obtaining subsequent grants from both national and international sources, and enhanced status and recognition. In particular the latter was attached great importance for grantees subsequent research. A high share also stated that grants have a positive impact on their research and had allowed them to conduct research that otherwise would not have been possible. Though, as for Norway, these survey results have been difficult to support through the bibliometric results. Overall, the present analyses indicate that research grants may have statistically significant impact on bibliometric productivity scores, both on the number of publications per year and the number of highly cited papers; though in many cases evidence of an impact is weak, and notably these scores increase both for successful and rejected project grant applicants the successful slightly significantly more than the rejected. Whereas the differences between successful and rejected project applicants are significant for the productivity scores, the increase in MNCS is not significantly higher for the successful than the rejected applicants; the average citation score of publications does not increase more for the successful than for the rejected applicants. A number of factors may account for these results. Firstly, the successful applicants are already very productive and highly cited, and what specific kind of grant they receive may have limited impact on their further success in terms of MNCS. On the other hand, the grant helps the PIs to add staff to their research team and hence the possibility to publish more papers (with the PI/grant holder as author/co-author). The slight impact found on number of papers above (and twice above) world average citation rate, but lack of impact on MNCS, could be understood as a difference between impact on productivity and impact on scientific importance. Grants primarily impact research productivity, whereas rejections imply postponed projects and or/smaller research groups and a lower number of parallel projects. Moreover, higher productivity implies higher likelihood of (some) highly cited papers and hence also (slightly) impact the number of papers above (and twice above) world average citation rate. MNCS, on the other hand, does not seem impacted by grants nor by productivity. Concerning the postdoc fellowships, the Norwegian and Danish results differ and no obvious reasons for these differences are found. Still, some general differences between impacts of project grants and (individual) postdocs fellowships should be mentioned. Firstly, the size and duration of projects and size of project groups are likely to impact productivity. Research grants facilitate increased research capacity and activity of the PIs but whereas this for postdocs fellowships mainly include own salary and is limited to 2 years, for the research projects the grant is larger, 3 4 years, and imply the ability to employ research staff (e.g. multiple PhDs or postdocs). Secondly, general conditions for obtaining high citation rates may differ between project grants and postdocs fellowships. Doing research with an international/broader orientation may yield high citation rates, and individual postdocs may be less likely (than PIs with larger grants/groups) to do research attracting wide scientific attention or to be affiliated with large-scale projects with a greater potential for high citation rates. Hence, the funding or rejection of the individual postdoc fellowships may have little impact on publication or citation rates. In fact, the rejection of individual postdoc fellowships may result in these young scholars being affiliated with larger research projects, instead of pursuing their own projects, and doing more collaborative research and (co-) authoring more publications than they would in their individual projects. Moreover, it might be more difficult to measure change in the scientific productivity of postdocs due to the early stage of their research career. 6 Finally, general measurement problems may limit the possibility of identifying and delineating the publications resulting from a project/grant, and give weak or varying results of impact studies. There are varying time-lags in publication and publications may be sponsored by several projects/funding sources. When multiple funding sources fund the same research, or PIs conduct multiple projects in parallel, the impact of a particular funding scheme is hard to trace, especially when the time-lag from the project period to publications may vary. Some analyses of this could be provided by developing databases containing adequate data for linking publications to specific funding schemes, but in order to compare the success of successful versus rejected applicants, we would need to include the total publication portfolio of both groups in the comparison. Hence, what are compared in the present analysis are not the results of the specific grant applications, but whether or not rejection/funding of the application impacts the publication or citation profile of the applicant. The cumulative, two-way effect between publications and grants deriving from the analyses should also be noted: PIs with high scores on publications and citations

11 266. L. Langfeldt et al. more easily succeed in grant competitions, and to some extent grants contribute to their publications/productivity (due to increased capacity to perform research). Even with such cumulative advantages, there is no evidence that the grants impact the mean normalized citation rate of the applicants. 5. Conclusions and implications A variety of impacts of grant schemes on research may be unrelated to the overall citation rate of the PIs. Results and added values emphasized in the survey replies of the Norwegian FRIPRO applicants and Danish research project and postdoc grant applicants included competence building/phd candidates, new insight and research results, new/extended research networks, and the exploration of new research topics. These survey results can partly be substantiated by bibliometric impact in terms of increased productivity, but not in terms of any increase in mean normalized citation rate (the increase in MNCS is not significantly higher for supported than rejected PIs). Hence, it could be concluded that the grant schemes increased the capacity to perform research and to publish more papers, and generally improved research conditions, but did not impact scientific importance as measured by the general citation rate (MNCS) of the PIs either because MNCS is an inadequate indicator of the scientific importance of a PI s research or because grants do not enable PIs to perform more important research. It should be added that the survey data still points to important career impacts of grants, especially for the younger researchers (Langfeldt et al. 2012: 51). These researchers are often dependent on external funding, and their ability to attract such funding may be important for obtaining a permanent position. Similar results were found in the Danish evaluation, both from interviews and the survey of grantees. In addition, Bloch et al. (2014) found that these grants from the Danish Council for Independent Research had a significant, positive impact on career progression for PIs. The findings bring up two general questions concerning the aims and evaluation of open mode funding schemes aiming at funding scientifically important projects (scientific value/relevance) and facilitating ground-breaking or frontier research. Firstly, as far as grant schemes support researcher-initiated projects, and do not demand any changes in research content, approaches or collaboration patterns of the PIs, is it reasonable to expect research grants to impact the scientific importance of the PIs research? Secondly, how may research councils design proper impact evaluations of open mode funding schemes, in order to assess whether they have managed to select and fund the most successful projects? The answers to these two questions are interlinked. It hardly seems reasonable to expect open mode research grants to increase the scientific importance of the research of already high performing PIs. Still, according to our analyses, providing more resources to those who already perform at the higher level (i.e. have a higher MNCS/more highly cited papers) tend to yield a higher number of highly cited papers than if the grants were allocated to PIs with less impressive track records as the score is increased from an already high level. Hence, grant allocation based on track records could be justified even without any general treatment effect on the scientific importance of the research; those with the higher level of past performance tend to perform better also for future research. It should be emphasized that the impact of the particular grant seems to principally be through increased productivity, not through increased scientific impact. Concerning implications for the design of impact evaluations, it seems useful to include a variety of indicators, in order to separate between productivity and impact measures, as well as using both bibliometric data and surveys to applicants. When combining reported added value for successful vs rejected applicants (survey data) and DiD analyses of publication and citation impacts, we can obtain a more comprehensive and nuanced picture of the applicants situation, funding sources, publication activity and citation profile, and the ways in which grants may impact research activities and scientific success. It should be noted that the measurable impact of grants on bibliometric scores is likely to be weak and large samples are needed in order to obtain significant DiD analysis results. Moreover, with a view to the complex, multiple funding structures of grantees and the lack of significant impacts found for the studied postdoc fellowships, it may be more promising to focus impact evaluations on large and longterm grants than smaller grants which are often combined with other funding. Bibliometric impacts of short-term grants (1 2 years) that account for a minor part of a PI s portfolio and/or do not significantly increase research capacity are hard to establish based on available data. 7 Notes 1 Moreover, some funding agencies develop systems for linking funding and outcome data to facilitate more robust evaluations of programme impacts (Haak et al. 2012). 2 Average project size increased from 2005 to Total amount per project varied from 0.01 to 2.5 mill Euro. 3 The bibliometric data for the two Danish evaluations are described in detail in Mortensen and Thomsen (2010) and Mortensen et al. (2011).

12 Options and limitations in measuring the impact of research grants Due to space constraints, box plots are not shown for the other indicators. 5 Using descriptive statistics, in the evaluation, significant differences between funded and rejected applicants were not found, neither in terms of increase in the number of publications per PI, nor in their citation rates (Langfeldt et al. 2012). In the present reanalyses of the data using more refined methods, impact is found on productivity (number of publications per PI) and measures dependent on productivity (number of publications per PI cited above world average), but not on MNCS. 6 In our case, as the smaller samples for analysis/limited numbers of awarded postdoc fellowships provide an additional limitation for measuring impact. 7 One alternative in order to demonstrate bibliometric impact of minor grants, would be to develop databases with complete and precise information of the specific funding schemes which have sponsored the publications (names sponsor organizations do not suffice), and to compare publication frequencies and citation scores between funding schemes and their total allowance, rather than to compare the successful and rejected applicants (cf. MacLean et al. 1998). References Benavente, J. M. et al. (2012) The impact of national research funds: A regression discontinuity approach to the Chilean FONDECYT, Research Policy, 41/8: Bloch, C., Graversen, E. K. and Pedersen, H. S. (2014) Competitive research grants and their impact on career performance, Minerva, 52, Bloch, C. et al. (2011) An Evaluation of Research Project Grants of the Danish Council for Independent Research, Main report and subreports. Copenhagen: Danish Agency for Science, Technology and Innovation (in Danish). Blundell, R. & Costa Dias, M. (2002) Alternative approaches to empirical evaluations in microeconomics, Portuguese Economic Journal, 1/2, Bornmann, L., Leydesdorff, L, and Van den Besselaar, P. (2010) A meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications, Journal of Informetrics, 4/3, Dehejia, R. H. & Wahba, S. (1999) Casual effects in nonexperimental studies: re-evaluating the evaluation of training programs, Journal of the American Statistical Association, 94/ 448, (2002) Propensity score-matching for nonexperimental studies, The Review of Economics and Statistics, 84/1, Donovan, C. (2011). State of the art in assessing research impact: introduction to a special issue, Research Evaluation, 20/3, Faber, S. T. et al. (2010) The Danish Research Council s Support of Female Researchers and Researchers at an Early Stage of their Career. Copenhagen: Danish Agency for Science, Technology and Innovation (in Danish). Haak, L.L. et al. (2012) The electronic Scientific Portfolio Assistant: Integrating scientific knowledge databases to support program impact assessment, Science and Public Policy, 39/4: Heckman, J. J. et al. (1998) Characterizing selection bias using experimental data, Econometrica, 66/5, HFSP (2009) Report on the citation database for the Human Frontier Science Program, Evidence/HFSP October < HFSP%20Bibliometrics% pdf> accessed 11 Nov Jacob, B. A. and Lefgren, L. (2011) The impact of NIH postdoctoral training grants on scientific productivity, Research Policy, 40/6, Langfeldt, L. et al. (2012). Evaluation of the Norwegian scheme for independent research projects (FRIPRO). Oslo: NIFU Report 8/2012 < NIFUrapport pdf > accessed 11 Nov Lewison, G. and Dawson, G (1998) The effect of funding on the outputs of biomedical research, Scientometrics, 41/1 2, MacLean, M. et al. (1998) Evaluating the research activity and impact of funding agencies, Research Evaluation, 7/1, Melin, G. and Danell, R. (2006) The top eight percent: development of approved and rejected applicants for a prestigious grant in Sweden, Science and Public Policy, 33/10, Mortensen, P. S. and Thomsen, G. S. (2010) Delrapport 4: Den bibliometriske undersøgelse blandt ansøgere til postdocstipendier og talentprojekter inden for natur-, sundheds- og teknisk videnskab, Aarhus: Dansk Center for Forskningsanalyse. Mortensen, P. S., Thomsen, G. S. and Kruuse, J. (2011) Evaluering af virkemidlet forskningsprojekter, Delrapport 4: Den bibliometirske undersøgelse. Aarhus: Dansk Center for Forskningsanalyse < 2011/5-bibliometri_delrapport.pdf> accessed 11 Nov Nedeva, M. et al. (2012) Understanding and Assessing the Impact and Outcomes of the ERC and its Funding Schemes (EURECIA). Final Synthesis Report. European Commission. < final_synthesis_report.pdf> Neufeld, J. and von Ins, M. (2011) Informed peer review and uninformed bibliometrics?, Research Evaluation, 20/1, Rigby, J. (2011) Systematic grant and funding body acknowledgement data for publications: new dimensions and new controversies for research policy and evaluation Research Evaluation, 20/5, Sandstro m, U. (2009) Research quality and diversity of funding: A model for relating research money to output of research, Scientometrics Volume, 79/2, Sivertsen, G. and Larsen, B. (2012). Comprehensive bibliographic coverage of the social sciences and humanities in a citation index: an empirical analysis of the potential. Scientometrics, 91/2, Smith, J. A. and Todd, P. E. (2005). Does matching overcome LaLonde s critique of nonexperimental estimators? Journal of Econometrics, 125/1 2, van Leeuwen, T. N. and Moed, H. F. (2012) Funding decisions, peer review, and scientific excellence in physical sciences, chemistry, and geosciences, Research Evaluation, 21/3, Vedung, E. (1997) Public Policy and Program Evaluation. New Brunswick, NJ: Transaction Publishers. Zhao, D. Z. (2010) Characteristics and impact of grant-funded research: a case study of the library and information science field, Scientometrics, 84/2,

13 268. L. Langfeldt et al. Appendix Figure A1. Postdoc fellowship applications: Box plot /quartiles distribution of field standardized citation rate (per paper, 1 = word average) before and after application, for successful and unsuccessful applicants, and the increase/reduction in citation rate from the period before to period after the application (outliers: 4 PIs with MNCS > 10 citations are excluded; years with 0 publ not included).