!"#$%&"'()*$+,-'./))0,'1234' '

Transcription

1 !"#$%&"'()*$+,-'./))0,'1234' ' Suzi Iacono, Head Office of Integrative Activities Philadelphia, PA April 20, 2016

2 NSF by the Numbers FY 2017 Budget Request

3 NSF Merit Review Criteria What is the intellectual merit of the proposed ac3vity? How important is the proposed acbvity to advancing knowledge and understanding within its own field or across different fields? How well qualified is the proposer (individual or team) to conduct the project? (If appropriate, the reviewer will comment on the quality of prior work.) To what extent does the proposed acbvity suggest and explore creabve, original, or potenbally transformabve concepts? How well conceived and organized is the proposed acbvity? Is there sufficient access to resources? What are the broader impacts of the proposed ac3vity? How well does the acbvity advance discovery and understanding while promobng teaching, training, and learning? How well does the proposed acbvity broaden the parbcipabon of underrepresented groups (e.g., gender, ethnicity, disability, geographic, etc.)? To what extent will it enhance the infrastructure for research and educabon, such as facilibes, instrumentabon, networks, and partnerships? Will the results be disseminated broadly to enhance scienbfic and technological understanding? What may be the benefits of the proposed acbvity to society?

4 Why Broader Impacts? Since the beginning of NSF, more excellent projects than there are funds. Non-technical considerabons come into play when there are many projects of quasi-equal scienbfic merit. Clear criteria are important In the past, there have been many more up to 11 at one point (1974). Now, there are 2 Intellectual Merit and BI. NIH sbll uses a larger number of criteria.

5 How does this work? Since 1998, more strict accountability for BI at every stage of the merit review process. BI must be discussed and evaluated at each stage: Proposal Reviews Panel Summary Review analysis Annual report & final report COVs are asked to assess BI treatment at each stage.

6

7 COV Reports on BI COVs, in general, have a very posibve view of NSF s merit review process and oben offer suggesbons for improvements. A majority of the COVs called for greater clarificabon about the meaning of the BI criterion. Since documentabon was inconsistent, COVs are concerned about unnecessary randomness in the merit review process. A number of COVs suggest more extensive training for reviewers.

8 COV Comments For some proposals, PO used a boilerplate template for the analyses, and these lacked sufficient details to understand the rabonale for the final recommendabon [DBI]. In a centers program, the individual reviews did not have a depth or breadth appropriate to the complexity and scale of center proposals [DBI]. RelaBve weighbng of IM and BI is not consistent across the secbon and variabons in expectabons of what consbtute quality BIs are seen in individual reviews, panel summaries and review analyses [SEP]. There conbnues to be confusion (among PIs, reviewers, and COV members) about what sorts of acbvibes consbtute broader impacts, and whether they need to be specifically related to the research aspects of the proposal [CISE].

9 COV Comments There is evidence of a wide range of interpretabon among both PIs and reviewers about what consbtutes broader impact and how it should be weighted in the proposal rabngs [CBET]. The broader impact criterion appears to be used by reviewers more as a Be-breaker rather than a more substanbal and equally weighted criterion [CBET]. That said, the lack of in-depth comments regarding the broader impacts of the proposed research is an indicabon that the overall broader impacts definibon is not well understood by many of the reviewers [CMMI]. Most reviewers made an effort to report on both intellectual merit and broader impacts. However, what consbtutes broader impacts and the relabve weighbng of these criteria in the review varies greatly [PHY-A].

10 Why BI Now? The long line of COV reports express confusion about how BI is handled during merit review. Persistent quesbons from everyone, really, about how BI is applied at NSF. QuesBons from our OMB examiner about what we are doing about all this.

11 A Good Look at BI last Fall A process Directorate-wide discussions Internal staff External ACs NSF Leadership Workshop in September Report to OMB in December Agency-wide strategic review in January - March Consensus hypotheses Hypothesis 1: The applicabon of BI has changed over Bme; we should expect it to conbnue to evolve. Hypothesis 2: The so-called confusion is really variabon in BI due to different award sizes, disciplines and solicitabon language Hypothesis 3: There are some common pracbces in panels, e.g., Be-breaking.

12 A Good Look at BI Last Fall (2) There is a lack of consensus within NSF on the following BI pracbces. 1. What is the expectabon of innovabon in BI? 2. How can the poten&al for societal impact of BI be reviewed? 3. Can/should BI be considered as insbtubonal/ community outcomes rather than at the individual level? Also, 1. Are the BI criteria clear? 2. Have reviewers received guidance regarding BI beyond the GPG?

13 Research Grants Awarded to PIs in Early & Later Stages of Career and Research Proposal Success Rates

14 Key AnalyIcal QuesIons for Strategic Review How do directorates differ in their understanding of BI? To what extent do awardees report the BI acbvibes they propose?

15 Data Analy3cs Documents analyzed Five groups of NSF documents related to NSF FY 2013 awards: Project summaries (N=8,042) Project descripbons (N=8,823) Panel summaries (N=1,264) Final Project reports (N=1,586) Public outcomes reports (N=1,812) Methods Analyzed BI content of all documents Each document was assigned to one of five categories +Awarded proposals, panel summaries, and post-award reports were assigned to 1 of 5 categories +Why different N s? Encoding issues; BI secbon cannot be found in project summary; panels not required for all proposals; not all awards expired yet.

16 Broader Impacts Categories Advance discovery and understanding while promoting teaching, training, and learning Broaden participation of under-represented groups Enhance infrastructure for research and education Broaden dissemination to enhance scientific and technological understanding Benefits to society

17 Broader Impacts Categories Archived Advance discovery and understanding while promoting teaching, training, and learning, e.g., by training graduate students, mentoring postdoctoral researchers & junior faculty, involving undergraduates in research experiences, & participating in the recruitment, training, & professional development of K-12 mathematics & science teachers. Broaden participation of under-represented groups, e.g., by establishing collaborations with students & faculty from institutions & organizations serving women, minorities, & other groups under-represented in STEM. Enhance infrastructure for research and education, e.g., by establishing collaborations with researchers in industry & government labs, developing partnerships with international academic institutions & organizations, & building networks of U.S. colleges & universities. Broaden dissemination to enhance scientific and technological understanding, e.g., by presenting results of research & education projects in formats useful to students, scientists & engineers, members of Congress, teachers, & the general public. Benefits to society may occur, e.g., when results of research & education projects are applied to other fields of science & technology to create startup companies, to improve commercial technology, to inform public policy, & to enhance national security.

18 Differences among Directorates 2013 Awards 100% Broader Impact From Project Description BI Category ClassiKication 8,823 2,138 1,253 1,030 1, % 80% 70% 60% 50% 40% 30% 20% 10% 0% NSF MPS CISE GEO ENG BIO SBE EHR Advance Discovery BeneKit To Society Broadening Participation Dissemination Enhance Infrastructure Clear differences across the directorates Strong focus on advancing discovery and teaching, learning

19 BI Assessment by Stage in Merit Review 100% 8,042 8,823 1,264 1,586 1,812 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Summary Description Panel FinalReport Project Outcome Advance Discovery BeneKit To Society Broadening Participation Dissemination Enhance Infrastructure PercepBons change as project progresses through merit review.

20 *&+(&,&'$X&':8&9I+&:/ $!!"# $!!"#,!"# $R%)'# $R%)'# 8,823 1,264,!"# +!"# +!"# *!"# *!"# )!"# )!"# (!"# (!"# '!"# '!"# &!"# &!"# %!"# %!"# $!"# $!"#!"#!"# 1<G>J# L=M<GA>#DFEA;M>:I# 7>G>KFB#N;#.;AF>BI# DFEE>?FG<BF;G# 1<G>J#

21 LiYle Difference between Proposed and Reported AcIviIes 100% 1,586 1,586 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Description Final Report Advance Discovery BeneKit To Society Broadening Participation Dissemination Enhance Infrastructure

22 IniIal ObservaIons Significant differences among the directorates, and some similaribes The distribubon of BI categories in proposals differs from that in panel summaries Similar distribubon of BI acbvibes in proposals and in final project reports

23 Recommended AcIons for Discussion Do more data analysis Share and improve the thesaurus associated with the five categories Confirm the consensus hypotheses and analyze the others Give addibonal guidance to PDs, reviewers, and COV members Describe the variability of responses for their directorate or division, e.g., award size, program and discipline Provide more explicit instrucbons on BI for program directors, reviewers and COVs Add guidance for PIs for project reports regarding addressing BI Provide a template for review analysis Assess BI in a separate secbon in the review analysis may facilitate more in-depth analysis Develop possible new reporbng requirement Require PIs to report on impacts in annual reports

24 Challenges for the Summit How do we beser understand what the impact of these investments are on people? grow the diversity of types of BI, if appropriate? develop meaningful impact of BI beyond the life of the project? best develop shared BI resources for the insbtubon and the community? develop a BI knowledge repository?

25 Thanks!

26 Five BI categories were idenified in 2012 merit review criteria update Advance Discovery Benefit Society Broaden Par3cipa3on Dissemina3on Enhance Infrastructure curriculum decision maker African American blogs collaborabon educabonal material environmental problem community college broader community educabonal plauorm doctoral student environmental blacks broader audience equipment elementary school management AGEP cibzen science industry transformabve economy disability database informabonal tool k-12 federal agencies female disseminabon infrastructure g-12 hazard HBCU diverse media instrumentabon graduate student local agencies Hispanic DLESE mulbdisciplinary masters student nabonal interest HSI exhibits interdisciplinary mentoring nabonal interests LaBno informal science mulb-user facility middle school policy analysis LSAMP knowledge transfer partnership ms student policy tool minority libraries partner insbtute ms students public health minoribes museum research center phd student public parbcipabon MSI nature center research plauorm phd students public policy nabve American public accessibility technology post doctoral public safety nabve Alaskan public engagement technology center post doc public welfare nabve Hawaiian public outreach transdisciplinary RET risk assessment OEDG radio show REU social impact pacific islander science center student assessment stakeholder TCU student evaluabon state agencies two-year-college teacher women undergraduate student underrepresented group

27 AddiIons to Five BI categories in 2012 merit review criteria update Advance Discovery Benefit Society Broaden Par3cipa3on Dissemina3on Enhance Infrastructure Junior Faculty Management Two year college Public engagement Non-ProKit Technology Well-being Training public awareness Technological Advances ScientiKic Literacy Target Audience Webpage Undergraduate Research National Security Journal articles Models Economic Competiveness Conferences Leveraging Innovation Creative Novel Evidence Based Resources

28 Par3cipa3on of Female Reviewers in Virtual, In Person, and Mixed Panels FY 2014

29 Team Leaders DepuBes Team Suzi Iacono, OIA Pramod Khargonekar, ENG Marge Cavanaugh, GEO Joan Frye, OIA Nicholas Daly, OIA/EAC Tasha Inniss, EHR/HRD Wyn Jennings, EHR/DRL Katherine Meyer, SBE/SES Jessica Rosenberg, OD Heather Watson, EHR/DRL