Understanding NIH Peer Review CPDD Grant Wri,ng Workshop June 17, 2015 Gerald McLaughlin, PhD Chief, Scien,fic Review Branch, DER Na,onal Ins,tute on Drug Abuse, NIH
The Grant Review Process Peer review is the cornerstone of the NIH research mission Center for Scientific Review YOU Initiate a Research Idea Submit Application Assigns IC and Study Section Study Section Reviews for Scientific Merit YOU Conduct the Research Allocate Funds Institute Evaluates for Programmatic Relevance I/C Advisory Council Evaluates reviews, appeals Institute Director Takes Final Action
Submi<ng an Applica@on to NIH To apply as PI, your organiza@on must give you permission to submit an applica@on as PI; if not, consider Co- inves@gator or other role. If permiied, register early in era Commons as PI: hip://commons.nih.era.gov Almost all applica@ons are electronically submiied NIH interacts with you online through the submission process Follow instruc@ons in the FOA Incomplete or inconsistent applica@ons cannot be changed aner submission deadline. All parts should be consistent with the theme and scope of work proposed. Seriously flawed/incomplete applica@ons are usually rejected at Grants.gov and/or by later administra@ve withdrawal by CSR/I/C Receipt and Referral, onen when no@ced by SRO, before peer review. Applica@ons to RFAs also have responsiveness evaluated by Program. Even if a deeply flawed applica@on makes it to reviewers, flaws may affect your reputa@on and that of the submi<ng organiza@on; many universi@es restrict permission and/or have quality control steps internally, and earlier- than- NIH deadlines for related documents.
Applica@on Groundwork New, interes@ng, feasible ideas are rare and are most welcome to reviewers. Inves@gate, read papers, think, generate many ideas, discard most, select and firm up plans for the best idea. Fortunately, even old ideas and problems have new variants and approaches. Which health- related issues are you prepared to help address? Discuss research and/or training ideas and plans with your mentors and colleagues; and have them read and re- read the applica@on in @me for you consider and edit before submi<ng. What research has NIH funded and by which Ins@tute(s)? NIH RePORTER includes all NIH funded grants since 1989 hip://projectreporter.nih.gov/reporter.cfm and search by keyword, ins@tute, inves@gator, study sec@on for projects in a similar areas to yours Surf NIH/I/C websites e.g. hip://www.nih.gov/ has all I/Cs; NIDA s home page is hip://www.drugabuse.gov/ Will the Ins@tute(s) be interested, and what are current priori@es? Email, then call NIH Program staff; typically Chief- level staff of each I/C Division have contact informa@on at websites, e.g. for NIDA at: hip://www.drugabuse.gov/about- nida/organiza@on who can discuss and/or refer you to experienced specialist extramural science administrators. Iden@fy Funding Opportunity Announcement(s) (FOA(s): hip://grants.nih.gov/grants/guide/; hip://www.drugabuse.gov/funding/funding- opportuni@es
Find Study Sec@on(s), Include in Cover LeIer Include a cover leier to give CSR and I/C Referral guidance, although their decisions are seldom reversed. This cover leier is seen by CSR and SRO, not by reviewers. FIND AND SUGGEST a Study Sec@on: - Search CSR site: hip://www.csr.nih.gov; the CSR study sec@on website is among NIH s most- visited websites. A drug abuse search yielded 14 standing integrated CSR review panels. Among these: Risk, Preven@on and Interven@on for Addic@ons: RIPA and IPTA Behavioral Gene@cs and Epidemiology: BGES Biophysical Pharmacological..Neuroscience Fellowship: F03B Drug Discovery for the Nervous System: DDNS Molecular Neuropharmacology and Signaling: MNPS HIV/AIDS Fellowship: AARR Fellowship SEPs Behavioral Neuroscience Fellowship: F02A - NIDA has FOA- specific SEPs, some@mes with recurring reviewers e.g. for RFAs, PARs and PASs. NIDA reviews K99/R00s and T32s. NIH RePORTER hpp://projectreporter.nih.gov/reporter.cfm
Find Study Sec@on(s), Include in Cover LeIer (cntd) Consider in cover leier: Sugges@ng possible study sec@on(s) Sugges@ng Ins@tute(s) for assignment Sugges@ng key themes/exper@se in cover leier You can iden@fy a few poten@al reviewers in your speciality area who you believe should not review your applica@on due to an non- obvious real or perceived conflicts of interest; provide related ra@onale(s) (e.g. long- standing personal or professional issues with you or your mentor document if feasible). SROs consider such poten@al conflicts but do not share final decisions. DO NOT suggest reviewers, this creates perceived conflicts of interest for NIH; this NIH guidance differs from that of some journals
NEW THIS Year: Modified BIOSKETCH FORMAT A new biosketch format is required for all applica@ons submiied for due dates on or aner May 25, 2015. NOT- OD- 15-085 includes prepara@on instruc@ons at SciENcv hip://www.ncbi.nlm.nih.gov/sciencv/ which are rela@vely complex. What s most different about the new biosketch format? Five- page limit, not four. New Sec@on C: Contribu@on to Science Briefly describe up to five of your most significant contribu@ons to science. For each contribu@on, indicate the historical background that frames the scien@fic problem; the central finding(s); the influence of the finding(s) on the progress of science or the applica@on of those finding(s) to health or technology; and your specific role in the described work. For each of these contribu@ons, reference up to four peer- reviewed publica@ons or other non- publica@on research products (can include audio or video products; patents; data and research materials; databases; educa@onal aids or curricula; instruments or equipment; models; protocols; and sonware or netware) that are relevant to the described contribu@on. The descrip@on of each contribu@on should be no longer than one half page including figures and cita@ons. Also provide a URL to a full list of your published work as found in a publicly available digital database such as SciENcv or My Bibliography, which are maintained by the US Na@onal Library of Medicine.
NEW BIOSKETCH FORMAT (cntd) Biosketch informa,on is now within the Instruc,ons associated with SF424 applica,ons at hpp://grants.nih.gov/grants/funding/424/index.htm and there are somewhat different guidelines and detailed examples for: General: Version C (regular applicants, and Fellowship Sponsor/Co- Sponsors) Fellowship: Predoctoral vs postdoctoral
NIH Biosketch Iden,fica,on
New NIH Biosketch, Sec,on A Example
New NIH Biosketch, Sec,on C Example
NEW BIOSKETCH FORMAT (cntd) Example, Postdoc Biosketch, from hpp://grants.nih.gov/grants/funding/424/index.htm: A. Personal Statement My long term research interests involve the development of a comprehensive understanding of key developmental pathways and how altera,ons in gene expression contribute to human disease. My academic training and research experience have provided me with an excellent background in mul,ple biological disciplines including molecular biology, microbiology, biochemistry, and gene,cs. As an undergraduate, I was able to conduct research with Dr. Xavier Factor on the mechanisms of ac,on of a new class of an,bio,cs. As a predoctoral student with Dr. Tan, Auguri, my research focused on the regula,on of transcrip,on in yeast, and I gained exper,se in the isola,on and biochemical characteriza,on of transcrip,on complexes. I developed a novel protocol for the purifica,on for components of large transcrip,on complexes. I was first author of the ini,al descrip,on of the Most Novel Complex. A subsequent first author publica,on challenged a key paradigm of transcrip,on elonga,on and was a featured ar,cle in a major journal. During my undergraduate and graduate careers, I received several academic and teaching awards. For my postdoctoral training, I will con,nue to build on my previous training in transcrip,onal controls by moving into a mammalian system that will allow me to address addi,onal ques,ons regarding the regula,on of differen,a,on and development. My sponsor Complete List of Published Work in MyBibliography: hpp://www.ncbi.nlm.nih.gov/sites/myncbi/collec,ons/public/1tay8xsxtexiw5r2sttcjhq5x/? sort=date&direc,on=ascending D. Scholas@c Performance Table with courses and grades
NEW BIOSKETCH FORMAT FROM REVIEW PERSPECTIVE Non-compliant biosketches: 1. SROs will instruct reviewers: Reviewers should make note of a noncompliant biosketch in the Addi@onal Comments sec@on of the cri@que template. Reviewers need not consider extraneous biosketch materials included in the grant applica@on. Reviewers should not consider lists of publica@ons beyond the acceptable four per descrip@on of a significant scien@fic contribu@on unless they are provided through a weblink to a publicly available digital database.
NEW BIOSKETCH FORMAT (cntd) 2. If the SRO or a reviewer no@ces a noncompliant biosketch, the following administra@ve note is added to the summary statement: ADMINISTRATIVE NOTE: During the review of this applica@on, reviewers and/or NIH staff noted that one or more biosketches did not comply with the required format (NOT- OD- 15-032). An electronic no@fica@on has been sent to the Signing Official for this applica@on, to ensure that future applica@ons use the correct biosketch format. NIH has the authority to withdraw such applica@ons from review or considera@on for funding. CSR will send an email to the Signing Official of the applica@on organiza@on.
Links Regarding New Biosketch My NCBI documenta,on: hpp://www.ncbi.nlm.nih.gov/books/nbk3843/ Movie tutorial on managing compliance: hpp://youtu.be/jyodiod_yye NIH Public Access Policy homepage: hpp://publicaccess.nih.gov/ SciENcv is available at: hpp://www.ncbi.nlm.nih.gov/account/ SciENcv documenta,on: hpp://www.ncbi.nlm.nih.gov/sciencv SciENcv video released: hpp://youtu.be/prwy- 3GXhtU Provide feedback to info@ncbi.nlm.nih.gov
ANer you Submit the Applica@on 70% of grant applica@ons are reviewed at the Center for Scien@fic Review (CSR) Specific mechanisms are reviewed at NIDA K99/R00 Pathway to Independence Award K12 Mentored Clinical Scien@st Development Program Award T32 Na@onal Research Service Awards (NRSA) Centers, PARs, PASs, RFAs, R25, U01s CSR makes the ini@al assignments for Funding IC, based on ins@tute mission and programma@c mandates Study Sec@on, based on primary topic of applica@on; some are reviewed at CSR, some are referred to and assigned within the Ins@tute s review office
Review Study Sec@ons At CSR, there are 255 standing study sec@ons defined by research topic or mechanism (e.g., fellowships). NIDA OEA has only Special Emphasis Panels (SEPs), recruited de novo for each review, about 100 specialty SEP reviews/year. The Scien@fic Review Officer (SRO) Manages the Study Sec@on and runs the review mee@ng Is usually a scien@st with experience in the area Iden@fies reviewers with the correct exper@se Ensures fair and unbiased scien@fic evalua@on Releases average scores (10-90), writes the overall paragraph resume, may also edit cri@ques, releases the summary statements results from review mee@ng)the assigned SRO is your point of contact aner submission through review; then the PO is your contact.
Who are the Reviewers? Experts in the area of science being reviewed Respected by their peers Can be from academics, NIH, industry They typically have broad scien@fic perspec@ve They are impar@al and without conflict of interest for the applica@ons they review and score The par@cular roster of reviewers for your applica@on can and should be accessed via era Commons about 30 days before the review. If worried about exper@se coverage, non- obvious conflicts of interest, or other issues, contact the SRO.
Process of NIH Reviews Large review mee@ngs are primarily face to face. Telereviews, Internet Assisted Reviews (chatroom- style) and audiovisual reviews are onen used for smaller reviews. Each applica@on is typically assigned 3 reviewers, depending upon the complexity of the applica@on and other variables. Assigned reviewers submit ini@al detailed cri@ques and 1-9 numerical scores about a week before the mee@ng in ecommons. Some@mes the third reviewer writes only an overall paragraph. Top scoring 50% are usually discussed from best to worst At the mee@ng: Assigned reviewers present their cri@ques of the applica@on Open discussion among the commiiee Each non- conflicted panel member gives a final overall impact score
Score Descrip@ons SCORE DESCRIPTOR Strengths/Weaknesses IMPACT 1 Exceptional Strengths 2 Outstanding HIGH 3 Excellent 4 Very Good 5 Good MEDIUM 6 Satisfactory 7 Fair 8 Marginal LOW 9 Poor Weaknesses
How Reviewers Evaluate an Applica@on Review Criteria are listed in each FOA Reviewers score each of the review criteria Make sure you understand what each criterion is asking F30, F31, F32 Fellowships Fellowship Applicant Sponsors, Collaborators and Consultants Research Training Plan Training Poten,al Ins,tu,onal Environment and Commitment to Training Candidate K01 / K99 Career Development Plan Research Plan Mentor(s) Co- Mentor(s) Consultant(s) Collaborator(s) Environment & Ins,tu,onal Commitment to the Candidate R01/R03/R21/U01.. Significance Inves,gator(s) Innova,on Approach Environment
Scoring Criteria Overall Impact The likelihood for a project to exert a sustained, powerful influence on the research field(s) involved For R- type OVERALL IMPACT HIGH MEDIUM LOW Score 1 2 3 4 5 6 7 8 9 Addresses a problem of high importance / interest in the field. May have some or no weaknesses. May address a problem of high importance in the field, but weaknesses in the criteria bring down the overall impact to medium. May address a problem of moderate importance in the field, with some or no weaknesses. May address a problem of moderate/high importance in the field, but weaknesses in the criteria bring down the overall impact to low. May address a problem of low importance in the field, with some or no weaknesses
What Reviewers Want They need to understand WHAT you want to do, WHY it is important, and can YOU do it? Clear rationale of proposed research and methods, supported by current knowledge of literature. Significant and feasible project. Tables and figures are informative and readable All review criteria fully addressed; for U/R mechanisms, typically Significance, Innovation and Approach are incorporated within Specific Aims/Research Plan; Investigators, Biosketches; Environment, organization resources and evidence of commitment to the project. For research training mechanisms, mentor/organization support, and a strong albeit brief research project. No typographical or grammatical errors or inconsistencies. Errors suggest lack of attention to detail and commitment that reviewers tend to think are likely to convey to the actual research.
ANer the Review Mee@ng Average overall Impact Scores are released 0-3 days aner the review ends in ecommons. A Summary Statement is generated for every applica@on and is generally available within 30 days aner a mee@ng The summary statement includes the assigned reviewer s wriien cri@ques regarding overall impact and each individual evalua@on criterion score; occasionally comments to applicant Non- discussed applica@on summary statements have (*.*), instead of a numerical overall impact score. Discussed applica@ons have: Resume of the discussion at the mee@ng wriien by the SRO Administra@ve comments regarding budget, human subjects, vertebrate animals, data sharing plans, etc. Overall Impact score and percen@le rank if the par@cular review is percen@led.
What The Overall Impact Priority Score Means Priority Score The average of the panel member s Overall Impact Score (1-9, 1 best, 9 worst) @mes 10. Percen@le Rela@ve ranking of an applica@on from the study sec@on provided for applica@ons in standing review commiiees It is the percentage of applica@ons with scores beier than or equal to that applica@on Allows direct comparison of applica@ons across mul@ple CSR study sec@ons Not all applica@ons are given percentages; those for Ins@tute- based reviews seldom are.
Reading the Summary Statement Great score? Take a deep breath and enjoy. Not so great score? Take a deep breath and pause Remember The review is not personal, its just the business of science The comments reflect the scien@fic opinions of the reviewers Read the Summary Statement again Talk with your colleagues about the comments Talk with your Program Officer about the comments and possible next steps for this or another project.
ANer the Review: Evalua@on by the Funding Ins@tute Scien@fic merit Is the applica@on asking an important ques@on? Program balance Are other funded grants asking the same ques@on? Is the applica@on addressing an unmet need? Ins@tute mission Is the topic a research priority? Availability of funds What is the priority rela@ve to other applica@ons? Ins@tute Director priori@es What direc@ons should the I/C be moving toward? If funded, be aien@ve to guidance from NIH s Program and Grants offices. Do and report on what is promised. P.S.: If asked to review for NIH try to do so, although do balance with other commitments. If you are an effec@ve reviewer, other reviewers and NIH staff no@ce. I/C review offices olen try out rela@vely junior reviewers; and CSR has an early career reviewer program: hop://public.csr.nih.gov/reviewerresources/becomeareviewer/ecr/pages/default.aspx
Example of Funding Timeline Submission Due Date Feb 5 (new R01) Feb 12 (new K) Apr 8 (F type) IC/Study Sec@on Assignment Soon aner submission Scien@fic Review June/July Summary Statement Released July/Aug Advisory Council Meets Sept/Oct Earliest Start Date December Up to 10 Months from Submission to Funding
Knowledge (e.g., good websites) is Power CSR Applicant Resources hpp://public.csr.nih.gov/applicantresources/planningwri,ngsubmiong/pages/default.aspx Videos on Peer Review and Applica@on Process hpp://public.csr.nih.gov/applicantresources/pages/default.aspx Reviewer Criteria and Templates hpp://grants.nih.gov/grants/peer/reviewer_guidelines.htm#general_guidelines CSR Early Career Reviewer hpp://public.csr.nih.gov/reviewerresources/becomeareviewer/pages/overview- of- ECR- program.aspx NIDA FAQ hip://www.drugabuse.gov/frequently- asked- ques@ons