Review of Small Business Applications at the National Institutes of Health Gagan Pandya, Ph.D. Scientific Review Officer, Infectious Diseases Microbiology, IRG Center for Scientific Review, NIH November 7, 2017 Milwaukee, Wisconsin NIH Peer Review System for Grant Applications First Level of Review Scientific Review Group (Study Section) Second Level of Review NIH Institute/Center Council 1
Timetable for SBIR and STTR Grant Applications Due Date Scientific Review (Study Section Meets) Advisory Council Review September 5 October -November January January 5 March-April May April 5 June-July August http://grants.nih.gov/grants/guide/notice-files/not-od-15-038.html PEER REVIEW Goal To provide thorough, fair, independent, expert & unbiased evaluation of scientific and technical merit of applications assigned to a panel : so the best applications receive the best scores Focus Impact: Will the research have a sustained effect on science or practice? 2
Scientific Review Officer (SRO) at the NIH Center for Scientific Review (CSR) Designated Federal Official with overall responsibility for the review process Selects reviewers based on broad input Have a broad range of scientific expertise and background Are experts in the field, but maybe not in exact area of your application Manages study section meetings Releases scores & prepares summary statements Provides any requested information about study section recommendations to Institutes/Centers and National Advisory Councils/Boards How Reviewers Are Selected for Study Sections Demonstrated scientific &/or technical expertise Research support preferably small business Doctoral degree or equivalent Mature judgment Breadth of perspective Impartiality Representation from both academia and small business. At least one member must be from small business. At least 25% small business or other industry members is encouraged. Representation of women and minority scientists Geographic distribution Avoid excessive service on a panel 3
Personal Reviewer Conflicts of Interest (COI) Family member/close friend Professional Collaborator Employees of companies in direct competition with applicant s company Financial Financial interest in company or competing companies Academic scientists that hold patents for competing technologies Institutional Longstanding scientific disagreement Personal bias Appearance of conflict Difference of scientific opinion is not a conflict Reviewers sign pre-meeting & post-meeting COI forms Confidentiality Review materials and proceedings of review meetings represent privileged information for reviewers and NIH staff. At the end of each meeting, reviewers must destroy or return all review-related material. Reviewers should not discuss review proceedings with anyone except the SRO. Questions concerning review proceedings should be referred to the SRO. Applicants should never communicate directly with any members of the study section about an application. It is critical to maintain confidentiality everything about the review is confidential before, during, and after the meeting. 4
Before the Study Section Meeting Each application is assigned to 3 or more reviewers. 5-6 weeks in advance Reviewer assess each application by providing: A preliminary Overall Impact score Criterion Scores for each of the 5 Core Review Criteria A written critique Phase I (R41 and R43) Applications No preliminary data required: purpose is to establish feasibility. Should include a rigorous experimental plan. Commercial potential should be evaluated but no detailed commercialization plan is required. This is part of the Significance criterion. Eligibility of applicant organization to receive small business funds is NOT a review criterion and should not be discussed at the meeting. 5
Phase II (R42 and R44) Applications Evaluation of Phase I research (milestones reached) Preliminary data required / progress of Phase I research Commercial Potential and Societal Impact (Commercialization Plan required, including intellectual property issues) Fast-Track Applications (Phase I and Phase II) Clear, measurable criteria for Phase I success (quantitative milestones) that must be achieved before initiating Phase II. Preliminary data are not required but expected. Commercialization Plan required. 6
Review Criteria Overall Impact Assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved Core Review Criteria Significance (includes commercialization potential & premise) Investigator(s) Innovation Approach (includes analysis of rigor, SABV) Environment Additional Review Criteria Commercial Potential (Phase I) Commercialization Plan (Phase II) Quantitative Milestones (Fast-Track) Review criteria each scored from 1-9 (best-worst) Four Rigor and Transparency Review Elements Can Affect Your Overall Impact Score! Rigor and Transparency Element What s added to the review criteria? Where in the application? 1. Scientific Premise Is there a strong scientific foundation for the project? Research Strategy (Significance) 2. Scientific Rigor Are there strategies to ensure a robust and unbiased approach? Research Strategy (Approach) 7
Four Rigor and Transparency Review Elements Projects with Vertebrate Animals and/or Human Subjects Can Affect Your Overall Impact Score! Rigor and Transparency Element Where in the application? What s added to the review criteria? 3. Consideration of Relevant Biological Variables, Such as Sex Research Strategy (Approach) Are adequate plans to address relevant biological variables, such as sex, included for studies in vertebrate animals or human subjects? Four Rigor and Transparency Review Elements Projects Involving Key Biological and/or Chemical Resources Rigor and Transparency Element Which applications? Where in the application? What s added to the review criteria? 4. Authentication of Key Biological and/or Chemical Resources Project involving key biological and/or chemical resources New Attachment Comment on plans for identifying and ensuring validity of resources. Won t Affect Your Overall Impact Score 8
Commercialization Plan Criteria (Evaluated as Part of the Significance Criterion) 1. Value of project, expected outcomes, societal and educational benefits 2. Company information 3. Market, customer, and competition information 4. Intellectual property protections 5. Finance plan 6. Production and marketing plan 7. Revenue stream generation Overall Impact Scoring Overall Impact: The likelihood for a project to exert a sustained, powerful influence on research field(s) involved Overall Impact High Medium Low Score 1 2 3 4 5 6 7 8 9 Evaluating Overall Impact: Consider the 5 criteria: significance, investigator, innovation, approach, environment (weighted based on reviewer s judgment) e.g. Applications are addressing a problem of high importance/interest in the field. May have some or no weaknesses. 5 is a good medium-impact application, and the entire scale (1-9) should always be considered. e.g. Applications may be addressing a problem of high importance in the field, but weaknesses in the criteria bring down the overall impact to medium. e.g. Applications may be addressing a problem of moderate importance in the field, with some or no weaknesses. e.g. Applications may be addressing a problem of moderate/high importance in the field, but weaknesses in the criteria bring down the overall impact to low. e.g. Applications may be addressing a problem of low or no importance in the field, with some or no weaknesses. 9
Clustering of Review At the Meeting o Phase I and Phase II applications are often discussed in separate clusters. Order of Review o The average of the preliminary Overall Impact score from the assigned reviewers determines the review order o Discussions start with the application with the best average preliminary Overall Impact score Not Discussed Applications o o About half the applications will be discussed Applications unanimously judged by the review committee to be in the lower half are not discussed What Reviewers Look for in Applications Significance and impact (includes commercial potential) Exciting ideas Clarity Ideas they can understand -- Don t assume too much Realistic aims and timelines -- Don t be overly ambitious Brevity with things that everybody knows Noted limitations of the study A clean, well-written application Insider s Guide to Peer Review for Applicants: http://www.csr.nih.gov/applicantresources/insider 10
Common Problems in Applications Lack of new or original ideas No Significance: unimportant problem, unconvincing case for commercial potential or societal impact Inadequate consideration of scientific premise & rigor Absence of an acceptable scientific rationale Questionable reasoning in experimental approach Diffuse, superficial, or unfocused research plan Lack of sufficient experimental detail Failure to consider potential pitfalls and alternatives Lack of knowledge of published relevant work &/or technologies Lack of experience in the essential methodology Unrealistically large amount of work After the Meeting Release scores Prepare and release summary statements Responds to queries from program 11
Your Summary Statement Scores for each review criterion Critiques from assigned reviewers Administrative notes if any If your application is discussed, you also will receive: An overall impact/priority score A summary of review discussion Budget recommendations Benefit from Becoming Review Panel Member Become an Early Career Reviewer (ECR): Increase Your Chances of Getting Your SBIR/STTR Grant Train and educate qualified scientists without significant prior review experience so that they may develop into critical and well trained reviewers Qualifications for the Early Career Reviewer Program Demonstrated training and experience in the scientific areas under review as evidenced by: Two or more years in faculty appointment or the equivalent An active independent program of research or product development At least 2 senior authored research publications in peer reviewed journals in the past 2 years or equivalent accomplishments working in a non-academic setting You cannot have an R01 grant or have served on a CSR study section. 12
How Can I Become an Early Career Reviewer? Apply at www.csr.nih.gov/ecr and follow instructions If you are accepted, we will: Place your name in our ECR database Invite you to serve if your expertise is needed to review particular applications Ask the Right NIH Person for Help Before You Submit Your Application A Program Officer at an NIH Institute or Center Scientific Review Officer After You Submit Your Assigned Scientific Review Officer After Your Review Who Can Answer Your Questions? Your Assigned Program Officer GrantsInfo: GrantsInfo@nih.gov 301 435-0714 13
Key NIH Review and Grants Web Sites NIH Center for Scientific Review http://www.csr.nih.gov NIH Office of Extramural Research http://grants.nih.gov/ 14