ONS Foundation Research Grant REVIEWER ORIENTATION
Assigned Applications Reviewers will identify the applications they would be able to review, based on the match with their content and/or methods expertise, identifying any conflicts of interest. The total number of applications per reviewer will depend on the number of applications received. A Reviewer Proposal Identification Worksheet, will be submitted by each reviewer with the proposals identified for their review. The completed worksheet will be sent to the ONS Foundation Research Department. The chair will finalize all assignments, ensuring that each application has a primary, secondary and collateral reviewer. Assigned applications will be reviewed through a link to the online review area. All components of the assigned applications should be read Major strengths and weaknesses should be identified for each criteria, using the Reviewer Comment-Scoring Form A preliminary overall impact score is to be assigned as well as scores to each of the five core criteria on the Reviewer Comment-Scoring form
Reviewer Comment-Scoring Form A Reviewer Comment-Scoring Form will be used by the three assigned reviewers (primary, secondary and collateral) for each application A form will be emailed with the link to the online review website, the reviewer s passwords and other review materials needed. All reviewers should use this form when reviewing the applications assigned. The completed Reviewer Comment-Scoring Forms are to be emailed to the ONS Foundation Research Department by a designated date. These forms must be submitted on time as they will be combined and emailed to the review team in preparation of the review call.
Preparation of Critique Comments Use bulleted points to make succinct, focused comments Short narratives may occasionally be appropriate, but should be rare Focus on major strengths and weaknesses (ones that impacted your overall rating of the application) Limit text to ¼ page per criterion
Reviewer Comment-Scoring Form Layout Criteria Description of scoring criteria (i.e., significance, investigators, etc.) Score: Strengths: - Bulleted Comments Weaknesses: - Bulleted Comments
Scoring Descriptions Impact Score Descriptor Additional Guidance on Strengths/Weaknesses High Medium Low 1 Exceptional Exceptionally strong with essentially no weaknesses 2 Outstanding Extremely strong with negligible weaknesses 3 Excellent Very strong with only some minor weaknesses 4 Very Good Strong but with numerous minor weaknesses 5 Good Strong but with at least one moderate weakness 6 Satisfactory Some strengths but also some moderate weaknesses 7 Fair Some strengths but with at least one major weakness 8 Marginal A few strengths and a few major weaknesses 9 Poor Very few strengths and numerous major weaknesses Minor Weakness: An easily addressable weakness that does not substantially lessen impact Moderate Weakness: A weakness that lessens impact Major Weakness: A weakness that severely limits impact
Scoring Criteria 1. Significance - Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field? 2. Investigator(s) - Are the PIs, collaborators, and other researchers well suited to the project? If Early Stage Investigators or New Investigators, or in the early stages of independent careers, do they have appropriate experience and training? If established, have they demonstrated an ongoing record of accomplishments that have advanced their field(s)? If the project is collaborative or multi-pd/pi, do the investigators have complementary and integrated expertise; are their leadership approach, governance and organizational structure appropriate for the project?
Scoring Criteria (Continued) 3. Innovation - Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed?
Scoring Criteria (Continued) 4. Approach - Are the overall purpose, aims, strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project? Are potential problems, alternative strategies, and benchmarks for success presented? If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed? If the project involves clinical research, are the plans for 1) protection of human subjects from research risks, and 2) inclusion of minorities and members of both sexes/genders, as well as the inclusion of children, justified in terms of the scientific goals and research strategy proposed?
Scoring Criteria (Continued) 5. Environment - Will the scientific environment in which the work will be done contribute to the probability of success? Are the institutional support, equipment and other physical resources available to the investigators adequate for the project proposed? Will the project benefit from unique features of the scientific environment, subject populations, or collaborative arrangements?
Where criteria are addressed Significance: Project narrative Investigator(s): Biographical sketch(es) Innovation: Environment: Innovation field, project narrative and overall Facilities and resources
Approach Project narrative Protection of human subjects or animals Women and minority inclusion References Timetable Letters of support Instruments Consent form Miscellaneous field
Project Narrative Length: >$25,000 (12 pages) / $25,000 (6 pages) Purpose and specific aims Significance, framework and review of literature Preliminary work Methods and design Design Sample and settings Experimental variables, if applicable Instruments Data collection schedule and procedures Data analysis and interpretation
Additional Review Criteria As applicable, reviewers will consider additional items in the determination of scientific and technical merit, but will not give separate scores for these items. Responses for Protections for Human Subjects and/or Vertebrate Animals are required for all applications A response for Inclusion of Women, Minorities and Children is required for applications proposing Human Subjects Research.
Additional Review Criteria Protection for Human Subjects Protections for Human Subjects Did the application describe how informed consent will be obtained and the steps taken to protect participants rights or the welfare of animals? Did the application identify any potential risks associated with participation in the project? Comments Data and Safety Monitoring Plan (Clinical trials only) Comments -
Additional Review Criteria Inclusion of Women, Minorities and Children Inclusion of Women, Minorities and Children (Applicable Only for Human Subjects Research) Did the application address the inclusion of women, minorities and children in developing a research design appropriate to the scientific objectives of the study. Inclusion is required unless a clear and compelling rationale shows that inclusion is inappropriate with the respect to the health of the subjects or that inclusion is inappropriate for the purpose of the study. Did the application provide information on the composition of the proposed study population in terms of sex/gender and racial/ethnic group and provide a rationale for selection of such subjects in terms of the scientific objectives and proposed study design. Comments -
Additional Review Criteria Vertebrate Animals Vertebrate Animals Did the application address the involvement of live vertebrate animals as part of the scientific assessment according to the following five points: 1) proposed use of the animals, and species, strains, ages, sex, and numbers to be used; 2) justifications for the use of animals and for the appropriateness of the species and numbers proposed; 3) adequacy of veterinary care; 4) procedures for limiting discomfort, distress, pain and injury to that which is unavoidable in the conduct of scientifically sound research including the use of analgesic, anesthetic, and tranquilizing drugs and/or comfortable restraining devices; and 5) methods of euthanasia and reason for selection if not consistent with the AVMA Guidelines on Euthanasia. Comments (Required unless not applicable)
Additional Review Criteria Budget and Period of Support Budget and Period of Support Is the budget and the requested period of support fully justified and reasonable in relation to the proposed research. For more details, please see Budget Information on the application. Recommended budget modifications or possible overlap identification -
Overall Impact Reviewers will provide an overall impact score to reflect their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) including consideration of the following five scored review criteria, and additional review criteria. An application does not need to be strong in all categories to be judged likely to have major scientific impact. Proposal #: Principal Investigator: Reviewers will provide an overall impact score to reflect their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved in consideration of the following five scored review criteria, and additional review criteria. An application does not need to be strong in all categories to be judged likely to have major scientific impact. Overall Impact - After considering all of the review criteria, briefly summarize the significant strengths and weaknesses of the application and state the likelihood of the project to exert a sustained powerful influence on the field. Preliminary Score: Final Score: Strengths Weaknesses -
Overall Impact Scoring Impact Score Descriptor Additional Guidance on Strengths/Weaknesses High Medium Low 1 Exceptional Exceptionally strong with essentially no weaknesses 2 Outstanding Extremely strong with negligible weaknesses 3 Excellent Very strong with only some minor weaknesses 4 Very Good Strong but with numerous minor weaknesses 5 Good Strong but with at least one moderate weakness 6 Satisfactory Some strengths but also some moderate weaknesses 7 Fair Some strengths but with at least one major weakness 8 Marginal A few strengths and a few major weaknesses 9 Poor Very few strengths and numerous major weaknesses Minor Weakness: An easily addressable weakness that does not substantially lessen impact Moderate Weakness: A weakness that lessens impact Major Weakness: A weakness that severely limits impact
Additional Comments to Applicant Reviewers may provide guidance to the applicant or recommend against resubmission without fundamental revision. Additional Comments to Applicant (Optional) Please provide any additional guidance to the application or recommend against resubmission without fundamental revision
Online Overall Impact Scores My Application Score 0 / 90.0 My Ratings Overall Impact Score 1 = Exceptional (High impact) 2 = Outstanding (High impact) 3 = Excellent (High impact) 4 = Very Good (Medium impact) 5 = Good (Medium impact) 6 = Satisfactory (Medium impact) 7 = Fair (Low impact) 8 = Marginal (Low impact) 9 = Poor (Low impact) 1 2 3 4 5 6 7 8 9 O O O O O O O O O After completing your review of each application you will need to enter a preliminary impact score in the online review system by clicking on one of the numerical ratings at the bottom of each application reviewed Once you have selected your preliminary impact score, click Submit Ratings to finalize your score
Online Comment Area Primary Reviewers Only Comments There are currently no comments. Add A Comment If you are the primary reviewer on an application, you will need to copy and paste a brief summary of the application (Do not include any evaluative or critique comments) into the Comment box and then click Submit Comment The summary will then appear above the comment box with your name and the date submitted *Note that clicking the "Submit Comment" button will not submit your ratings as they are two separate forms. This area of the online review system is at the bottom of each application and will be used to display ONLY the primary reviewer s summary of each application No other comments are to be entered in this area
Prior to Review Conference Call Completed worksheets are to be emailed to research@onsfoundation.org by the designated due date Primary, secondary and collateral comment-scoring forms are combined by the ONS Foundation Research Department for each application The combined forms for all applications are emailed to all reviewers several days before the call for your preparation for the call Primary, secondary and collateral preliminary impact scores are averaged and the Preliminary Mean Scores are ranked and emailed to all reviewers The Grant Review Team Chair contacts any reviewers with variant scores and determines any proposals that are non-competitive and do not need to be discussed
ONS Foundation Research Grant Program ONLINE Review Instructions
Logging In with the Temporary Password Provided Access the website (https://admin.closerware.net/gm_onsf/ page.jsp?pagename=evaluatorarea ) Enter the username provided in the email Enter the password provided in the email (please do not change the password provided) Click Login
My Applications Evaluator Extranet My Applications Applications Pending Your Review Research Grants Program (RE##): Type of Grant Applicant Name (Username) Applicant Name (Username) Applicant Name (Username) Pending Applications Approved Applications Applications you have reviewed All of the applications you have been assigned to review will appear below the Applications Pending your Review area Click on the applicant s name to view the application Applications will remain in this area until a score is assigned Admin Copyright 2003-2010 Closerware, LLC. All rights reserved.
Pending Applications Evaluator Extranet My Applications Application Label Applicant Name (Username) Pending Applications Approved Applications Grant: Grant Opportunity Research Grants Program (RE##): Type of Grant Admin Copyright 2003-2010 Closerware, LLC. All rights reserved. As you complete your review of each application and the score is assigned, the application will move to the Pending Applications area To view these applications again, simply click on the applicant name in that area to access that application again, as needed
Grant Application Main > Grant Application Accessing Applications Evaluator Extranet * Administrative Index * Grant Opportunity Main * Grant Application Main * Grant Application Grant Application Created: 9/6/10 5:29:36 PM EDT Applicant Name (Username) Grant Opportunity: Research Grants Program (RE##): Type of Grant User Queue: #1 Application Status: Pending Research Grants: Research Team Research Team: Name Credentials Institution Res Team Role Name Credentials Institution PI Name Credentials Institution Co-investigator The applicant name and application number will appear in the blue area followed by: Type of Grant User Queue Number Application Status (Pending) Research Team
Conflict of Interest Review Check the names shown in the Research Team area of the Applicant Login Worksheet to see if you have any conflicts of interest with anyone on the team or their institution If you identify anyone or any institution that you may be perceived as having or have a conflict with, notify research@onsfoundation.org immediately
Abstract: Project Narrative: Reference List: Timetable: Uploaded Documents Abstract.pdf Project Narrative.doc Reference List.doc Timetable.pdf Human Protection Education Certificates of Completion: Certificates.pdf Support Letters: Letters of Support.pdf Biographical Sketches: Biosketches.pdf Instruments: Itemized Budget: Budget Justification: Instruments.pdf Budget Worksheet.doc Justification.doc Documents uploaded by the applicant will appear as links which can be opened and either reviewed online or printed The uploads will be in PDF formats The system will not allow the applicant to upload more than one document in any area (i.e., letters of support, biosketches, instruments, etc.) so the applicants have been asked to scan the multiple documents into one document and upload in a PDF format.
Resubmission / Special Grants Is this application a resubmission from a previous ONS Foundation research grant cycle: Year of Grant Type of previous grant Resubmission from a previously non-funded ONS Foundation research grant cycle - year and type of grant will be shown Resubmission Cover Letter-uploaded The previous reviewer s critiques will have been added to the end of the resubmission cover letter by the Research Department. Special Grants ($25,000 or less Only) Applicant will indicate if they are applying for one of the ONS Foundation special grants Special Grants ($25,000 or less Only): Neuro-Oncology Research Grant ONS/STTI Research Grant Trish Greene Pain Research Grant Certification, Education and Outcomes Grant
Project Title and PI Information Title of Project: PI First Name: PI Last Name: PI Credentials: PI Title: PI Institution: Following the project title is the principal investigator s name, title and institution Review both the research team members (at the top of the application page) as well as the PI information to be sure you do not have a conflict of interest with this applicant
IRB Submission Human Subjects?: Yes / No / Pending Animal Subjects?: Yes / No / Pending IRB to be submitted upon notification of funding?: If human or animal subjects will be included, the applicant must submit to their IRB for approval prior to their application submission They will indicate if it is pending or approved and proof of submission or approval will be uploaded as a Word or PDF document Some institutions do not allow IRB submission until notification of funding is received and the applicant will indicate this and upload their institution s policy on this
There are currently no comments. Protection of Human Subjects and Inclusion of Women and Minorities Protection of Human Subjects or Animals Used for Research: How informed consent will be obtained Steps taken to protect participant s rights or the welfare of animals Potential risks associated with participation in the project Women and Minority Inclusion in Clinical Research: Information on the composition of the proposed population in terms of sex/gender and racial/ethnic group Rationale for selection of subjects in terms of scientific objectives and design The applicant is asked to submit 1-2 paragraphs on the following areas Protection of human subjects or animals used for research Women and minority inclusion in clinical research
Innovation and Facilities and Resources Innovation How the project challenges existing paradigms or clinical practice How the project addresses an innovative hypothesis or critical barrier to progress in the field If applicable, how it develops or employs novel concepts, approaches, methodologies, tools or technologies in the area Facilities and Resources (Environment): Facilities and resources available to carry out the project at all research sites such as: Computers Statistical and data management support, Office space Equipment, etc. The applicant is asked to submit 1-2 paragraphs on the following areas Innovation Facilities and Resources
Implications for Practice Implications for Practice and Research: Implications for oncology nursing practice Identify future research that may develop from this project Describe how this project will provide the groundwork for seeking additional funding in the future The applicant is asked to submit 1-2 paragraphs on the Implications for Practice and Research Describe when and how the study findings will be disseminated
Review Call Discussion 1) Primary, secondary, and collateral reviewers will begin by stating their preliminary impact scores for the proposal. 2) Primary reviewer provides a brief summary of the project (included in the comment section of the online review page). 3) Primary reviewer presents a brief 2 3 minute critique highlighting strengths and weaknesses in each criteria. 4) Next, Secondary Reviewer adds additional strengths and weaknesses. 5) Collateral reviewer follows with any additional comments not previously made.
Review Call Discussion 6) After the critiques have been presented, the review team will discuss the proposal for up to 2 3 minutes. 7) Primary, secondary and collateral reviewers are then asked if their preliminary impact scores stand as submitted 8) All remaining reviewers will assign a final Overall Impact Score to the application in private on the online application review page. 9) Any concerns about the budget are discussed at this time.
Comment Revisions You may have changes to your Reviewer Comment- Scoring form after the conference call. Please make these changes and e-mail the revised form to research@onsfoundation.org within 48 hours of the conference call. If you do not have changes to your critiques, please e-mail the ONS Foundation Research Department to let them know. The chair or co-chair will read the critiques. You may be contacted if there are any questions.
Summary Statements Overall impact scores of applications will be the mean of scores entered by all eligible reviewers, multiplied by 10 Final scores will range from 10-90, in whole numbers The primary reviewer s summary for ALL applications will include the criterion scores and critiques posted by assigned reviewers
Recipient Selection After the review all of the proposals, the group s average scores will be rank ordered and used to guide the assignment of grant awards. The ONS Foundation Executive Director, the ONS Research Director and the review team chair and co-chair will identify the recommended impact score funding range and the applications within that range. The group s recommendations are presented to the ONS Foundation Board of Trustees for their approval. Once the award and non-award letters are mailed, the list of funded proposals will be emailed to you.
THANK YOU! Questions? Please e-mail research@onsfoundation.org with any questions during the review process