Grant Writing Workshop and Practicum Stan Matwin, Paola Flocchini, Eric Dubois Faculty of Engineering Universit[é y] [d of] Ottawa Summer-Fall 2010
Acknowledgement Many thanks to Professor Evangelos Milios, Dalhousie University, for the permission to use some of the slides from his presentation How To Succeed In The New NSERC Discovery Grant Competition Model Professors Pierre Berini and Murat Saatcioglu for the permission to use their DG applications as a model 2
Some stats - 2010 Early career researchers (ECR): success rate 58%, avg. grant $24K Established career researchers: success rate 72%, avg. grant $36K; new applicants (includes RU): 29%! Civil, Industrial, Systems Engineering has the lowest Eng success rate 52% uottawa success rate = 55% (less than Montreal, Alberta, UBC, Toronto, McGill, Western) Faculty of Engineering ECR success rate = 20% 3
Much more at Much more at http://www.nserc-crsng.gc.ca/_doc/professors-professeurs/2010-dg-compstat_e.pdf 4
Who, why, what Group of professors, experienced with grant evaluation and selection (NSERC) Our experience indicates that oftentimes applicants could significantly improve their chances by Understanding how the system has changed and how selection process works Applying a set of simple principles of grantsmanship Sharing this knowledge and experience with colleagues, and working one-on-one with a group of applicants between now and the NSERC submission time 5
Plan Recent changes in the NSERC Discovery Grants selection process: the conference model Description of the grant evaluation process in the DG conference model Detailed discussion of do s and don t s in the context of The form 180 The form 100 The form 101 Models: an excellent and a not-so-good application Research tools and instruments (RTI) applications Comments on team grants, adjuncts and special cases Practicum Q&A and evaluation 6
The conference model Groups of reviewers meet to combine expertise necessary to evaluate a given application (like sessions in a conference). They can go across disciplines (e.g. EE and physics) Proposal is evaluated on Excellence of Researcher Merit of Proposal Highly Qualified Personnel all three weighed equally Combined into a score The score determines grant amount Memory-less system 7
Timeline of evaluation August 1: Submission of Form 180 (Proposal Intent) September/Early October: Internal Reviewers are picked External referees are selected November 1: November/December: February: Submission of Application Package (Form 101,100, pubs) Readers are picked Candidate s application package is sent to external referees and committee readers Grants Competition s1 March/April: Announcement of Results 8
Slide 8 s1 With the conference model, the readers are only selected in November, at the time of scheduling. sbo, 5/19/2010
Evaluation Process Your application will be reviewed by: 1 st Internal Reviewer 2 nd EG members Internal Reviewer 3 Readers (was 4) External Referees (5 requests, a mix of applicant and committee choices, typically 2-3 referee reports received) 1 st /2 nd /readers chosen based on comfort ratings submitted by each committee member in the fall 9
Evaluation Process September/ Early October: Each application is assigned by the Chair to a 1 st /2 nd internal reviewer (readers are selected later, in November) Conflicts taken into consideration 1 st internal selects external reviewers From mid-december to early February, each member reviews 20-30 applications --> as 1st or 2nd Internal 30-40 applications --> as Reader 20-25 RTI (equipment) applications External Referee reports trickle in during January, incorporated into member evaluations 10
Rating of proposals Each proposal is rated on: - Excellence of Researcher - Merit of the proposal -HQP Each aspect receives one of the following six ratings by choosing the median among five ratings from the reviewers: Exceptional (E) Strong (S) Outstanding (O) Moderate (M) Very strong (V) Insufficient (I) Example: V S S M M S The three ratings are added, proposal placed into one of 16 bins, e.g. VSM in same bin as SSS. 11
Excellence of the researcher Acknowledged leader E O VS S M I Influential, exceptional, to a broad community Outstanding, to a broad community Very strong Quality, impact and/or importance of accomplishments Comparable Moderate Lower 12
Merit of the Proposal E O VS S M I Research program, impact Clearly presented, extremely original and innovative, leading to groundbreaking advances or technology/ policy Clearly presented, highly original and innovative, contributing to groundbreaking advances or technology/ policy Clearly presented, original and innovative, leading to advances Clearly presented, original and innovative, likely to have impact Clearly presented, original and innovative aspects, could have impact Lacks clarity, limited originality and innovation, impact not convincing -ly described. Objectives Long-term vision and short-term objectives clearly defined Long-term goals are clearly defined and short-term objectives are well planned Long-term goals are defined and short-term objectives are planned Long-term goals and short-term objectives are clearly described Long-term objectives and shortterm objectives are described Objectives are not clearly described and/or likely not attainable 13
Merit of the Proposal (2) E O VS S M I Method -ology Clearly defined and appropriate Clearly described and appropriate Clearly described and appropriate Described and appropriate Partially described and appropriate Not clearly described and/or appropriate Budget Clearly demonstrates how the research activities to be supported are distinct from and complement other sources Clearly demonstrates how the research activities to be supported are distinct from and complement other sources Demonstrates how the research activities to be supported are distinct from and complement other sources Demonstrates how the research activities to be supported are distinct from and complement other sources Demonstrates how the research activities to be supported are distinct from and complement other sources Does not demonstrate how the research activities to be supported are distinct from and complement other sources 14
Training of HQP E O VS S M I Level Exceptional, HQP contributing to high-quality research Outstanding, HQP contributing to quality research Very strong, HQP contributing to quality research Comparable Moderate Low Positions Most move on to positions that require highly desired skills obtained through training received Most move on to positions that require highly desired skills obtained through training received Many move on to positions that require desired skills obtained through training received Many move on to positions that require desired skills obtained through training received Some move on to positions that require desired skills obtained through training received 15
Training of HQP (2) E O VS S M I Research plans Appropriate and clearly defined. Appropriate and clearly defined. Appropriate and clearly described. Appropriate and described. Described Insufficiently described HQP success Highly likely Highly likely Likely Likely Maybe. Limited information to predict. 16
17
How does the Committee deliberate? 1 st Internal presents his/her ratings on applicant/proposal/hqp with detailed rationale 2 nd Internal adds to the summary, and own ratings Each reader adds further comments, and own ratings Discussion among the five reviewers follows. (Electronic voting - secret) on each rating one of E, VS, S, M, I (applicant/proposal/hqp) Program officer announces auction outcome All of the above take at most 15 minutes All ratings of moderate or insufficient receive comments from the committee, reflecting the consensus of the reviewers (highly focused) 18
Lack of Memory Current (new) system Previous grant amount not discussed Applications grouped by topic Previous system Minibudget for each member, based on: incoming grants of returning applicants an allocation for new applicants Grant amounts were explicitly discussed and adjusted up or down Applications discussed in this order: new/senior new/renewals by increasing previous grant amount 19
Funding decisions Subcommittee consisting of current chairs, with the assistance of Program Officers, decides on: Exact grant amounts per bin Handling of early career applicants (e.g. fund a lower bin than normal applicants) Decision based on: Available budget Desired success rate range for early career and normal applicants Other considerations (e.g. whether to differentiate between proposals in the same bin) 20
Discovery Accelerator Supplements (DAS) - Provide timely resources to excellent researchers who have a well-established research program, and who show strong potential to become international leaders. - $120,000 over three years - The Evaluation Groups nominate candidates during the deliberations. A ranked list is produced by the end of the week. The executive committee ranks and rates nominees after the competition. 21
Your form 180 Think carefully about the keywords in your abstracts, they determine who will review your proposal Think carefully about five potential referees Ask advice (collegaues, former supervisors, etc) Identify referees who will be recognizable to the reviewers (at least their institutions) Will be willing to supply a referee report if approached by NSERC in the requisite timeframe 22
Your form 100: the goal Demonstrated expertise in the field Quality and impact of research accomplishments Publication record in high impact journals and conferences Industry impact - patents and collaborations Continued progress Most significant research contributions (important!) Stature in the field Invited lectures Review articles Program committees and service involvement 23
Your form 100 Make sure your HQP statistics on p. 1 are consistent with the HQP details on the HQP part of form 100 Work on the Most Important Contributions: Not limited to six years Be positive about them, point to the impact. Do not limite yourself to papers, if you have other accomplishments 24
An exceptional (top bin) researcher Strong research record (most important) Several of the following: Interest in applications Professionally active Journal editor Conference organization NSERC committees Significant research contributions in the last 6 years, or prior to this but with continuing impact Strong HQP record 25
Assessing strength of publications Strong publications are those that are known to and used by the peer community, for example: In journals that are recognized by peers as competitive and widely read. In conferences with competitive acceptance Used by industry Used by the peer community as open source software The onus is on the applicant to make a case for the strength of his/her publications. 26
Quality Metrics NSERC discourages the look-up of impact factors / citation indices / numbers of citations by committee members. If they are brought in by the applicant or external referees, they will be scrutinized by the committee. 27
External referee reports NSERC made a successful effort to increase the response rate of external referees (4 and 5 reports not rare, one even got 6!) Quality of external reports varies widely Committee members are supposed to only judge based on the content of the proposal and the external referee reports, not look up the Web, except to confirm the validity of information provided External referee reports very important, especially for the members who are not experts in the area of the proposal 28
A moderate researcher Many papers in unknown conferences and journals (quantity instead of quality) Research lacks focus (too diverse to be credible) Publication output insufficient in terms of significance 29
Examples of comments to M/I researchers Publications not in high-impact venues Referee points out that publications have had limited impact. Applicant did not take advantage of available space to explain the significance of his/her research record 30
3. HQP Past record Joint publications with students (roles explained) Students moving on to positions requiring the training they received (mention current employment) Evidence of impact of students (e.g. startup companies) HQP training potential Role of future students explained well in the proposal Thesis topics defined 31
More on HQP List all HQP trained PhD, Masters, Postdocs, undergraduates, research assistants, technicians Just a list of names is not enough, nature of contributions must be explained, especially for undergraduates (e.g. USRA) 32
Examples of comments to M/I HQP Too few students supervised Has not published with students Lacking plan of how students will engage in the proposed research Applicant should strengthen number and quality of HQP Very few graduate students trained (in a school with strong graduate program) 33
Relation to other support If you have other grant support, explain: how this research is different and how it ties in to the other research Example: An NSERC CRD supports applied research in X of interest to industrial partners A CANARIE contract supports software development of a system incorporating contributions in X The DG supports basic research and conceptual innovation in X 34
Form 100 do s and don t s Be careful when listing your publications: 6 yr rule Do not mix journal and conference papers, follow NSERC guidelines Do not mix publications submitted (or in preparation) with accepted or published Inlude page numbers, or number of pages Bold names of graduate students/hqp who are coauthors Include, if available, evidence of impact of given publications (e.g. Citation count, most downloaded paper, Best paper) and calibration of the venue 35
Your form 101 - presentation Strike a balance between Explaining the proposal to someone who is not ncecessarily expert in your exact research area, but knowledgeable in areas adjacent to yours Convincing the reader that you know, at the advanced level, what to do, and how to do it: Provide details to satisfy the expert; convince the non-expert about impact and importance - from NSERC web site Try to convey a sense of excitement Cite recent literature, try to be complete 36
Your form 101 - presentation Argue significance of the proposed research Remember, it s a proposal for a five year research program, not a research project make sure the reader has a sense of focus, be realistic, do not try to propose too much If empirical research, remeber to discuss evaluation of results Map into HQP Avoid unncecessary buzzwords 37
Your form 101 - structure Specify the overall objective in a nutshell Specify short-term and long-term goals Indicate why the research is important --> context! Provide up to date literature review Describe the methodology in detail Outline a research plan and milestones Conclude with impact and outcome Use space effectively avoid duplication Describe training plans clearly Address EG comments (if resubmission) 38
An exceptional proposal Fundamental theory or system or application Coherent and focused research direction Clear evaluation plan Essence of proposal explained in intuitive terms, and theory and applications nicely weaved into it Gets to the objectives within the first couple of paragraphs Maximum 1 page of highly technical stuff Why is the proposed work significant? 39
An insufficient proposal Vague goals Lack of focus: too many distinct subproblems Claims to attack unrealistic sized problems Confusing to read, overuse of acronyms Lack of evaluation strategy Unclear that applicant has the expertise to do the proposed research 40
Budget Budget is one of the factors in evaluating the proposal Even though applicants get less than what they ask for, a detailed and well-justified budget is a plus! Identify all personnel and their salaries (funding/student/year) Identify all equipment and their costs Specify travel costs (which conference, where?) Does the budget match the proposed work? An inflated/ill-justified budget could result in a lower rating for the proposal 41
Examples of comments to M/I proposals Lack of an evaluation methodology The applicant s prior research record does not include contributions in the area of the proposal Literature review did not include significant relevant work, e.g. Did not discuss how the proposed research will advance the state of the art Proposal did not have clear objectives, hence feasibility is questioned 42
Examples of comments to M/I proposals The applicant has not provided sufficient technical detail to allow the committee to assess the feasibility of their proposal Methodology was too general, making it hard to see how the potential contribution will generalize Methodology too sketchy. Not clear how the proposal will compete with established methods Scope of proposed research too broad Applicant does not have the experience needed to carry out the proposed research 43
Example applications An excellent application from an experienced researcher Fictitious Insufficient application, made up to illustrate some of the common problems 44
Adjuncts Funding for students only Make sure to separate the project applied for from the projects at work Address HQP 45
Research Tools and Instruments Every proposal gets 5 readers, one lead Committee member evaluates ~20 applications, rates 1-10 A flat distribution is enforced Three bins are determined In the meeting, application in the middle bin are presented by the lead, discussed. Some applications from top and bottom bin can be flagged for discussion. Members revise (or not) their ratings, still with flat distributions Rating are added, applications are ranked by ratings, cumulative budget and total RTI budget determines who is successful 46
Research Tools and Instruments Apply for equipment necessary for a research programme, not for infrastructure Tie the equipment to the programme Justify need If possible, present two (or more) configurations at different price levels: Ideal Sufficient Equipment shared by several researchers and used by students has better chance 47
Appeals Only on grounds of procedure: Wrong committee Committee ignored something in the application Cannot be based on disagreemnt with opinions of the reviewers or the referees 48
Practicum One-on-one work with a limited number of people on their applications Priority to ECR and RU Staggered over July and August Those interested please contact Margaret McKenna before June 15. 49
Conclusion Allocate sufficient time to all three parts of the application Make sure that your proposal is read by a colleague Start early 50
51