2016 NSF CAREER Proposal Writing Workshop March 21-22, 2016 St Louis, MO Post Workshop Evaluation - Initial Report 1. How do you feel about what you have learned from this workshop? # Answer Response % 1 Very unsatisfactory 0 0% 2 1 1% 3 2 1% 4 28 19% 5 Very satisfactory 116 79% Total 147 100% Statistic Value Min Value 2 Max Value 5 Mean 4.76 Variance 0.25 Standard Deviation 0.50 Total Responses 147 2. Will you recommend this workshop to other future CAREER proposal submitters? # Answer Response % 1 Definitely no 0 0% 2 1 1% 3 2 1% 4 14 10% 5 Definitely yes 130 88% Total 147 100% Statistic Value Min Value 2 Max Value 5 Mean 4.86 Variance 0.19 Standard Deviation 0.43 Total Responses 147 3. What is your overall rating for this workshop? # Answer Response % 1 Poor 0 0% 2 1 1% 3 2 1% 4 36 24% 5 Excellent 108 73% Total 147 100%
Statistic Value Min Value 2 Max Value 5 Mean 4.71 Variance 0.30 Standard Deviation 0.54 Total Responses 147
Number of votes Number of votes 2. Will you recommend this workshop to other future CAREER proposal submitters? 150 130 120 90 60 30 0 1 2 1 2 3 4 5 14 Definitely no <------------------------------------------> Definitely yes 3. What is your overall rating for this workshop? 150 120 108 90 60 30 0 1 2 1 2 3 4 5 36 Poor <-------------------------------------------------------> Excellent
4. What activities in the workshop are the most helpful? Text Responses Expert and peer review of project summaries Mock panel review; summary review with group The breakout session discussing our project summaries Mock panel review; hearing from NSF PD. The panel review experience. (Day1); The summary review with panel (Day 2). In general it is a very good workshop. All sessions. Panel reviews specific to CAREER proposal are great; NSF director presentation about CAREER is also very helpful. George's lecture. George's talk. Breakout sessions; replication of panel evals. Review of our summary in group and the initial presentation. Also the conversations with NSF PMs. Mock proposals. Presentations of recent CAREER awardees; Mock reviews; Reading our summaries and getting feedback from moderator. Discussion of the summary. The presentations; The mock proposals review panels; Presentation by recent CAREER awardee. PD's presentation and mock review. George s presentation; Mock proposals; summary peer review feedback. Reading proposals and discussing with PM. Awardees' presentation; George's presentation on what CAREER is; Presentations of CAREER proposal tips at the beginning; review project summary within each panel. Editing my proposal and critiquing was supper useful. Mock panel reviews of the proposals. Talking to program directors informally. All activities were helpful. Talks by NSF directors. Mock panel reviews. All are helpful! Panel reviews; Interacting with PDs; Presentations by NSF PD and CAREER proposal awardees. Talks given by NSF directors. Mock panel discussion of my project summary. George's presentation on key points; Mock panel reviewers; Homework, feedback on project summary. George's presentation thought I also find it very helpful to hear perspectives from the PM who facilitated our panel review; Proposal summary critique. Mock review panels, advices from awardees, and exposure to NSF PMS were very helpful. George's presentation, chatting with PDs. Breakout sessions and individual discussions George's presentation. The panel work was helpful, but none of us had any knowledge of the research area of the others on the panel.
Mock panel. Discussions of summary statements. George's presentation. CAREER awardee presentations. George's presentation, panel review and comparing and listening other thoughts from different background and expertise. The strategy of having a life (research) plan before starting thinking about a CAREER proposal. Discussion of a recommended and very direct approves to.that this is a "career" proposal not a research proposal. Mock panels were also useful. George's presentation Review by colleagues of proposal summary and also feedback from program officers on the same. Giving feedback for my summary and discussing the summary of other folks in the group. Discussing previous proposals. Panel discussion on the proposal review and summary review. George's presentation Revision of modified project summaries from each participant in the panel. George's presentation, Mock panel, talks by prior career awardees. The panel review process. Mock panel review and panel review of our own project summaries. Mock panel, homework and panel feedback about proposal summary. Mock panels, summary reviews. The presentation by Dr. Hazelrigg is very helpful to understand the procedure, dos, don'ts for NSF CAREER projects, the panel review is very helpful in understanding how the review process goes and how proposal is evaluated from different perspectives. Work in groups. Mock panel process and the presentation from funded PIs. George's presentation, NSF CAREER awardees presentation, Mock panel Panel George's presentation, mock panel review--> specifically getting 6 prior proposals to read. Review of panel's own proposal Review proposals and revise summary page. Successful NSF CAREER presentation. George's presentation Panel discussion Project summaries discussion, presentation by George, George's presentation. It redefined my understanding of a career award proposal. Proposal and project summaries reviews, Feedback session. The 1st session for NSF program General by George. Mock panel, review of other's summaries; their reviews of mine Morning 'top level' lecture. Mock panels. The panel review process of proposals and our project summary. The panel discussions, talking with PMs and George's presentation. The feedback on our revised projects summaries was very helpful. George's presentation, mock panel, direct feedback on project summary by attendees and PD. Mock panel reviews and panel discussion, George's presentation.
Defining objectives and hypothesis. Review of personal summary, mock panel review, George's presentation. George's presentation, specifically formulation. Mock review Reviewing proposals. Discussing project summaries George's presentation Panel Reviewing each other's proposals. Presentation of previous awardees. Interaction with program managers. I enjoyed all activities, as well as the way they were structured and their sequence. Grant writing presentation, panel review, summary editing. Mock panel workshop for reviewing grant. Presentation from NSF PD feedback session. Panel summary, mock-up reviews. Mock review and panel discussion of summary revision. Mock proposal review and panel discussions. All sessions Review panel and project summary review. The lecture from George was also remarkable! George's presentation. Discussion of summary Everything. Very useful and engaging workshop. Thank you! Mock panel Panels The first day presentation by George and panel mock review with Z.J. Pei. George's presentation. Panel discussion about previous proposals. George's presentation. Presentations by recent awardees, review revised projected summaries on Day 2. George's presentation. I learned a lot about CAREER specific proposal writing in comparison to the regular NSF proposals. All the sessions Peer review of my own project summary, reviewing other proposals before the workshop, then seeing how my review changes after the mock panel. Panel summaries; George's presentation; networking The panel with individual feedback and comments along with a program director and seeing the proposals in a new light following the great morning presentation. The mock panel review is very helpful. George's presentation is impressive and very informative. Reviewing abstract from CAREER proposals. The mock panel Most helpful: discussion and feedback on our own project summaries. Mock proposal review George's presentation. Mock panel review. Panel discussion on revised project summary. Reading other proposals. George's presentation. Mock panel and revision of individual own summaries
Career Awardees' presentation; George's presentation; Project summary session. The Monday morning presentation was the most helpful to me. All activities were helpful. Project summary review and presentation George's presentation. Getting the chance to read proposals from outside my field had a significant impact on how to write my proposal. Lecture from George is extremely helpful. Panel review of proposals is also informative. Feedback on my own project summary is beneficial. George's presentation. Previous NSF CAREER awardees' presentations. The interactive panel discussions about previous proposals. The presentation of the past awardees. The discussion on the proposals. George's presentation. The mock review of past proposals was the best. George's presentation. Mock review panel. Mock panel In the order of helpfulness. 1. George's presentation. 2. Review of mock proposals. 3. review of own project summaries. 4. Presentation for previous career awardees. George's presentation. The panels were very helpful. Two recent CAREER awardees presentation. NSF program director NSF CAREER proposal guideline presentation. Everything was very good. If there was a way to pair about the young faculty with a PM they want to talk to during lunch or dinner would be helpful. Review and comments on revised summary. Presentation on general aspect of CAREER grant. I think it was a good balance of presentations, working together on the mock proposal panels and discussing project summaries. Having a smaller group panel and one on one time was very helpful. The mock panel was the best by far. Review of our own summary is very good and helpful. Meeting new people outside my area was a good bonus. Panel activities and discussions. In the order of usefulness. 1. Break panel sessions! 2. Working meals. 3. Stories from previous winners. 4. George talking for 3 hours. NSF CAREER proposal guideline presentation More review and presentation Mock panel reviews The panel summary discussions were very helpful. Review professor/ panel discussion I felt like George's presentation was very helpful to me. The mock panel was great as well. Presentations and mock panel Review of previous proposal and reworking you own project summary. Panel review, overview presentation on guidelines.
The mock panel review was the most useful to me. Similarly, the discussion on proposal summary was excellent. I really liked the additional comments/insights make by the PD (George) time to time on the proposal summary. George's presentation interactions with PD, networking with colleagues. Honestly all of it. This is one of the only workshop I've attend where everything was useful. The timeline was also good. Panel reviews. George's presentation. Presentation by CAREER recipients one just being able to interact with NSF personnel. The Panel review activities. The presentation. In fact, everything was perfect! Face to face access with PD and NSF officials is by far the most valuable. George's presentations are phenomenal but could also be video-taped and pre-screened by attendees. The feedback session (review/edits on our career summaries) was extremely helpful! The discussion sections was very helpful. The feedback from the peer helped me a lot to understand how to write a good proposal what to expect from the reviewers. Mock panels and comments on the revised project summaries. Presentations regarding writing a successful CAREER proposal. Seeing the successful and unsuccessful proposals. The presentation by George. The revised summary discussion. The mock panel. Statistic Value Total responses 147
5. What activities in the workshop are not helpful? Text Responses Mock panel was only guesswork based on previous lecture rather than real expertise. Not surprising results. Not interesting or insightful process. Non engineering seemed very out of place and different. Prospective from award winners The food was bad. The meals were tasteless and sparse. The first presentation was very specific and explicit a/b not using words like "design" and "develop". But these are words successful career awardees and words my NSF PD supports. Everything was helpful. The working dinner and lunch could be praised to insure actual interaction between program managers and directors and attendees. In the lunch P.M. were clustered and some attendees did not benefited from the potentials interactions. Not sure if I did get much from the two presenters that their proposals were reviewed for the conference. Awardee talks. The mock panel was less useful to me but I have served on two panels. It was very good, Time for lunch and breakfast was long. It could be better if the recent awardees or NSF members could be introduced in the front or had name tags with a different color so that it was easier to find during lunch or dinner. Also if they could talk about their programs that could be helpful. The irrelative proposals. Reviewing proposals way different from our expertise. Experience previous awardees. The presentations and Q&A from previous awardees. Mock review panels were difficult for me because the area was not close to mine at all. The mock panel was OK, but I think more time on critiquing would be better. Maybe less time on overview of NSF. Former CAREER winners All are helpful! The mock panel results are different from the real panel results. I wish we could know which mock proposals got awarded right before or right after the mock panel discussion. If the mock panel have 15 mins to discuss why they were awarded, I would have learnt more from the mock panel. All are helpful! George's presentation makes research seem prescribed. Would be good to hear other opinions. Base information could be shortened, such as submitting in time, font sizes, etc. The project summary panel was very informal and was helpful only because our facilitator was very knowledgeable. Talks by previous CAREER award recipients. It just repeated what are have already known. Hot breakfast please. I found the mock panels to be less useful to me personally as I have already served on a panel at NSF. It reinforced some of those lessons learned, however.
... review of submitted proposals First 2 hours of going over GPG. Everything is very helpful. Mock review too many. 1-3 would be enough. All good. Individual experience are good but not necessarily helpful in that they cannot cover a wide range of scenarios because they are every specific with their own experience. I have no complaints. I didn't learn very much from the previous award winner talks, most was repeated. Panel reviews were not as smooth as could be. Many people not manufacturing. All are helpful! Panel and previous award winners' talk.. N/A Everything was helpful. Previous awardees. Welcome party Every activity has its purpose. Mock review for faculty in their final stage of their tenure-track position Presentation N/A The peer review of our summaries was helpful, but less so than the mock reviews. Maybe assign NSF staff to tables to avoid empty tables during meals. Nothing particular, but perhaps a more spacious hotel conferences room would be good. none They were all useful and interesting. I feel like the topics covered in presentation from past awardees was already covered extensively in previous presentations and mock proposal review. No
The work lunch does not seem to add much to the workshop. Could try to arrange tables and seats and define a theme for discussion over the lunch. Would be helpful to tell us to re-read all 6 proposal the night before for the mock review panel.. I understand why it is done this way, but the first day was very long. I only wanted some more time to "fix" my summary since I had lots of thinking on my part. (this is all still helpful) George's talk is great but it could be given as a podcast (youtube video) before the conference to give more time to other activities. Lengthy presentation on the structure of NSF and the GPG (also from George). I know this is important but it is covered at every single NSF workshop. N/A not really The food was a challenge. I requested gluten-free food but got sick after eating it. CAREER awardee. The speeches from past winners were good and reinforced concepts, but least essential. They are all good, Maybe not give homework after an intensive first day. N/A The presentations are a bit long, adding more hands on session and group discussions will be great., we could have worked on less proposals. I thought the panel discussions could have been a bit shorter to add a bit more formal presentations. N/A N/A N/A noted. The food section is too far from the discussion room! The presentations by previous CAREER recipients was very repetitive from the main presentation, but specific insights would be useful or additional comments on the process maybe more beneficial. Statistic Value Total responses 92
6. What activities would you suggest to be included in future workshops? Text Responses More interaction of project summary reviews writing workshop. Perhaps have more diverse set of NSF speakers rather than single multi hour lecture; Consider panel of program managers to get different insights; More interactive panels/discussion, proposal development. A panel discussion with program directors. One -on-one time with PDs. If there was a way to get technical feedback from someone in my area, it would help. Develop more program-specific activities (if possible). Reviewed proposals all belong to the same CMMI area (very manufacturing oriented), with more samples could be made available (e.g. parts of proposal) for discussion of particular strong points and covering broader set of CMMI programs. Direct discussion with relevant program managers. Maybe make the presenters (whose proposals were reviewed) present as a panel with other people whose proposals not also awarded w/ lots of questions from the floor. Additional review of participant career proposal; more on how to find core flows in proposals. A social aspect to encourage other participants to continue and submit their proposals after the workshop is over. Put the attends in the right group with the same background. I liked all the activates included. More space and more interactions with PDs. See official NSF awards/rejection with panel feedback. Breakout groups with people more closed related to your field. The panels are useful because you see how different areas review proposals but additions similar groups would also be nice. Some protein (eggs) items at breakfast. Select proposals more related to the attendee's fields, maybe more than six proposal samples that are the same for all attendees. More proposal editing, Maybe 2 rounds instead of one. Overnight homework should be moved to a day-time activity. Schedule begins too early. A little more time to work on project summaries. More recent awardees to present their stories. Examples of full proposals including budget, data manage plan, etc. Divide participants into panels based on fields. In the mock review, the moderator should be more involved. More carefully select attendees, those who did not submit a CAREER proposal should not be selected!! Longer panel review sessions and better/larger rooms for panel reviewers. Bigger rooms for presentation with tables and electric outlets. Longer sessions for review and feedback on attendees' project summaries. Separate and dedicated time that attendees can talk to the PDs. A full week workshop with focused exercise (writing) and developing project description also along with summaries. We ended up with extra time after my proposal summary critique. Though we moved through those really quickly. I benefited even more from having time to talk more with our
group's facilitator about general NSF structure, funding metis, and proposal tips. This somewhat formal bur organic interaction could be encouraged for the groups. Can't think of any Orient groups more towards research subjects. Smaller group activities, less lectures. Actual breakfast. Optional breakout sessions with each current and former NSF PD so that workshop attendees who will be submitting to those programs can meet with them and have a Q&A session. Would be very helpful and perhaps decrease how much NSF folks are mobbed during breaks. Having PDs from various programs. If at all possible it would be nice to have more time to revise summaries before discussion with the mock panels. It would be better if the mock proposals could be grouped into specific expertise of participants. I would like to see the process of a winning proposal being developed and see the transition between an unfunded draft and a funded draft and a later submission. Can you organize the group such that the mentor from the NSF has overlap with most of the members? I think it s really help. Invite the latest CAREER awardees from other divisions, e.g. DMR. It may be helpful to know a little more about why proposals 3,4,5,6 were not funded that would be a good learning experience. More opp for networking w/ participants. More PDs. More info/advice on education section. Attendees should be required to submit a document or figure that outlines their 5, 10, 15 year CAREER plans. Including: big idea, research objectives. more structured interaction time with PM... PM introduce their perspectives? One or two presentations by PD. More awarded proposals. Fitting attendees with collaborators, people in the same area in several panels. Bring more people from past NSF CAREER received. More people from other field. More panel discussions! More CAREER panels with actual awardees. More time is required to spend with actual CAREER awardees. More interaction for people in the same field. More time for project summaries review. 1 more feedback session would be helpful after the 1st feedback. Could there be some discussion of differences in institution type? R1 vs PUI vs others and how to convince people that research can be done effectively at a non-r1. Newer proposals. Can we see actual panel comments from the example proposals? Individual/group meeting with respective PDs.
see actual reviews from panels for a proposal we reviewed. One meal could arrange tables by research area to help us network with one another. More awardee presentations. More emphasis on integrating research and education. Provide references in mock panel review. I would have liked to meet one-on-one or one on small group (i.e. pane; review) with the PM's I am submitting to. Share some good project summaries. Share some good hypothesis statements or research objectives. More time to revise the project summaries. Maybe a 2 draft review and revise opportunity. More directors talking about the programs Should include PD from all field. Have a chance to interact with PD in your field. Interactive communications and/or Q/A between PDs and attendees. Hope it could be a little longer and provide a follow-up session before the submission of CAREER proposals. Share specific literature and references. Same again. Maybe roaring panel chairs or one chair in the first day and another in the second day. Breakout with specific PM to describe expectations within each program followed up with Q&A. More guidance in the panel reviews [a final comment form the moderator] [some comments on proposals are not relevant but hard to tell].. Great, efficient program. I have learned a lot. I wish other directors ran program like this one! The proposals selected were somewhat narrow. Proposals from the MME/MEP programs were selected. Some proposals from the SMOR program would help! The writing exercise and feedback was good -- maybe forcing a bit more reflection on what we learned as a function of the workshop (minor point) Maybe a fun activity in the local area. More focused proposal review for individuals. more focused group when organizing panels, e.g. a panel group with an civil engineering domain, etc. Have a "meeting PD" session for one hour or so. Select six proposal from different disciplines (computing, materials, etc) A short presentation on background of how NSF work. Use proposals of participants for the mock panels, then we could obtain more feedback. sessions with specific PM. Mock proposal review and presentation. Keep doing the good work. Budget planning details. Could it be beneficial to split the discussions into individual programs for more specific Q&A
More presentations by past awardees. Presentation by PD explaining their programs. In this case, faculty can attend their desired session and group more information about their desired/targeted programs. Networking opportunity for people in similar field. The workshop already seems well optimized for the time available. Make it a 2 day activity and add activities. Maybe wide range of mock proposals. Career awardees presentation. NSF Career proposal guideline presentation. Mock panels. See above.. Providing photos of PD and other session leaders? It can be hard to find specific people during breaks. I would make the due date closer to the workshop so that the proposals are more "recent" in my mind. I needed to catch up on these while in the mock panel. More networking time. Actually good proposals to review from recent years, not 10 years ago. Note: the confidentiality of the identity of proposers that we read was weakened as you could delete the black highlights in most of the PDFs. More breakout session. More discussions on how to pick a research topic, maybe breakout sessions on this? More PDs from other divisions e.g. DMS. More examples on the guidelines. It would be great if we can see some real comments for the proposals for mock review. Formal Q&A session between a program director and participants. Proposal summary discussion could be longer. More speakers who are former CAREER award winners. Expose more opportunity of interacting with current and past program director. It may be good to discuss overlap and how to navigate. Open solicitations us CAREER which to choose when to submit each how to differentiate. A panel of PD from NSF for Q&A. come to mind. Workshop components were all helpful. Having program mangers present was a great way to connect. It would be great to take 10-15 minutes to have all attending PMs and PDs introduce themselves in front of the group, the room format make it difficult to see people from across the room. More group discussion. It would be great to set up the panels according to expertise to have more interaction with the PM that will review your proposal. If there is an opportunity to schedule time with PMs that would helpful. However, this is not necessary because I did reach out to a PD directly and received valuable feedback. Perhaps a second panel review as the first was very helpful. More time to talk to program directors. Some of them were not present. Do at least two passes on summary reviews. Give attendees a chance to incorporate feedback into summary before a 2nd review. Send a video of George's talk out and require that attendees review it before submitting their first draft or a revision of it once accepted into the workshop.
Maybe getting feedback from the peers on the fail draft CAREER proposal. It should not be perfect, but at least, the comments would be very helpful to some who has never submitted a CAREER proposal before. Have specific table assignments to ensure panelists have a program director to talk to during lunch and dinners sessions (maybe based on the area they are submitting to) I would suggest including a presentation from different PMs so that we may get different opinions about what is acceptable and what is not. Thank you for a great workshop! Statistic Value Total responses 119