2017-18 Associated Medical Services Peer Review Guidelines
Table of Contents Purpose of this Document... 1 About Associated Medical Services (AMS)... 2 AMS Project Grant... 2 AMS Postdoctoral Fellowship... 4 Peer Review Committee Structure... 6 Committee Membership: Roles & Responsibilities... 6 Chair... 7 Scientific Officers... 7 Committee Members... 8 NSHRF Staff... 9 Observers... 9 Confidentiality and Conflict of Interest... 10 Peer Review Process... 12 Prior to the Peer Review Committee Meeting... 12 Rating Scale and Scoring... 12 Reviewer Reports... 13 At the Committee Meeting... 14
Purpose of this Document The purpose of this document is to provide information about the Associated Medical Services (AMS) funding opportunities, peer review committee structure, roles and responsibilities, processes and procedures, and evaluation criteria. 1 Access to Information and Protection of Privacy The NSHRF is subject to provincial legislation in regards to privacy, as outlined in Freedom of Information and Protection of Privacy Act SNS 1993, c. 5 (the FOIPOP Act). We are committed to openness and transparency, while ensuring the protection of private information and intellectual property of Nova Scotia researchers. Collection, Use, Disclosure and Retention of Personal Health information The NSHRF is subject to provincial legislation governing collection, use, disclosure and retention of personal health information, as defined by the Personal Health Information Act, SNS 2010, c. 41 (the PHI Act). We are committed to openness and transparency, while ensuring the protection of personal health information in accordance with the PHI Act. Personal information collected by the NSHRF about applicants is used to: review applications; administer and monitor grants and awards; compile statistics; and support health research in Nova Scotia. Integrity in Research and Scholarship The NSHRF is guided by national standards and procedures related to integrity in research as described in the Tri-Agency Framework: Responsible Conduct of Research (2011). These guidelines are followed for promoting integrity in research and scholarship and in investigating any allegations of misconduct in research and scholarship ( http://www.rcr.ethics.gc.ca/eng/policypolitique/framework-cadre/). For all of our policies on conflict of interest and confidentiality, legislation and ethics requirements please see: http://www.nshrf.ca/programs-services/generalprogram-requirements/policies-and-ethics 1 These guidelines are largely based on the practices followed by Canadian Institutes for Health Research (CIHR) and various provincial health research foundations. 1
About Associated Medical Services (AMS) Associated Medical Services (AMS) is a small Canadian charitable organization with an impressive history as a catalyst for change. AMS has, and continues to have, profound impact on the healthcare of Canadians through its support of the history of medicine and healthcare, health professional education, compassionate patient care and bioethics. AMS Project Grant Objective The AMS project grant is designed to support small budget proposals on projects in history of healthcare/disease/ medicine with special interest in, but not limited to Canadian history of healthcare/ disease/medicine. Such projects might include requests for seed money to develop research initiatives. Proposals are expected to advance an aspect of the history of health, health care or disease or the history of education of health professionals that could benefit the broader community of practitioners in the field. There is an expectation that during the period of these grants or within a year of their completion, one or more conference papers or publications/presentations will result from this funding. Funding and Duration Funding is available up to a maximum of $10,000 per grant. Funds will only be issued to institutions located in Canada. The grant is for a period of one year. The number of grants allocated will be based on the number of applications received and available funding. Funded applicants must demonstrate the ability to generate financial statements which comply with the NSHRF Program Requirements. Eligibility Principal Investigator must meet one of the following requirements: be employed as a health professional in Canada by a health care organization/ health authority/ provincial/national government; be a researcher with an academic appointment at a Canadian university or affiliated research institution or hold an appointment (with research responsibilities) at an affiliated hospital/research centre/institute; or be employed by an organization such as a professional organization, eg., CMA, CNA, MCC, or a health professions regulatory college located in Canada. Please note: if ethics is required to complete the suggested project the principal investigator must be able to provide documentation from a Tri-Council Compliant Ethics Board demonstrating that an acceptable review has been completed and the project approved. Evaluation Criteria The following criteria will be used to evaluate the AMS project grant applications: Objectives The proposal must: List the main project objectives 2
Ensure listed objectives cover all main activities planned using the project funds Please note: Funded applicants will be expected to report on progress related to the listed objectives. Project Description The proposal must demonstrate: Clarity, scope and appropriateness of objectives Clarity of method, including limitations Appropriateness of method to the discipline of History of Medicine Appropriate consideration of chronology, change over time and periodization, as appropriate Potential contribution to History of Medicine and advanced of the field in Canada or elsewhere, if relevant. The quality and appropriateness of the kind of sources being consulted The appropriateness and impact of the scholarly questions/themes/debates being engaged. Appropriateness of the plan for presentation and publication. Work Plan The proposal must demonstrate: Clarity and feasibility of project timeline Clarity and feasibility of project activities Alignment of timeline and work plan with objectives. Team Composition - The proposal must demonstrate: Clarity and appropriateness of team member roles and responsibilities Relevance and scope of expertise and experience of the PI and team members, including peer reviewed publications, presentations and previous research awards or grants Appropriateness of the PI s environment to enable the conduct and success of the research project Potential of team to successfully conduct proposed research Merit of the Proposal The proposal must show: Significance of proposed research and expected contribution to understanding, interpreting, or creating knowledge in the field of the history of medicine The potential of the investigator/research team to succeed in the activities proposed Budget - The proposal must demonstrate: Appropriateness and adequacy of budget expenses and their justification Availability of other sources of funding, if any 3
Special needs related to the project (i.e., meetings, teleconferences or infrastructure costs, such as user fees) AMS Postdoctoral Fellowship Objective The AMS Project Grant is designed to support small budget proposals on projects featuring the history of medicine. AMS defines the history of medicine as the study, analysis, and interpretation of past practices, philosophies, and/or epistemologies related to human health, health care, and/or disease using methods appropriate to the discipline of History. Projects that consider the Canadian context are particularly welcome. Projects might include requests for seed money to develop research initiatives. Proposals are expected to advance an aspect of the history of health, health care or disease or the education of health professionals that could benefit the broader community of practitioners in the field. There is an expectation that during the period of these grants or within a year of their completion, one or more conference papers or publications/presentations will result from this funding. Funding Details and Duration Value and Duration: The maximum amount awarded is a $45,000 stipend for one (1) year plus a $2500 research and travel allowance. Eligibility To apply for the AMS Postdoctoral Fellowship, the Principal Investigator (PI) must: Meet minimum commitment of 75% of your time in the research fellowship role proposed in the application. Identify a primary supervisor. The supervisor must be located at the university where the applicant will be based during the fellowship. At the time of application, must hold a PhD or expect to complete all of the requirements of the degree before September 1, 2017. Candidates who have obtained a doctoral degree more than three years prior to the competition deadline are not eligible. NSHRF uses the date of degree completion to determine applicant eligibility. This is the date on which all requirements of your degree have been met, including successful defense and submission of the corrected copy of your dissertation. If prospective candidates have experienced a serious Interruption or delay in pursuing their career since completion of the PhD that would make them ineligible for this fellowship based on the three year limit, exemptions will be considered on an individual basis on provision of relevant information including specific dates. Contact the Manager, REAL Knowledge Program, for more information. Fellowships are normally held at Canadian institutions. However, Canadian citizens and permanent residents are eligible to hold their awards outside Canada. Candidates who are 4
neither Canadian citizens nor permanent residents of Canada, are eligible for awards to be held in Canada only. For individuals who are seeking support to hold their award outside of Canada, the award will only begin after the recipient has provided proof of Canadian Citizenship or Canadian Permanent Residency. For international candidates coming to Canada, awards will only begin after proof of full time registration at a Canadian university is provided. Evaluation Criteria The following criteria will be used by the peer review committee to evaluate AMS Postdoctoral Fellowship applications: Challenge The aim and importance of the endeavor (20%) Originality and potential significance of the proposed program of work. Feasibility The plan to achieve excellence (30%) Feasibility of the proposed program of work Appropriateness of the intended place of tenure Effectiveness of the overall support provided by the host institution for research training and career development Appropriateness of the supervisor identified Capability The expertise to succeed (50%) Fellowships, scholarships or other awards obtained Previous research experience and/or publications Timely completion of doctoral studies, taking into account the nature of the program and any related personal circumstances that may have delayed the applicant s academic career 5
Peer Review Committee Structure The NSHRF AMS peer review committee is made up of health researchers from across Canada. The committee consists of a Chair, Scientific Officers (generally 2), and usually at least 8 Peer Reviewers. Committee membership will be reviewed on an annual basis using a roster system to optimize the alignment of peer reviewer expertise with applications received and to minimize reviewer fatigue while also building reviewer capacity. Committee member turnover will be staggered to ensure decision-making continuity; any given year, 2/3 s of the committee will have served previously and the remaining 1/3 will consist of new or returning members. 2 The identity (names) of reviewers assigned to specific applications will not be revealed to applicants. However, following the completion of the review process the peer review committee membership will be made available to the public and to applicants on the NSHRF website, in the NSHRF Annual Report, and other appropriate communication mechanisms. Committee Membership: Roles & Responsibilities Membership of the peer review committee is established to ensure there are appropriate reviewers for all of the applications. Peer Review Committee Membership as a Whole: all members must have some level of knowledge regarding medical history and/or historical methodologies must cover the range of research areas (history of healthcare/disease/ medicine/anthropology) represent the range of research methodologies attempt to achieve gender balance provide institutional and regional representation (to avoid conflict of interest situations) Individual Peer Review Committee Members: are established health/historical researchers: o they hold, or have held, peer reviewed research funding preferably at the national/international level o have published in peer reviewed journals are experienced peer reviewers (i.e., current or past experience with another agency, preferably at the national/international level) have a breadth of knowledge and a specific area of expertise are objective, fair minded, and have maturity of judgment 2 The NSHRF will revise its Review Committees when new programs are added to ensure the appropriate complement and number of reviewers based on specific program requirements and review expertise required. 6
are able to appreciate innovative research in the areas of history of health care, disease and/or medicine or the history of the education of health professionals and the development of new innovative research proposals are willing to make the time commitment to participate in the review process must agree to adhere to the NSHRF confidentiality policy to ensure all the information related to the review process and application remains confidential Specific roles for individuals on the peer review committee are outlined below. Chair: is directly responsible for ensuring that the committee functions smoothly, effectively and objectively, according to the NSHRF's requirements helps to identify and assign reviewers to specific applications based on fit with content and expertise between the reviewers and the applications establishes a positive, constructive, fair-minded environment in which applications are to be evaluated ensures that each application receives equal attention by the committee and that all committee members are involved in the discussion reviews applications works with the Scientific Officer(s) to provide a verbal summary of the discussion of each application before the final rating is determined ensures that specific concerns of ethics, and of other NSHRF requirements, are addressed is not a Principal Investigator or Co-Principal Investigator on an application in the competition when necessary, asks the a delegate to act as the Chair or Scientific Officer when the Chair or Scientific Officer has a conflict with a specific application or when the Chair or Scientific Officer is unable to attend committee meetings Scientific Officers: support the Chair in her/his role during the committee meeting help to identify and assign reviewers to specific applications based on fit with content and expertise between the reviewers and the applications review applications ensure that issues of ethics, eligibility and other concerns are raised at the committee meeting are not a Principal Investigator or Co-Principal Investigator on an application in the competition take notes of the committee discussion and provide these to the NSHRF in a timely fashion after the meeting concludes (these notes are sent to applicants along with their letter of decision and reviewer reports) 7
Scientific Officer Notes 3 The purpose of Scientific Officer (SO) Notes is to inform the applicant of the determining factors, from the point of view of the committee, in the rating of the application. The notes provide the applicant with insights into the committee discussion of the application which are not available through the comments of the reviewers. The SO Notes should explain how the consensus score was reached by capturing the tone of the committee discussion. The Scientific Officer Notes should include: major strengths and weaknesses of the application pivotal points determining the rating (i.e., comments about the application relative to the evaluation criteria). Not every point needs to be covered in every note; only those which had the greatest impact on the evaluation, or where the committee has important advice for the applicant, need be included salient aspects of the committee s discussion, for example: o Level of enthusiasm o The committee was divided as to o The committee had difficulty arriving at a rating that would reconcile the perceived stature of the applicant and the applicant s past productivity with the apparent quality of the current application. resolution of reviewer disagreement (which view expressed in the reviewers reports did the committee favour?) explanation of significant budget and/or term reductions encouragement or discouragement in relation to resubmission, as applicable - explain how the applicant could improve the proposal if resubmission is encouraged The Scientific Officer Notes should not include: a summary of the proposal repetition of internal or external reviewer comments a recommendation for funding the rating of the grant (number or descriptor) Committee Members Internal Reviewers 4 : are responsible for reviewing their assigned applications (either as first reviewer or second reviewer) prior to the committee meeting and completing their reviewer reports present summaries of their assigned applications in the committee meeting participate in the review of all other (non-assigned) applications at the committee meeting Readers: 5 Committee members are also assigned to be "readers" on other applications. Readers are responsible 3 Adapted from CIHR Instructions for Scientific Officers http://www.cihr-irsc.gc.ca/e/40216.html 4 Investigators working in clinical or industrial settings and policy makers may sometimes be included as appropriate. 5 Adapted from CIHR Instructions for Readers http://www.cihr-irsc.gc.ca/e/44083.html 8
only for reading an application so that they may serve as a discussant and aid the committee in reaching a consensus rating. Readers are not required to submit a written review. NSHRF Staff: Observers provide advice to the committee on the NSHRF requirements and policies ensure review materials are sent to reviewers in advance of the committee meeting keep notes on procedural aspects of the committee's function record the consensus rating and budget recommendations made by the committee for each application record concerns raised by the committee on issues requiring later attention by staff (e.g., overlap of funding, ethics, eligibility) ensure the peer review process is run effectively and efficiently attend peer review meetings to enhance their understanding of the peer review process, with the intent that they will: o gain an appreciation for the rigor by which applications are evaluated o learn from the discussion of applications what applicants can do to strengthen applications as well as how to address weaknesses in applications that may be identified through the peer review process o increase the research enterprise s understanding of the value peer review brings to the strength of evidence that can be used for making decisions 9
Confidentiality and Conflict of Interest All information contained in applications, reviews by internal/external peer reviewers and committee discussions are strictly confidential. Peer review committee members and observers must not discuss with applicants, or any individual outside the peer review process, any information relating to the review of a specific application, or offer opinions on the chances of success or failure. All requests for information on an application should be referred to the NSHRF. All reviewers (internal and external) and observers are required to adhere to the NSHRF Confidentiality and Conflict of Interest Policies outlined below. Confidentiality Policy All documents and information provided for the purpose of peer review by the Nova Scotia Health Research Foundation (NSHRF) are subject to the conditions of the Privacy Act and the Access to Information Act. Therefore, they and any discussions thereof must be treated as strictly confidential and may not be used for any purpose beyond that for which they are originally intended. All materials related to the review process must be stored in a secure manner to prevent unauthorized access. When documentation is no longer required, it must be destroyed using a secure method such as burning or shredding or returned to the NSHRF for destruction. It is the intention of the Foundation that all information gained by our reviewers by reason of their participation in this program is to remain absolutely confidential. This would include information as to who may or may not be in a list of competitors for grants and awards, as well as information concerning applicants who have submitted proposals for review. All inquiries received by committee members, reviewers, or scientific officers concerning the review of any application should be referred to the NSHRF. Conflict of Interest Policy NSHRF makes every effort to ensure not only that its decisions are fair and objective, but also that they are seen to be so. No committee member with a conflict of interest may participate in the review of an application. The best reviews are carried out by people who are in a relevant (to the application) discipline, understand the methodology and content of the research, and have other relevant expertise. Canada s, and Nova Scotia s, health research communities are relatively small and conflict of interest situations do arise that might (or could be perceived to) compromise impartial review. The NSHRF staff, together with the peer review committee Chair will make every effort to identify and avoid situations in which conflict of interest might arise. It is also the responsibility of individual Reviewers to declare the existence of any conflict of interest. It is the peer review committee s duty to decide whether the individual may participate in the discussion. The Chair should rule in ambiguous cases and resolve areas of uncertainty, in consultation with NSHRF staff as necessary. If a reviewer has been asked to review an application for which s/he feels a conflict exits (or may exist), s/he must declare the conflict immediately to the Manager, Research Programs. In addition, the NSHRF staff may declare a perceived or real conflict of interest on behalf of any peer reviewer and this information will be provided to the committee Chair prior to the meeting. The Chair may need to 10
reassign the application to another reviewer and /or ask the reviewer in conflict of interest to leave the room when the application is reviewed. Conflict of interest situations are considered to exist for reviewers under circumstances where a reviewer 6 : is from the same department as the applicant (or that of any team member including: Co- Principal Investigator, Co-Investigator, Associate, or Partner); has collaborated, published or been a co-applicant (or a team member, as outlined above) with the applicant, within the last five years; has been a student or supervisor of the applicant within the last ten years; is a close personal friend or relative of the applicant (or any team member); has had long-standing scientific or personal differences with the applicant; is in a position to gain or lose financially from the outcome of the application (e.g., hold stock in the company of an industry partner or a competitor); or for some other reason feel that s/he cannot provide an objective review of the application. The Chair and Scientific Officers are subject to the same conflict of interest guidelines as the other committee members. These guidelines are not intended to cover all possible situations. It is the responsibility of individual reviewers to declare the existence of any conflict of interest. No applicant should contact a member of the peer review committee regarding the review of his/her application. Peer review committee members are instructed to advise the NSHRF if such contact should take place. 6 These are adopted from CIHR Peer Review Guidelines. 11
Peer Review Process The NSHRF undertakes an initial screening of the registrations and applications to determine an applicant s eligibility, including the completeness of his/her application. On occasion reviewers identify eligibility concerns during their detailed review of an application. Please contact the Program Manager to discuss any concerns in advance of the committee meeting so that issues may be screened prior to discussion at the meeting. Prior to the Peer Review Committee Meeting Dates of the peer review committee meetings are established; The Chair, Scientific Officers and NSHRF staff meet to assign internal reviewers (a first and second reviewer for each student and grant application) and external reviewers for Establishment grants (preferably two per grant). The Chair and the Scientific Officers are provided summary information for all applications assigned to their committee one to two weeks before the meeting; Internal Reviewers are forwarded a copy of the applications they have been assigned to review four to six weeks before the committee meeting and are asked to complete an internal reviewer report for each application they review, including suggestions for improvement (as appropriate). Each committee member reviews approximately 6-10 applications. Internal reviewers use the scale provided below to score grant applications and are encouraged to use the full scale as appropriate. The internal reviewer provides an initial score for each grant application that he/she reviews. Rating Scale and Scoring The NSHRF is committed to research excellence and will fund only proposals that achieve an overall committee rating of 3.0 or higher (subject to budget availability). The following scales are used by peer reviewers to rate applications: Fundable: 4.5-4.9 4.0-4.4 3.5-3.9 3.0-3.4 Not fundable: 2.5-2.9 2.0-2.4 1.0-1.9 0 Range Descriptors Operating Grants Descriptors Scotia Support Grants Outstanding Excellent Very good Solid/significant research Needs revision Needs major revision Seriously flawed Not acceptable Outstanding Excellent Very good Good Average Below average Not acceptable Unable to judge 12
Reviewer Reports All applications to be considered for Research Program funding will be electronically submitted and evaluated in the NSHRF online Grants Management System (GMS). The NSHRF GMS Reviewer Portal can be found at the following link: https://gms.nshrf.ca/ Reviewers evaluations should include the following information as prompted for within the GMS Reviewer Portal: Comments related to criteria and overall impression of the application: o a brief summary of the project in point form (~ 60 to no more than 100 words), using objective language include strengths and weaknesses (e.g., originality, potential impact of the research described in the application, past productivity of the applicants) o comments may be provided on the ethical acceptability of the research (e.g., adequacy of consent forms, appropriateness of techniques), possible overlaps of the application with other sources of funding (as described in the CV Module), possible issues of eligibility or any other issues o Internal reviewers (Establishment Grants Only): include views of the external reviewers evaluations pointing out areas of agreement/disagreement. Guideline for Presenting Project Reviews at the Committee Meeting The initial review sets the stage for the discussion of the project. It flags what points need to be discussed in greater detail. Reviewer 1 Provide a 1-2 sentence summary of the project so that the rest of the panel knows what it is about Using the scoring criteria as a guide, describe the strengths and weaknesses of the proposal. Summarize the strengths, expand on the weaknesses. Incorporate the external review where appropriate. EXAMPLE: This project examines the impact of a disruptive scientific officer on the scoring of a grant review panel. It will analyze video recorded panel proceedings and use qualitative interviews and surveys of panel members. This proposal has an excellent team with appropriate set of skills, time, strong track records and mentorship. While it s somewhat original with clear objectives, it raises a number of ethical concerns. For example, the study will be conducted at an actual grant review session with real grants and the intervention may impact the scoring of the grants. Although they provide a rationale for hiding the study from the panel members, it still raises concerns about informed consent. The methods, analysis plans, limitations and timelines are clearly described. The dissemination plan is superior and innovative. Budgeting is adequate. The relevance is weak. While they do a good job stating why the study is important, its impact or link to the health of Nova Scotians is not clearly demonstrated. Reviewer 2 Presents new points or points of disagreement including any issues raised by the external reviewers (Establishment Grants only) 13
EXAMPLE: I agree with the points you ve raised. Although there is evidence of a mentor with a stellar track record, I question the level of mentorship given the ethical issues around informed consent that have already been raised by Reviewer 1. Tips Do not read your prepared review; rather summarize your detailed comments. Do not describe the entire proposal - all committee members have summary books in front of them. They will ask you for more description if needed. Reviewer 2 does not describe the application. At the Committee Meeting Peer review committee meetings are usually held between two to four months after the application deadline. The meeting takes place over a maximum of two days. Normally, a maximum of 30 applications 7 are reviewed per day. The NSHRF staff provides the committee with a list of applications to be reviewed. An initial mean score is provided for all applications - applications with mean scores above 2.9 will be scheduled to be discussed at the committee meeting. 8 Assigned internal reviewers announce their initial score (to one decimal place) to the group. The first internal reviewer presents his/her assessment of the application, describing the strengths and weaknesses relative to the academic excellence of the researcher, the merit of the proposal, the budget and the relevance to Nova Scotia. The second internal reviewer addresses the same issues, concentrating on points of agreement or disagreement with the first, and elaborating points that may not have been addressed by the first internal reviewer. The reader is then called upon by the Chair to offer his/her comments. The Scientific Officer notes areas of agreement and disagreement with the internal reviewers. The Chair leads discussion among all committee members, which should consider the strengths and weaknesses of the proposal; the quality, originality and potential impact of the research proposal; and the recent productivity of the applicant(s). 7 This is based on practices at the national level. 8 The Committee may not discuss applications with scores below 3.0 (scores of both internal reviewers are below 3.0 and the external reviewer comments are consistent with the scoring of the internal reviewers) unless a committee member requests it be reviewed by the Committee. In these streamlined or triaged cases, the applicant receives the mean of the two internal reviewer initial scores. Applicants receive a copy of the internal and external reviews; there are no Scientific Officer notes and Committee Members do not vote on the rating in these cases. 14
The Scientific Officer or Chair briefly summarizes the discussion. As the Chair and Scientific Officer are not voting members they can challenge the committee if they disagree with the discussion but if the committee maintains its position on an application the summary of the discussion must reflect the position of the committee in a neutral way. The Scientific Officer then reads the SO Notes out loud to the committee before the consensus rating is determined. The committee can request adjustments be made to the notes if necessary. After the discussion, the Chair seeks a "consensus rating" from the two internal reviewers. If a consensus cannot be reached, the mean value of the ratings of the two internal reviewers initially announced scores are used (rounded up, if necessary, to one decimal point). All committee members, including the two internal reviewers but excluding the Chair and Scientific Officer(s), then cast individual confidential votes within +/-0.5 of the consensus rating. The internal reviewers are not bound to the consensus score; they can still vote within +/-0.5 of the consensus score. The final rating assigned to the application is the average of all committee member scores. The committee then discusses the budget, the term of the award, and any other special concerns related to the application. Applications are rank ordered, in descending order, which serves as the committee s funding recommendation. The NSHRF strives to fund applications with scores of 3.0 or higher (budget permitting). An essential component of any committee meeting is the final review of the committee s effectiveness and functioning, and a discussion of policy issues that may have arisen in the course of its deliberations. Committee members should remain for the entire meeting and make travel arrangements accordingly. Staff and the Chair will provide advance guidance on the probable length of the meeting. 15
After the Peer Review Committee Meeting Funding decisions are based on final ratings from the peer review committees, NSHRF policies, and funding availability. Applicants are notified directly of the funding decisions and are provided with copies 9 of the following: o Final score o External reviewers assessments (Establishment Grants only) o First and second internal reviewers reports (comments only) o Scientific Officer Notes NSHRF hosts an annual event to celebrate applications funded within the fiscal year. 9 Names are removed from the forms to ensure anonymity. 16