Manual. For. Independent Peer Reviews, Independent Scientific Assessments. And. Other Review Types DRAFT

Transcription

1 Manual For Independent Peer Reviews, Independent Scientific Assessments And Other Review Types DRAFT International Center for Regulatory Science George Mason University Arlington VA

2 TABLE OF CONTENTS INTRODUCTION... 1 DEFINITION OF TERMS... 1 APPLICATIONS... 2 Independent Peer Review...2 Independent Scientific Assessment...3 Technical Review...4 STRUCTURE OF A PROJECT SPECIFIC TECHNICAL REVIEW PROCEDURE... ERROR! BOOKMARK NOT DEFINED. STRUCTURE OF PEER REVIEW AND SCIENTIFIC ASSESSMENT PROCESSES... 4 Peer Review Oversight Committee...4 Qualification of Panel Members...5 Conflict Of Interest...5 Criteria For Peer Review...6 Criteria For Scientific Assessments...7 Peer Review and Technical Assessment Panels...7 Procedures...8 Peer Review and Scientific Assessment Reports...8 Stakeholder Participation...9 REFERENCES... 9

3 INTRODUCTION The purpose of this manual is to describe peer review, scientific assessment, and other review processes developed to support activities performed by government agencies at federal, state, regional, or local levels; industry; and other organizations. It is based on the policies, statements, and traditions of various segments of the scientific and engineering community. This manual is largely based on several major sources: 1. Numerous reports of the National Academies (National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council). The reports of National Academies are typically published by the National Academy Press; 2. The Information Quality Bulletin of the Office of Management and Budget (OMB 2005) resulting from the passage of the Information Quality Act; 3. Activities of the Institute for Regulatory Science and the American Society of Mechanical Engineers (ASME/RSI 2002); and, 4. A textbook on peer review (Moghissi et al 2013). These documents are based on the results of about 300 peer review and scientific assessment reports performed for various governmental agencies at federal state and local levels, the Congress and others. DEFINITION OF TERMS There has been major disagreement on what constitutes peer review and scientific assessment, the implications of independency, and the difference between independent peer review and peer review. The term science as used in this manual includes natural sciences, engineering, and all other scientific disciplines. This section is not intended to thoroughly review the literature; instead it provides definitions that are generally accepted and routinely used. Independent Peer Review and Independent Scientific Assessment: Peer review provides a critical evaluation of a scientific product. The product may be a completed study; a paper; the outline of study; a research program; scientific foundation of a regulation or a standard; or any other activity that has a product that can be evaluated. In contrast to independent peer review, independent scientific assessments provide a product to the sponsoring agency. Often the sponsoring agency needs advice on a specific subject. Consequently, during the scientific assessment process, available existing information is gathered and critically assessed. Simply stated, whereas a peer review evaluates a technical product, a scientific assessment generates a new product. The prerequisite for independency of peer review or scientific assessment is avoidance of conflict of interest. The rule governing the criteria of independency (ASME/RSI 2002) is: Those who have a stake in the outcome of the review may not act as a reviewer or participant in the selection of the reviewers. Merit Review: This term is used by the National Science foundation (NSF 2013) to describe a review that addresses scientific issues in a peer review and includes related programmatic requirements of the National Science Foundation. A similar process is used by the National Institutes of Health (NIH 2004). In principle, merit review is based on independent peer review. 1

4 Technical Review: As used in this manual, this term covers a large number of activities that are not independent peer review, independent scientific assessment or (independent) merit review. Sometimes called external panel, technical advice, or simply review, they describe a process that responds to specific needs. Many organizations require assistance in evaluating activities and seek advice. The individuals chosen for this type of panel are not required to meet the stringent standards of panel selection for independent peer review and the review may cover subjects unrelated to science. APPLICATIONS Government agencies are the primary users of these processes. However international organizations, industry, academia, associations,and foundations are increasing their use of peer review, scientific assessment, merit and technical reviews. Independent Peer Review Consistent with the historic tradition of science, the peer review process is intended to provide an unbiased, independent, accurate, economical, and timely response to those organizations needing support on specific actions. There is a long tradition in peer review, performed routinely by many scientific and engineering organizations for their technical publications. Activities that clearly benefit from peer review include the following: 1. Government agencies, industrial concerns, and foundations support research and development in specific areas of science. Often, the amount of funding requested by applicants is larger (in some cases, significantly larger) than available funds. Peer review provides a mechanism not only to evaluate the technical acceptability of specific proposals but also to rank them in accordance with predetermined criteria. 2. Some programs are developed to accomplish a certain goal. Peer review provides a reasonable and defensible method to ensure that the approach and implementation of the program are consistent with the desired goals. On occasion, the agency identifies the parameters of a program and asks for a review of existing approaches for development of its desired program. 3. A major function of government agencies at the federal, state, regional, and local levels is the promulgation of regulations. A number of these regulations are based or erected on the evaluation of available scientific information. Peer review provides the necessary tool to ensure that the foundation of the regulation is based on appropriate scientific and engineering principles. 4. The regulated community must often comply with requirements that are subject to different scientific interpretations. Potentially available options and their respective performance; cost-effectiveness; and numerous other topics benefit from peer review or technical assessment. Because specific needs are not always foreseeable, the peer review and scientific assessment systems must be flexible enough to accommodate these needs as they arise. 5. In the development of large-scale projects, peer review ensures that during the design and construction of the project, the underlying science is sound, the chosen technologies are appropriate, the road map is reasonable, and correct economic principles are used. 2

5 6. Agencies often support competing technologies to ensure the availability of an option if one technology fails to meet its predicted performance. Depending upon the desire of the sponsoring agency, the evaluation of competing technologies and the selection of the most promising technology can benefit from peer review. 7. Once a technology reaches a certain level of maturity, the supporting agency must make a decision on whether it should continue funding the work. The potential for success of that technology, based on specific parameters, is subject to peer review. 8. The results of research and development are often published in the form of internal reports. The scientific acceptability of information included in these reports can be peer-reviewed much like publications of professional societies. 9. Agencies routinely prepare requests for proposals (RFPs) and requests for applications (RFAs). Peer review provides a reasonable method for evaluation of the validity of the technical criteria of RFPs and RFAs; responses to them; and the prioritization of various responses based on specific technical criteria. Independent Scientific Assessment As stated above, the scientific assessment process consists of gathering and critically evaluating existing information. Often a sponsor plans to undertake an activity and is seeking advice on how to proceed. Activities that would benefit from independent scientific assessment include the following: 1. Government agencies, industrial concerns, and foundations support research and development in specific areas of science and engineering. Scientific assessment provides a mechanism to implement the requirements imposed by legislation, charter, and other legal mandates. In particular, scientific assessment can provide the sponsor with review criteria, (questions to be used by peer reviewers), criteria for selection of reviewers, and other critical guides. 2. A major function of certain government agencies at the federal, state, regional, and local levels is the promulgation of regulations. Regulations require gathering and evaluation of available scientific and engineering information; scientific assessment provides the necessary tool to perform this important task. 3. Many organizations have or must develop programs to accomplish a certain goal. Scientific assessment provides a reasonable method to describe how these goals are to be accomplished. 4. Government agencies and several other organizations routinely undertake the construction of largescale projects. The objective of these projects may be to advance science and technology or to improve the quality of life. Scientific assessment can be used to provide guides to the agency on how to proceed 5. Many government agencies, notably Department of Energy and the Department of Defense, have facilities requiring environmental restoration. There are other government agencies at the state or local level, as well as industrial facilities, facing the same problems. Evaluation of technological 3

6 needs; available technologies and needed technology developments; and optimization of engineering processes are subject to scientific assessment. 6. Agencies routinely prepare requests for proposals (RFPs) and requests for applications (RFAs). Scientific assessment provides a reasonable method for the development of RFPs and RFAs including criteria for the evaluation of submissions; and the prioritization of various responses based on the evaluations criteria. Technical Review Activities that would benefit from technical review cover virtually the entire fields of science, as defined above; business; and commerce. For example, many agencies do not use independent peer review or independent scientific assessment in preparing or evaluating RFPs or RFAs. The sponsoring agency may decide to use technical review because the size of the contract is too small to justify the time and the cost of independent peer review. Similarly, the organization may find it appropriate to use reviewers who have a conflict of interest or may not necessarily be qualified as peers. In all of these cases, technical review provides a mechanism to accomplish the stated goal of the organization. STRUCTURE OF PEER REVIEW AND SCIENTIFIC ASSESSMENT PROCESSES The peer review and scientific assessment processes are typically based on a tiered system. The elements of the process are: 1. A review panel (RP) or an assessment panel (AP) performs independent peer review or independent scientific assessment. For the sake of simplicity, members of the RP or AP will be referred to as panel members. The requirements for selection of panel members include relevant qualifications and independency, the latter demonstrated by lack of conflict of interest. (These qualifications are explained in detail in the next section.) 2. The RP (or AP) is provided with review or assessment criteria (questions) constituting their charge. The panel is expected to respond to these criteria. 3. The Peer Review Oversight Committee advises the performing organization on the selection of the panel members and other parts of the process. 4. Various types of panels have been developed to ensure timely response to the needs of potential sponsors. 5. In order to optimize the process, a structure for the report of panels has been developed that provides sufficient flexibility to respond to variable sponsor needs. 6. Finally, provisions have been made for stakeholder participation. Peer Review Oversight Committee Most organizations that perform peer review, scientific assessment, and related activities including ICRS have made provisions for a Peer Review Oversight Committee (PROC) to perform certain functions that substantially enhance the peer review and scientific assessment processes. The members of PROC are chosen based on their qualifications including their education, experience, contribution to their respective field, and peer recognition. In organizing PORC an attempt is made to ensure that all needed competencies and diversity of technical views are represented in the committee. The exact number of individuals who constitute the 4

7 Committee depends upon the nature of the reviews and assessments, and numerous other parameters. Specific functions of the PROC include the following: Review and approval of relevant manuals notably Manual on Peer Review and Scientific Assessment describing the process of independent peer review and independent scientific assessment. Review and approval of the following key issues of each planned peer review: 1. Review criteria provided by the sponsors to ICRS 2. Evaluation and approval of qualifications of the members of the RP and AP as identified by the ICRS staff. 3. Evaluation and approval of independency (lack of conflict of interest) of the members of the Review Panel as identified by the ICRS staff. Qualification of Panel Members The key to the success of every peer review or scientific assessment is the selection of qualified Panel members. The selection of a Panel member must be based on the totality of that individual s qualifications. However, there are several generally recognized and fundamental criteria for evaluating qualifications of a member of an AP or RP as follows: 1. Education: A minimum of a B.S. degree in scientific, engineering, or a relevant field would be required for any Panel member. In practice, the Panel members are likely to have advanced degrees. 2. Professional Experience: Because of the rapid advancement of science and engineering, often relevant professional experience is as important as or more important than earned degrees. Consequently, significant experience in the area that is being reviewed or assessed is necessary. 3. Peer Recognition: Election to office of a professional society; serving on committees of scholarly organizations; relevant awards; and similar activities are considered a demonstration of peer recognition. 4. Contribution to the Profession: The individual s contribution to professional advancement may be demonstrated by publications, particularly those in peer-reviewed journals. In addition, patents and similar activities are also considered. Conflict Of Interest One of the most complex and contested issues in peer review and scientific assessment is the independency of panel members collectively called conflict of interest. The ideal member of the RP or AP is an individual who is intimately familiar with the subject and yet has no monetary interest in it. Despite this apparent difficulty, the scientific and engineering communities have successfully performed peer reviews and technical assessments without having a real or an apparent conflict of interest. As stated above, the guiding principle for conflict of interest in peer reviews is as follows: Those who have a stake in the outcome of the review may not act as a reviewer or participant in the selection of the reviewers. 5

8 This principle applies not only to peer reviews and scientific assessments but also to the members of the PROC. In order to insure the independency of panel members they must sign a statement indicating a lack of personal or financial interest in the outcome of a specific review or assessment. For obvious reasons, conflict of interest is most severe for members of the PROC. Because of the large number of projects that may be reviewed by PROC, it is not always predictable that a member may have a conflict of interest in the review or assessment of a future activity. Accordingly, the program must make provisions for a member of the PROC to be recused from participation in a specific project. The management of conflict of interest is significantly simpler for members of the RPs or APs because the projects are reasonably well known and narrowly defined in advance of formation of the RP or AP. However, the process of the RP or AP members to be recused is identical to that for members of the PROC. Criteria for Peer Review The findings of the RP are responses to specific review criteria, sometimes referred to as the charge of the panel, review questions, or lines of inquiry. Experience has shown that sponsoring agencies would benefit from the availability of general guidelines for selection of project specific review criteria. The following general criteria provide guidance to the sponsoring agency for preparation of project specific review criteria: 1. Scientific and engineering validity: By far the most important aspect of any project is its consistency with established scientific and engineering principles and industry standards. 2. Relevancy: All projects supported by a sponsoring organization must be relevant to its needs. 3. Competency: Those who propose to perform a project must have sufficient competency to be able to accomplish the proposed tasks. 4. Facilities: Those who propose to perform a study must have demonstrated access to facilities commensurate with the requirements of the study. In addition to these criteria, other criteria may be considered as follows: 5. Regulatory acceptability: If a study includes a subject that requires regulatory compliance, this criterion must be considered during the peer review. 6. Reducing human health and ecological risks: If a project is associated with a potential exposure to materials that pose a human health risk, reduction of risk to the general public or the workers is an important parameter in peer review. Similarly, a reduction of risk to the ecological system can be a parameter in peer review. 7. Cost-effectiveness: Cost assessment of a decision is an exceedingly important and often neglected parameter. The cost of a decision for a given level of risk is subject to peer review. Similarly, the life cycle cost of a technology is an important criterion for its desirability. 6

9 Criteria for Scientific Assessments Much like peer review, the findings of the AP are responses to specific questions called assessment criteria, assessment questions, or lines of inquiry. In contrast to review criteria, it is difficult to provide guidance to the sponsoring agency for preparation of specific assessment criteria. However, as a general rule, scientific assessments deal entirely with scientific and engineering questions and exclude societal decisions. Peer Review and Technical Assessment Panels The number of individuals constituting a panel depends upon the complexity of the subject to be reviewed or assessed. However, typically, a panel consists of at least three individuals and, depending upon the nature of the subject, may be significantly larger (ASME/RSI 2002, Moghissi et al 2013). Members of a panel must have appropriate qualifications in the area being reviewed or assessed and their selection is to be based on the competencies required for the review or assessment. Although every review or assessment is unique, it is desirable to establish a guide on the nature of the process. Accordingly, the following types of panels are established: Type I: Manuscript review: This type is the simplest form of peer review. It consists of asking individuals to review a document. The comments of the reviewers are subsequently evaluated by someone other than peer reviewers. This type of peer review is used primarily by journals but is also occasionally used by others. Type II: (AP or RP): This panel type reviews a document or a single project. They may also be asked to assess a relatively small or narrowly defined scientific subject. The panel may be asked to meet at a location or perform its task via virtual meetings or teleconference Type III: (AP or RP): This panel type consists of at least five individuals who perform a review or assessment of a complex project, several similar projects, or a scientific assessment. Again, the panel may be asked to meet at a location or perform its task via virtual meetings or teleconference, although meeting significantly expedites the completion of the task. Type IV: (RP): This panel type reviews competing submissions such as grant proposals. The number of individuals constituting this type depends upon the number and nature of submissions. However, each submission must be reviewed by at least three individuals who perform the review. Depending upon the desire of the sponsor, part or all of the report resulting from the peer review can be designated as proprietary. Type V Review of Competing Submissions (RP): This type of peer review is commonly used to review competing submissions such as grant proposals. The number of individuals constituting the RP for this type depends upon the number and nature of submissions. However, each submission must be reviewed individually by at least three reviewers. Depending upon the desire of the sponsor, either a part of, or the entire, report resulting from the peer review will be designated as proprietary. Type VI Rapid Response Review (AP or RP): Panels established for this purpose provide a review or an assessment of a limited technical issue that is needed in a short time frame. This type of review is a modified version of type II or type IV, but may require access to unique infrastructure and specific personnel. 7

10 Type VII: (AP or RP): Panels are established for this type to visit specific facilities and review or assess the planning, operation, and other aspects of specific projects at that facility. Type VIII: (AP or RP): Panels of this type review or assess classified information. For obvious reasons, the meeting of the panel is closed to all but those with appropriate security clearances. Furthermore, depending upon the desire of the sponsor, a part or all of the report resulting from the effort can be designated as classified. Procedures The reviews described in this manual require procedures that describe how to perform the process. Although some of these procedures are generic, others are by necessity specific to a program or a sponsor. Peer Review and Scientific Assessment Reports Each member of the RP or AP is expected to participate in the preparation of a report containing the outcome of the review. The comments of the panel are subsequently combined into a report containing some or all of the following parts: Introduction: The introduction or preface describes activities that led to the preparation of the report. This section is typically short and no more than 1-2 pages Executive Summary: This part briefly describes the project or subject, and a summary of criteria, findings and recommendations, typically 1 page. The Process: A description of the peer review or scientific assessment process is included in this part. The Project or Subject: For peer review this part describes a summary of the subject that was reviewed. For scientific assessment this part consists of several sub-parts describing relevant information on the topic that is being assessed. Criteria and Findings: This part of a peer review report contains the review criteria and findings of the RP, reporting the shortcomings and meritorious aspects of the project. This section of a scientific assessment report contains the findings of the AP responding to the assessment criteria. Recommendations: The recommendations of the panel (RP or AP) are derived from the findings of the panel and are the fundamental outcome of the process. References: Much like all scientific documents, this part includes references to documents used during the review or assessment. Biographical Summaries: The credibility of review or assessment depends greatly upon the individuals involved in the review or assessment process. This part includes biographical summaries of members of the panel and others who were involved in the process. Appendix: On occasion it becomes necessary to include information in the report that did not result from the review or assessment process. These include the text of relevant regulations, other documents found to be helpful to the sponsor, or comments by each reviewer upon which no consensus could be reached. 8

11 Stakeholder Participation There is ample evidence suggesting that participation of stakeholders enhances the appreciation of the decision process. In particular, the participation of stakeholders in peer reviews and scientific assessments increases the probability of their acceptance of solutions resulting from the peer review [cite(s) needed here then]. The example of a peer review addressing plans to contain radioactive materials at the Nevada Test Site can be used to demonstrate the point. During the cold war several hundred nuclear weapons were tested at that site including those below, at or above the groundwater; or even on the surface. Stakeholders participated in the Las Vegas City Council where the peer review was planned, Peer Review Meeting, and the meeting where the results of the peer review (ASME/RSI2001) were reported. The only meeting where stakeholders were not present was the executive session of the Review panel. The Report describing the stakeholder participation (Love and Moghissi 2002) demonstrate the effectiveness of the process. The review or assessment criteria are the technical issues of concern to the stakeholders. Consequently, these criteria should consider stakeholder concerns. Experience shows that comments by stakeholders are taken seriously by the panel members and thus provide a powerful incentive for stakeholder participation. The impact of comments by the stakeholders is the major reason for their acceptance of the results of review or assessment. REFERENCES ASME/RSI (American Society of Mechanical Engineers/Institute for Regulatory Science). Assessment of technologies supported by the Office of Science and Technology, Department of Energy; Results for Fiscal Year 2002.vol. 2. New York: ASME; ASME/RSI (American Society of Mechanical Engineers/Institute for Regulatory Science). Technical Peer Review Report; Report of the Review Panel; Strategy for remediation of groundwater contamination at the Nevada Test Site. New York: ASME; Moghissi AA, Love BL, Straja SR. Peer review and Scientific Assessment: A Handbook for funding organizations, regulatory agencies, and editors. Alexandria, VA 2013 (Available from Amzon.com) Love BL, Moghissi AA. Stakeholder participation in the ASME/RSI peer review Strategy for remediation of groundwater contamination at the Nevada Test Site. Alexandria, VA Institute for Regulatory Science 2002 NIH (National Institutes of Health) Scientific peer review of research grant applications and research and development contract projects. 42 CFR part 52h. Fed. Reg. 69: ; 2004 NA National Academies/Research Council) Peer review in environmental technology development programs. Washington DC, National Academy Press 1998 NA (National Academies/National Research Council). Strengthening Peer science at U.S. Environmental Protection Agency: Research management and peer review practices. Washington DC, National Academy Press 2000 NA (National Academies/National Research Council). Review procedures for water resources project planning. Washington DC, National Academy Press

12 NSF (National Science Foundation) National Science Board Summary Report of the May 9-10, 2013 Meeting. OMB (Office of Management and Budget). Final Information Quality Bulletin for Peer Review. Fed. Reg. 70: ;