Questions for the Patient-Centered Outcomes Research Institute Peer Review Process Webinar (8/26/13)

Transcription

1 Questions for the Patient-Centered Outcomes Research Institute Peer Review Process Webinar (8/26/13) Clarification of Patient-Centeredness and Stakeholder Engagement Can PCORI provide more guidance on patient-centeredness: how can topics concerning health policy, health disparities, or healthcare quality which do not directly relate to individual patients be addressed? Response: For PCORI, patient-centered comparative effectiveness research answers questions that are important to patients and addresses outcomes that are meaningful to patients. Projects can address questions and outcomes relevant to patients whether they focus on the patientphysician interaction or on systems-level or policy-relevant questions. Researchers need to think through why the questions they are posing are important to patients and, whether systems- or policy-related, why it is important for patients to have those questions answered. Can you clarify the definition of sufficient stakeholder engagement? Response: PCORI has scheduled a webinar on meaningful engagement of patients and other stakeholders in research and to present examples. Title: Promising Practices of Meaningful Engagement in the Conduct of Research Date: Thursday September 19, 2013; 1:00 pm (ET) 2:00 pm (ET) Register here. 1 Webinar Objectives: 1) To feature successful practices of patient and stakeholder engagement in PCORI funded projects. 2) To explore approaches to engaging patients and stakeholder partners in all stages of the research process, from helping craft the research question to ensuring dissemination of the research results. 1 Questions for PCORI Peer Review Process Webinar 1

2 Application Process Could the web application system be enhanced to allow for saving of applications during the process? Response: We will be using different software with this new round. It will save partial applications, so that applicants can return to complete the application at a later time. We recognize that some applicants had challenges working with the previous software. Review and Scoring Are there plans to calibrate reviewer scores? Response: Reviewer training includes several examples, and reviewers are asked to complete a mock review and discuss it with PCORI staff. No further calibration is planned. Can PCORI change the requirement to list 3+ strengths and weaknesses in each category regardless of the score? This is excessive and not possible for all scores, e.g. high score with no apparent weaknesses. Response: We have revised the critique structure and while we still request that strengths and weaknesses be described, we no longer instruct reviewers to provide three or more strengths and weaknesses. How are discrepancies between patient and scientist reviewers accounted for within a review? Response: We realize that there is concern about potential discrepancies between scores of patient and stakeholder reviewers and those of the scientists on the panels. However, the critiques that are shared with applicants at the conclusion of the merit review were written before the panel discussions. Thus, the written comments provided on critiques do not reflect the conclusions of the reviewers following discussion. We have observed convergence in scores between the different categories of reviewers following the in-person discussion. The Summary Statement is provided to communicate the comprehensive evaluation of the application, including the in-person discussion among the reviewers. We have been collecting feedback from reviewers on multiple aspects of the review process, including receptiveness of reviewers to input from others as noted in the webinar presentation. On the basis of this input, we have been revising both the review criteria and the process for collecting and discussing application reviews. We do, however, value the range of perspectives obtained through merit review. Data regarding scientist, patient, and Questions for PCORI Peer Review Process Webinar 2

3 stakeholder agreement will be presented and discussed at the upcoming Peer Review Congress 2 in September. Reviewer Selection To what extent are reviewers familiar with governance, ethical and regulatory challenges raised by these types of projects? Response: Patient-centered outcomes research may raise unique issues related to project governance, ethical concerns, and human-subjects protection. Future trainings will incorporate more detailed information about governance, as more guidelines are provided regarding meaningful engagement in PCORI projects, and about human-subjects protections, as more information is obtained from PCORI awardees regarding specific human-subjects protection actions. We discuss these issues routinely at PCORI, and we continue to work with reviewers and applicants to highlight and address these issues. To what extent are the reviewers familiar with peer review, and are there plans to increase the number of reviewers familiar with the process? Response: PCORI is moving toward standing panels for the broad PFAs, to help ensure that the majority of panel participants have substantial background with the PCORI merit review process. For all panels, and particularly for our National Patient Centered Clinical Research Network review, we have worked to enhance the expertise of the review panel. It is our intention to include breadth of representation and perspectives to enhance the evaluation of the applications. Are there plans to include more community-based patients as reviewers rather than those with specific disease-focused advocacy interests? Response: The reviewer application process is open to all, and we encourage community-based patients to apply. We seek a mix of reviewers who bring a range of perspectives to the review process. Review Process The review process can be too prescriptive and burdensome with multiple criteria, the current interface with the review system and a large number of applications to review are there plans to revise? 2 Questions for PCORI Peer Review Process Webinar 3

4 Response: We have asked for and received reviewer feedback after the review sessions and are working on streamlining the process. For broad PCORI Funding Announcements (PFAs), we have instituted several process changes through the summer to reduce applicant and reviewer burden, including reducing the number of PCORI merit review criteria and streamlining the written critique process. For the CDRN and PPRN applications, we are reducing the reviewer burden and asking each reviewer to evaluate only three or four proposals. Can you explain how the 2-stage review approach works? Can you provide feedback to those who participate in stage 1, regarding the stage 2 process? Response: PCORI has discontinued the two-stage review process. Are there plans to clarify the guidelines for reviewers regarding discussion of other sources of funding, commercial and legal matters during the review? Response: Reviewer training addresses a range of issues relevant to application evaluation including resource source and use. Staff from PCORI s Contracts Management Department will attend the town hall session and answer questions related to these issues. Similar to the merit review process, will the patient and stakeholder reviewer s focus on only a select number of the review criteria? Response: We no longer have a two-stage approach to review. It was in place only for the first cycle of reviews. Reviews are now done by a single set of reviewers. This is a topic of ongoing discussion, but for now, all reviewers are invited to review across all criteria areas for the broad PFAs. However, they are not required to score on the basis of all criteria. The group discussion drives the scores toward appreciation of all criteria. For the CDRN and PPRN review, in light of the large number of review criteria, we are asking reviewers to limit, based on their areas of expertise, the number of criteria on which they review. Questions: Are there plans to revise the criteria to more closely match the NIH model of five criteria? The involvement of stakeholders and patients in the review process is novel. How will this involvement be evaluated as to changing the quality and impact of the research funded? Response: We have revised the number of criteria, so there are now five review criteria. However, they are not the same as those of NIH or AHRQ because our aims are distinct. Depending on the topic, some of our criteria may be modified, as we did for the Improving Methods Request for Applications (RFA), or expanded, as we did for the Clinical Data Research Networks RFA, which includes 14 criteria. We are tracking the evaluation process and will make information available at the next session. Questions for PCORI Peer Review Process Webinar 4

5 Can PCORI clarify and provide data regarding the triage process? Some applicants sent for review didn t seem to follow PCORI guidelines. Response: All applications are screened for responsiveness to the call. The highest-scoring applications are forwarded to in-person discussion at the panel meeting. Feedback to Applicants Are there plans to provide feedback and critiques for Letters of Intent (LOI)? Response: There are currently no plans to provide detailed feedback on Letters of Intent. What percentage of the LOI s received were invited to apply? Response: The Application Guidelines provide information about program area priorities. We departed from the typical process for the infrastructure awards in response to the complexity and precision required for those applications. Out of respect for the applicants, we wanted to provide go/no go feedback earlier in the process. As PCORI is still adding staff, and the Clinical Data Research Networks Patient-Powered Research Networks (CDRN-PPRN) applications were complex and specific, we did not have the staff available to provide feedback to individuals. We will provide specific information at a later date about the application process, including success rates. For those who have been asked to submit an application, the odds of receiving funding are much higher than for submissions for a typical PCORI announcement. Is it possible to give investigators feedback in the review letters and the range of scores, where their score fits into the distribution, and on those proposals determined to be high quality but not selected for funding? Response: The revised summary statements provide additional information about the evaluation of applications. More detailed information will not be provided for this round of review. Communication with PCORI Staff Are there plans to expand communication with portfolio managers beyond only? Response: We prefer to have individuals questions to us so that we have a record of them. We try to respond promptly to questions submitted to us. However, we now also accept phone calls when the question is complex. We have been adding staff to assist us with this process. Questions for PCORI Peer Review Process Webinar 5

6 CDRN and PPRN Networks Will PPRN s work with specific CDRN s or will there be a matrix how will they be interacting and will there be a steering committee? Response: There will be a steering committee that will be comprised of representatives from PCORI staff, CDRN and PPRN awardees, and key federal agencies. We anticipate that numerous relationships will develop between CDRNs and PPRNs. Though the steering committee will oversee the CDRN-PPRN relationships, we expect that these interactions will depend on the conditions represented by the PPRN, the awardees, as well as the cohorts represented in the CDRNs. Part of the role of the Coordinating Center will be to help facilitate these relationships. Who will sit on the coordinating center and how much power will the center have? Response: PCORI issued a Request for Proposal (RFP) for the Coordinating Center. The RFP on our website gives a detailed description of the duties and responsibilities of the Coordinating Center. The Center will provide support for the CDRNs and the PPRNs, and the Steering Committee will also function as a source of expertise for the networks; one of its roles is to provide technical assistance. Answer: How many applicants have been invited to submit a CDRN? We do not feel that it is constructive to state the actual numbers of applicants invited to submit full proposals. However, we assure all those who were invited that we consider your LOIs to have merit and promise. The payline will be considerably higher than the percent rate for the broad PCORI funding announcements. Further details regarding the application process will be disseminated as part of our Board of Governors reporting process. Consistent with PCOR and community-engaged research, are reviewers aware that their comments will be shared with patient and community groups, who can be disenfranchised by certain comments? Response: Reviewers are aware that their critiques will be made available to the applicants, and beyond that, confidentiality of the review process is stressed. Questions for PCORI Peer Review Process Webinar 6

7 We have been led to understand that the successful Network Coordinating Center (NCC) site may not be eligible to also participate in a CDRN application. Is this correct? If so, what impact will this have on CDRN proposal invitees that include a NCC? Response: Applicants are invited to apply for both, but if they are selected to be a Network Coordinating Center, they will have to choose whether they want to go that route or decline that opportunity in order to remain eligible for a CDRN contract. Will future "regular" PCORI contract awardees be required to do their work through the networks in the current call? Response: No, future recipients of PCORI funding will not be required to complete their work through the networks, although we plan to have specific money available for research conducted through the networks. Also, networks will be expected to collaborate with organizations outside the network to make certain research opportunities available. Will reviewers differentiate in the roles (as participants in the CDRN) of individual patients vs. patient advocacy groups? Will the patient reviewers be independent (unaffiliated with any group) or representatives of patient advocacy groups? Response: We are looking to get a range of perspectives, and so we mix individual patients with representatives of patient groups, which could include those from advocacy organizations. Are you planning to use the same combination of reviewers for the CDRN who were involved in reviews for the standing announcements? Response: Yes, we will use the same ratios of reviewers (two scientists, one patient, and one other stakeholder) for both the CDRN and the PPRN. To make sure we have the right reviewers at the table, we have spent significant staff time in selecting whom to invite to review these proposals. Will the Clinical Research Data networks include Practice Based Research Networks that would plan to expand to include patients? Response: Yes, the CDRNs will be able to include Practice Based Research Networks. Questions for PCORI Peer Review Process Webinar 7

8 CDRN and PPRN Networks and Letter of Intent Questions: Since the reviewers of CDRNs and PPRNs will have the LOIs or will have read them, what is advised approach as we make changes from LOI into full application? Will reviewers of the CDRN proposals see the LOI responses in addition to the full proposal? Response: The LOIs will not be made available to reviewers. Our expectation is that the proposal will expand upon the LOI. We expect the executive summary of the proposal to be similar to the LOI, although we recognize that proposals may change between the two submissions. Bio-Specimens Questions: A question regarding bio specimens: as I read the review criteria, you are asking that we have the capacity to manage bio specimens for prospective CER trials more so than the ability to reuse our existing bio specimen repositories for comparative effectiveness research of retrospective data (which may be harder to harmonize and reuse nationally). Can you discuss? You mentioned you are very explicit about what your expectations are from a CDRN. Between 1-10, 10 being extremely important, how important is having a Bio-repository capability to your expectations? Response: PCORI is interested in both the capability to manage new collections of bio-specimens for prospective trials and to reuse bio-specimens already in repositories. We recognize that there will be wide variation among applicants. We have not weighted the importance of any one criterion over another. Our overarching criteria as we evaluate each application will be an assessment of whether the applicants have explicitly described how they will make meaningful progress in developing and enhancing their infrastructure beyond its current state capacity, especially in regard to fulfilling the characteristics of the ideal network, as described in the funding announcements. We are not interested in funding organizations to continue what they are already doing. Applicants must show that PCORI funding presents an opportunity to support transformative development toward building a national patientcentered clinical research network. A number of questions are about the surveys, for example: Is a survey of a sample of cohort eligible patients likely to meet the requirement for that element? Response: As mentioned in the FAQs on our website, during the course of phase one, each of the three cohorts must be contacted and recruited to participate in the cohort and in a brief baseline survey. A high response rate to the survey is desirable, and a variety of approaches to conducting the survey may be proposed, including traditional mail or phone surveys or surveys using , social media, or mobile devices. Questions for PCORI Peer Review Process Webinar 8