Final Accreditation Report - for guidance and advice for use in decision support systems

Transcription

1 Guidance producer: NHS Direct Guidance product: The content of the Call Streaming Prioritisation Tool and Teleguides Date: 2 January 2013 Version: 1.4 Final Accreditation Report - for guidance and advice for use in decision support systems Page 1 of 24

2 Contents Introduction... 3 Accreditation recommendation... 3 Background to the guidance producer... 3 Implementation... 5 Appendix A: NICE Accreditation analysis... 6 Appendix B: Bibliography Appendix C: NICE Accreditation Advisory Committee, external advisers and NICE Accreditation team Page 2 of 24

3 Introduction The NICE Accreditation Scheme recognises organisations that demonstrate high standards in producing health or social care guidance. Users of the accredited guidance can therefore have high confidence in the quality of the information. Organisations may publicly display a seal of approval called an Accreditation Mark for 5 years after their processes have been accredited. The process for accrediting producers of guidance and recommendations for practice is described in the process manual on the NHS Evidence website. Accreditation recommendation NICE has accredited the process used by NHS Direct to produce the content of the Call Streaming Prioritisation Tool and Teleguides. Accreditation is valid for 5 years from January Background to the guidance producer NHS Direct provides a telephone service for people in England and Wales to get health advice based on their symptoms. Calls are handled by healthcare advisers and nurses, based on algorithms developed by NHS Direct. The algorithms are presented through a clinical decision support system (CDSS) comprising the Call Streaming Prioritisation Tool (CSPT) and teleguides. Referral recommendations are reached by progressing through the algorithm by answering questions about symptoms. Summary The accreditation programme is piloting the use of the accreditation criteria and process to assess how content is developed for a range of clinical decision support systems. NICE has accredited the process used by NHS Direct to develop the content of the CSPT and teleguides. The Accreditation Advisory Committee considered that the Page 3 of 24

4 process complied with all 25 criteria for accreditation. Because the content is used in a CDSS it is not available through NHS Evidence. Development of the content for both the CSPT and teleguides follows the same process, documented in CL055 NHS Direct Guidance clinical content development (2012) and CL056 NHS Direct Guidance for taking best practice clinical guidelines into account when developing CDSS (2012). The process involves a systematic search for evidence and input from a range of internal and external healthcare specialists, in addition to patient representation. There is careful consideration of the benefits, risks and side effects of recommendations and all content is checked for clinical safety before sign-off and testing. The process accounts for the possibility of bias through systematic methods, multidisciplinary stakeholder involvement, transparency over funding, and the requirement for all those involved to declare any conflicts of interest. Suggestions for improving the process used by NHS Direct to produce the content used in the CSPT and teleguides include providing more detail in the process documentation about the potential reasons for exclusion of evidence, the process of external peer review and the mechanisms for including patients views and preferences during the updating of existing content. Dr Carl Parker Vice Chair, Accreditation Advisory Committee December 2012 Page 4 of 24

5 Implementation Following accreditation, guidance from the accredited producer will be identified on NHS Evidence by the Accreditation Mark. The accredited guidance producer is also granted a royalty-free, worldwide licence to use the NICE Accreditation Mark in accordance with the Conditions and Terms of Use. Providing these conditions are met, a guidance producer's accreditation will last for 5 years from publication of approval on NHS Evidence website. Accredited guidance producers must take reasonable steps to ensure the accredited processes are followed when generating the type of evidence for which they are accredited. Accredited guidance producers should have quality assurance mechanisms in place and must inform NICE Accreditation within 30 days if any significant change is made to a process. Figure 1: The Accreditation Mark Page 5 of 24

6 Appendix A: NICE Accreditation analysis The Accreditation Advisory Committee considered the following analysis of the guidance producer s compliance with NICE Accreditation criteria, which covers 6 discrete domains. The full analysis leading to the accreditation decision is shown below. Criterion Evidence for meeting the criterion Accreditation decision Does the guidance producer have a policy in place and adhered to that requires them to explicitly detail: Scope and purpose 1.1 Overall objective The overall objective of the guidance is described in the document CL055 NHS Direct Guidance clinical content development a, and in the CDSS factsheet b. Healthcare advisers and nurses trained to use the CSPT or teleguides are familiar with the aims and objectives of the system to facilitate an assessment of patients symptoms. 1.2 The clinical, healthcare or social questions covered There is a process c to document the questions used to develop the guidance, and evidence of implementation. Within the guidance it is clear to the healthcare advisers and nurses which clinical questions are being addressed at any point, and a clinical rationale is provided for further information. Page 6 of 24

7 Criterion Evidence for meeting the criterion Accreditation decision 1.3 Population and/or target audience to whom the guidance applies The target population for the CDSS is very broad, being anyone who uses the telephone to call NHS Direct for advice. Specific populations are defined by symptoms as progress is made through the algorithms. The target audience for the CSPT is non-nurse healthcare advisers. The target audience for teleguides is NHS Direct nurses. 1.4 Guidance includes clear recommendations in reference to specific clinical, healthcare or social circumstances CL055 NHS Direct Guidance on clinical content development a ensures recommendations are given in reference to specific clinical or healthcare circumstances, as the content forms an algorithm in which healthcare advisers and nurses must answer questions on symptoms in order to progress to an appropriate recommendation. Does the guidance producer have a policy in place and adhered to that means it includes: Stakeholder involvement 2.1 Individuals from all relevant stakeholder groups, including patient groups, in developing guidance 2.2 Patient and service user representatives and seeks patient views and preferences in developing guidance There are processes a,d in place to ensure a range of stakeholder views are taken into account during development of the content for the CSPT and teleguides. This includes both professional and patient groups. The process a,d requires the involvement of patient groups during the development of new content. There are also mechanisms for gathering patients views and preferences during reviewing and updating of content, through feedback mechanisms and through requests for change. Page 7 of 24

8 Criterion Evidence for meeting the criterion Accreditation decision 2.3 Representative intended users in developing guidance. Target users are NHS Direct healthcare advisers and nurses, who are included as stakeholders and involved in development and testing of content a,d. They may also submit requests for change once the content has gone live on the system. Does the guidance producer have a clear policy in place that: Rigour of development 3.1 Requires the guidance producer to use systematic methods to search for evidence and provide details of the search strategy The process requires that systematic search methods are used for both new and updated content, and that the details of the search strategies are recorded. CL055 NHS Direct Guidance on clinical content development a states that searches should be systematic, carried out by staff with specialist research skills, and that the results including the search strategies should be documented on a research results template c. 3.2 Requires the guidance producers to state the criteria and reasons for inclusion or exclusion of evidence identified by the evidence review The process e requires that relevant NICE guidance, or NICE accredited guidance, is considered for inclusion. The reasons for exclusion of any evidence are required to be documented in the research results template c, and a summary of research is made available to end users. Page 8 of 24

9 Criterion Evidence for meeting the criterion Accreditation decision 3.3 Describes the strengths and limitations of the body of evidence and acknowledges any areas of uncertainty The research results template c requires the researcher to describe any strengths and limitations of the body of evidence and acknowledge any areas of uncertainty. The levels of evidence specified by the Scottish Intercollegiate Guidelines Network are provided to assist this. The documentation of the strength of the evidence informs the development of recommendations. 3.4 Describes the method used to arrive at recommendations (for example, a voting system or formal consensus techniques like Delphi consensus) 3.5 Requires the guidance producers to consider the health benefits against the side effects and risks in formulating recommendations The process a for reaching recommendations incorporated into the CDSS involves iterative review by different stakeholders before sign-off for testing. A process is described for reaching recommendations in the case of uncertainty in the evidence base. CL055 NHS Direct Guidance on clinical content development a provides a process requirement to ensure the risks of the recommendations are carefully considered before implementation. Examples of guidance f demonstrate discussion of the risks or side effects associated with different drugs. Benefits of treatment or intervention are considered during the appraisal of evidence. Page 9 of 24

10 Criterion Evidence for meeting the criterion Accreditation decision 3.6 Describes the processes of external peer review The testing stage, described in CL055 NHS Direct Guidance on clinical content development a, represents a process of external peer review. During this stage the clinical safety of the content is verified, and the content is tested with end-users to see if it works and to gather their feedback. Additionally there is a request for change process that allows end-users to suggest changes based on operational use. 3.7 Describes the process of updating guidance and maintaining and improving guidance quality The processes for scheduled and unscheduled updating of guidance are described in CL055 NHS Direct Guidance on clinical content development a. Specific sources of evidence are searched on a monthly basis and any new evidence is recorded on the evidence tracker g and assessed for its impact on existing content. Changes can be made during urgent, unscheduled updates or as part of a standard annual review for all content. Does the guidance producer ensure that: Clarity and presentation 4.1 Recommendations are specific, unambiguous and clearly identifiable There are processes in place that help to ensure the content is specific, unambiguous and clearly identifiable. Content is specific to patients circumstances because of the algorithmic nature of the guidance. The user testing phase and the request for change process help to ensure specificity and lack of ambiguity in the recommendations. Page 10 of 24

11 Criterion Evidence for meeting the criterion Accreditation decision 4.2 Different options for the management of the condition or options for intervention are clearly presented The CDSS is a large algorithm with many possible end points representing different options for treatment, intervention or referral. Progressing through the algorithm narrows down the options to the one that is most applicable given the answers provided at each stage, following a logical flow. Hard copies h display the algorithms so users are able to see the options available, which are clearly presented. 4.3 The date of search, the date of publication or last update and the proposed date for review are clearly stated The date ranges of searches are documented internally on the research results template c, and research summaries are provided on the NHS Direct intranet for users. The CSPT and teleguides f display the version number of the algorithms at the top of the screen, along with the date that they were last updated. The review dates for content are documented on the project initiation documents i and development plans j. 4.4 The content of the guidance is suitable for the specified target audience. If patients or service users are part of this audience, the language should be appropriate. Development of content for the CDSS involves clinicians, patients and target users a,d. User testing is also undertaken. There are processes to ensure that the content is suitable for the target audience and their needs. It can be seen from the screenshots of guidance f that the statements viewed by the healthcare advisers and nurses are clear and jargon free, although more specialist information is provided in a rationale section. Applicability Does the guidance producer routinely consider: Page 11 of 24

12 Criterion Evidence for meeting the criterion Accreditation decision 5.1 Publishing support tools to aid implementation of guidance There are a range of supporting tools and processes to aid implementation of the recommendations, by supporting staff to use the CDSS. These include clinical release notes k that describe significant changes to content for each version, educational materials l, hard copies of algorithms h, and the training that all healthcare advisers and nurses receive to use the system. 5.2 Discussion of potential organisational and financial barriers in applying its recommendations It is appropriate that the guidance producer considers the organisational and financial barriers of recommendations prior to incorporating them into the CDSS, as they have an impact on the wider health system. Barriers may include the capacity and resources of the ambulance service, accident and emergency department and primary care. There is evidence that these barriers are considered through monitoring of referral levels to these services m, with a drive to reduce referrals to emergency and urgent primary care, where clinically appropriate. 5.3 Review criteria for monitoring and/or audit purposes within each product. It is not applicable for this type of guidance to provide monitoring or audit criteria to the end-user, as they are not in a position to monitor or audit the patient beyond the length of the call. However there are processes to monitor service usage n and obtain feedback from users a. There are also processes to review and audit the guidance a. Page 12 of 24

13 Criterion Evidence for meeting the criterion Accreditation decision Does the guidance producer: 6.1 Ensure editorial independence from the funding body The funding body is the Department of Health. The multidisciplinary development group and a range of external stakeholders a,d in development helps to maintain independence from the funding source. Editorial independence 6.2 Demonstrate transparency about the funding mechanisms for its guidance 6.3 Record and state any potential conflicts of interest of individuals involved in developing the recommendations The funding source is identified as the Department of Health and the service is provided by the NHS, which is a public body. The annual reports and accounts of the guidance producer are publically available online o. Individuals involved in developing content are required to declare any conflicting interests as defined by the NHS Direct Gift and Hospital Policy p. This applies to both NHS Direct staff and external stakeholders. The policy describes how conflicts are recorded and managed. 6.4 Take account of any potential for bias in the conclusions or recommendations of the guidance There are processes in place to minimise and account for the potential for bias. These include systematic searching for evidence a,e, a multidisciplinary development group including external stakeholders a,d, the requirement for all those involved to declare any conflicts of interest p, and transparency over funding o. Page 13 of 24

14 Criterion Evidence for meeting the criterion Accreditation decision a CL055 NHS Direct Guidance clinical content development process document b CDSS factsheet process document c Research results template used to collate and described the results of searches d CL055 Stakeholder Lists details of stakeholder involvement e CL056 NHS Direct Guidance for taking best practice clinical guidelines into account when developing CDSS process document f Screenshots of guidance example screenshots of teleguides and the CSPT g Evidence tracker spreadsheet used to record and assess new evidence h Hard copies of algorithms paper versions of the algorithms used when the computer system is unavailable i Project initiation documents internal project documents detailing key milestones in development j Development plans internal project document detailing key milestones in development k Clinical release notes summary of important changes in each version of the algorithms to update users l Educational materials a range of training materials provided to users of the CDSS on specific clinical topics m Clinical sorting session slides slides demonstrating monitoring and consideration of referral levels to different services n Performance monitoring report usage statistics for NHS Direct o Annual reports and accounts available on the NHS Direct website Page 14 of 24

15 Appendix B: Bibliography Appendix B lists the additional information taken into account in the analysis and considered by the Committee. Document name Description Location Memo - CAS Planned Maintenance Down Memo for staff describing scheduled downtime and relevant procedures Time.doc Service Quality Service performance review Review Urgent document Contacts fullreport v0 10.doc Gift Policy including hospitality Hospitality Policy.pdf and declaring conflicts of interest NHS Direct Outlines best practice in the Clinical Audit Guidance management and delivery of v2 0.doc clinical audit within NHS Direct Summary of A monthly summary of best Best Practice Clinical practice clinical guidance Guidance Published published and details of SCDT July 12 v1.0.doc collaboration between NHS Direct and NICE NHSDirectRe NHS Direct records cordsmanagementpolic management policy y.pdf NHSDirectCo NHS Direct feedback and mplaintsfeedbackpolic complaints policy Page 15 of 24

16 Document name Description Location y.pdf NHSDirectPol icyproceduraldocument s.pdf Sets out the policies, principles and processes that NHS Direct uses to develop and manage all of its procedural documents NHSDirectAI NHS Direct adverse incident MPolicy.pdf management policy CAG Terms Terms of reference for the of Reference v3 0.doc Clinical Audit Group CDSS Describes the CDSS factsheet (CAS) V1.1.doc CL055 Details internal and external stakeholder lists.doc stakeholders slide - Slide explaining benefits to Commissioner Value commissioners of NHS Direct Breakdown (web and phone).ppt ACE findings and Findings of testing NHS recommendations slide Direct recommendations pack v0_30.pdf against clinicians advice ACE Focussed Exec Executive summary of above Summary v0_20.pdf report Adverse Incident NHS Direct adverse incident Management management operational Operational guidance Guidancev1 2 3 NP008B.pdf CA Draft Service performance review Page 16 of 24

17 Document name Description Location Interim Report P1 appropriateness v0 1.doc document CAS Best Practice Guidance Document.pdf Identification of potential problems with the service and best practice for addressing them CL NHS Process document governing Direct Guidance Clinical development of clinical Content Development content v3.doc CL NHS Process document for Direct Guidance for identifying and using taking clinical best evidence practice into account when developing CDSS v3.2 final.doc Clinical authority to Pre-implementation checklist deploy DSCN18 NHSD to ensure clinical safety prior template.xls to release Clinical Content Terms of reference for clinical External Clinical Peer peer review group (external Review Terms of stakeholders) Reference.doc Clinical release archive Checklist for closing a clinical checklist v1.0.doc release Clinical release notes Example of supporting notes V19.3.doc to explain changes in a clinical release Diabetes Conference Feedback from diabetes Page 17 of 24

18 Document name Description Location Success Flyer.doc stakeholder event Diabetes current and future content draft.doc Summary of current and proposed content for diabetes Diabetes Interim Report An overview of the findings for NHS Direct from a national stakeholder Conference 13 Feb event sponsored by NHS 2009 v3.doc Diabetes Diabetes Learning Support tools to aid learning Resource.doc about diabetes Diabetes Project Project planning document Plan.doc Diabetes steering group Agenda of diabetes steering agenda 9 Dec 08.doc group Diabetes Steering Membership list of diabetes Group members list.doc steering group Diabetes steering group Terms of reference for Terms of Reference diabetes steering group v2.doc Earache self care warm Example of completed or cold.doc research results template Evidence tracker Details of evidence-based V19.3.xls clinical information considered for version 19.3 Example of combined Tracker for issues raised by issue log.xls identification of new guidance Example of RFC log.xls Tracker for requests for change Example of tracker.xls Tracker for identified guidance Front Line Staff Internal newsletter article Page 18 of 24

19 Document name Description Location Influence Clinical Development.doc Internal peer review invitation sent to invitation.doc Directors of Nursing and senior clinicians regarding opportunity to participate in content development KH PT195 RFC3476 Completed research results apnoea.doc template Learning and Development CAS V.19.3 v0.8.doc Learning resource which contains some of the changes to version 19.3 clinical content Managing Support learning resource Contraceptive Enquiries about contraceptives Guidance.doc Meningicoccal disease Service performance review CA interim report document v1.doc P1 performance Service performance review reporting.doc document PID 1183 NHSD v18 Project planning document Algorithms v1 2.doc for version18 clinical release QIP Terms of Terms of reference for the Reference v1.1.doc quality improvement panel Request for Approval Internal document seeking clinical releases approval for upcoming clinical 2009.doc releases Res886 - common Checklist for assessing the Page 19 of 24

20 Document name Description Location cold.doc impact of identified guidance Res1072 vaginosis.doc Gap analysis for identified guidance research results Blank research results templatec.doc template Risk of Example of questions and deterioriation.doc additional background information diseases where a patient s condition can deteriorate rapidly SCDT Terms of Terms of reference for the Reference 2010 FINAL Senior Clinical Directorate V2.2.doc Team Scorecard Template Service performance review July 2012.xls document Self care for An example of the self-care management of statements for the abdominal pain.doc management of abdominal pain Sudden death peer Correspondence between the review.doc internal and external peer reviewers as part of the sudden death content development System Management Terms of reference for the Steering Board Terms System Management of Reference - V1.0.doc Steering Board Terms of reference - Terms of reference for the change review Change Review Group group.doc Page 20 of 24

21 Document name Description Location Terms of reference - Terms of reference for the clinical panel.doc Clinical Panel TOR PPIWG VO2 Draft Terms of reference for the July 2011 (3).doc Patient and Public Involvement Working Group Toxic Substance Briefing for clinical audit to Clinical Audit measure the extent to which Briefing.doc NHS Direct staff achieve best practice in the management of toxic substance calls Toxic Substance Clinical audit to measure the Clinical Audit - Interim extent to which NHS Direct Report.doc staff achieve best practice in the management of toxic substance calls Users request for Outlines the escalation and change v4 (3).pdf change control process for dealing with a user s request for change V11 hot fix development Tracker for hot fix issues plans.xls V19.3 Change Version control document Sheet.xls V20 Change Sheet.xls Version control document v20 development Project planner for plans.xls development of version 20 Page 21 of 24

22 Appendix C: NICE Accreditation Advisory Committee, external advisers and NICE Accreditation team NICE Accreditation Advisory Committee The NICE Accreditation Advisory Committee operates as a standing advisory committee of the Board of the National Institute for Health and Clinical Excellence (NICE). The Committee provides advice to NICE on a framework for accrediting sources of evidence that should be recognised as trusted sources of information for the NHS. The Chair of the Committee is appointed by NICE Board and the meetings are conducted by the chair or in his/her absence the vice chair. The current Chair is David Haslam. A full list of the Accreditation Advisory Committee membership is available on the NICE website 1. Members are appointed for a period of 3 years. This may be extended by mutual agreement for a further 3 years, up to a maximum term of office of 10 years. The decisions of the Committee are arrived at by a consensus of the members present. The quorum is set at 50% of committee membership. The Committee submits its recommendations to the NICE Publications executive which acts under delegated powers of the NICE Board in considering and approving its recommendations. Committee members are asked to declare any interests in the guidance producer to be accredited. If it is considered that there is a conflict of interest, the member(s) is excluded from participating further in the discussions. Committee members who took part in the discussions for this accreditation decision are listed below. Title Name Surname Role Organisation Ms Judy Birch Lay Member Mr Jim Blair Consultant Nurse Learning Disabilities St. George's Healthcare NHS Trust 1 Page 22 of 24

23 Mr Richard Brownhill Unscheduled Care Lead Stepping Hill Hospital Stockport Foundation Trust Professor Ann Caress Professor of Nursing/Director of postgraduate research programmes University of Manchester Ms Ailsa Donnelly Lay Member Ms Amanda Edwards Deputy Chief Executive Social Care Institute for Ms Joyce Epstein Lay Member Excellence Dr Bobbie Jacobson Director London Health Observatory Dr Carl Parker Primary Care Medical Advisor North Tees and Hartlepool Foundation Trust Dr Mahendra Patel Research Fellow, Senior Lecturer and Practising Community Pharmacist Professor Sasha Shepperd Professor of Health Services Research Universities of Huddersfield and Bradford University of Oxford Dr Pete Smith Vice President National Association of Primary Care Dr Mark Strong Medical Research Council Fellow Ms Gill Swash Head of Knowledge and Library Services Dr Sara Twaddle Head of Evidence & Technologies School of Health and Related Research (ScHARR) University of Sheffield NHS Western Cheshire Healthcare Improvement Scotland External Advisers for the content of the NHS Direct Call Streaming Prioritisation Tool and Teleguides accreditation application Dr Carol Bond, Senior Lecturer, Health Informatics, School of Health and Social Care, Bournemouth University, UK Mandy Sainty, Research and Development Manager, College of Occupational Therapists, London, UK Page 23 of 24

24 NICE Accreditation team for the content of the NHS Direct Call Streaming Prioritisation Tool and Teleguides accreditation application James Stone, Accreditation Technical Analyst, National Institute for Health and Clinical Excellence, Manchester, UK Stephanie Birtles, Accreditation Technical Analyst, National Institute for Health and Clinical Excellence, Manchester, UK Page 24 of 24