Guidance producer: Egton Medical Information Systems Limited (EMIS) Guidance product: Clinical Immediate Reference articles Date: 17 August 2010 Final Accreditation Report
Contents Introduction... 3 Accreditation recommendation... 3 Reapplication for accreditation... 6 Appendix A: NHS Evidence accreditation analysis... 7 Appendix B: Bibliography... 20 Appendix C: Guidance subject to accreditation decision... 21 Appendix D: Advisory Committee members, external advisers and NHS Evidence accreditation team... 22
Introduction The NHS Evidence Accreditation Scheme recognises organisations that demonstrate high standards in producing health or social care guidance. Users of the accredited guidance can therefore have high confidence in the quality of the information. Organisations can publicly display a seal of approval called an Accreditation Mark for three years after their processes have been accredited. The process for accrediting producers of guidance and recommendations for practice is described in the process manual 1. Accreditation recommendation NHS Evidence has not accredited the process used by Egton Medical Information Systems Limited (EMIS) to produce Clinical Immediate Reference articles. Background to the guidance producer Egton Medical Information Systems Limited (EMIS) is a UK-based clinical information systems supplier to primary healthcare. EMIS produces two separate summary guidance streams: clinical immediate reference information patient information This accreditation decision applies to EMIS Clinical Immediate Reference (ECIR) articles, which summarise clinical information for primary care users. 1 http://www.nice.org.uk/nhsevidence/aboutaccreditation/aboutaccreditation.jsp?domedia=1&mid= 27C232A0-19B9-E0B5-D4A11FA899F4C219
ECIR articles are reproduced as Medical Immediate Reference (MIR) guides in the Mentor component of the Egton Medical Information System, which is sublicensed to other clinical information systems. The articles are also known as Patient Plus articles published on Patient UK (www.patient.co.uk), which is a joint venture between EMIS and Patient Information Publications (PiP). Summary The process for the development of ECIR articles are described in the manual, EMIS Clinical Immediate Reference Production, Jan 2010. EMIS s effort to improve the processes for developing ECIRs is recognised. The overall objective and recommendations for each ECIR are clearly stated. ECIR articles have been developed with the input of general practitioners who are the primary audience for the guidance. EMIS also provides support mechanisms for the implementation of ECIR articles such as flow charts, algorithms and calculators. The process meets 20 of the 25 accreditation criteria. However, two of the most critical criteria, those addressing the processes used to include or exclude evidence and how its strengths and weaknesses are taken into account, were only partially met. EMIS s process does not assess and appraise the primary evidence in enough detail which, combined with insufficient evidence of implementation of the new processes, is the main reason for the decision not to accredit. Although EMIS has a new mechanism for providing patient feedback on ECIR articles, it is not clear whether patients are included in the guidance development group and there is a lack of evidence of how this feedback has been implemented in guidance examples. Similarly, although EMIS has recently modified its process to ensure that organisational and financial barriers are explicitly taken into account when developing ECIR articles, and a new monitoring and audit system has been introduced, there is insufficient evidence of implementation of these promising developments.
The progress made by EMIS is acknowledged but principally because of the weaknesses in the rigour of development coupled with a lack of evidence of implementation of recent developments, the process cannot yet be recommended for NHS Evidence accreditation. Areas that did not meet the required standard for NHS Evidence accreditation and where improvement is needed are: EMIS needs to undertake its own independent, robust evaluation of the full primary evidence base for each ECIR article and include this information in an updated process manual The process document needs to incorporate full critical appraisal methods to evaluate the strengths and weaknesses of the evidence base The input of patients needs to be included in the guidance development group Full implementation of the new process described by EMIS needs to be demonstrated across ECIR articles. David Haslam Chair, Advisory Committee July 2010.
Reapplication for accreditation Following a final decision where accreditation has not been granted, guidance from the non-accredited producer will still be available on the NHS Evidence site but will not be identified by the accreditation mark graphic. Guidance producers that are not accredited following the accreditation process have the opportunity to reapply from one year after the previous assessment. It is assumed that the organisation will have addressed any concerns highlighted in the original assessment before reapplying. The NHS Evidence team will provide detailed feedback and advice on areas where improvement is required to meet the criteria in a future application.
Appendix A: NHS Evidence accreditation analysis The Advisory Committee considered the following analysis of the guidance producer s compliance with NHS Evidence accreditation criteria, which covers six discrete domains. The full analysis leading to the accreditation decision is shown below. Criterion Evidence for meeting the criterion Accreditation decision Does the guidance producer have a policy in place and adhered to that requires them to explicitly detail: Scope and purpose 1.1 Overall objective The overall objective is clearly described in the process document 1. It states under 1a, Role of ECIR that The purpose of ECIR articles is to provide front-line healthcare workers with an up-to-date, reliable, easily accessible and easily digestible source of information that they can refer to before, during or after consultations. ECIR articles 2 3 do not detail individual objectives but this is considered acceptable given the concise nature of the summaries.
Criterion Evidence for meeting the criterion Accreditation decision 1.2 The clinical, healthcare or social questions covered Section 1c of the process document 1 states ECIR clinical summaries are designed to provide answers to clinical questions from GPs, nurse practioners, casualty officers and specialists working outside their area of experience or are likely to be asked in consultations with patients. ECIR articles 2 3 contain specific clinical questions for which recommendations are suggested. 1.3 Population and/or target audience to whom the guidance applies Section 1b of the process document 1, ECIR Target Audience states that the patient population is primarily the UK general population while the target audience for ECIR articles is doctors and primary care clinical staff. ECIR articles 2 3 contain a generic statement explaining that articles are primarily written for general practioners. Each defines demographics of patients commonly affected, along with risk factors, investigational methods, treatment and management. 1.4 Guidance includes clear recommendations in reference to specific clinical, healthcare or social circumstances The process document 1 explains within section 1d that summary articles are organised into chapters mainly dealing with conditions and their treatments. The aim is to be as comprehensive as possible and to cover the information requirements of clinicians while they are consulting with patients in the limited time available. ECIR articles 2 3 provide clear and specific recommendations to manage each condition.
Criterion Evidence for meeting the criterion Accreditation decision Does the guidance producer have a policy in place and adhered to that means it includes: Stakeholder involvement 2.1 Individuals from all relevant stakeholder groups, including patient groups, in developing guidance 2.2 Patient and service user representatives and seeks patients views and preferences in developing guidance The process document 1 addresses the criterion in section 1f. ECIR articles 2 3 contain a generic statement acknowledging the lead author of the authoring team. The process document 1 states that feedback is actively encouraged and received from patients. However it is not clear that patients are included in the guidance development group. EMIS has a new mechanism for providing patient feedback on ECIR articles. EMIS supplied samples for ECIRs showing feedback from Not fully met patients is now being collected. This is stored until the author reviews the article and decides whether the patient opinions necessitate a change in the article. Authors have to confirm that patient opinion has been considered when articles are revised. However, as the process has only recently been revised guidance examples 2 3 have not yet considered patient feedback, as they have not yet undergone their scheduled update.
Criterion Evidence for meeting the criterion Accreditation decision 2.3 Representative intended users in developing guidance. Section 1f of the process document 1 supports this criterion. ECIR articles have a wide audience across general practice and hospital clinicians (via Patient UK through Patient Plus articles). Articles can be used by nonmedics if people want more detailed information. Users of guidance (both doctors and patients) are encouraged to provide feedback and comments which are fed back to the authoring team. Via a generic statement, ECIR articles 2 3 declare that general practioners are the primary target users and are included in the authoring and reviewing teams. The Acknowledgements section of each article states that the guidance has been scrutinised by an independent mentor GP reviewing team.
Does the guidance producer have a clear policy in place that: Rigour of development 3.1 Requires the guidance producer to use systematic methods to search for evidence and provide details of the search strategy A minimum generic search strategy to author ECIR articles is outlined on page 12 of the process document 1. Criteria are systematic and the suggested sources include guidance from the National Institute for Health and Clinical Excellence (NICE), the National Service Frameworks (NSFs), the Cochrane database, current Clinical Knowledge Summaries (CKS) information and a number of grey literature sources. The search strategy focuses on identifying a hierarchy of reliable evidence from existing primary guidance. Authors are instructed to perform a standard evidence, best-practice search strategy when researching or revising ECIR articles. ECIR articles 2 3 do not detail search strategies but do list articles used to produce them in the section Document references. This is considered acceptable given the concise nature of the summaries.
3.2 Requires the guidance producers to state the criteria and reasons for inclusion or exclusion of evidence identified by the evidence review 3.3 Describes the strengths and limitations of the body of evidence and acknowledges any areas of uncertainty The new process specifies that ECIR authors should critically appraise primary guidance and study articles for quality. However the methods appear to rely too much on the peer review evaluation undertaken by journals prior to publication. EMIS needs to undertake its own independent evaluation of the full primary evidence base for each ECIR article. Without this thorough approach, primary recommendations are prone to selection bias which is not in accordance with the development of high quality guidance. ECIR articles 2-3 do not demonstrate implementation of EMIS s new process because it has only been introduced very recently. The process manual 1 states that authors usually summarise existing guidance although sometimes use primary guidance. Where there is conflicting evidence concerning a treatment, EMIS states that all high quality evidence-based viewpoints are represented. The process document recommends that authors check the correspondence papers of ejournals as they can give insight to current areas of dispute and uncertainty. EMIS state that the authors of primary studies, systematic reviews or guidance on which ECIRs are based should have already considered the evidence base and methodology used. Examining journal correspondence papers is not considered a viable substitute for robust critical appraisal of the primary evidence base. This is a fundamental requirement for guidance. No evidence of implementation of the new process was evident from ECIR articles 2-3 as it has only been introduced very recently. Not fully met Not fully met
3.4 Describes the method used to arrive at recommendations (for example, a voting system or formal consensus techniques like Delphi consensus) According to the process manual 1, consensus is used as a method of making recommendations. In exceptional circumstances, the EMIS Knowledge Based Systems (EKBS) Head of Department is approached to resolve conflicts surrounding the evidence base and come to a consensus. ECIR articles are not published until medical reviewers, the proof reader and author are all in agreement regarding recommendations. 3.5 Requires the guidance producers to consider the health benefits against the side effects and risks in formulating recommendations Section 5 of the process document 1 recommends that the strengths and limitations of a treatment, management method or drug needs to be discussed between clinician and patient. ECIR articles provide adequate detail to enable a full discussion to take place. Detailed information about the likely benefits and risks are available through web links to the primary source article. Information is clearly evident from the example ECIR articles 2-3. 3.6 Describes the processes of external peer review The process document 1 describes the peer review policy. The authors of ECIR guidance and the team that reviews it are separate. 3.7 Describes the process of updating guidance and maintaining and improving guidance quality The section entitled Authoring, Advertising, and Funding in the process document 1 states that articles are routinely reviewed every two or three years. If there is no change in the way a condition is treated or managed then an article does not have to be reviewed until a maximum of five years has passed. ECIR articles 2 3 stipulate a review date and, if applicable, their last update.
Does the guidance producer ensure that: 4.1 Recommendations are specific, unambiguous and clearly identifiable Guidance documents 2 3 present recommendations for a treatment or delivery of a procedure or drug in a clear and concise manner. The process document 1 states that...summaries are designed to be both concise and readable without creating misunderstanding. Clarity and presentation 4.2 Different options for the management of the condition or options for intervention are clearly presented ECIR articles 2 3 show that alternatives for the management of a condition or health concern are discussed. The process document 1 makes numerous references to the concept that different options are included in guidance. 4.3 The date of search, the date of publication or last update and the proposed date for review are clearly stated Date information is provided in the ECIR articles 2 3 and defined in the process document 1.
4.4 The content of the guidance is suitable for the specified target audience. If patients or service users are part of this audience, the language should be appropriate. ECIR articles are primarily designed for clinicians according to the process document 1 in sections 1b and 1c. Example ECIR articles 2 3 state that they have been written for doctors via a generic statement at the start of each article. They are clearly suitable for this audience. Patients and service users are not the primary users of the guidance but if they require detailed clinical information, they can freely obtain it from Patient UK. The content is identical to that of ECIR articles. Some of the guidance has a patient or parallel version available specifically tailored for a lay audience. Patient information leaflets are available for both example articles.
Does the guidance producer routinely consider: 5.1 Publishing support tools to aid implementation of guidance The process document 1 states that ECIR articles are designed to be easily accessed and digested rapidly by busy clinicians during consultations with patients. It stipulates on page 13 that Where relevant, flowcharts and tables should be included when this aids understanding and decision making. Images, sound and video, tables and calculators maybe available as support tools. Additional processes supporting the criterion are stated within the Introduction (Appendix E, process document). Sources such as algorithms, tables and diagrams can be included in ECIR articles. Applicability 5.2 Discussion of potential organisational and financial barriers in applying its recommendations EMIS has modified its processes to ensure that organisational and financial barriers are explicitly taken into account. Patient groups and health professionals have mechanisms to provide feedback on potential barriers to implementation. A sample was provided for a specific ECIR Not fully met article that demonstrates where responses are captured. EMIS points out that as ECIRs are designed to be concise they do not have space to include feedback about potential organisational and financial barriers in ECIRs explicitly. However, as the process has only recently been revised and guidance examples 2 3 have not yet undergone their scheduled updates (2010 and 2012) feedback on potential barriers will not yet have been considered.
5.3 That their guidance is current, with review criteria for monitoring and/or audit purposes within each product. A new monitoring and audit system keeps documents up to date. The authoring notes on Patient UK have been updated to describe how feedback is evaluated, audited and monitored. To keep ECIR articles concise feedback is not included as a larger document would not be fit for purpose. Not fully met However, as the process has only recently been revised guidance examples 2 3 will not yet have incorporated any changes. Guidance examples have not yet undergone their scheduled updates (scheduled for 2010 and 2012).
Does the guidance producer: 6.1 Ensure editorial independence from the funding body EMIS states that it has a policy of not allowing external commercial organisations to sell or fund changes to the design or function of its clinical system. Authors are required to use their best efforts to reflect current evidence-based and best practice medicine. Editorial independence 6.2 Demonstrate transparency about the funding mechanisms for its guidance Creation and maintenance of ECIR articles 2 3 is entirely funded by EMIS. Section 2a, of the process document 1 is relevant to this criterion. The Patient UK web pages carry external adverts as a means of revenue which assists with the costs of supplying content, including ECIR articles, for free. The advertisements on Patient UK are clearly marked as such and are separate to the clinical summaries. Advertisements are made available by external suppliers and are not approved by Patient UK. Patient UK states that it is not part of the National Health Service (NHS). Income generated from advertising is forwarded to EMIS. 6.3 Record and state any potential conflicts of interest of individuals involved in developing the recommendations The process document 1 states that all team members are required to confirm their independence from any commercial pressures that may influence their writing. Appendix A details a template for writing a conflict of interest email, which is sent to members of the authoring and reviewing teams.
6.4 Take account of any Section 1f of the process manual 1 explains that authoring and reviewing potential for bias in the teams research and summarise information in an unbiased and conclusions or transparent manner. recommendations of the guidance 1 EMIS Clinical Immediate Reference Production, January 2010; 2 Breast Cancer, September 2009; 3 Hodgkin s Lymphoma, March 2008
Appendix B: Bibliography Appendix B lists the additional information taken into account in the analysis and considered by the Committee. Document name Description Location EMIS Clinical Immediate Reference Authoring. Version 1, January 2010. EMIS Clinical Immediate Reference Production. Version 1, January 2010. Breast Cancer, 2009 Process document (summary) Process document (full document) ECIR / MIR / Patient Plus summary guidance article (for GPs and primary care staff) Document provided for assessment Document provided for assessment http://www.patient.co.uk/doctor/bre ast-cancer.htm Breast Cancer, 2009 Patient information leaflet http://www.patient.co.uk/health/can cer-of-the-breast.htm Hodgkin s Lymphoma, 2008 ECIR / MIR / Patient Plus summary guidance article (for GPs and primary care staff) http://www.patient.co.uk/doctor/ho dgkin's-lymphoma.htm Hodgkin s Lymphoma, 2008 Patient information leaflet http://www.patient.co.uk/health/ly mphoma-hodgkin's.htm About us webpage on Web page containing process http://www.patient.co.uk/about.asp Patient UK information Patient Plus access to more in-depth information webpage on Patient UK Web page containing process information http://www.patient.co.uk/patientplus.asp Style guide (authoring hints and tips medical articles) Conflict of Interest Email Content Creation Editorial Independence (editorial disclaimer) Document supplied by EMIS detailing their ECIR / MIR / Patient Plus article authoring style guide Document supplied by EMIS detailing email sent to authoring and reviewing team etc to ascertain annual conflict of interest status Web page containing process information directly relating to conflict of interest issues Document provided for assessment Document provided for assessment http://www.patient.co.uk/editorial_di sclaimer.asp
Appendix C: Guidance subject to accreditation decision Guidance not accredited.
Appendix D: Advisory Committee members, external advisers and NHS Evidence accreditation team NHS Evidence Advisory Committee Members The NHS Evidence Advisory Committee operates as a standing advisory committee of the Board of the National Institute for Health and Clinical Excellence (NICE). The Committee provides advice to the Institute on a framework for accrediting sources of evidence that should be recognised as trusted sources of information for the NHS. The Chair of the Committee is appointed by the Institute s Board and the meetings are conducted by the Chair or in his/her absence the vice chair. The current Chair is David Haslam. A full list of the Advisory Committee membership is available on the NICE website 2. The members have been appointed for a period of 3 years. This may be extended by mutual agreement to a further term of 3 years and up to a maximum term of office of 10 years. The decisions of the Committee are arrived at by a consensus of those members present. The quorum is set at 50% of committee membership. The Committee submits its recommendations to the Institute s Guidance executive which acts under delegated powers of the Institute s Board in considering and approving its recommendations. 2 http://www.nice.org.uk/nhsevidence/nhseac.jsp
Committee members are asked to declare any interests in the guidance producer to be accredited. If it is considered that there is a conflict of interest, the member(s) is excluded from participating further in the discussions. A list of the committee members who took part in the discussions for this accreditation decision appears below. Title Name Surname Role Organisation Ms Judy Birch Lay member Mr Jim Blair Mr Richard Brownhill Consultant Nurse Learning Disabilities Clinical Development and Nurse practitioner St. George s Healthcare NHS Trust Calderdale and Huddersfield NHS Trust and Kirklees PCT Ms Parul Desai Director of Population Health NHS Information Centre Ms Joyce Epstein Lay member Dr Brian Fisher General Practitioner NHS Alliance (GP and national patient/public lead) Professor David Haslam National Clinical Advisor Care Quality Commission Dr Bobbie Jacobson Miss Ruth Liley Professor Stuart Logan Dr Donal O Donoghue Director of London Health Observatory, Vice Chair of Association of PH Observatories Assistant director of Quality Improvement Professor of Paediatric Epidemiology National Clinical Director for Kidney Care and Consultant Renal Physician London Health Observatory Marie Curie Cancer Care Peninsula College of Medicine and Dentistry Salford Royal NHS Foundation Trust Dr Carl Parker PEC Chair Stockton-on-Tees Teaching PCT Dr Mahendra Patel Senior lecturer in pharmacy practice, school of applied University of Huddersfield
sciences/consultant Pharmacist Mr Adrian Reyes- Hughes Associate Clinical Director NHS Direct Dr Karen Ritchie Ms Sasha Shepperd Ms Ann Slee Lead Health Services Researcher Senior Research Scientist, Department of Public Health Director of Pharmacy and Medicines Management NHS Quality Improvement Scotland University of Oxford University Hospitals Birmingham NHS Foundation Trust Dr Peter Smith Vice President National Association of Primary Care Dr Mark Strong Ms Gill Swash MRC Fellow, Section of Public Health Head of knowledge and Library Services School of Health and Related Research (ScHARR) University of Sheffield NHS Western Cheshire Dr Sara Twaddle Director Scottish Intercollegiate Guidelines Network Advisory Committee Deputies Title Name Surname Role Organisation Deputising for Ms Diane Gwynne- Smith Head of Knowledge Management Social Care Institute for Excellence (SCIE) Amanda Edwards Ms Rebecca Rees RCUK Academic Fellow Social Science Research Unit, University of London Sandy Oliver
External Advisers for Egton Medical Information Systems Limited; Clinical Immediate Reference articles accreditation application Dr Daniele Bryden, Consultant in Intensive Care Medicine / Anaesthesia, Sheffield Teaching Hospitals NHS Trust, Sheffield, UK Stephen Pilling, Director - CORE, Director - National Collaborating Centre for Mental Health Research, Department of Clinical, Educational and Health Psychology, University College London, London, UK NHS Evidence accreditation team for Egton Medical Information Systems Limited; Clinical Immediate Reference articles accreditation application Mr John Huston, Accreditation Technical Analyst, NHS Evidence, National Institute for Health and Clinical Excellence, Manchester, UK Dr Paul Chrisp, Associate Director Accreditation, NHS Evidence, National Institute for Health and Clinical Excellence, Manchester, UK