HEALTH TECHNOLOGY ASSESSMENT

Transcription

1 HEALTH TECHNOLOGY ASSESSMENT VOLUME 19 ISSUE 13 FEBRUARY 2015 ISSN The cinica effectiveness and cost-effectiveness of teephone triage for managing same-day consutation requests in genera practice: a custer randomised controed tria comparing genera practitioner-ed and nurse-ed management systems with usua care (the ESTEEM tria) John L Campbe, Emiy Fetcher, Nicky Britten, Coin Green, Tim Hot, Vaerie Lattimer, David A Richards, Suzanne H Richards, Chris Saisbury, Rod S Tayor, Raff Caitri, Vicky Bowyer, Katherine Chapin, Rebecca Kandiyai, Jamie Murdoch, Linnie Price, Juia Roscoe, Anna Varey and Fiona C Warren DOI /hta19130

2

3 The cinica effectiveness and cost-effectiveness of teephone triage for managing same-day consutation requests in genera practice: a custer randomised controed tria comparing genera practitioner-ed and nurse-ed management systems with usua care (the ESTEEM tria) John L Campbe, 1* Emiy Fetcher, 1 Nicky Britten, 2 Coin Green, 2 Tim Hot, 3 Vaerie Lattimer, 4 David A Richards, 2 Suzanne H Richards, 1 Chris Saisbury, 5 Rod S Tayor, 1 Raff Caitri, 1 Vicky Bowyer, 6 Katherine Chapin, 2 Rebecca Kandiyai, 1 Jamie Murdoch, 4 Linnie Price, 1 Juia Roscoe, 6 Anna Varey 4 and Fiona C Warren 1 1 Primary Care Research Group, University of Exeter Medica Schoo, Exeter, UK 2 Institute of Heath Service Research, University of Exeter Medica Schoo, Exeter, UK 3 Department of Primary Care Heath Sciences, University of Oxford, Oxford, UK 4 Schoo of Nursing Sciences, Facuty of Medicine and Heath Sciences, University of East Angia, Norwich, UK 5 Centre for Academic Primary Care, Schoo of Socia and Community Medicine, University of Bristo, Bristo, UK 6 Department of Heath Sciences, Warwick Medica Schoo, University of Warwick, Coventry, UK *Corresponding author Decared competing interests of authors: none Pubished February 2015 DOI: /hta19130

4

5 This report shoud be referenced as foows: Campbe JL, Fetcher E, Britten N, Green C, Hot T, Lattimer V, et a. The cinica effectiveness and cost-effectiveness of teephone triage for managing same-day consutation requests in genera practice: a custer randomised controed tria comparing genera practitioner-ed and nurse-ed management systems with usua care (the ESTEEM tria). Heath Techno Assess 2015;19(13). Heath Technoogy Assessment is indexed and abstracted in Index Medicus/MEDLINE, Excerpta Medica/EMBASE, Science Citation Index Expanded (SciSearch ) and Current Contents / Cinica Medicine.

6

7 Heath Technoogy Assessment HTA/HTA TAR ISSN (Print) ISSN (Onine) Impact factor: Heath Technoogy Assessment is indexed in MEDLINE, CINAHL, EMBASE, The Cochrane Library and the ISI Science Citation Index and is assessed for incusion in the Database of Abstracts of Reviews of Effects. This journa is a member of and subscribes to the principes of the Committee on Pubication Ethics (COPE) ( Editoria contact: nihredit@southampton.ac.uk The fu HTA archive is freey avaiabe to view onine at Print-on-demand copies can be purchased from the report pages of the NIHR Journas Library website: Criteria for incusion in the Heath Technoogy Assessment journa Reports are pubished in Heath Technoogy Assessment (HTA) if (1) they have resuted from work for the HTA programme, and (2) they are of a sufficienty high scientific quaity as assessed by the reviewers and editors. Reviews in Heath Technoogy Assessment are termed 'systematic' when the account of the search appraisa and synthesis methods (to minimise biases and random errors) woud, in theory, permit the repication of the review by others. HTA programme The HTA programme, part of the Nationa Institute for Heath Research (NIHR), was set up in It produces high-quaity research information on the effectiveness, costs and broader impact of heath technoogies for those who use, manage and provide care in the NHS. Heath technoogies are broady defined as a interventions used to promote heath, prevent and treat disease, and improve rehabiitation and ong-term care. The journa is indexed in NHS Evidence via its abstracts incuded in MEDLINE and its Technoogy Assessment Reports inform Nationa Institute for Heath and Care Exceence (NICE) guidance. HTA research is aso an important source of evidence for Nationa Screening Committee (NSC) poicy decisions. For more information about the HTA programme pease visit the website: This report The research reported in this issue of the journa was funded by the HTA programme as project number 08/53/15. The contractua start date was in November The draft report began editoria review in November 2013 and was accepted for pubication in Apri The authors have been whoy responsibe for a data coection, anaysis and interpretation, and for writing up their work. The HTA editors and pubisher have tried to ensure the accuracy of the authors report and woud ike to thank the reviewers for their constructive comments on the draft document. However, they do not accept iabiity for damages or osses arising from materia pubished in this report. This report presents independent research funded by the Nationa Institute for Heath Research (NIHR). The views and opinions expressed by authors in this pubication are those of the authors and do not necessariy refect those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Heath. If there are verbatim quotations incuded in this pubication the views and opinions expressed by the interviewees are those of the interviewees and do not necessariy refect those of the authors, those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Heath. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. Pubished by the NIHR Journas Library ( produced by Prepress Projects Ltd, Perth, Scotand (

8 Editor-in-Chief of Heath Technoogy Assessment and NIHR Journas Library Professor Tom Waey Director, NIHR Evauation, Trias and Studies and Director of the HTA Programme, UK NIHR Journas Library Editors Professor Ken Stein Chair of HTA Editoria Board and Professor of Pubic Heath, University of Exeter Medica Schoo, UK Professor Andree Le May Chair of NIHR Journas Library Editoria Group (EME, HS&DR, PGfAR, PHR journas) Dr Martin Ashton-Key Consutant in Pubic Heath Medicine/Consutant Advisor, NETSCC, UK Professor Matthias Beck Chair in Pubic Sector Management and Subject Leader (Management Group), Queen s University Management Schoo, Queen s University Befast, UK Professor Aieen Carke Professor of Pubic Heath and Heath Services Research, Warwick Medica Schoo, University of Warwick, UK Dr Tessa Criy Director, Crysta Bue Consuting Ltd, UK Dr Peter Davidson Director of NETSCC, HTA, UK Ms Tara Lamont Scientific Advisor, NETSCC, UK Professor Eaine McCo Director, Newcaste Cinica Trias Unit, Institute of Heath and Society, Newcaste University, UK Professor Wiiam McGuire Professor of Chid Heath, Hu York Medica Schoo, University of York, UK Professor Geoffrey Meads Professor of Heath Sciences Research, Facuty of Education, University of Winchester, UK Professor John Powe Consutant Cinica Adviser, Nationa Institute for Heath and Care Exceence (NICE), UK Professor James Raftery Professor of Heath Technoogy Assessment, Wessex Institute, Facuty of Medicine, University of Southampton, UK Dr Rob Riemsma Reviews Manager, Keijnen Systematic Reviews Ltd, UK Professor Heen Roberts Professor of Chid Heath Research, UCL Institute of Chid Heath, UK Professor Heen Snooks Professor of Heath Services Research, Institute of Life Science, Coege of Medicine, Swansea University, UK Pease visit the website for a ist of members of the NIHR Journas Library Board: Editoria contact: nihredit@southampton.ac.uk NIHR Journas Library

9 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Abstract The cinica effectiveness and cost-effectiveness of teephone triage for managing same-day consutation requests in genera practice: a custer randomised controed tria comparing genera practitioner-ed and nurse-ed management systems with usua care (the ESTEEM tria) John L Campbe, 1* Emiy Fetcher, 1 Nicky Britten, 2 Coin Green, 2 Tim Hot, 3 Vaerie Lattimer, 4 David A Richards, 2 Suzanne H Richards, 1 Chris Saisbury, 5 Rod S Tayor, 1 Raff Caitri, 1 Vicky Bowyer, 6 Katherine Chapin, 2 Rebecca Kandiyai, 1 Jamie Murdoch, 4 Linnie Price, 1 Juia Roscoe, 6 Anna Varey 4 and Fiona C Warren 1 1 Primary Care Research Group, University of Exeter Medica Schoo, Exeter, UK 2 Institute of Heath Service Research, University of Exeter Medica Schoo, Exeter, UK 3 Department of Primary Care Heath Sciences, University of Oxford, Oxford, UK 4 Schoo of Nursing Sciences, Facuty of Medicine and Heath Sciences, University of East Angia, Norwich, UK 5 Centre for Academic Primary Care, Schoo of Socia and Community Medicine, University of Bristo, Bristo, UK 6 Department of Heath Sciences, Warwick Medica Schoo, University of Warwick, Coventry, UK *Corresponding author john.campbe@exeter.ac.uk Background: Teephone triage is proposed as a method of managing increasing demand for primary care. Previous studies have invoved sma sampes in imited settings, and focused on nurse roes. Evidence is imited regarding the impact on primary care workoad, costs, and patient safety and experience when triage is used to manage patients requesting same-day consutations in genera practice. Objectives: In comparison with usua care (UC), to assess the impact of GP-ed teephone triage (GPT) and nurse-ed computer-supported teephone triage (NT) on primary care workoad and cost, patient experience of care, and patient safety and heath status for patients requesting same-day consutations in genera practice. Design: Pragmatic custer randomised controed tria, incorporating economic evauation and quaitative process evauation. Setting: Genera practices (n = 42) in four regions of Engand, UK (Devon, Bristo/Somerset, Warwickshire/ Coventry, Norfok/Suffok). Participants: Patients requesting same-day consutations. Interventions: Practices were randomised to GPT, NT or UC. Data coection was not binded; however, anaysis was conducted by a statistician binded to practice aocation. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. vii

10 ABSTRACT Main outcome measures: Primary primary care contacts [genera practice, out-of-hours primary care, accident and emergency (A&E) and wak-in centre attendances] in the 28 days foowing the index consutation request. Secondary resource use and costs, patient safety (deaths and emergency hospita admissions within 7 days of index request, and A&E attendance within 28 days), heath status and experience of care. Resuts: Of 20,990 eigibe randomised patients (UC n = 7283; GPT n = 6695; NT n = 7012), primary outcome data were anaysed for 16,211 patients (UC n = 5572; GPT n = 5171; NT n = 5468). Compared with UC, GPT and NT increased primary outcome contacts (over 28-day foow-up) by 33% [rate ratio (RR) 1.33, 95% confidence interva (CI) 1.30 to 1.36] and 48% (RR 1.48, 95% CI 1.44 to 1.52), respectivey. Compared with GPT, NT was associated with a margina increase in primary outcome contacts by 4% (RR 1.04, 95% CI 1.01 to 1.08). Triage was associated with a redistribution of primary care contacts. Athough GPT, compared with UC, increased the rate of overa GP contacts (face to face and teephone) over the 28 days by 38% (RR 1.38, 95% CI 1.28 to 1.50), GP face-to-face contacts were reduced by 39% (RR 0.61, 95% CI 0.54 to 0.69). NT reduced the rate of overa GP contacts by 16% (RR 0.84, 95% CI 0.78 to 0.91) and GP face-to-face contacts by 20% (RR 0.80, 95% CI 0.71 to 0.90), whereas nurse contacts increased. The increased rate of primary care contacts in triage arms is argey attributabe to increased teephone contacts. Estimated overa patient cinician contact time on the index day increased in triage (GPT = 10.3 minutes; NT = 14.8 minutes; UC = 9.6 minutes), athough patterns of cinician use varied between arms. Taking account of both the pattern and duration of primary outcome contacts, overa costs over the 28-day foow-up were simiar in a three arms (approximatey 75 per patient). Triage appeared safe, and no differences in patient heath status were observed. NT was somewhat ess acceptabe to patients than GPT or UC. The process evauation identified the compexity associated with introducing triage but found no consistency across practices about what works and what does not work when impementing it. Concusions: Introducing GPT or NT was associated with a redistribution of primary care workoad for patients requesting same-day consutations, and at simiar cost to UC. Athough triage seemed to be safe, investigation of the circumstances of a arger number of deaths or admissions after triage might be warranted, and monitoring of these events is necessary as triage is impemented. Tria registration: Current Controed Trias ISRCTN Funding: This project was funded by the NIHR Heath Technoogy Assessment programme and wi be pubished in fu in Heath Technoogy Assessment; Vo. 19, No. 13. See the NIHR Journas Library website for further project information. viii NIHR Journas Library

11 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Contents List of tabes List of figures Gossary List of abbreviations Pain Engish summary Scientific summary xiii xvii xix xxi xxiii xxv Chapter 1 Introduction 1 Scientific background and review of current iterature 1 Feasibiity of teephone triage 1 Primary care workoad 1 Cost 1 Patient experience of care 2 Patient safety 2 Patient heath status 2 Rationae for the research 2 Aims and objectives 3 Chapter 2 Methods 5 Study design 5 Piot study 5 Ethica and governance arrangements 5 Patient and Pubic Invovement 5 Heath technoogies assessed 6 Triage interventions 6 Core triage processes 6 Areas of fexibiity 7 Training practices in the two triage systems 7 Computer decision support in nurse triage 8 Usua care comparator 10 Incusion and excusion criteria 10 Practices 10 Patients 11 Recruitment and randomisation procedures 11 Practices 11 Patients 13 Outcome measures 16 Describing practices and patients 16 Primary outcome measure 16 Secondary outcome measures 17 Sampe size 18 Feasibiity of patient recruitment 19 Secondary outcomes 19 Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. ix

12 CONTENTS Tria deivery in participating practices 20 Run-in period 20 Data coection period 20 Data coection and management 21 Cinician Form 22 Patient questionnaire 22 Case note review 22 Binding 22 Data entry 22 Monitoring intervention fideity 23 Integrity checks 23 Use of computerised decision support software in nurse-ed teephone triage practices 23 Statistica anaysis 24 Practice and patient characteristics 24 Primary outcome 24 Secondary outcomes 25 Economic evauation 26 Aim 26 Method 26 Process evauation 29 Aims and objectives 29 Methods 29 Data coection 31 Data anaysis 32 Chapter 3 Resuts 33 Fow of practices and patient participants in the tria 33 Practices 33 Patients 33 Patient oss to foow-up 36 Describing usua care 36 Practice descriptions and patient demographic characteristics 38 Practice descriptors 38 Patient demographics 38 Cinica outcomes 40 Primary outcomes 40 Additiona anayses based on primary outcome 42 Safety outcomes 42 Patient management on the index day 47 Resource use 49 Patient sef-reported heath status (European Quaity of Life-5 Dimensions) 52 Patient experience 52 Case compexity 52 Economic evauation 54 Estimate of the intervention cost (unit cost) for the triage interventions 54 Economic anayses 59 Process evauation 68 Structure of the resuts 68 How the tria was experienced 68 How the teephone triage interventions are impemented in different practice settings 76 How triage was experienced 79 Acceptabiity of triage 91 What infuences the extent to which the teephone triage interventions are seen to work or not work? 92 x NIHR Journas Library

13 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Chapter 4 Discussion and concusions 95 Summary of findings 95 Primary care contacts in the 28 days foowing a same-day consutation request 95 Care on the index day 96 Resource use 97 Economic anaysis 97 Safety 98 Patient heath status 98 Patient experience of care 99 Research findings in context 99 Workoad 99 Safety 100 Patient experience 101 Process evauation 101 Strengths and imitations 102 Impications for future research 105 Impications for heath care 107 Acknowedgements 111 References 115 Appendix 1 ESTEEM tria: piot study report 121 Appendix 2 Summary of changes to origina ESTEEM protoco 135 Appendix 3 Audit Log Sheet 137 Appendix 4 Receptionist fow charts 139 Appendix 5 Questionnaire for withdrawn practices 143 Appendix 6 Covering invitation etters 145 Appendix 7 Patient information eafet 147 Appendix 8 Patient questionnaire 153 Appendix 9 Receptionist Tria Log Sheet 157 Appendix 10 Adverse events reporting procedure 159 Appendix 11 Cinician Form 163 Appendix 12 Case note review 165 Appendix 13 Overview of practice integrity checks 167 Appendix 14 Assessment of nurses use of computerised decision support software during ESTEEM 169 Appendix 15 Practice Profie Questionnaire 173 Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xi

14 CONTENTS Appendix 16 Avaiabiity of patient information throughout the tria, consent status and initia patient management 179 Appendix 17 Practice Profie Questionnaire: additiona data 183 Appendix 18 Predictors of case note review and questionnaire return 185 Appendix 19 Tota primary care and accident and emergency contacts within 28 days of index day (primary outcome) across tria arm: per-protoco anaysis 189 Appendix 20 Interactions between intervention and practice-eve variabes, and between intervention and age category/gender/ethnic group, for primary outcome 191 Appendix 21 Patient-reported heath status (European Quaity of Life-5 Dimensions): individua question frequencies 193 Appendix 22 Patient experience of care: individua question frequencies 195 Appendix 23 Case compexity 197 Appendix 24 ESTEEM intervention resource use and cost estimates 199 Appendix 25 Data used to estimate staff contact time for triage contact (source: Cinician Form data) 203 Appendix 26 Mean triage contact time by practice 205 Appendix 27 Aternative unit costs for genera practitioner and nurse consutations 207 Appendix 28 Sensitivity anayses using aternative unit costs for out-of-hours contacts 209 Appendix 29 Tota number of contacts by consutation type (from case note review) 211 xii NIHR Journas Library

15 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 List of tabes TABLE 1 Sampe size cacuation 19 TABLE 2 Outcome data, timing of data coection and source of data 21 TABLE 3 Inter-rater reiabiity of case note review data extraction 22 TABLE 4 Unit cost data (2012) with data sources for the base case 28 TABLE 5 Process evauation: participating practices by region 29 TABLE 6 Process evauation participating practice staff 31 TABLE 7 Process evauation participating patients 31 TABLE 8 Estimate of the proportion of patients seeking same-day consutations who were ineigibe for the tria 36 TABLE 9 Practice Profie Questionnaire 37 TABLE 10 Practice demographics and custer size coefficient of variation 38 TABLE 11 Baseine patient demographics 39 TABLE 12 Primary outcome measure: tota primary care contacts (incuding A&E) within 28 days of index consutation request across tria arms 41 TABLE 13 Sensitivity anayses based on the primary outcome 43 TABLE 14 Comparison of safety outcomes based on contacts across tria arms 46 TABLE 15 Patient management on index day 47 TABLE 16 Descriptive summary of individua contact types comprising the primary outcome and patient-reported use of NHS Direct 50 TABLE 17 Duration of consutations during tria 51 TABLE 18 Patient-reported heath status (EQ-5D): ITT anaysis 52 TABLE 19 Patient experience of care: ITT anaysis 53 TABLE 20 Training and set up schedue for triage interventions 54 TABLE 21 Organisationa training for triage interventions, staff attending (mean count by grade) and mean duration of training 54 Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xiii

16 LIST OF TABLES TABLE 22 Triage-specific training for triage interventions, staff attending (mean count by grade) and mean duration of training 55 TABLE 23 Cinician contact time for triage intervention, by intervention type 55 TABLE 24 Unit costs used in estimate of triage intervention costs 56 TABLE 25 Mean cost of per-practice training and software required for triage interventions, and estimate of mean cost per same-day appointment request 58 TABLE 26 Estimated unit cost for triage interventions per same-day appointment request 58 TABLE 27 Summary tabe of sensitivity anayses and impact of different assumptions on estimate of unit cost for triage intervention (for same-day request) 58 TABLE 28 Description of primary cost data (summary view) and primary anayses with comparison by intervention 60 TABLE 29 Description of primary cost data (summary view) primary cost anayses with comparison by intervention type 61 TABLE 30 Secondary anayses and sensitivity anayses against primary cost anayses, tota 28-day cost 62 TABLE 31 Same-day contacts: description of data, base case (detaied view) and exporatory economic anayses on the same day as appointment request 64 TABLE 32 Same-day contacts: exporatory anaysis and costs for primary outcome contacts on the same day as appointment request 65 TABLE 33 Duration of face-to-face consutations foowing triage for ESTEEM patients 66 TABLE 34 Cost consequences matrix, UC costs and consequences compared with triage interventions 67 TABLE 35 Overview of the process evauation practices 69 TABLE 36 Summarising evidence from ESTEEM considered in reference to UC 109 TABLE 37 Summary tabe of piot study outcomes against stopping rues 123 TABLE 38 Staff interview participants in the piot study 131 TABLE 39 Summary of practices and patients for whom CDSS data were avaiabe 169 TABLE 40 Average ength of ca, number of questions, intensity score and percentage of patients for whom CDSS was used by practice 171 xiv NIHR Journas Library

17 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 TABLE 41 Patient data avaiabiity, intervention status and consent status through the tria 179 TABLE 42 Baseine patient demographics by avaiabiity of case note review data and return of competed questionnaire 185 TABLE 43 Demographic associations with consent to case note review and questionnaire response 186 Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xv

18

19 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 List of figures FIGURE 1 Practice training structure 9 FIGURE 2 Practice recruitment procedure 12 FIGURE 3 Patient recruitment and consent procedure 14 FIGURE 4 Timeine of interventions 20 FIGURE 5 Consoidated Standards of Reporting Trias diagram 34 FIGURE 6 Patient recruitment by practices 35 FIGURE 7 Primary outcome resuts by arm 40 FIGURE 8 Redistribution of within-practice workoad over (a) the 28-day workoad (primary outcome measure) and (b) the index day 45 FIGURE 9 Summary, by arm, of within-practice patient management on the index day 49 FIGURE 10 Main tria practice recruitment at end of piot (Juy 2010 to February 2011) 124 FIGURE 11 Fow diagram representing the adverse events reporting procedure for the ESTEEM tria 159 FIGURE 12a Descriptive fow of Cinician Form timings data for heath economics anaysis for GP triage arm 203 FIGURE 12b Descriptive fow of Cinician Form timings data for heath economics anaysis for nurse triage arm 204 Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xvii

20

21 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Gossary Contact A consutation within primary care of any of the types specified within the primary outcome (e.g. a consutation with a genera practitioner or nurse, face to face or over the teephone, within surgery hours or out-of-hours primary care). A contacts are successfuy attended by patients (i.e. excudes patients who do not attend) uness stated in the text. Did not attend Use of the term did not attend appies to both face-to-face and teephone contacts that are unsuccessfu (i.e. patient fais to attend face to face or answer the teephone). Genera practitioner face-to-face contact A face-to-face consutation with a genera practitioner occurring at the practice within working hours. Genera practitioner face-to-face out-of-hours primary care contact An out-of-hours, face-to-face consutation with a genera practitioner occurring within a cinic setting (i.e. not a home visit). Index consutation request The patient s request for a same-day consutation which served as the patient s point of entry to the tria and beginning of the 28-day foow-up period. Index consutation/contact The triage or first management contact foowing the index consutation request. Index day The day of the index same-day consutation request. Per protoco Patients were considered to have been managed per protoco in each tria arm as foows: Genera practitioner triage intervention: patients receiving a genera practitioner teephone contact as their first contact on the same day as the index consutation request. Nurse triage intervention: patients receiving a nurse teephone contact as their first contact on the same day foowing the index consutation request. Usua care: a patients, as no changes had been made to practices system for managing same-day consutation requests. Triage-bookabe (appointment) An appointment reserved excusivey for the use of a genera practitioner or nurse conducting triage to book for a patient who needs to be seen foowing the triage consutation. Triage/first management A triage/first management contact was defined in each tria arm as foows: Genera practitioner triage intervention: a genera practitioner teephone contact on the day of the index consutation request. Nurse triage intervention: a nurse or genera practitioner teephone contact on the day of the index consutation request (recognising that the genera practitioners may occasionay assist with the triaging). Usua care: the first contact occurring within 7 days of the index consutation request. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xix

22

23 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 List of abbreviations A&E accident and emergency LSOA ower super output area APHO Association of Pubic Heath Observatories NIHR Nationa Institute for Heath Research CDSS computerised decision support software NT nurse-ed (computer-supported) teephone triage CONSORT DMC DNA EMIS EQ-5D GLM GP GPT HTA ICC IMD ISRCTN ITT Consoidated Standards of Reporting Trias Data Monitoring Committee did not attend Egton Medica Information Systems European Quaity of Life-5 Dimensions generaised inear mode genera practitioner genera practitioner-ed teephone triage Heath Technoogy Assessment intracuster correation coefficient Index of Mutipe Deprivation Internationa Standard Randomised Controed Tria Number intention to treat OR PCRN PCT POM PPI PSSRU R&D RCT REC RM&G RR SD SF-36 UC odds ratio Primary Care Research Network primary care trust primary outcome measure Patient and Pubic Invovement Persona Socia Services Research Unit research and deveopment randomised controed tria Research Ethics Committee Research Management & Governance rate ratio standard deviation Short Form questionnaire-36 items usua care Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xxi

24

25 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Pain Engish summary Demand for primary care has increased in recent years. Teephone triage invoves cinicians assessing patients needs over the teephone, giving advice or arranging an appointment. We compared genera practitioner- and nurse-ed triage (GPT and NT, respectivey) with usua care (UC) for patients seeking same-day GP appointments, examining patterns of patients service use, safety and experience of care, and UK NHS costs. Forty-two practices participated: 15 (7012 patients) were randomy aocated to NT with computer decision-supported software, 13 (6695 patients) to GPT and 14 (7283 patients) to continue UC. We incuded a patients who were teephoning and seeking a same-day GP appointment. Patients were asked about their opinion of the system via a posta questionnaire 4 weeks ater. We interviewed some patients and staff. Information was coected from the records of patients who gave permission; we documented how often patients were seen over the 28 days foowing their same-day request. Sufficient practices and patients took part to give us confidence in our resuts. Both types of triage increased primary care contacts in the 28-day foow-up, but, overa, triage cost amost the same as UC over the 28 days. Across a range of measures, triage appeared to be safe when compared with UC; there were no significant differences between tria arms in the number of deaths, the number of emergency hospita admissions or the number of accident and emergency department attendances. Patients were sighty ess happy with NT than with GPT or UC. Interviews identified that the success of triage depended on individua practice cuture, and highighted the compexity of introducing a major appointment system change. We found that GPT or NT for patients seeking same-day GP consutations is potentiay a usefu approach to support the effective deivery of NHS primary care. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xxiii

26

27 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Scientific summary Background Demands on UK primary care are increasing, prompting an exporation of aternative ways of managing patients in an attempt to respond to government and societa expectations whie continuing to deiver safe, high-quaity care. When combined with teephone consutation, it has been proposed that teephone triage improves the management of demand for primary care by providing more rapid access to heath-care advice for patients, reducing non-attendance rates and freeing up opportunities for face-to-face consutation. The majority of previous research reates to modes of triage that invove nurses, with itte research addressing the impact of genera practitioner (GP) teephone triage. To date, there has been no arge-scae randomised controed tria (RCT) comparing the potentia benefits and harms of GP- or nurse-ed teephone triage (NT) of patients requesting same-day consutations. Objectives The overarching aim of this tria was, in comparison with usua care (UC), to assess the impact of NT- and GP-ed teephone triage (GPT) on primary care workoad and cost, patient experience of care, and patient safety and heath status, for patients requesting same-day consutations in genera practice. Specific objectives were to compare the effects on primary care workoad and cost, and patient experience of care, patient safety and heath status, of (1) NT vs. UC; (2) GPT vs. UC; and (3) NT vs. GPT. We aso expored the experiences and views of patients and members of practice staff on the acceptabiity of teephone triage. Methods Design Pragmatic custer RCT incorporating economic evauation and parae quaitative process evauation. A preiminary piot RCT was conducted in six practices to (1) confirm the impementation of GP- and nurse-ed triage systems as feasibe; (2) confirm the proposed recruitment of practices and refine data coection systems; and (3) confirm the assumed eve of custering of outcomes. Setting and participants Forty-two genera practices from four regions of Engand, UK (Devon, Bristo/Somerset, Warwickshire/ Coventry and Norfok/Suffok). Participants were consecutive patients (aged 16 years or < 12 years) seeking a same-day face-to-face consutation with a GP. Patients aged years were excuded owing to concerns regarding confidentiaity of a maied questionnaire, as were patients with heath-care needs that were deemed too urgent to wait for triage (e.g. difficuties breathing, chest pain) and patients who were unabe to communicate in Engish by teephone. Randomisation Individua patient-eve randomisation was deemed impractica as it does not refect the practice-wide reaity of triage system impementation and is vunerabe to contamination. Consenting practices (custers) were randomised in a 1 : 1 : 1 ratio using a secure remote automated aocation system designed by a statistician who was independent of the research team. The aocation sequence was computer generated, and minimised for geographica ocation, practice deprivation and practice ist size. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xxv

28 SCIENTIFIC SUMMARY Interventions Patients from practices aocated to either of the triage arms who requested a same-day face-to-face consutation with a GP were advised that they woud be caed by the cinician (GP or nurse using computer decision support software) ater that day to discuss their needs and to discuss the most appropriate management option. The options avaiabe incuded (1) sef-care advice; (2) a further within-practice consutation on the same day; (3) a further within-practice consutation on a subsequent day; and (4) referra to another NHS service. Triage practices received standardised training on how to impement triage, athough some taioring of the intervention was permitted to suit oca needs. Patients in UC practices were managed foowing the standard protocos for that practice. Outcome measures Our primary outcome measure (POM) is the tota number of primary care contacts occurring within a 28-day period foowing a patient s same-day consutation request. We defined primary care as incuding consutations within genera practice (incuding the triage contact), wak-in centres, out-of-hours primary care services and attendance at accident and emergency (A&E). A POM data were coected by trained researchers conducting case note reviews of eectronic practice records. Other process and outcome data were coected to document NHS resource use, appointment non-attendance and consutation ength. Secondary outcomes of patient safety (number of deaths and emergency hospita admissions within 7 days of the index request and A&E attendance at foow-up), NHS resource use and costs, appointment non-attendance and consutation ength were coected from case note review and primary care Cinician Forms. Patient-reported heath status [European Quaity of Life-5 Dimensions (EQ-5D)] and experiences of care (modified Engish nationa GP Patient Survey items) were coected by posta questionnaire. Given the nature of the interventions, it was not possibe to bind patients or practitioners to treatment aocation. Athough the custer design of the tria might theoreticay aow researchers who are undertaking case note reviews to be binded, our piot study showed this was not possibe in practice. The data anaysis was carried out by a statistician who was binded to treatment aocation. Sampe size Based on a previous UK study comparing NT with UC for handing same-day consutation requests, we powered our study to detect a between-arm difference of 1.02 [standard deviation (SD) 0.78] vs (SD 1.79) at foow-up between the triage arms and UC [based on 90% power, intracuster correation coefficient (ICC) 0.05, two-sided apha 0.05]. Assuming 47% non-coection of the POM, it was necessary to recruit 7046 patients from 14 practices across each arm to reach our target of 3751 patients per arm for anaysis (i.e. a tota of 21,138 patients from 42 practices). The piot study provided confirmation of our assumed ICC of 0.05 (i.e. 0.03, 95% CI 0.00 to 0.08). Furthermore, the piot study ed to a change in the method of patient consent to case note review from written consent (obtained from competed patient questionnaires) to incude initia verba consent obtained from the treating cinician. Statistica methods The primary anaysis of the POM took the form of a regression anaysis using a hierarchica mode to take account of the custer aocation, using a random effect to adjust for potentia custering effect by practice, and aowing for adjustment for practice-eve minimisation variabes and patient-eve covariates shown to differ at baseine. Modes were performed twice, initiay using the UC arm as reference, and then using the GPT arm as reference, to derive comparison between the two triage arms. Investigation of the effect of missing POM data (owing to ack of avaiabiity of a case note review) was undertaken using mutipe imputation methods, based on the assumption that missing case note review data were missing at random. Additiona anayses were conducted on the POM, and on secondary measures derived from the POM, using the hierarchica generaised inear mode methods described above. Some of these anayses were determined a priori; others were determined post hoc foowing initia inspection of the data. xxvi NIHR Journas Library

29 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Economic methods Primary economic (cost) anayses were undertaken using data coected on the POM contacts taking pace over 28 days, and conducted from the perspective of the NHS. Economic anayses were conducted in accordance with the statistica methods above. The primary economic anaysis estimated the mean cost of care across each of the tria arms, to incude triage (where used) and the items in the primary outcome. Anaysis was based on a microeve costing estimate for the triage intervention, and the use of pubished unit cost data for other eements of resource use. Estimates of the cost associated with triage interventions were based on incrementa costs when compared with UC, with any capita costs and/or training costs depreciated/ spread over an appropriate time period in the primary anayses (with other time horizons for these costs expored in sensitivity anayses). Sensitivity anayses were undertaken against the primary anayses to expore the impications of uncertainty in data used and the assumptions made within the primary anayses. Resuts Participating practices (n = 42) and patients were we baanced across the three tria arms with respect to key characteristics (practice ist size, setting and deprivation; patient age, gender and deprivation). In tota, 20,990 patients were eigibe for the tria (UC n = 7283; GPT n = 6695; NT n = 7012). POM data were anaysed for 16,211/20,990 (77%) participants (UC n = 5572; GPT n = 5171; NT n = 5468). There was some evidence of participation bias, with young aduts (aged years) being ess ikey than the reference age group (25 59 years), and women being ess ikey than men, to have POM data avaiabe. The mean number of POM contacts in the 28-day foow-up period was 1.91 in UC (SD 1.43; tota 10,616), 2.65 in GPT (SD 1.74; tota 13,720) and 2.81 in NT (SD 1.68; tota 15,400). Compared with UC, there was an increase in POM contacts of 33% in GPT [rate ratio (RR) 1.33, 95% CI 1.30 to 1.36] and 48% in NT (RR 1.48, 95% CI 1.44 to 1.52). There was a sma increase of 4% in NT (RR 1.04, CI 1.01 to 1.08) compared with GPT. Triage-arm patients had more diverse patterns of management when compared with UC. For GP face-to-face and teephone contacts combined across the 28-day foow-up period, the RR was 1.38 (95% CI 1.28 to 1.50) in GPT compared with UC, and 0.84 (95% CI 0.78 to 0.91) in NT compared with UC. GP face-to-face contacts decreased by 39% during the 28-day foow-up period in GPT compared with UC (RR 0.61, 95% CI 0.54 to 0.69) and in NT by 20% compared with UC (RR 0.80, 95% CI 0.71 to 0.90). Foowing the impementation of triage, no impact was observed on contacts with other services outside of the practice (out-of-hours primary care, wak-in centres or A&E) in either intervention arm. Changes were aso observed in the distribution of estimated patient cinician contact time on the index day foowing the introduction of triage. Introducing GPT was associated with a sma increase in patient nurse contact time, whereas introducing NT was associated with both a decrease in patient GP contact time and a substantia increase in overa patient nurse contact time. The estimated heath-care costs over the 28-day foow-up were simiar across a three arms, at a mean cost of approximatey 75 (US$120, 88) per patient. There was no evidence of differences across the three tria arms with respect to patient safety (patient mortaity, emergency hospita admissions and A&E attendance rates) and patient heath status did not vary between triage arms and UC, or between GPT compared with NT. NT was somewhat ess acceptabe to patients than GPT or UC. Data from 84 quaitative interviews with patients and staff (samped from 10 practices) found no strong, compeing or consistent narrative about what works and what does not work when impementing teephone triage in primary care. Rather, the quaitative data highighted the compexity of primary care organisations and the significance of individua practice cuture. Both triage modes were sometimes experienced positivey by staff and patients, whereas others viewed it negativey. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. xxvii

30 SCIENTIFIC SUMMARY Concusions We beieve ESTEEM to be the first RCT to assess both nurse- and GP-ed teephone triage of patients requesting same-day consutations in primary care. The ESTEEM tria achieved its recruitment target of 22,000 patients across 42 genera practices in four regions in Engand. Contrary to suggestions that triage can reduce primary care workoad, we found that both GPT and NT increased the number of primary care contacts in the 28 days foowing a patient s same-day consutation request compared with maintaining a UC approach to such requests. Foowing the introduction of GPT, we observed a redistribution of GP workoad from face-to-face to teephone consutations. Foowing the introduction of NT, we observed a redistribution of workoad from GPs to nurses. Triaged patients had more diverse patterns of management than those in UC, possiby indicating more fexibe approaches to patient management in the triage arms. There was evidence of increased nurse depoyment in both triage arms, with this being substantia foowing the introduction of NT. When considering the differing patterns and duration of patient care contacts, both forms of triage were cost neutra to the NHS compared with UC. However, we found no important gains in patients safety, heath status or experience of care with triage. Our process evauation identified no strong, compeing or consistent narrative about what works and what does not work when impementing teephone triage in primary care. It did, however, identify key issues on which any practice considering impementing triage might want to refect before adopting a triage system. These incuded consideration of issues reating to individua practice cuture and capacity, and the forward panning, with the whoe staff team, of any major changes in access arrangements. Our resuts do not support a definitive poicy recommendation to ro out across the NHS either nurse or GP triage for the management of same-day appointments. Triage, whether impemented by a GP or by a nurse using decision support software, shoud be introduced with fu awareness of the whoe-system impications arising from the decision to impement such a process. Notwithstanding this, cinician triage of patients seeking same-day consutations may offer advantages in supporting the fexibe deivery of patient care, and potentiay offers a usefu approach in the armamentarium of toos faciitating the deivery of effective NHS primary care. Tria registration This tria is registered as ISRCTN Funding Funding for this study was provided by the Heath Technoogy Assessment programme of the Nationa Institute for Heath Research. xxviii NIHR Journas Library

31 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Chapter 1 Introduction Scientific background and review of current iterature Demands on UK primary care are increasing. The introduction of a new Genera Medica Services contract in was foowed by an estimated 25% increase in primary care workoad. 2 For many practices and staff, addressing this increase in workoad has invoved an exporation of aternative ways of managing patients, in an attempt to respond to government and societa expectations whie continuing to deiver safe, high-quaity care. The introduction of UK NHS wak-in centres, the 24-hour nurse-ed teephone advice service NHS Direct and the more recent NHS 111 service, the increased diversity of staff ski mix and the use of remote consutations in primary care a represent organisationa responses aimed at increasing the range of services avaiabe to patients and improving access to primary heath care. When combined with teephone consutation, teephone triage is beieved to provide rapid access to heath-care advice for patients whie freeing up opportunities for face-to-face consutation. Previous research 3 has demonstrated the utiity of nurse-ed teephone triage (NT) of patients requesting same-day appointments in UK genera practice. An average practice (7000 patients) might be expected to manage around 20 patients each day requesting a same-day appointment request, representing around 35% of genera practitioner (GP) workoad. 4 Some research evidence exists regarding the feasibiity, workoad impications and cost of teephone triage, and patient experience of care, safety and heath status foowing teephone triage. Most evidence derives from modes invoving nurse triage; ess research has been carried out to address the impact of GP teephone triage to date. There have been no arge-scae mutipractice studies examining the potentia vaue of genera practitioner-ed teephone triage (GPT) or NT of patients requesting same-day consutations. Feasibiity of teephone triage Previous studies suggest that around 50% of nurse triage cas in out-of-hours primary care settings may be handed by teephone advice aone. 5 8 However, such studies have been sma or focused soey on out-of-hours primary care. Use of teephones (fixed or mobie) is now amost universa in the UK, 9 and recent years have seen a near quadruping in the proportion of GP consutations conducted on the teephone. 10 Primary care workoad In the short term, teephone triage, whether by doctor or nurse, appears to reduce GP contacts by around 40%, 3,11 but it coud be that this shifting of GP workoad may resut in the work undertaken by GPs comprising the more compex cases. 12 The reduction aso appears to be associated with an increase in ater return consutations of a roughy simiar magnitude (30%, 3 50% 13 between 2 and 4 weeks foowing a same-day appointment request), in effect smoothing out the peaks and troughs of GP workoad that are associated with same-day appointment requests. Athough this eve of return consutations foowing triage may raise concerns regarding patient safety, convenience of care and cost-effectiveness, it has been suggested that a proportion of return consutations may be panned routine appointments, resuting from a downgrading of urgency foowing triage. Cost Equivoca resuts on costs of teephone triage and associated resource service use have been reported across three trias. NHS Direct nurse triage was more expensive than practice nurse triage of patients making same-day consutation requests 14 but simiar costs have been reported esewhere between standard management and practice NT of same-day consutation requests. 3 NT in out-of-hours primary care may reduce ong-term NHS costs but may not be cost-effective at a times of the day. 15,16 Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 1

32 INTRODUCTION Patient experience of care Equivoca resuts on patient acceptabiity and satisfaction have been derived from sma studies. One study 13 reported no difference in satisfaction between teephone and face-to-face consutations. Jiwa et a. 11 reported that 80% of patients were satisfied with GP teephone management of same-day consutation requests, and Brown and Armstrong 17 have suggested that patients who eect to use GP teephone consutations may do so as an aternative to face-to-face consutations in primary care. Patient safety Teephone consutation appears safe and effective. 18,19 A UK equivaence tria (in which death within 7 days of contact was the primary outcome) estabished the safety of out-of-hours primary care NT by experienced nurses using computerised decision support software (CDSS) in comparison with usua care (UC). 5 This is supported by work in the UK by the Richards et a. 20 study, in which audio tapes of nurse triage consutations to assess decision-making reveaed that decision-making was rated by GP or nurse practitioner review to be mosty good, with minima risk from poor nurse triage decisions. However, one Swedish study 19,21 noted that nurses often used sef-care advice and aso over-rode software-determined recommendations on management. A recent Dutch study 22 highighted concerns regarding information-gathering in teephone triage when deivered without the use of CDSS, reying ony on cinica protocos. Studies 8,21,23,24 have highighted the importance of training in the use of CDSS to address patient safety issues. Nurse teephone triage deivered within a framework of nationa guideines (but not with CDSS) was judged to be efficient, athough some concerns were raised with respect to patient safety. 25 One study, 3 adopting a triage system invoving computerised management protocos deveoped by the practice identified a substantia increase in accident and emergency (A&E) attendance, athough actua numbers were sma. Athough computerised, such a system did not provide CDSS (DA Richards, Institute of Heath Research, University of Exeter Medica Schoo, 2008, persona communication) such as is now avaiabe within a number of NHS primary care computer systems, and which we propose to examine in this study. 26 The other tria by Richards et a. 14 used CDSS for NHS Direct triage nurses, but not for nurses acting in primary care. A systematic review 27 of nine studies of teephone consutation and triage noted that it is uncear if teephone management simpy deays visits and thus aso the provision of definitive care. Patient heath status Severa randomised studies (but none invoving teephone triage) have compared the heath status of primary care patients foowing consutations with a doctor or a nurse by patients with minor probems or after a same-day consutation request. One study 28 identified no difference in heath status [Short Form questionnaire-36 items (SF-36) scores] between the intervention groups when foowed up after 2 weeks. Simiar findings have been reported with respect to resoution of patients symptoms and concerns after 2 weeks, 29 or in the proportion of patients reporting themseves as cured or improved 2 weeks after a consutation with either a doctor or a nurse. 30 Rationae for the research The four UK-based trias 3,5,11,13 of primary care teephone consutation and triage have been conducted in reativey sma popuations and/or in imited numbers of practice settings (i.e. urban, rura), and most studies examining NT without the use of CDSS, athough research undertaken by Lattimer et a. 5 did invove out-of-hours primary care nurse teephone consutations using the CDSS Odyssey TeeAssess, provided by Pain Heathcare Ltd. Despite uncertainty about the benefits and costs, many practices operate GPT or NT systems as a way of providing fast access to care for patients and in order to manage practice workoad, as demonstrated by the amost fourfod increase in proportion of consutations conducted over the teephone. 10 As an exampe, in 2008, the NHS Institute for Innovation and Improvement 31 promoted a mode of GPT the Stour Access system within which GPs triage a patient requests for care by teephone but without any robust evidence about benefits. We therefore proposed to address this important agenda in a arge-scae experimenta study of two forms of triage currenty being promoted by the NHS for use in UK primary care. Our findings may be generaisabe to other heath settings, especiay those with strong primary care-based heath-care systems. 2 NIHR Journas Library

33 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Aims and objectives The overarching aim of this tria was to assess, in comparison with UC, the impact of NT and GPT mechanisms on primary care workoad and cost, patient experience of care, and patient safety and heath status for patients requesting same-day consutations in genera practice. The specific research objectives were as foows: Piot and feasibiity study To undertake an externa piot randomised controed tria (RCT) in six practices to: 1. confirm the abiity of practices to impement the GPT and NT systems 2. confirm the proposed process for recruitment of practices 3. review the assumed eve of custering of outcomes 4. check data coection systems, and 5. identify potentia difficuties in impementing the triage systems. Main tria To undertake a three-arm pragmatic custer RCT comparing, for patients requesting a same-day consutation in genera practice, the effects on primary care workoad and cost, and on patient experience of care, safety and heath status of: i. GPT compared with UC ii. NT compared with UC iii. GPT compared with NT. The funding arrangements for this tria through the Nationa Institute for Heath Research (NIHR) Heath Technoogy Assessment (HTA) programme invoved the deivery of the 1-year piot study and satisfying a number of key stopping rues (see Appendix 1), before progression to the main tria phase. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 3

34

35 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Chapter 2 Methods Study design Consistent with the Medica Research Counci framework for evauating compex interventions, 32 a two-stage mixed-methods study was conducted, comprising an externa piot tria, foowed by a definitive custer RCT. The ESTEEM tria was designed as a pragmatic three-arm, muticentre custer RCT. Practices were randomised 1 : 1 : 1 to receive GPT, NT or UC as comparators. The main tria incuded a parae economic evauation to examine the cost-effectiveness of the two triage interventions and a process evauation to assess the intervention acceptabiity from the perspectives of patients and practice staff. Piot study A 12-month, externa piot custer RCT and parae process evauation was undertaken in six practices across Devon and Bristo. This work informed aspects of the main tria protoco pubished in We present the methods impemented within the main tria, reating to the triage interventions, practice and patient incusion and excusion, practice and patient recruitment, randomisation, and the outcome measures and sampe size. A fu account of the changes made to the main tria methods can be found in Appendix 2. Ethica and governance arrangements Muticentre ethica approva for the study was obtained from South West 2 NHS Research Ethics Committee (REC) in October 2009 (reference 09/H0202/53). Loca NHS research governance approvas were obtained from the primary care trusts (PCTs) in Devon, Somerset, North Somerset, Bath and North East Somerset, Bristo, South Goucestershire, Warwickshire, Coventry, Norfok, Suffok and Great Yarmouth, and Waveney. At the time this work was undertaken, PCTs were the main unit of administration of primary care, with a tota of 152 PCTs in Engand, each covering an average popuation of around 330,000 individuas. PCTs were aboished under major changes to the NHS, introduced in 2012, with cinica commissioning groups taking over former PCT functions. The Research Management & Governance Unit, Devon PCT, acted as the study sponsor. A Tria Management Group and an independent Tria Steering Committee and Data Monitoring Committee (DMC) ensured that the study was conducted within appropriate NHS and professiona ethica guideines. The tria was registered with the Internationa Standard Randomised Controed Tria Number (ISRCTN) Register (reference ). Patient and Pubic Invovement The Patient and Pubic Invovement (PPI) group supporting ESTEEM initiay met to review the study protoco and methods. Practice service users (nurses, GPs, managers and administrative staff) were aso consuted to inform the tria design. It was panned that patient service users woud be invoved at a stages of the piot study and main tria. Patient representatives were recruited through an existing GP practice patient group oca to the Exeter site (not an ESTEEM tria practice). Eeven patient representatives were recruited and were invited to contribute to some or a of the various tasks as they preferred. Athough our intention was to recruit representatives on the basis of their demographic characteristics, to refect variations in the peope most ikey to be requesting same-day consutations in genera practice (e.g. parents of young chidren, retired peope) the majority of the patient representatives were of retirement age. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 5

36 METHODS There were a number of PPI tasks, as isted beow, and not a patient representatives wished to contribute to every task over the duration of the study. i. Most of the patient representatives contributed to the deveopment of study materias in preparation for ethica review before the piot study began, and then again at the end of the piot study for use in the main tria. This invoved heping to design the patient information sheets, consent forms, topic guides for patient interviews, and patient questionnaire. Comments were obtained through a combination of face-to-face meetings and via e-mai and post. ii. Advice was aso sought on ways to improve patient participation in interviews for the process evauation. This ed to the introduction of an option for teephone interviews during the main tria. iii. (The same) two patient representatives sat on the Tria Steering Committee for the whoe period of the tria and one, if not both, attended every meeting. iv. A sma number of patient representatives contributed to the process of anaysing quaitative data from the process evauation during the piot study. v. A sma number of patient representatives attended meetings and received posta materia during the running of the tria in order to be kept informed of study progress. vi. A patient representatives were invited to contribute to the dissemination of the study resuts at the end of the project, by way of providing input to the pain Engish summary of the tria findings. Heath technoogies assessed Triage interventions The GPT and NT interventions were compex interventions that invoved staff training (cinica and technoogy based); CDSS to support the deivery of NT; process and organisationa change in practices regarding reception activity and appointment system management; and accommodation of patient expectations. Some core eements of triage deivery were common to, and adopted by, a practices in both intervention arms. However, some organisationa fexibiity was permitted because of the compex nature of the intervention. Fu detais of the interventions are avaiabe from the research team. Core triage processes A patients contacting the practice initiay spoke to a receptionist. Once the receptionist estabished that the patient (or a proxy asking on their behaf) was requesting a same-day, face-to-face appointment with a GP, the patient was asked to provide a contact teephone number and was advised that the cinician (GP or nurse, according to the practice s aocation) woud ca them back within around 1 2 hours. This timescae was suggested as a guide for practices but was not considered mandatory. On teephoning the patient ( index consutation ), the cinician discussed the presenting condition and had a range of management options at their disposa: i. give the patient sef-care advice ii. book the patient into a triage-bookabe face-to-face or teephone appointment with the reevant heath professiona ater the same day iii. book the patient into a triage-bookabe face-to-face or teephone appointment with the reevant heath professiona on another day iv. book the patient into any routine appointment avaiabe v. refer the patient to other NHS services where appropriate, incuding those outside the practice. Practices kept a number of triage-bookabe appointments with GPs and nurses, reserved excusivey for triaging cinicians to book. When patients required face-to-face appointments, they coud be triaged to any avaiabe, appropriatey timed, triage-bookabe appointment sot; this may or may not have been with the patient s usua doctor (at the discretion of the practice) or on the same day as the patient s index consutation. 6 NIHR Journas Library

37 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Areas of fexibiity Athough the core triage processes were specified, practices had some areas of fexibiity in impementing triage interventions, in order to accommodate the compexities of individua practice organisation. These modes used were informed by individua practices audits of appointment requests, GP and nurse capacity information, and guidance and training given to practices prior to starting the tria (see Training practices in the two triage systems). Fexibiity was aowed in terms of the triaging cinician, triage avaiabiity, the triaging of babies, and same-day requests for nurse practitioner consutations. Triaging cinician Three aternative staffing modes emerged regarding which cinician (GP or nurse) deivered the triage: (1) one cinician (GP or nurse) at a time conducted a triage session (on a rotating basis); (2) a number of cinicians (GPs or nurses) conducted triage sessions on the day (on a rotating basis); or (3) in GPT practices ony, a of the GPs conducted triage each day, with GPs triaging their own patients to triage-bookabe appointments. Triage avaiabiity Practices were asked to triage a consecutive eigibe patients during opening hours to ensure the east disruption and shortest time of tria data coection possibe. In the event that imited staff resources prevented triage of a eigibe patients each day, practices were permitted to operate triage within specific agreed sessions ( research window ), amounting to no ess than 50% of the week, encompassing a five working days and both mornings and afternoons. These arrangements were agreed with the research team in advance. Practices were advised of the importance of a eigibe patients being triaged during an entire session, as opposed to receptionists booking in a set number of triage patients within a session and then cosing the triage once the aocated sots were taken. Practices operating triage within research windows had their data coection period extended unti the patient recruitment target was reached. Triaging babies Practices were advised that babies (i.e. 2 years of age) were eigibe for tria entry according to the incusion and excusion criteria, with the parent or guardian identified as the individua who participated in the triage ca. As some practices routiney see babies as soon as possibe on the same day of the appointment request, it was expected that some practices woud opt to book these patients in for a face-to-face appointment rather than add them to a triage ist. Athough a of the intervention practices were encouraged to offer triage to babies during intervention training, some practices eected to offer face-to-face appointments in ine with their existing practice poicy. When this happened, the babies were sti incuded in the tria anaysis on an intention-to-treat (ITT) basis. Requests for same-day, face-to-face nurse practitioner consutations Some practices operated a nurse practitioner-ed cinic for acute or minor inesses, for which the nurse typicay managed same-day consutation requests. This request was deemed equivaent to a same-day GP appointment request, as the patient woud see a GP if the nurse practitioner was unavaiabe. Such patients were deemed eigibe for tria entry and managed as per the treatment arm aocation. Training practices in the two triage systems Roe of commercia providers in triage training To prepare GPT and NT practices for incorporating teephone triage within their appointment systems, we worked with Productive Primary Care Ltd, a Leicestershire-based, NHS-focused commercia organisation that works nationay across the UK with NHS organisations. Pain Heathcare s Odyssey PatientAssess (derived for genera practice use from Odyssey TeeAssess or TAS ) CDSS was used to support the nurses in NT practices. Pain Heathcare, then part of the Avia Heath Informatics PLC group, suppies the NHS with CDSS systems (e.g. out-of-hours primary care services). Pain Heathcare iaised directy with NT practices to discuss IT and training requirements for the instaation of the software. Foowing a site initiation ca between Pain Heathcare and each practice, instaation and testing took pace over a Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 7

38 METHODS 6-week period. The CDSS was fuy embedded within the Egton Medica Information System (EMIS, Yeadon, Leeds) PCS and LV, and instaed as a separate stand-aone window tab to the cinica notes within a other types of practice IT system [e.g. Microtest Evoution (Microtest, Bodmin, Cornwa), Vision (In Practice Systems Ltd, Battersea, London), SystmOne (The Phoenix Partnership, Horsforth)]. Nurses received CDSS training and training in teephone consutation skis. Foowing this, there was a pretria period of 1 month during which nurses practised using CDSS in simuated patient scenarios during their daiy work. Towards the end of this period, the nurses use of the system was assessed by Pain Heathcare trainers, with a nurses needing to demonstrate proficiency in its use before being recruited into the tria. It is aso important to note that the CDSS is designed to support the nurses cinica decision-making; there was no requirement in the tria that the recommended advice the CDSS generated had to be foowed. Computer decision support in nurse triage Odyssey PatientAssess [ (accessed 5 November 2014)] is a UK product, deveoped to support nurses and paramedics to assess the cinica needs of patients. It is aready being used by severa out-of-hours primary care and NHS wak-in services, and is aso the subject of the Department of Heath-funded SAFER1 tria focusing on the care of oder peope who have caed the 999 service foowing a fa, and the HTA-funded SAFER3 tria focusing on the care of aduts who have caed the 999 services and are not in need of transfer to an emergency department. 34 Odyssey PatientAssess has been evauated in a number of other RCTs, and its cinica effectiveness and cost-effectiveness in settings other than in-hours genera practice is aready estabished. For instance, it has been demonstrated to be safe and cost saving in the ong term in one study 5,16 invoving a co-investigator compared with GP teephone triage in a tria of out-of-hours primary care consutations. That study remains the argest tria of nurse teephone triage to date. 5,16 Furthermore, in excess of 60% of PCTs currenty commission out-of-hours primary care services that use nurses to triage patients by teephone, supported by Odyssey PatientAssess (Chris Coyne, Pain Heathcare, 2008, persona communication). However, the findings and experience of out-of-hours primary care (providing care for around 10.8 miion contacts per year in UK) cannot necessariy be generaised to the very different system providing in-hours primary care and experiencing around 1 miion contacts per working day. Odyssey PatientAssess provides the user with a network of assessment prompts and guided responses reating to over 465 presenting compaints. It aows for mutipe symptoms to be evauated simutaneousy, supported by evidence-based frameworks for referra and sef-care, with a assessment data remaining visibe at a times. It supports the cinician s judgement and expertise through enhancing norma consutation processes. The cinica database comprises severa hundred assessment and examination guideines and protocos, each inked to triage, treatment and advice guideines, differentia diagnoses, patient information and education. These are maintained by an in-house cinica deveopment team, which reviews the entire cinica content at east annuay to ensure that it refects current best practice, incuding Nationa Institute for Heath and Care Exceence (NICE) guidance. The assessment screens incude drop-down menus that provide reguary updated referenced information on differentia diagnoses and rationaes for ines of enquiry for each type of presentation so reminding the user about the importance of different ines of enquiry. For the purposes of the ESTEEM tria, Odyssey PatientAssess was either embedded within GP computing systems (EMIS and SystmOne) or instaed to run in parae aongside a other systems (Synergy, Microtest, Vision). Odyssey PatientAssess guides and stores documented records of the assessment, advice and/or referra of each patient producing a fuy auditabe record. Based on the data eicited during the teephone assessment, Odyssey PatientAssess suggests an appropriate care pan (e.g. patient advice, same-day appointment, home visit, routine appointment, 999 referra). 8 NIHR Journas Library

39 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Overview of practice training Practices were trained in research procedures and the deivery of triage interventions, foowing a mode (Figure 1) devised during the piot study. Training was deivered in six stages, and by a combination of Productive Primary Care (GPT and NT ony), Pain Heathcare (NT ony) and tria researchers (a arms). Triage training workshops were organised for GPT and NT practices (training components 3 5, described beow) and deivered as a package to staff from a number of practices at a centra ocation in each of the recruitment areas. If practices were unabe to attend a centra training event, the various components were deivered on site at the practice. 1. Inception meeting An initia whoe-practice meeting with the tria researcher was hed to outine the tria, staff group invovement and timeine of training. The meeting was convened for the practice manager, at east three receptionists, at east two GPs to incude the two GPs identified as cinica eads for the tria and up to four nurses (NT ony). 2. Appointment request audit To assess patient demand for same-day and prebookabe appointments, a 1-week audit was conducted by the practice reception team. Capacity information was coated reating to the number of GP and nurse sessions provided per week and the number and typica duration of appointments offered within each session (incuding the proportion reserved for same-day and prebookabe use). The tria researcher provided the reception team with Audit Log Sheets (see Appendix 3) and asked them to record a incoming cas each day for a week (Monday to Friday), indicating whether each ca was a same-day or prebookabe appointment request. The researcher maintained contact with the practice during the first day or two of the audit to ensure that the procedure was foowed appropriatey. GPT NT UC Inception meeting Appointment request audit over 1 week Triage training Organisationa training GP triage skis Professiona issues and teephone consutation skis Research procedures training Run-in period Data coection period FIGURE 1 Practice training structure. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 9

40 METHODS 3. Organisationa training and guidance in organising the triage system Productive Primary Care trained the practice manager, at east two receptionists, at east two GPs (GPT) and up to four nurses and one GP (NT). The training incorporated advice for the reception teams on how to communicate the introduction of the triage system to patients. Practica guidance on setting up the appointment system to incorporate triage consutation sots and face-to-face appointment sots for the use of the triaging cinician was aso given. Using the data coected during the appointment request audit, Productive Primary Care used a spreadsheet (the Appointment Redesign Too) to estimate the number of appointments that the practice needed to provide and the proportion that shoud be triage bookabe based on their previous experience of working with a range of practices. The detais of the cacuations for these estimates are the inteectua property of Productive Primary Care. 4. GP triage skis In GPT practices, Productive Primary Care deivered training to GPs on how to communicate the operation of the new triage system to patients, and genera guidance for consuting over the teephone. At east two GPs at each practice were required to receive this training and were expected to disseminate the information to their other GP coeagues. 5. Professiona issues and teephone consutation skis A nurses at each NT practice who woud undertake triage were required to receive this training. A earning and deveopment adviser from Pain Heathcare presented information and advice on the professiona issues that may arise from nurses consuting over the teephone. Guidance on teephone consuting skis, incuding roe-pay scenarios, was provided. In addition, nurses received one-to-one remote training from an advisor whie they practised using the CDSS on computerised simuated patient scenarios in the weeks eading up to the tria, cuminating in a proficiency assessment. 6. Research procedures training Tria researchers provided briefing sessions to practice staff in data coection procedures. Staff groups incuded the reception team, practice manager, key IT and administrative staff and the GPs and nurses to be invoved in triage or deaing with patients requesting same-day consutations. Usua care comparator Practices were asked to continue with their standard consutation management systems for handing same-day consutation requests. UC practices aso (1) received training through the inception meeting; (2) performed the appointment request audit; and (3) received the standard research procedures training, as outined for the intervention arms. To describe the range of systems that constitute UC, a 42 practices were asked to compete a Practice Profie Questionnaire before randomisation to coect detais on their current staffing and appointment system arrangements to manage same-day consutation requests, incuding the extent to which they aready used triage of any sort. Incusion and excusion criteria Practices Practices aready impementing a documented triage system for routiney handing same-day GP consutation requests were excuded. We defined a documented triage system as a system invoving teephone triage (by a GP or a nurse) managing > 75% of a same-day consutation requests received. This definition recognised that many GP practices aready undertake some form of teephone triage. Practices aso had to be wiing to be randomised to one of the three tria arms. 10 NIHR Journas Library

41 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Patients Incusion criteria A consecutive patients making teephone requests for a same-day, face-to-face consutation with a GP during the data coection period were potentiay eigibe. When an individua patient had mutipe same-day consutation requests during the recruitment period, ony the first contact was incuded in the study to avoid confusion in foowing up the contact (athough subsequent cas woud be triaged in the same way, according to tria arm). A patients aged < 12 years and 16 years requesting eigibe appointments were incuded with respect to the primary and secondary outcome measures. Parents or guardians of chidren aged < 12 years provided consent on behaf of the chid. Excusion criteria Temporary residents or young peope aged years were excuded, as the study invoved receipt of a posta questionnaire aong with written consent to review case notes. We beieved that receipt of the questionnaire at the young person s address may have inadvertenty ed to a breach of confidentiaity shoud third parties have access to the young person s mai. Athough excuded from tria participation, both temporary residents and young peope received the intervention as per tria aocation. Aduts aged 16 years were incuded, uness the practice wished to screen out patients for whom they fet it woud be inappropriate to send a questionnaire (e.g. patients with recent bereavement, vunerabe aduts). Patients seeking urgent or emergency care on account of the foowing conditions were excuded: 30 too i to participate (severe chest or abdomina pain or severe difficuty breathing; vomiting bood; atered consciousness; seizures; pregnancy-reated probems; or severe psychiatric symptoms); unabe to speak Engish or difficuties with communicating on the teephone (hearing or speech). Reception staff members were trained to refer to a standardised Receptionist fow chart to ascertain patient eigibiity (see Appendix 4). Patients ringing on a second occasion were not incuded in the tria again (i.e. their point of entry into the tria was on the first occasion they rang the practice seeking a same-day consutation). However, in practice, receptionists woud have recorded the patient as being eigibe and sent their information to the research team. Our database woud have fagged them as a dupicate and excuded them from being entered into the tria database again. Recruitment and randomisation procedures Practices We approached a practices within our four participating centres (Devon, Bristo/Somerset, Warwickshire/ Coventry and Norfok/Suffok). To maximise recruitment we ran the tria in conjunction with the NIHR Primary Care Research Network (PCRN), incuding the networks in the South West, West Midands (South) and East of Engand. In recognition of the chaenges of recruitment into arge-scae cinica trias of compex interventions, 35 a two-stage recruitment process was adopted (Figure 2). A written etter inviting participation was sent to a practices in the four geographica areas. This etter was co-signed by the tria Chief Investigator, the oca principa investigator and oca PCRN cinica ead. Practices expressing preiminary interest were sent an information pack and offered a practice visit by a researcher. At this recruitment meeting (for which the practices were remunerated), the tria design and methods were expained, and staff were given the opportunity to ask questions. We prioritised recruiting practices in workabe proximity to recruitment centres (taking account of reevant samping issues). Randomisation procedure Individua patient-eve randomisation was deemed impractica as it was unabe to refect the practice-eve triage system impementation and was vunerabe to contamination. A custer RCT design was adopted, with practices that agreed to take part subsequenty randomised to one of the three tria intervention Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 11

42 METHODS Invitation etter sent Foow-up ca after 10 days Repy sheet received Information sent again Agree a date for meeting Await practice to set date Practice decines Practice meeting Further correspondence Decision FIGURE 2 Practice recruitment procedure. arms (GPT, NT, UC). 36 To manage the recruitment process, practices were randomised in three distinct waves. Wave 1 took pace during spring and summer 2011, wave 2 during winter 2011/12 and wave 3 during spring and summer The sequence of practice randomisation was computer generated and minimised for geographica ocation (Devon, Bristo, Norwich, Warwick), practice deprivation [based on the data provided by the Association of Pubic Heath Observatories (APHO) 37 at time of randomisation, practices were cassed as deprived (i.e. score is above the average for Engand) and non-deprived (score is average or beow average) and practice ist size of sma (< 3500 patients), medium ( patients) and arge (> 8000 patients)]. A stochastic eement was incuded within the minimisation agorithm to maintain conceament. Practice aocation was undertaken using a password-protected web porta (deveoped and vaidated by the UK Cinica Research Coaboration (UKCRC)-accredited Peninsua Cinica Trias Unit) by a statistician who was independent from the tria statistica team. Aocation was communicated by e-mai to the tria manager who then informed the oca researchers, who, in turn, informed practices of their aocation. Repacing practices that withdrew post randomisation To maintain baance between groups, practices that withdrew post randomisation were repaced with a practice from a waiting ist in the same geographica area, and of a simiar size and deprivation eve, where possibe. Owing to the imited number of waiting ist practices, it was not possibe to seect a matched practice randomy. Instead, repacement practices were purposivey aocated to a tria arm based on ocaity, practice size and deprivation index. However, other than knowing that the practice was matched for the above characteristics, the researcher making the practice seection was unaware of any other characteristics of that practice. Repacement practices were not aware of their potentia aocation when the researcher checked that they were sti wiing to enter the tria. A questionnaire exporing reasons for withdrawa was sent to practice managers (see Appendix 5). 12 NIHR Journas Library

43 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Patients During the intervention period a practices were provided with a poster to be dispayed in the practice to inform participants of the tria and its purpose, and some practices incorporated a voice message onto their teephone systems to expain that the triage system was being triaed. A incoming cas were received by a practice receptionist, who (referring to the Receptionist fow chart ) ascertained whether the patient was requesting a same-day, face-to-face consutation with a GP at the surgery. Reception staff asked patients to briefy outine the nature of their probem in order to faciitate timey care (but patients did not have to discose this information). If the patient was not requiring urgent or emergency care (see Incusion and excusion criteria), the receptionist informed the patient of the current practice consutation arrangements (according to tria arm) and managed the request foowing the standard operating procedure for that practice. Patient consent The patient recruitment and consent procedure is summarised in Figure 3. Consent to tria participation Consistent with the custer RCT design, a of the eigibe patients were managed according to the practice s tria aocation. Patients were not asked to give individua consent on the basis that a UK practices are free to organise patient care to fit their circumstances. Reception staff simpy expained the current consutation arrangements to patients, and informed patients that they woud receive a posta questionnaire in 4 weeks time regarding their experience of care and that their competion of the questionnaire woud be greaty appreciated. Consent to the case note review At the end of the cinica interaction during the index triage or UC consutation, the cinician reminded the patient that they may receive a questionnaire in 4 weeks time regarding their experience of care that day. The cinician referred to standard text, asking the patient for their initia verba consent to the case note review aspect of the study. Twenty-eight days after the index consutation, patients were sent a questionnaire pack by the practice. Each pack incuded a covering invitation etter (see Appendix 6), an information sheet describing the study (see Appendix 7), a questionnaire (see Appendix 8) and a prepaid return enveope. The pack aso contained a fyer providing detais of an incentive to return the questionnaire: entry to a prize draw to receive one of 20 prizes of 25 worth of shopping vouchers. Patients written consent to a case note review was sought on the fina page of the questionnaire. Patients who had given initia verba consent at the index consutation coud opt out of the case note review at this stage if they wished, or opt in if they had initiay decined, with written consent aways taking priority over the previous verba response. A name and contact detais were provided if patients required any further information about the tria. Patients who did not wish to compete a questionnaire were encouraged to return a bank questionnaire in the prepaid enveope in order to receive no further contact in reation to the study. Non-responders after 2 weeks were sent a reminder questionnaire pack. Capturing the patient sampe The receptionist fagged a triage or UC appointment as being aocated for a study participant by entering a free-text comment ( ESTEEM ) and the date of the appointment request as a note on the appointment sot. Within a week, a dedicated member of the practice staff appied a study-specific Read Code to each ESTEEM -fagged patient, thus defining the eigibe patient sampe. This Read Code was dated as per the receptionist s free-text comment, aowing the practice to run eectronic searches to generate an ESTEEM patient ist that was passed to the research team as the denominator for the study database. In the triage practices, if for any reason an eigibe patient was booked a face-to-face or any other type of appointment instead of triage (e.g. a patient being unabe to receive a teephone ca whie at work), such patients were sti captured as part of the eigibe sampe and foowed up according to the study protoco. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 13

44 METHODS Patient teephone ca Eigibe for ESTEEM Not eigibe for ESTEEM Receptionist handes ca according to standard procedure for the intervention Deat with according to practice usua protoco Cinician Cas patient for triage/sees patient in UC Competes Cinician Form Requests verba consent to case notes review Verba consent yes Verba consent no Practice admin staff Adds study Read Code to patient s eectronic record Questionnaire Sent to patients at 4 weeks Written consent yes Written consent no Questionnaire bank/ not returned Written consent yes Written consent no Questionnaire bank/ not returned Case note review Performed at 12 weeks FIGURE 3 Patient recruitment and consent procedure. 14 NIHR Journas Library

45 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Simiary, if a patient was not booked into an appointment (of any type), despite having made an eigibe request (e.g. a patient faied to compete the teephone ca with the receptionist), the patient s record was sti fagged by reception staff as eigibe and incuded within the study denominator. It was not possibe to routiney record eigibiity for a patients teephoning the practice. However, on two occasions during run-in and once during data coection (see Integrity checks) receptionists ogged a incoming cas and recorded patient eigibiity, fagging those patients who were requesting same-day appointments and who were too unwe or coud not communicate ceary (see Appendix 9). We used these data to estimate the proportion of patients making same-day requests who were ineigibe for the tria. However, receptionists did not compete the og sheets as fuy as we woud have iked. For exampe, in some cases a ca had been recorded, but there was no record of whether or not it was a same-day request. Additionay, for other patients recorded as requesting a same-day appointment but who were ineigibe, the reason why was sometimes omitted. Confidentiaity A persona information obtained about patients or staff for the purposes of recruitment or data coection (e.g. names, addresses, contact detais, persona information) was kept confidentia and hed in accordance with the Data Protection Act. Each patient incuded in the tria was assigned a research number, and a data were encrypted and stored without the subject s name or address. Eectronic data were hed on a secure database accessibe by ony members of the study team, and paper-based information was hed in a ocked fiing cabinet in the research team office. Names and participant detais were not passed on to any third parties and no named individuas have been incuded in the reports. Regarding case note review, the cinica record was viewed ony when patients had given verba or written consent. Patient records were accessed by researchers hoding reevant approvas and etters of access granted by PCT research governance units and NHS staff. Case note reviews took pace in the practice and no patient records were removed from the premises. Any data extracted were identified by the research number; a other patient identifiers were removed. When data were temporariy stored on aptops, memory sticks or attached to e-mais prior to transfer to the research offices, these media were encrypted as per NHS security requirements. Researchers adhered to confidentiaity agreements stipuated by the practices concerned. Safety of participants There were not thought to be any significant risks to patients or staff arising from the tria methods. Triage itsef may carry risks but the approaches to triage in this tria were aready in routine use in the NHS in other settings. Notwithstanding this, we identified two areas of potentia minor risk, and deveoped systems to ensure that these were minimised. Minimising risk of deays to care for emergency cases or inappropriate triaging Eigibe patients are those requesting same-day consutations at their practice. Consequenty, some patients were ikey to perceive themseves to have urgent or emergency heath-care needs. Where emergencies were identified (see Incusion and excusion criteria), patients were to be deat with according to the practice s usua emergency protoco and excuded from tria participation. Teephone triage means that some patients who might otherwise have been seen in person are managed over the teephone. It was essentia that this did not ead to a patient s care being deayed when same-day care was needed. The triage systems were designed to provide safeguards against inappropriate triaging. Both have been tested in previous studies and have been found to be safe (see Heath technoogies assessed), and are aready used within some GP practices in the UK. Cinicians and receptionists received training in the system that they were using, and were provided with ongoing support from a cinica ead in the practice and from the system designers. The NT intervention, which reied on CDSS, required nurses to pass a proficiency test (see Training practices in the two triage systems). As the GPT intervention draws on cinica skis that are routiney offered by GPs, no proficiency test was required, athough guidance in the practicaities of teephone triage were provided. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 15

46 METHODS Minimising risk associated with unexpected software or triage probems Steps were taken to minimise the risks associated with any unexpected CDSS or triaging probems. Each study practice identified two named cinica eads to be the first point of contact, and who were abe to dea with any immediate probems and take corrective action. A form was estabished for use in practices to aert the cinica eads and the tria manager to any such probems arising. A run-in period of 4 weeks was aso buit in to aow the interventions to be propery set up, and to aow staff to begin operating the interventions hands on before patient recruitment and data coection began. Finay, for NT practices, the CDSS was instaed and thoroughy tested by Pain Heathcare before the tria began. Outcome measures The seection of reevant outcomes has been a contentious issue in previous evauations of triage systems in primary care. 27 Most studies assessed primary care and hospita service use and workoad. However, these outcomes may not fuy capture the aim of triage and more broady the aim of primary care. We proposed that the aim of a primary care consutation management system is to provide an administrative framework for practices to faciitate (1) the safe, timey and definitive ( first pass ) management of patients and (2) the timey and efficient management of primary care consuting time resource. We coected descriptive data on practices and patients, and for the foowing outcome domains: (1) tota primary care workoad [primary outcome measure (POM), incuding out-of-hours primary care, A&E and wak-in centre attendance]; (2) patient safety; (3) NHS resource use and cost, and non-attendance rates in primary care; (4) patient heath status and experience of care; and (5) case compexity. Describing practices and patients Practice-eve descriptive data (e.g. ist size, ocation, rura or urban nature, staffing) were coected on tria entry via the Practice Profie Questionnaire competed by the practice manager. This questionnaire aso captured information on the typica management of patients before the tria began, providing a description of UC. Practice-eve deprivation was coected from the APHO website. 37 We coected patient-eve descriptive data (age, gender, deprivation) from practice records. Patient sef-reported ethnicity, presence of ong-standing heath conditions and ease of taking time away from work (where reevant) were derived from the posta questionnaire. Primary outcome measure The POM is the tota number of NHS primary heath-care contacts that took pace in a 28-day period, commencing with the day of a patient s index teephone request for a same-day consutation. This number incudes the index consutation (initia heath-care contact resuting from the index teephone ca) and a subsequent NHS primary heath-care contacts (as defined beow) over the foowing 28 days, which may or may not be reated to the nature of the index consutation. A POM data were coected via a case note review (see Data coection and management). The primary heath-care contacts incuded in the POM were: 1. GP practice contacts: i. GP face-to-face consutation (in surgery) ii. GP teephone consutation iii. GP home visit (within surgery hours) iv. GP unspecified v. nurse face-to-face consutation (in surgery) vi. nurse teephone consutation vii. nurse home visit (within surgery hours) viii. nurse unspecified ix. genera unspecified 16 NIHR Journas Library

47 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO out-of-hours primary care contacts: i. GP face-to-face out-of-hours consutation ii. GP out-of-hours home visit iii. GP teephone out-of-hours consutation iv. nurse face-to-face out-of-hours consutation v. nurse out-of-hours home visit vi. nurse out-of-hours teephone consutation vii. unspecified 3. wak-in centre contacts: i. wak-in centre attendance (doctor) ii. wak-in centre attendance (nurse) iii. wak-in centre attendance (unspecified) 4. A&E contacts. The focus of the tria is on primary care workoad; however, it is recognised that changes in access to practice appointments may have a knock-on effect on A&E contacts and, therefore, these contacts were aso incuded within the POM. Secondary outcome measures Patient safety The number of deaths and unpanned emergency hospita admissions (and associated number of bed-days) within 7 days, and A&E attendances within 28 days of the index consutation request were coected at case note review. An unpanned emergency hospita admission was defined when there was no evidence in the notes of advance panning, even on the day the admission occurred. The number of panned hospita admissions was aso coected. An adverse events reporting procedure (see Appendix 10) was deveoped to monitor patient safety. The cinica ead(s) at each practice were contacted weeky by the tria researcher and asked to return a og sheet competed with any events arising during the intervention period, such as tria patient deaths, emergency hospita admissions or A&E attendances, as we as any probems with the triage system itsef or patient compaints. It became evident that the number of deaths recorded in the weeky og sheets of adverse events from practices did not aways correspond to that identified at case note review. On discussion with the DMC, practices were asked to perform a search of their eectronic records at the end of the tria to produce a report of a deaths occurring during the data coection period. This became the definitive source of data on patient deaths. Patient management on the index day A description of the management of patients on the index day of the same-day consutation request was based on patients first two primary care contacts coected for our POM. NHS resource use The NHS resources that were combined into the POM were aso reported separatey as secondary outcomes. Non-attendance rates for aocated appointments in the 28 days foowing the index request were aso coected by case note review. It was not possibe to capture patients contacts with NHS Direct from practice records. Patients use of NHS Direct during the 28-day foow-up period was assessed through sef-report by posta questionnaire. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 17

48 METHODS We coected actua consutation ength for the index triage and UC (face to face or teephone) consutations on the Cinician Form (see Data coection and management). We aso estimated the ength of subsequent face-to-face consutations, for both same-day and 28-day foow-up periods. The former was based on the recorded start and end times of a sma sampe of consutations captured over two randomy seected days in each practice during the intervention period. The atter was based on consutation ength used in pubished unit costs reported within the NHS Persona Socia Services Research Unit (PSSRU 38 ). Patient-reported outcomes A patient-reported outcomes were coected by the posta questionnaire. Heath status Heath status was assessed using the EuroQo Group European Quaity of Life-5 Dimensions (EQ-5D 39 ) questionnaire, and a question on probem resoution 28,29 (five-point Likert scae). We had originay intended to use the SF-36; 40 however, pioting emphasised the need to shorten the overa ength of the patient questionnaire for the main tria, and thus the shorter EQ-5D measure of heath status was adopted (see Appendix 1). Patient experience of care Patient experience in the context of this tria reates to an episode of care deivered foowing a same-day consutation request, potentiay invoving mutipe heath-care contacts during the index day. Survey instruments suitabe for use in a UK setting ask individuas to evauate a specific consutation 41,42 or aggregate (practice-based) care. As no vaidated instruments assessing the patient s experience of an overa episode of care were identified, we used reevant questions from the nationa GP Patient Survey instrument, 43 modifying the questions to focus on the patient s recent experiences of care rather than over the ast 6 months. Items seected incuded the responsiveness of the consutation management system (e.g. how quicky care was provided, overa satisfaction) and patient evauations (using a five-point Likert scae) 44,45 of the timeiness and convenience of the response 46 to the same-day consutation request. Case compexity To define patient case mix, the cinician conducting the index consutation captured the compexity of the case (after Howie et a.) 47 using an eight-point scoring schedue incorporated on the Cinician Form (see Data coection and management). Each consutation was scored as having either substantia (2 points), attributabe (1 point) or no content (0 points) with respect to each of four domains: physica, socia, psychoogica or other (e.g. administrative) components to the consutation. Sampe size Our sampe size estimate was based on our POM, combined with a methodoogy for gaining patients consent to case note review that was not eventuay operationaised in either piot or main trias. A UK study comparing NT with standard practice for handing same-day consutation requests 3 reported the number of NHS consutations based on genera practice (GP and nurse), A&E and out-of-hours primary care contacts. We beieved this to be a good proxy for our POM. Over the 28-day foow-up, that tria reported a mean number of NHS consutations of 1.02 [pooed standard deviation (SD) 0.78] in UC compared with 1.38 (pooed SD 1.79) in the NT arm. Using these data, we estimated tria sampe size requirements under four scenarios shown in Tabe 1, based on 80% or 90% statistica power, and intracuster correation coefficients (ICCs) of 0.01 or 0.05 (chosen from a range of ICCs reported from a survey across a basket of outcome measures coected in trias from primary care settings) NIHR Journas Library

49 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 TABLE 1 Sampe size cacuation n per group a ICC Design factor n patients per group Fina n patients per group c n practices per group 305 a a b b a At 90% power and 5% apha. b At 80% power and 5% apha. c See text beow for patient recruitment and attrition assumptions. Seecting the most conservative sampe size scenario (i.e. 90% power, ICC 0.05) we required 3751 patients for anaysis and 14 practices per group to detect a difference in means of 1.02 (SD 0.78) compared with 1.38 (SD 1.79) in our POM at foow-up between NT and UC arms. To inform the study power, and prior to undertaking the piot study that woud yied supporting data, we estimated that of a patients requesting a same-day consutation, 6% 3,28,29 woud be deemed ineigibe, 10% 14,29,30 woud initiay decine participation in the tria and/or foow-up of their notes when taking to the receptionist, and 30% woud not respond to the request to return the questionnaire. Of those who did respond in the questionnaire survey, we estimated 10% (conservative) woud decine notes review. Thus, a tota of 7046 patients seeking same-day consutations across 14 practices over a 5-week period woud be required in each of the three tria arms. In the absence of information about a minimum cinicay important difference, we used the same sampe size as outined for the NT and UC comparison above for the comparison between GPT and UC. Therefore, we needed 21,138 patients in tota (i.e per arm) from 42 practices. It is important to note that our patient recruitment process was atered before conducting the main tria (see Appendix 1) in two ways that both impact on our origina sampe size estimate. First, we did not incorporate a stage at which patients coud initiay decine participation in the tria and/or foow-up of their notes when speaking to the receptionist (estimated 10%, above). Second, the piot study aso ed to a change in method of patient consent to case note review from written consent ony (obtained from competed patient questionnaires) to incude initia verba consent obtained from the treating cinician. As a resut, we anticipated that, post piot study, the proportion of patients agreeing to case note review woud be around 78%, a much higher estimate than in our origina pre-piot estimate. The piot study provided confirmation of our assumed ICC of 0.05 [i.e. mean 0.03, 95% confidence interva (CI) 0.00 to 0.08]. Feasibiity of patient recruitment An average practice (7000 patients) accommodates around 714 consutations per week, 49 approximatey 142 consutations per day. Some have estimated 4 that up to one-third of these are patients seeking same-day consutations; case definition is, however, important and we proposed a more conservative estimate of around 20 patients per day (100 per week, 14% of consutations). This figure was found by the piot study to be reaistic, confirming that to achieve our sampe size for the main tria we woud need to recruit patients for a 5-week period in each of the 42 practices. Secondary outcomes Patient heath status and perception of access to care were to be coected by the posta questionnaire with an estimated response rate after one reminder of 70%; after pioting this process was atered to incude two reminder packs in the main tria. The questionnaire pack aso incorporated written consent for case note review, through which our POM is derived. Before the piot study, we estimated that at 80% power and 5% apha, our sampe size of 7046 per group woud aow us to detect an effect size of Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 19

50 METHODS 0.18 of a SD at an ICC of 0.01, and an effect size of 0.34 of SD at an ICC of Thus, the proposed sampe size woud aow us to be abe to detect a sma-to-moderate effect size in our patient-reported secondary outcomes. Tria deivery in participating practices Foowing standardised training, practice systems were aowed to stabiise during a run-in period prior to a period of ive data coection (Figure 4). Run-in period A minimum run-in period of 2 weeks was required before ive tria data coection coud proceed. During this time a triage and data coection activities were introduced and aowed to bed in. Practices remained in run-in unti their performance met predefined tria criteria indicating protoco compiance (see Integrity checks). UC practices required ony a 2-week run-in period to famiiarise themseves with research procedures. As triage practices used this period to adjust the number and distribution of triage sots and protected appointments for excusive use by the triaging cinicians, most were in run-in for 4 weeks, athough some took onger to demonstrate protoco compiance. Data coection period Data coection was panned to ast approximatey 5 weeks in each practice. Medium-sized practices ( patients) were expected to recruit 500 unique patients, whereas sma (< 3500 patients) and arge practices (> 8000 patients) were expected to recruit 350 and 550 patients, respectivey. Practices ceased data coection when the target number of unique patient records was entered into the study database; if the target was not reached within 5 weeks then data coection continued. A GPT and NT practices were asked to revert to their UC arrangements on competing the data coection period for a minimum of 4 weeks. This was for the purpose of preventing differentia continuation with a triage management system (shoud practices wish to continue using triage) by practices during the 28-day period in which we examined patients use of primary care services. GPT NT UC Inception meeting and audit of appointment Training (4 5 weeks) Run-in period (4 weeks) Data coection Foow-up FIGURE 4 Timeine of interventions. 20 NIHR Journas Library

51 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Data coection and management The sources of outcome data and the timing of data coection are summarised in Tabe 2. TABLE 2 Outcome data, timing of data coection and source of data Data Timing of data coection Source of data Practice characteristics (ist size, ocation, deprivation) Practice staffing and other cinica information Patient age, gender Prior to randomisation Prior to randomisation At entry to tria (i.e. on the day of index same-day request) List size and ocation derived from practice; deprivation derived from APHO website a Practice Profie Questionnaire Practice records Duration of first management/ Patient s index contact during tria Cinician Form triage consutation b Case compexity b Patient s index contact during tria Cinician Form Duration of face-to-face contacts Two randomy seected days during tria Recording by GPs and nurses of subsequent to index consutation request c durations of face-to-face appointments Adverse events Patient deprivation data Patient satisfaction outcomes EQ-5D Patient sef-reported NHS direct use Patient ethnicity, presence of a ong-standing heath condition, ease of taking time away from work (if reevant) Patient contacts in a 28-day period; incudes the date of the index consutation request, deaths and emergency hospita admissions within 7 days of the index consutation request During the data coection period of the tria During the data coection period of the tria Approximatey 28 days after date of index consutation request Approximatey 28 days after date of index consutation request Approximatey 28 days after date of index consutation request Approximatey 28 days after date of index consutation request Approximatey 12 weeks after the index consutation request (to aow time for contacts to be recorded at practice) Practice staff weeky og Patients residentia postcode data provided by practices and mapped to d IMD via LSOA Patient questionnaire Patient questionnaire Patient questionnaire Patient questionnaire Case note review by researchers Patient deaths within 7 days of the index consutation request End of tria (i.e. foowing end of data coection period) Practice records LSOA, ower super output area. a Association of Pubic Heath Observatories website: (accessed 5 November 2014). b The Cinician Form shoud reate to the first management/triage contact after the index consutation request. c Face-to-face contact durations were recorded for a sampe of ESTEEM patients on 2 days during the tria. d Index of Mutipe Deprivation 2010 source: (accessed 5 November 2014). Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 21

52 METHODS Cinician Form A suppy of Cinician Forms (see Appendix 11) was avaiabe in each consuting room for every day of the patient recruitment and data coection period. Cinicians competed this form at the time of the initia index consutation foowing a patient s same-day request. This index consutation was usuay a teephone triage ca in the NT or GPT arms, or a face-to-face consutation in the UC arm. The form captured consutation detais, incuding which heath professiona had been consuted; whether the patient did not attend (DNA); a measure of case mix; treatment and management options chosen (e.g. ordering of tests, recommending subsequent appointment or referra); and the start and end time of the consutation. The cinician aso recorded whether the patient had verbay consented to case note review. Patient questionnaire Patients were sent a posta questionnaire 28 days after their index request (see Appendix 8). The questionnaire measured patient experience and heath status, and use of NHS Direct during the 28-day foow-up period (see Secondary outcome measures). The questionnaire offered patients the opportunity to confirm their initia verba consent to the case note review, or to withdrawa if they wished. Patients were asked to send back a bank questionnaire in a prepaid enveope if they did not wish to be contacted again. Patients who did not respond were sent two reminder etters with another copy of the questionnaire, at 2 and 4 weeks after the initia mai-out. Case note review Researchers undertook a review of patient s medica records 12 weeks after patients index request, competing a case note review form (see Appendix 12). This deay was to aow time for patients medica records to be updated with any information on contacts with services outside the practice. The purpose of the review was to extract information regarding our POM (see Primary outcome measure) and safety data reating to hospita admissions (see Patient safety). A standardised operating procedure was deveoped to support the four researchers who were conducting these reviews. The inter-rater reiabiity of the data extraction process was assessed at three time points during the tria. Any divergence between researchers was documented and the standardised procedure updated as appropriate. The inter-rater reiabiity was assessed by each researcher reviewing the same random set of 30 case notes. Case notes were seected from different practices, tria arms and practice IT systems (Vision, EMIS LV and SystmOne). The inter-rater reiabiity resuts are dispayed in Tabe 3. Binding Given the nature of interventions it was not possibe to bind patients, cinicians or researchers (conducting case note reviews) to treatment aocation. However, data anaysis was carried out by a statistician who was bind to treatment aocation. Data entry Eectronic records were stored in a bespoke Microsoft SQL Server 2008 database (Microsoft Corporation, Redmond, WA, USA), hosted on a restricted-access secure server maintained by Pymouth University. Data entry was performed via a website encrypted using Secure Sockets Layer (SSL). TABLE 3 Inter-rater reiabiity of case note review data extraction Assessment Location of practice Tria arm of practices notes Cohen s kappa Pre-wave 1 Devon One UC and one NT, practice foowed by one GPT practice 0.64 foowed by 0.92 a Pre-wave 2 Warwick One NT practice 0.89 Pre-wave 3 Devon One NT practice 0.82 a The initia Pre-wave 1 resut of 0.64 prompted a re-assessment foowing more rigorous researcher training. 22 NIHR Journas Library

53 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Monitoring intervention fideity Integrity checks Researchers monitored patient recruitment and adherence to research procedures during both the run-in and data coection periods. Receptionists kept a handwritten og (see Appendix 9) of a teephone cas received during three haf-day sessions. Integrity checks were prepanned with practices, covering two morning sessions during the atter haf of the run-in period and one morning session during the first week of data coection. For each patient s ca that was recorded, the Receptionist Tria Log Sheet captured whether the patient was eigibe for the tria, and, if eigibe, seected one of the foowing disposition options: triage or teephone appointment booked for today face-to-face appointment booked for today face-to-face or teephone appointment booked for another day patient asked to ca again another time (i.e. no appointment booked at a) other outcome. The integrity check documented the foowing performance indicators: at east 75% of patients identified as eigibe on the og sheet were subsequenty added to that day s triage ist at east 75% of eigibe patients had the study Read Code added to their medica record at east 75% of eigibe patients had a competed Cinician Form. Practices satisfying these performance indicators during the run-in period proceeded to data coection. Practices that faied any of the indicators remained in an extended run-in period unti another integrity check coud be arranged. When practices moved into the data coection period, they were required to undertake one fina integrity check during the first week. A practice that faied this fina check was permitted to continue with data coection but the tria researcher woud have further communication with the reevant members of practice staff regarding the aspect of research procedures on which the integrity check had been faied. An overview of how practices performed at these checks can be found in Appendix 13. In addition, we used the Receptionist Tria Log Sheet competed during the integrity checks to estimate the proportion of a incoming cas to reception which were received from individuas presenting a request for a same-day consutation. We used the same data set to estimate the proportion of individuas presenting same-day consutation requests who met our excusion criteria (see Incusion and excusion criteria). Use of computerised decision support software in nurse-ed teephone triage practices A further approach to monitoring intervention fideity was put in pace for NT practices. We assessed the nurses use and extent of use of CDSS during each triage consutation. Nurses were advised that the CDSS was intended to act as a support too, and, athough ESTEEM did not aim to evauate the CDSS itsef, as a minimum it shoud be opened for a triage consutations with ESTEEM patients. The information required for an assessment of use was extracted once the tria was competed by utiising reporting functions within the CDSS. Further detais can be found in Appendix 14. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 23

54 METHODS Statistica anaysis The methods and reporting of statistica anayses were in accordance with the Consoidated Standards of Reporting Trias (CONSORT) guideines for custer randomised trias and pragmatic trias. 55,56 The combined statistics and heath economics pan 33 was reviewed by the Tria Steering Committee and DMC in advance of the anayses. Practice and patient characteristics The recruitment of tria practices and disposition of patients during the tria was summarised using a CONSORT fow diagram. Practice characteristics (ocation, ist size and eve of deprivation) were compared descriptivey across the three tria arms. Practices that withdrew from the study at any point (foowing initia consent to participate) were scrutinised with regard to their key characteristics, as we as to the tria arm to which they were aocated. The stage of the tria at which they withdrew was recorded, as we as any specific reason for withdrawa (e.g. as the resut of staff changes). The resuts of the baseine Practice Profie Questionnaire (see Appendix 15) were aso reported descriptivey. Patient demographic characteristics (age, gender, ethnicity and deprivation) were reported descriptivey by arm. Age was reported both as a continuous variabe and divided into six categories for reporting of frequencies and for use in inferentia anayses: 0 4 years; 5 11 years; years; years (reference category); years; and 75 years. Patients deprivation status was based on the Index of Mutipe Deprivation (IMD) score and rank of patients residentia ower super output area (LSOA), obtained by mapping the patient s residentia postcode to the reevant LSOA. Avaiabiity of patient information throughout the tria was reported descriptivey, as was information regarding consent status and initia patient management (specificay, whether a patient in a triage arm was regarded as having been managed per protoco (see Appendix 16). Demographic differences between patients who did and did not have case note review performed, and did or did not return a competed questionnaire with at east one competed question (excuding the consent to case note review), were aso expored. Potentia associations between age, gender and deprivation status, and avaiabiity of case note review were described using ogistic regression anayses; equivaent anayses were performed for avaiabiity of a competed questionnaire. Presence of a ong-standing heath condition and ease of taking time away from work, if reevant, were aso reported descriptivey by arm. Primary outcome The primary anaysis was based on an ITT principe, i.e. anaysis of a tria patients in practices according to random aocation. The primary anaysis took the form of a regression anaysis, using a hierarchica mode to take account of the custer aocation, utiising a random effect to adjust for potentia custering effect by practice, and aowing for adjustment for practice-eve minimisation variabes (geographica ocation, deprivation eve and ist size of practice). For a inferentia anayses, patient demographic factors were adjusted for (in conjunction with minimisation variabes) if baseine descriptive anayses indicated imbaance of demographic factors across arms. Modes were performed twice, initiay using the UC arm as reference and then using the GPT arm as reference, to derive comparison between the two triage arms. A generaised inear mode (GLM) was fitted with the appropriate choice of famiy and ink function, according to the type of data and its properties. As the POM (and a secondary outcome measures based on number of patient contacts) was a count, the most appropriate mode woud be either a Poisson or a negative binomia mode, depending on the degree of dispersion in the data. Custer-eve SDs were reported where appropriate, as this parameter approximates the coefficient of variation in underying custer rates in certain modes. 57,58 To derive an ICC for the primary outcome, a inear hierarchica mode was aso fitted. 24 NIHR Journas Library

55 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Additiona anayses were conducted on the POM, and on secondary measures derived from the POM, using the hierarchica GLM methods above. Some of these anayses were determined a priori; others were determined post hoc foowing initia inspection of the data. A priori anayses 1. A per-protoco anaysis incuding ony those patients in the triage arms who received a teephone triage contact by the appropriate cinician type on the index day (a patients in UC were considered to be per protoco). 2. Excusion of data from two practices in GPT that did not revert to UC as requested foowing their period of impementing triage and recruiting patients to the tria. 3. Investigation of the effect of missing primary outcome data (due to ack of avaiabiity of a case note review) using mutipe imputation methods, 59 based on the assumption that missing case note review data were missing at random. 4. Anayses to investigate interactions between treatment arm and practice characteristics (deprivation, ocation and ist size) on the POM, using interaction terms within the regression modes. A series of modes were fitted, each mode investigating interactions between treatment arm and a specific covariate. Age (as a proxy for case compexity), gender and ethnicity (dichotomised into White and other ethnic group, comprising Mixed or mutipe ethnic groups, Asian or Asian British, Back/African/Caribbean/back British and Other ethnic group ) were aso investigated for potentia interaction with the treatment arm. A significance threshod of 0.01 was used for hypothesis tests for interaction terms. However, the tria was not powered to investigate subgroup interactions and resuts shoud be interpreted with caution. Resuts of inferentia anayses were presented with 95% CIs (with the exception of inferentia anayses incuding interaction terms, where goba p-vaues for the individua arm by covariate interaction are aso stated). Post hoc anayses 1. Combination of a within-practice contacts on the index day into one overa contact, with contacts on the subsequent days counted individuay, as for the primary outcome [e.g. a GP teephone contact and a GP face-to-face contact within the practice on the index day woud be counted as one contact ony, whereas a GP teephone contact and an A&E contact (or any other contact outside the practice) on the index day woud count as two contacts]. 2. Anaysis of POM contacts on the index day ony, and anaysis of POM contacts occurring during the 27 days subsequent to the index day ony. 3. Anaysis of GP face-to-face contacts ony, during the fu 28-day foow-up period, on the index day ony, and on days subsequent to the index day ony. 4. Anaysis of GP face-to-face and GP teephone contacts ony, during the fu 28-day foow-up period, on the index day ony, and on days subsequent to the index day ony. Secondary outcomes The foowing secondary outcomes were anaysed inferentiay (as we as being reported descriptivey) in accordance with the principes of the anayses of the POM, i.e. using a hierarchica regression mode appropriate to the nature of the outcome data, adjusting for custer (practice) effects and with adjustment for minimisation variabes and baseine demographic variabes. Again, modes were fitted twice: once with UC as the reference arm and once with GPT as the reference arm. The ICC was reported for a hierarchica ogistic, inear and Tobit regression anayses. 1. Emergency hospita admissions (overa and divided into panned and unpanned emergency admissions) within 7 days of the index day (dichotomised by whether or not a patient had an admission). Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 25

56 METHODS 2. A&E contacts within 28 days of the index day (dichotomised by whether a patient had at east one contact). 3. EQ-5D status (patients aged 16 years ony). 4. Patient experience aspects derived from the patient questionnaire (for inferentia anayses, inearised on a scae of 0 100). 60 The foowing secondary outcomes were reported ony descriptivey: 1. patient deaths within 7 days of the index day 2. number of bed-days resuting from emergency hospita admissions within 7 days of the index day 3. individua components of the primary outcome (i.e. individua contact types) 4. non-attended contacts (DNAs) 5. patient-reported NHS Direct contacts within the 28-day period foowing, but not incuding, the index day 6. duration of first management/triage consutations 7. duration of GP and nurse face-to-face consutations 8. case compexity. Patients who DNA were recorded for each contact type and recorded descriptivey. Owing to differences between cinicians in how non-attended teephone contacts were recorded, and in how many attempts may be made to contact a patient, repeated unsuccessfu attempts at contacting a patient were recorded as one DNA (if occurring on the same day and if not foowed by a successfu teephone contact on that day). Patient management by the practice on the index day was reported descriptivey by arm, using the first patient contact recorded on the index day, and the second patient contact on the index day if present. A composite patient cinician contact duration on the index day was estimated for each arm. This was cacuated using mean contact durations (defined by cinician type and contact type) derived from appropriate sources, incuding the Cinician Form, the audit of face-to-face consutation durations and standard unit durations, 38 and the proportion of patients foowing specific management pathways on the index day (using pathways that constituted at east 1% of the tota patients with avaiabe data). This overa patient cinician contact duration was then subdivided by GP and nurse contact time. Economic evauation Aim The primary aim of the economic evauation was to compare the cost incurred over 28 days with respect to primary care contacts pus A&E contacts (using primary outcome data) for GPT, NT and UC, in peope requesting same-day appointments in genera practice. The perspective of the anaysis is that of the UK NHS, i.e. third-party payer. Method Intervention cost Resource use associated with the set-up and deivery of the triage interventions (GPT, NT) comprised initia training costs (set-up), CDSS and icence costs, and staff time spent on deivery of the triage contact. These resource use areas were identified during the piot study. Training Staff time spent at training events, required for set up of the triage intervention, was captured by a within-tria data coection form (competed by the tria researcher). This was a written record of a staff 26 NIHR Journas Library

57 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 time (incuding trainers and trainees) at a reevant training events by staff grade or type, and by practice (see Training practices in the two triage systems, above, for an outine of the training schedue in each intervention). Software and equipment When comparing triage interventions with UC, piot study research identified the ony additiona resource here (i.e. software and equipment) to be the CDSS for the nurse-ed, computer-supported triage intervention. The costs of the CDSS and icence fees (required for first year and subsequent years) in each of the NT practices were documented (with information from suppier) within tria. Triage staff time Staff time (GP, nurse) used on the deivery of the triage contact (patient contact time) was recorded using the within-tria Cinician Form (see Data coection and management). Data coection incuded staff type by grade, and start time and end time for each triage contact was recorded. Resource-use data were combined with unit cost data, and market prices (software, icence fees) to estimate the mean cost per triage contact. Costs were reported using 2012 cost data, or with cost data uprated to 2012 costs where required. Costs associated with training and other set-up costs (computer support system) for triage interventions were estimated at a mean cost per practice, and an estimate of the expected number of same-day contacts per practice per year was used to spread the costs across an expected patient group (number of patients requesting same-day appointment per practice). Assumptions on these data and other areas of uncertainty are tested using sensitivity anayses. Economic outcome: costs of primary care contacts (pus accident and emergency) over 28 days Primary economic (cost) anayses are undertaken using data coected on the primary outcome, contacts taking pace in primary care over 28 days, coected within tria at participant eve using a case note review. Primary care and reated contacts by type of contact, as incuded in the POM, are detaied above (see Primary outcome measure). Tria data on service use were combined with unit cost data (Tabe 4) to estimate a mean 28-day cost for primary care service use, for each of the tria arms. Most unit cost data were taken from those reported by Curtis 38 (PSSRU unit costs), with other cost data sourced from credibe nationa data sources (see Tabe 4). Contacts recorded as GP or genera unspecified (n = 30) are treated as GP in surgery consutation, for the purpose of cost anayses. Contacts recorded as Nurse contact unspecified (n = 15) were treated as Nurse in surgery consutation for the purpose of cost anayses. Exporatory anayses report data (primary outcome) on the resource use and costs for same-day care for participants by intervention arm. This anaysis incudes the data on index contact and other resource use on the same day as the index contact. These exporatory anayses used unit costs for triage contacts as derived from tria data (as above), and pubished unit cost data on other contacts by type. A sampe of data was coected within the tria to provide information on the duration of GP and nurse face-to-face consutations foowing a triage contact. These data were considered in the context of exporatory anayses of same-day care, and costs of same-day care. Data anaysis: presentation of anaysis Economic anayses were consistent with the methods described for the main statistica anayses (effectiveness outcomes data). The primary economic anayses were based on the ITT tria data (as described above). Data are presented descriptivey, and thereafter cost anayses use a random-effects regression mode taking account of the hierarchica nature of the study design (i.e. aocation by practice) and aowing for adjustment for practice-eve minimisation variabes (geographica ocation, deprivation eve and size of practice) and participant-eve covariates for age and gender. Data were initiay expored using a GLM fitted with the appropriate choice of famiy and ink function according to the type of data Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 27

58 METHODS TABLE 4 Unit cost data (2012) with data sources for the base case Resource use unit Unit cost ( ) Source GP teephone triage intervention Nurse teephone triage intervention ESTEEM tria (see Tabe 26) 7.62 ESTEEM tria (see Tabe 26) GP consutation (in surgery) Per contact unit cost from Curtis, 38 based on consutation asting 11.7 minutes, incuding direct care staff costs with quaifications GP teephone consutation Per contact unit cost from Curtis, 38 based on teephone consutation of 7.1 minutes, incuding direct care staff costs with quaifications GP home visit (within surgery hours) Practice nurse consutation (in surgery) Practice nurse teephone consutation Practice nurse home visit (within surgery hours) Wak-in centre attendance (doctor/nurse/unspecified) Per contact unit cost from Curtis, 38 based on home visit asting 23.4 minutes (incuding trave time/cost) Derived using cost per minute from Curtis. 38 Assumes 15.5-minute nurse consutation unit cost per hour/minute ( 53/ 0.88, incuding quaifications costs) for face-to-face contact 5.28 Derived using cost per minute from Curtis. 38 Assumes 6 minutes/ teephone consutation cost [duration from advance nurse] unit cost per hour/minute ( 53/ 0.88, incuding quaifications costs) for face-to-face contact Derived using cost per minute from Curtis. 38 Assumes 25 minutes/home visit cost [duration from advanced nursing professiona] unit cost per hour/minute ( 53/ 0.88, incuding quaifications costs) for face-to-face contact Per contact unit cost from Curtis 38 A&E wak-in service (not admitted) Out-of-hours contact Primary Care foundation, 2013 ( (doctor/nurse/unspecified) a A&E: doctor/nurse/unspecified Per A&E attendance, not admitted nationa average (weighted averages irrespective of occupation) from Curtis 38 a Unit cost based on average cost per case, incuding a overheads and oncosts associated with provision of out-of-hours care. Obtained from survey of primary care trusts in Engand. URL: (accessed 5 November 2014). and its properties. Based on findings from these anayses (using GLM methods), main anayses are presented using a hierarchica mutieve mode, assuming normay distributed tota cost data. The ICC is reported for primary cost anayses. The primary economic anayses present estimates of the mean cost of care across each of the tria arms, as above. Primary anayses report on participants with data on the POM, i.e. a compete case anaysis. Regression-based methods (as above) were used to estimate difference in costs for care between tria arms, based on the 28-day data incuded in the primary outcome. CIs (95%) are estimated using parametric methods. Typicay, where a sampe has a arge number of observations, as in this case (with > 16,000 participants in ESTEEM), incorporating centra imit theorem impies parametric tests are appropriate and may be used for anaysis of resource use and cost data. Item-eve costs are presented descriptivey, consistent with the data presentation in the effectiveness anayses. In secondary anayses, a per-protoco anaysis has been performed (for the primary economic outcome ony), incuding ony patients who received the triage intervention, this being consistent with the main statistica anaysis pan and effectiveness anayses. As no difference is reported on EQ-5D singe index vaues (quaity-adjusted ife-year weights), by treatment aocation, no exporatory anayses are considered on cost-effectiveness anayses using this outcome. 28 NIHR Journas Library

59 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Sensitivity anayses were undertaken against the primary anayses to expore the impications of uncertainty in data used and the assumptions made within the primary anayses. Sensitivity anayses incuded an anaysis of primary outcome data (tota 28-day cost), with missing data imputed via mutipe imputation methods (as used in effectiveness anayses). Resuts from exporatory anayses are presented descriptivey, and with regression-based methods used to provide comparative anayses where appropriate. Resuts are presented in tabuar format using mean estimates of resource use and cost, with summary measures on the distribution around the mean. A broader presentation of findings are presented in tabuar format consistent with the approach described as cost consequences anayses, presenting estimates of costs aongside the expected impacts associated with interventions, for exampe safety outcomes, heath status and measures of patient experience. Cost consequences anayses are regarded as a form of fu economic evauation, even though the costs and outcomes are not brought together in a cost-effectiveness ratio. 61 Process evauation Aims and objectives The process evauation for the ESTEEM tria took pace aongside the main tria. Its aims were to: describe how the tria was experienced and how the teephone triage interventions were impemented in different practice settings describe the experience and acceptabiity of teephone triage for staff and patients eicit patients and staff views on what infuences the teephone triage being seen to work or not to work. Methods The process evauation was composed of semistructured, quaitative interviews conducted in a subsampe of tria practices. This was suppemented by a imited amount of observation and interviews with researchers from two of the study sites. Samping, recruitment and consent procedures Process evauation data were coected from a purposive sampe of 10 of the main tria practices (four practices in each triage arm and two in UC) shown in Tabe 5. The practices were samped from three regions (Devon, Bristo and Warwick) and varied in terms of their ist sizes and ocations (inner city, urban, suburban and rura). Patient recruitment Practices provided the research team with ID numbers for the first 20 consecutive patients eigibe for tria entry, within 3 days of a same-day consutation request. Eigibe patients were aged < 12 years or 16 years, with proxy participation by parents or guardians of those patients aged < 12 years. Practices provided the research team with an anonymised ist of the patients approached, incuding patients TABLE 5 Process evauation: participating practices by region Region GPT practices NT practices UC practices Tota Devon Bristo Warwick Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 29

60 METHODS interna practice ID number, patients initias, age, gender and ethnicity, as we as an indication of their anticipated views on the triage systems if this was known but practices did not aways provide this atter information. Cinicians who managed same-day consutations were asked to review the samping frame and to remove any patient where they beieved interviews woud be cinicay inappropriate and/or ikey to cause distress, either to patients themseves and/or to their famiies (e.g. end-stage termina iness or death within the immediate famiy). We aimed to recruit 15 patients per tria arm, to secure a tota of 45 patient interviews. Patients were to be seected purposivey to represent as diverse a sampe as possibe, based on age, gender, ethnicity and anticipated views (where possibe) on the triage system. However, suggish patient recruitment necessitated revising the target sampe size downwards, which meant that purposive samping was compromised. Practices were provided with etters to send, inviting patients to interview (encosing patient information eafets, repy sheets and repy-paid enveopes). Where possibe, interviews were secured within 2 weeks of patients index same-day consutation requests. Patients who were potentiay wiing to be interviewed returned the study repy sheets containing their preferred contact detais directy to the research team. It was intended that five of the first patients in the main tria who repied from each practice woud be contacted by the process evauation researchers to discuss the study further and to agree mutuay convenient interview dates and times with those who wished to proceed. This step was to be repeated unti four wiing patients from each practice, with diverse demographic characteristics, were found. The sow response necessitated revising the recruitment process, so that a patients returning their contact information and consent to be contacted were approached. The recruitment criteria were aso revised to provide patients with the option of a teephone interview. This proved critica in improving the response rate, and the majority of the interviews with patients were conducted by teephone. This seems an appropriate method to use in a study of teephone-based interventions. Interviewees written consent to participate was obtained by the researchers either at interview, if face to face, or by post if interviews were conducted by teephone. Written consent to teephone interviews was obtained retrospectivey in some cases, athough a patients were asked to provide verba consent prior to commencing the teephone interview. Staff participants We aimed to recruit five staff members from each practice, to secure a tota of 50 interviews. A practice staff within samped practices were invited to participate in the process evauation; those who responded to the invitation and went on to participate were sef-seected. A purposive samping strategy was designed, seecting potentia staff interviewees for occupationa diversity to ensure that the views of GPs, nurses, practice managers and reception staff were represented. Staff members of both sexes and of varied ages were approached. Each staff member received a personay addressed etter signed by the investigators in their respective recruitment sites, inviting them to participate in a study interview. The invitation etters were accompanied by staff participant information sheets, repy sheets and repy-paid enveopes. Staff members who were potentiay wiing to be interviewed returned the study repy sheets containing their preferred contact detais directy to the research team. Potentia participants were contacted by the researcher to discuss the study further and to agree a mutuay convenient interview date and time with those who wished to proceed. This step was repeated unti five staff members, representing each occupationa group at each practice, were found. The sma number of staff members who expressed an interest, but were not seected for interview, received etters of thanks. Interviewees written consent to participate was obtained by the researchers at interview. Tabes 6 and 7 show the numbers of patient and staff participants recruited. 30 NIHR Journas Library

61 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 TABLE 6 Process evauation participating practice staff Tria arm GPs Practice nurses or nurse practitioners Managers Receptionists Tota GPT NT UC Tota TABLE 7 Process evauation participating patients Tria arm Mae Femae Tota GPT NT UC Tota Data coection One-to-one semistructured interviews of minutes duration were hed with seected patients and practice staff. Staff interviews were conducted face to face within practice premises. Interviews with patients were conducted either face to face in patients homes, or by teephone, according to participants choices. Purpose-designed interview schedues were deveoped by the study team and the PPI group. They guided the interview process throughout and were further deveoped to refect emerging interview topics. Patient interviews expored the patient s index consutation (whether on the teephone in the triage conditions or face to face in the UC arm) and specificay their story of that consutation, and its antecedents and consequences. Views on the acceptabiity, convenience and speed of patients access to the new teephone triage systems were aso sought. Staff interviews expored preparation and training for the tria; expectations, experiences and views hed by practice staff of setting up and running the triage systems; the systems perceived acceptabiity, probems, soutions (or not) and staff members hypothetica wiingness (or not) to use triage systems post study. In contro practices, staff members were asked about the usua management of same-day patient consutation requests at their practice. Interviews were audio-recorded with the permission of the interviewees. No staff member refused permission to record the interview. However, it was suggested to the researcher that the knowedge that the interview woud be recorded had inhibited recruitment in a sma number of cases. Letters were sent to both patient and staff interviewees to thank them for their participation. Both patients and staff were aso given the opportunity to confirm and correct descriptive summaries of the preiminary findings (main themes) from the thematic anaysis of the study s patient and staff interviews conducted, if they wished to do so. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 31

62 METHODS Data anaysis The 99 interview audio tapes were transcribed verbatim by a professiona transcriber and checked and anonymised by the process evauation researcher. It became cear that UC practices had imited usefuness as a contro or comparator in a quaitative study of experience of triage. They were not experiencing triage and UC was not standardised. For various pragmatic and methodoogica reasons data were anaysed from the intervention practices ony (n = 84). The transcripts were anaysed thematicay, drawing on the grounded theory techniques of constant comparison 62 using the quaitative data anaysis software package NVivo 8 (QSR Internationa, Warrington, UK). A deductive coding frame based on the process evauation research questions was agreed by the research team. Within this structure data were coded inductivey to aow participants accounts to inform the anaysis to capture the subteties and subjectivities of a mutipicity of experiences in a range of settings. It aso permitted the identification of structura, cutura and organisationa factors within practices that framed these experiences, and how they were described and interpreted. A framework approach 63 was then adapted to enabe both within-case (individua practices) and cross-case (between practices) anayses. The practice emerged as the unit of anaysis, as it became evident that data coud not be interpreted without considering aspects of individua practices cuture and ways of working. Groups of coded materia were summarised and charted onto a matrix to map the range of phenomena emergent from the data, and to enabe reationships and interreationships to be conceptuaised in a systematic way. The anaysis of interviews was iterative, with knowedge gained from staff providing insights into patient interviews and vice versa. It aso refected differences in the impementation of the interventions and contextua factors in individua practices, as we as investigating the effect of how the tria itsef was experienced and how the intervention was impemented and experienced. Both patient and staff interviews were interrogated in terms of any areas of emerging agreement or disagreement about what worked, or did not work, and any observabe conficts and differences of opinion between and within staff groups. Coding and anaysis was undertaken by one researcher (LP) and vaidated by a second researcher (NB), who reviewed a sampe of the transcripts and tested these against the coding frame. 32 NIHR Journas Library

63 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Chapter 3 Resuts Fow of practices and patient participants in the tria The fow of practices and patients through the ESTEEM tria is summarised in Figure 5. Practices Foowing an approach to 388 practices, 42 (10.8%) agreed to participate and were randomised to one of the three tria arms (GPT 13, NT 15, UC 14). The 12 post-randomisation withdrawas (UC 2, NT 10) were repaced. Of 42 practices randomised, 11 withdrew and were repaced in a purposefu process in which we drew from a sma poo of reserve practices, matching as cosey as possibe to ocaity, size and deprivation profie of the origina withdrawn practice, and with aocation conceament preserved (see Repacing practices who withdrew post randomisation). One of the substitute practices itsef subsequenty withdrew from the tria and had to be repaced making a tota of 12 repacement practices. Finay, of the 12 practices withdrawing from the tria, 11 withdrew in the very eariest stages post randomisation, usuay prior to research procedures training or the run-in period of 4 5 weeks (see Figure 1); ony one practice withdrew during the tria data coection period. Reasons for practice dropout A of the practices that withdrew did so prior to patient recruitment, athough one practice in NT did begin the run-in period and had commenced managing patients with a triage system. This practice had two nurses deivering the triage but fet that it coud not manage the demand. The practice hated the triage and decided to use the funds made avaiabe through the tria to recruit ocum nurses to cover the triage. Unfortunatey, the practice was unabe to find any suitabe candidates and fet, therefore, that it coud not continue with the tria. The post-withdrawa questionnaire was returned by ony one of the two UC practices that withdrew the practice reporting dissatisfaction with the tria arm to which they had been randomised. We are unabe to comment on why the other UC practice withdrew. Nine of the ten practices that withdrew from NT returned a post-withdrawa questionnaire. Athough one practice was dissatisfied with the randomisation to NT, the majority of reasons for withdrawa revoved around the practice feeing under-resourced to cope with the demands of the tria: two practices had difficuty recruiting additiona staff, four practices had difficuty extending staff hours as needed, and, in one case, nursing staff had eft the practice (for reasons other than the tria). Other reasons for practice withdrawa from the tria (each reason cited by ony one practice) incuded retirement of key staff members, key staff on ong-term sick eave, insufficient commitment from practice staff, ack of support from the ESTEEM research team, and introduction of a triage system between recruitment and the start of the tria. Patients Recruitment of patients by practices as potentiay eigibe for the tria (i.e. requesting a same-day consutation) over the course of the study, against targets, between May 2011 and December 2012, is shown in Figure 6. Data on the numbers of patients excuded from the tria at the point of requesting a same-day consutation as a resut of being too unwe or unabe to communicate without difficuty were not routiney coected. The Receptionist Log Sheet competed for the integrity checks (twice during the run-in period and once during the data coection period) provided an estimate of the proportion of these patients from the tota number of same-day requests (Tabe 8). Missing data from the Receptionist Log Sheet on whether or not the incoming ca was a request for a same-day appointment ranged from around 19% in UC to 25% and 29% in GPT and NT, respectivey. For a proportion of patients who did make a same-day request we were unabe to determine whether they were eigibe or ineigibe for the tria. The reason for this, as noted in Chapter 2, was that receptionists did not reiaby record the reason why patients requesting a same-day appointment were ineigibe. Missing data from this group of patients ranged from < 1% in UC to 4% in GPT. Avaiabe data indicated that between 9% and 19% of patients making same-day requests were ineigibe for the tria. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 33

64 RESULTS Enroment Approached for incusion (number of practices) (n = 388) Positive response (number of practices) (n = 103) Meetings hed (number of practices) (n = 71) Randomised (number of practices) (n = 42) Did not respond, n = 129 Decined, n = 133 Ineigibe, n = 23 Ineigibe, n = 5 Decined, n = 27 Ineigibe, n = 4 Decined, n = 13 Waiting ist, n = 12 a GPT NT UC Withdrawn (n = 10) Withdrawn (n = 2) Repaced (n = 10) a Repaced (n = 2) a Aocation (n = 22,261) Practices Practices Practices Practices receiving intervention b (n = 13, M = 540, SD = 69.5) Patients Patients requesting same day appointment (n = 7017) Patients excuded aged years, n = 236 Patients withdrawn, n = 84 Patients eigibe for intervention, n = 6697 % patients receiving intervention: 93% c Practices receiving intervention b (n = 15, M = 502, SD = 133.5) Patients Patients requesting same day appointment (n = 7525) Patients excuded aged years, n = 261 Patients withdrawn, n = 250 Patients eigibe for intervention, n = 7014 % patients receiving intervention: 89% c Practices receiving intervention b (n = 14, M = 551, SD = 61.6) Patients Patients requesting same day appointment (n = 7719) Patients excuded aged years, n = 255 Patients withdrawn, n = 177 Patients eigibe for intervention, n = 7287 % patients receiving intervention: 100% c Foow-up (n = 20,990) Anaysis (n = 16,211) Patients Patients Patients Patients sent a questionnaire, n = 6695 Overa patients consenting to CNR, d n = 5202, 78% Practices Practices anaysed (n = 13, M = 515, SD = 69.5) Patients Primary outcome Notes reviewed, n = 5171, 77% Notes not reviewed, n = 31 Secondary outcomes Compete questionnaires returned, n = 4113, 61% Patients with a cinician form, n = 6032, 90% Patients sent a questionnaire, n = 7012 Overa patients consenting to CNR, d n = 5488, 78% Practices Practices anaysed (n = 15, M = 467, SD = 115.6) Patients Primary outcome Notes reviewed, n = 5468, 78% Notes not reviewed, n = 20 Secondary outcomes Compete questionnaires returned, n = 3837, 55% Patients with a cinician form, n = 6598, 94% Patients sent a questionnaire, n = 7283 Overa patients consenting to CNR, d n = 5589, 77% Practices Practices anaysed (n = 14, M = 520, SD = 59.6) Patients Primary outcome Notes reviewed, n = 5572, 76% Notes not reviewed, n = 17 Secondary outcomes Compete questionnaires returned, n = 4182, 57% Patients with a cinician form, n = 5984, 82% FIGURE 5 Consoidated Standards of Reporting Trias diagram. a, Withdrawn practices were purposivey repaced (with practices from the waiting ist ) whie maintaining aocation conceament; b, n (number of practices), mean number patients (SD) per practice; c, given the nature of the tria it was not possibe to determine exacty the number of patients who received the intervention. Assessment of whether patient was treated per protoco was dependent upon the patient having a competed Cinician Form or having their medica notes reviewed. Here, % receiving intervention is based upon case note reviews; % receiving the intervention based upon Cinician Forms; and d, CNR = case notes review. Reproduced with permission from Campbe et a NIHR Journas Library

65 24,000 22,000 20,000 18,000 16,000 14,000 12,000 10, Week Target Expected Actua 0 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Number of patients FIGURE 6 Patient recruitment by practices. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 35

66 RESULTS TABLE 8 Estimate of the proportion of patients seeking same-day consutations who were ineigibe for the tria a Receptionist og sheet data UC GPT NT Tota proportion of patients requesting a same-day consutation (from tota recorded teephone cas) 48% (1964/4090) 36% (1382/3865) 37% (1896/5115) Proportion of patients who were ineigibe due to being unwe or unabe to communicate without difficuty (from tota requesting a same-day consutation) 10% (189/1960) 9% (119/1320) 19% (358/1843) a These data were based on og sheets competed by reception staff. Expanations other than data inaccuracy do exist there may be heightened awareness of the nature of a consutation request in intervention practices, or there may indeed be a true difference in the proportion of same-day requests incurred by chance in the UC practices. We can ony specuate on this matter, however, and to extend beyond what we have said woud be unwarranted. Of the patients identified by receptionists as requesting a same-day consutation, who were both we enough and abe to communicate without difficuty, a tota of 20,990 patients [6695/7017 (95%) in GPT, 7012/7525 (93%) in NT and 7283/7719 (94%) in UC] were eigibe for the tria, foowing excusion of patients aged years and withdrawa of a number of patients for various reasons. Reasons incuded administrative errors in appying Read Codes to patients records, patients being entered into the tria mutipe times and incorrect incusion of temporary residents, patients making non-same-day requests and patients with hearing impairment or dementia. In the NT arm, 211 patients were withdrawn after being identified for the tria by practices during periods in which no triage system was operating. The proportion of patients who received the intended intervention (thus treated per protoco) was simiar across the intervention arms [4796/5171 (93%) in GPT and 4860/5468 (89%) in NT]. Patient oss to foow-up A eigibe patients (n = 20,990). Eight patients withdrew (various reasons) prior to being sent a questionnaire. Of a patients sent a questionnaire (excuding post-questionnaire withdrawas), consent to case note review (providing primary outcome data) was given by 5202/6695 (78%) in GPT, 5488/7012 (78%) in NT and 5589/7283 (77%) in UC. The procedure for obtaining a patient s consent to case note review was compex. For ease of presentation we have reported the fina decision foowing verba and written consent. The specific fow the numbers of patients providing verba consent and then opting out via written consent, and the numbers of patients refusing (or not having any record of) verba consent but opting in via written consent can be seen in Appendix 16. In tota, 68 patients who provided consent to case note review did not have primary outcome data extracted (< 0.01% of patients providing consent: 68/16,211). This was because these patients returned a questionnaire incuding written consent to case note review (which was not obtained verbay) after the research team had competed case note reviews for the reevant practice. The case note review foow-up period ended in January Describing usua care Information regarding the practices, their staff numbers and use of teephone triage (prior to the commencement of the tria) is set out in Tabe 9. In genera, across a of the arms, a arge proportion of practices did not aocate time for teephone triage (26/42). Of those that did, teephone triage was mainy used to manage patients seeking same-day consutations once face-to-face appointment sots had been taken (19/22). The majority of practices tended to manage < 25% of patients seeking same-day consutations via teephone triage (13/22). Staff composition and services provided by the practice can be seen in Appendix NIHR Journas Library

67 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 TABLE 9 Practice Profie Questionnaire Practice Profie Questionnaire items UC (N = 14) GPT (N = 13) NT (N = 15) List size: mean (SD), n 8257 (3792), (3619), (3600), 15 Number of GPs at practice (incuding fu-time and part-time): a mean (SD), n Number of nurses at practice (incuding fu-time and part-time): a mean (SD), n 5.9 (2.6), (3.0), (2.8), (1.8), (1.6), (1.4), 15 Practice ruraity: n (%) Rura 7 (53.9) 7 (58.3) 6 (42.9) Urban 6 (46.2) 5 (41.7) 7 (50.0) Inner city 0 (0) 0 (0) 1 (7.1) Tota 13 (100.0) 12 (100.0) 14 (100.0) Does the practice aocate time for doctors or nurses to carry out teephone triage? n (%) Yes 3 (21.4) 4 (30.8) 9 (60.0) No 11 (78.6) 9 (69.2) 6 (40.0) Tota 14 (100.0) 13 (100.0) 15 (100.0) Which option best describes how teephone triage is used by doctors/nurses at your practice? n (%) Patients seeking a same-day consutation are triaged on the teephone ony if a of the appointment sots are taken Teephone triage is not used across the whoe practice but at east one doctor triages his/her own patients 3 (100) 6 (75.0) 10 (90.9) 0 (0) 2 (25.0) 1 (9.1) Tota 3 (100.0) 8 (100.0) 11 (100.0) What proportion (%) of patients seeking a same-day consutation are triaged on the teephone? n (%) < 25 2 (66.7) 6 (75.0) 5 (45.5) (0) 1 (12.5) 4 (36.4) (0) 0 (0) 1 (9.1) (0) 0 (0) 0 (0) Cannot estimate 1 (33.0) 1 (12.5) 1 (9.1) Tota 3 (100.0) 8 (100.0) 11 (100.0) a If data were missing for either fu- or part-time cinicians, the number was assumed to be 0 and the tota cacuated using ony the avaiabe data. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 37

68 RESULTS Practice descriptions and patient demographic characteristics Practice descriptors Given the reativey sma number of practices in each arm, the practice characteristics (ocation, ist size and deprivation) were generay we baanced across the arms (Tabe 10). Patient demographics Patient age, gender and deprivation status were we baanced across tria arms (Tabe 11). In a three arms, femae patients constituted the majority (approximatey 60%): patients in GPT were sighty oder than in UC or NT [mean age (years) in GPT was 44.7 (SD 25.0), in UC 41.6 (SD 23.7) and in NT 41.5 (SD 25.2)]. Among questionnaire respondents, across the three arms, 95 97% were white peope and 46 50% reported that they had a ong-standing heath condition. Based on the resuts of anayses of predictors of avaiabiity of case note review (see Appendix 18), it was decided to conduct mutipe imputations anayses for the primary outcome (and tota costings) ony, on an ITT basis. TABLE 10 Practice demographics and custer size coefficient of variation Practice demographic variabes UC (N = 14) GPT (N = 13) NT (N = 15) Location: n (%) Tota incuded (N = 42) Withdrawn (N = 12) a Bristo 3 (21.4) 4 (30.8) 3 (20.0) 10 (23.8) 2 (16.7) Devon 4 (28.6) 3 (23.1) 4 (26.7) 11 (26.2) 3 (25.0) Norwich 4 (28.6) 3 (23.1) 3 (20.0) 10 (23.8) 4 (33.3) Warwick 3 (21.4) 3 (23.1) 5 (33.3) 11 (26.2) 3 (25.0) Deprivation: b n (%) Deprived 5 (35.7) 3 (23.1) 3 (20.0) 11 (26.2) 5 (41.7) Non-deprived 9 (64.3) 10 (76.9) 12 (80.0) 31 (73.8) 7 (58.3) List size: c n (%) Sma 1 (7.1) 1 (7.7) 2 (13.3) 4 (9.5) 1 (8.3) Medium 6 (42.9) 5 (38.5) 4 (26.7) 15 (35.7) 4 (33.3) Large 7 (50.0) 7 (53.9) 9 (60.0) 23 (54.8) 7 (58.3) Coefficient of variation for custer size a A withdrawn practices withdrew post randomisation: 2 in UC, 10 in NT. b Obtained from Pubic Heath Engand Nationa Genera Practice Profies: (accessed 5 November 2014) Deprived, above average deprivation for Engand; Non-deprived, average/beow-average deprivation for Engand. c Sma, < 3500 patients registered; medium, patients registered; arge, > 8000 patients registered. 38 NIHR Journas Library

69 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 TABLE 11 Baseine patient demographics Individua patient characteristics for tota cohort UC (N = 7283) GPT (N = 6695) NT (N = 7012) Gender, n (%) Mae 2920 (40.1) 2735 (40.9) 2774 (39.6) Femae 4363 (59.9) 3960 (59.2) 4238 (60.4) Age (years): mean (SD) 41.6 (23.7) 44.7 (25.0) 41.5 (25.2) Age by category (years), n (%) < (9.5) 605 (9.0) 830 (11.8) (6.5) 379 (5.7) 463 (6.6) (11.5) 675 (10.1) 726 (10.4) (46.2) 2875 (42.9) 3058 (43.6) (18.5) 1317 (19.7) 1204 (17.2) (7.8) 844 (12.6) 731 (10.4) Deprivation (IMD score a ): mean (SD), n 17.6 (10.3) (11.8) (11.7) 6930 Deprivation (IMD quintie, based on rank), n (%) Quintie 1 (most deprived) 460 (6.4) 524 (7.9) 653 (9.4) Quintie (23.4) 995 (14.9) 1348 (19.5) Quintie (25.7) 1992 (29.9) 1673 (24.1) Quintie (26.0) 1916 (28.7) 1783 (25.7) Quintie 5 (east deprived) 1345 (18.6) 1244 (18.7) 1473 (21.3) Tota N Individua patient characteristics for questionnaire respondents ony UC (N = 4182) GPT (N = 4113) NT (N = 3837) Ethnicity by ethnic group, n (%) White 3956 (96.5) 3876 (96.0) 3573 (95.3) Mixed/mutipe ethnic groups 33 (0.8) 36 (0.9) 27 (0.7) Asian/Asian British 82 (2.0) 79 (2.0) 110 (2.9) Back/African/Caribbean/back British 15 (0.4) 34 (0.8) 24 (0.6) Other ethnic group 15 (0.4) 12 (0.3) 17 (0.5) Tota N Abe to attend surgery during work hours, n (% tota N;% tota N reevant b ) Yes, easiy 794 (19.6; 38.2) 790 (19.8; 41.2) 736 (19.8; 39.7) Yes, with difficuty 883 (21.8; 42.5) 830 (20.8; 43.3) 778 (21.0; 42.0) No 402 (9.9; 19.3) 296 (7.4; 15.4) 340 (9.2; 18.3) Tota N reevant Not reevant, n (% tota N) 1974 (48.7) 2068 (51.9) 1857 (50.0) Tota N Long-standing heath probems n (%) Yes 1940 (48.0) 1985 (50.0) 1716 (46.4) No 2101 (52.0) 1983 (50.0) 1985 (53.6) Tota N a IMD score and rank derived from residentia postcode data mapped to LSOA. b Excudes chidren and non-working aduts. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 39

70 RESULTS Cinica outcomes Primary outcomes The proportion of patients having zero or more primary care contacts is dispayed as a function of tria arm in Figure 7. The moda number of contacts under UC was one, whereas the moda number of contacts under both GP- and nurse-ed teephone triage was two. Intention-to-treat anayses incuded 16,211 patients (UC: n = 5572; GPT: n = 5171; NT: n = 5468) and showed an increase in the rate ratio (RR) of contacts comparing GPT with UC (RR 1.33, 95% CI 1.30 to 1.36) with adjustment for practice characteristics (ist size, ocation and deprivation status) and patient demographic characteristics [gender, age (categorised) and deprivation quintie; Tabe 12]. The same mode comparing NT with UC yieded a RR of 1.48 (95% CI 1.44 to 1.52); comparing NT with GPT, the RR was 1.04 (95% CI 1.01 to 1.08). The equivaent per-protoco anaysis showed an intensification of the observed effects of both GPT and NT (see Appendix 19). A sensitivity anaysis excuding two GPT practices that did not revert to UC foowing the end of the tria period yieded resuts simiar to those of the primary ITT anaysis (resuts not shown). The observed ICC derived from a hierarchica inear mode was (95% CI to 0.025), ower than the ICC of 0.05 in the origina sampe size cacuations (see Sampe size). Using data derived from mutipe imputations combined with observed data where avaiabe, the resuts of ITT anayses were simiar to those derived from observed data ony (see Tabe 12) and hence the compete case resuts appeared to be robust compared with anayses incuding imputed data. There was some evidence of interactions between treatment arm and practice eve covariates (ocation and practice deprivation), and between treatment arm and age category; however, no cear patterns were evident (see Appendix 20). 60 Proportion of patients (%) UC GPT NT Number of contacts FIGURE 7 Primary outcome resuts by arm. 40 NIHR Journas Library

71 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 TABLE 12 Primary outcome measure: tota primary care contacts (incuding A&E) within 28 days of index consutation request across tria arms UC (N = 5572) GPT (N = 5171) NT (N = 5468) Tota primary care contacts: a,b tota, mean (SD) 10, (1.43) 13, (1.74) 15, (1.68) Tota primary care contacts: a,b n (%) 0 45 (0.8) 14 (0.3) 16 (0.3) (51.4) 1179 (22.8) 670 (12.3) (26.6) 1902 (36.8) 2391 (43.7) (10.9) 966 (18.7) 1146 (21.0) (5.2) 526 (10.2) 591 (10.8) (2.4) 255 (4.9) 287 (5.3) (2.4) 300 (5.8) 342 (6.3) (0.2) 29 (0.6) 25 (0.5) ITT anayses using compete case data, RR (95% CI) GPT vs. UC NT vs. UC NT vs. GPT Poisson GLLAMM mode c with robust residuas: adjusted d (n = 16,095), RR (95% CI) 1.33 (1.30 to 1.36) 1.48 (1.44 to 1.52) 1.04 (1.01 to 1.08) Custer-eve SD e 0.09 Mean difference: adjusted d (n = 16,095) 0.69 (0.52 to 0.85) 0.90 (0.73 to 1.06) 0.21 (0.05 to 0.37) ICC (95% CI) (0.009 to 0.025) ITT anayses using compete case data and imputed data a,b (UC: N = 7281, GPT: N = 6689), NT: N = 7012), RR (95% CI) GPT vs. UC NT vs. UC NT vs. GPT Poisson regression mode c with ordinary 1.37 (1.28 to 1.46) 1.48 (1.38 to 1.58) 1.08 (1.01 to 1.15) residuas; adjusted d Custer-eve SD e 0.08 GLLAMM, Generaized Linear and Latent Mixed Modes. a Patients who died within 28 days of index day excuded. b Incudes a 20 contact types constituting the primary outcome. c A modes incude random effect on practice. d Adjusted modes adjust for practice variabes: ocation (Devon, reference; Bristo; Warwick; Norwich), practice ist size [arge (> 8000 patients), reference; medium ( patients); sma (< 3500 patients)], practice deprivation [non-deprived (at or beow average deprivation for Engand by APHO ratings), reference; deprived (above average for Engand by APHO ratings)] and patient variabes: age (categorised, 0 4; 5 11; 16 24; 25 59, reference; 60 74; 75 years), gender (reference: femae), IMD deprivation based on residentia postcode categorised into quinties by rank (reference: east deprived, quintie 5). e Approximates coefficient of variation for Poisson mode with og ink. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 41

72 RESULTS Additiona anayses based on primary outcome Adjusting the primary outcome so that a within-practice contacts on the index day were combined into one overa contact, an increased rate of contacts was sti observed when comparing GPT with UC (Tabe 13), with an RR of 1.10 (95% CI 1.04 to 1.17), and when comparing NT with UC (RR 1.12, 95% CI 1.00 to 1.26). Considering primary outcome contacts on the index day ony, the RRs for both GPT and NT compared with UC were significanty increased (RR 1.51, 95% CI 1.43 to 1.59 and RR 1.72, 95% CI 1.63 to 1.82), respectivey; see Tabe 13). Aso, the RR of index day contacts was increased in NT compared with GPT (RR 1.14, 95% CI 1.09 to 1.20). Further anayses incuded the primary outcome contacts on days 2 28 of the foow-up period (i.e. incuding ony those contacts occurring subsequent to the index day); whist the increased rate of contacts continued to be apparent in the GPT and NT arms vs. UC (RRs 1.46, 95% CI 1.34 to 1.59, and 1.34, 95% CI 1.22 to 1.48, respectivey), there was itte difference in the rate of contacts comparing NT vs. GPT (RR 0.97, 95% CI 0.93 to 1.00). Anaysis of GP face-to-face contacts ony, on the index day ony, demonstrated significant reductions in the rate of contacts for both GPT (RR 0.45, 95% CI 0.37 to 0.55) and NT (RR 0.69, 95% CI 0.57 to 0.84) compared with UC (see Tabe 13). A simiar anaysis using GP face-to-face contacts across the fu 28-day foow-up period yieded contact rates that were again reduced in comparison with UC, but to a esser extent than when considering ony the index day (RR 0.61, 95% CI 0.54 to 0.69 in GPT; 0.80, 95% CI 0.71 to 0.90 in NT). Incuding GP face-to-face and teephone contacts ony, across the fu 28-day foow-up period, the index day ony, and the days subsequent to the index day ony, aso demonstrated increased contact rates for the GPT arm compared with UC (see Tabe 13). Triage resuted in a redistribution of primary care contacts. Athough GPT, compared with UC, was associated with an increased rate of overa GP contacts (face to face and teephone) over the 28 days of 38% (RR 1.38, 95% CI 1.28 to 1.50), the rate of GP face-to-face contacts was reduced by 39% (RR 0.61, 95% CI 0.54 to 0.69). NT was associated with a reduction in the rate of overa GP contacts of 16% (RR 0.84, 95% CI 0.78 to 0.91), incuding a reduction in GP face-to-face contacts of 20% (RR 0.80, 95% CI 0.71 to 0.90), which shoud be viewed in the context of increased numbers of nurse contacts incurred by the NT process (see Tabe 16). Figure 8a presents the redistribution of workoad across the 28 days evident in the three tria arms, highighting the magnitude of the redistribution of workoad from face to face to teephone (in GPT) and from doctor to nurse (in NT). Figure 8b presents the redistribution of workoad on the index day as a hepfu comparison with 28-day workoad redistribution. Safety outcomes There were eight deaths within 7 days of the index consutation request across a three treatment arms (Tabe 14): one in UC (0.1/1000 patients), five in GPT (0.7/1000 patients) and two in NT (0.3/1000 patients). Owing to the sma number of deaths within 7 days during the tria, no forma inferentia anaysis between groups has been performed. Two independent adjudicators deemed that the circumstances of the deaths were not associated with the tria group or procedures (Campbe et a. 64 ). No patient had more than one emergency hospita admission within the 7-day foow-up period. ITT anayses indicated no evidence of increased risk of emergency admission in either of the triage arms compared with UC (see Tabe 14). Durations of emergency admissions were aso simiar in a arms. Furthermore, there was no evidence of an increased risk of at east one A&E admission within the 28-day foow-up period in GPT or NT compared with UC. 42 NIHR Journas Library

73 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 TABLE 13 Sensitivity anayses based on the primary outcome UC (N = 5572) GPT (N = 5171) NT (N = 5468) Primary care and A&E contacts within 28 days of index day, with combination of a within-practice contacts on index day into one contact a,b Tota primary care contacts: c tota, mean (SD) 10,450, 1.88 (1.39) 11,189, 2.16 (1.65) 11,592, 2.12 (1.60) GPT vs. UC NT vs. UC NT vs. GPT Poisson GLLAMM with robust residuas; adjusted d (n = 16,095), RR (95% CI) 1.10 (1.04 to 1.17) 1.12 (1.00 to 1.26) 1.01 (0.87 to 1.18) Custer-eve SD e 0.08 Primary care and A&E contacts on index day ony a,b Tota primary care contacts: c tota, mean (SD) 5490, 0.99 (0.31) 7705, 1.49 (0.57) 9276, 1.70 (0.55) GPT vs. UC NT vs. UC NT vs. GPT Poisson regression mode with ordinary residuas; adjusted d (n = 16,095), RR (95% CI) 1.51 (1.43 to 1.59) 1.72 (1.63 to 1.82) (1.09 to 1.20) Custer-eve SD e 0.05 Primary care and A&E contacts on days 2 28 foowing index day ony a,b Tota primary care contacts: c tota, mean (SD) 5126, 0.92 (1.39) 6015, 1.16 (1.64) 6124, 1.12 (1.60) GPT vs. UC NT vs. UC NT vs. GPT Poisson GLLAMM with robust residuas; adjusted d (n = 16,095), RR (95% CI) 1.46 (1.34 to 1.59) 1.34 (1.22 to 1.48) 0.97 (0.93 to 1.00) Custer-eve SD e 0.18 GP face-to-face within-practice contacts on days 1 28 Tota contacts, mean (SD) 8113, 1.46 (0.85) 4766, 0.92 (0.91) 6496, 1.19 (0.89) GPT vs. UC NT vs. UC NT vs. GPT Poisson regression mode with ordinary residuas; adjusted d (n = 16,095), RR (95% CI) 0.61 (0.54 to 0.69) 0.80 (0.71 to 0.90) 1.30 (1.15 to 1.46) Custer-eve SD e 0.15 GP face-to-face within-practice contacts on index day ony Tota contacts, mean (SD) 5091, 0.91 (0.30) 2195, 0.42 (0.50) 3598, 0.66 (0.48) GPT vs. UC NT vs. UC NT vs. GPT Poisson regression mode with ordinary residuas; adjusted d (n = 16,095), RR (95% CI) 0.45 (0.37 to 0.55) 0.69 (0.57 to 0.84) 1.54 (1.27 to 1.87) Custer-eve SD e 0.24 continued Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 43

74 RESULTS TABLE 13 Sensitivity anayses based on the primary outcome (continued) UC (N = 5572) GPT (N = 5171) NT (N = 5468) GP face-to-face within-practice contacts on days 2 28 ony Tota contacts, mean (SD) 3022, 0.54 (0.82) 2571, 0.50 (0.77) 2898, 0.53 (0.80) GPT vs. UC NT vs. UC NT vs. GPT Poisson regression mode with ordinary residuas; adjusted d (n = 16,095), RR (95% CI) 0.87 (0.77 to 0.99) 0.95 (0.83 to 1.07) 1.09 (0.96 to 1.23) Custer-eve SD e 0.15 GP face-to-face and teephone within-practice contacts on days 1 28 Tota contacts, mean (SD) 8718, 1.56 (1.01) 11, (1.29) 7341, 1.34 (1.08) GPT vs. UC NT vs. UC NT vs. GPT Poisson regression mode with ordinary residuas; adjusted d (n = 16,095), RR (95% CI) 1.38 (1.28 to 1.50) 0.84 (0.78 to 0.91) 0.61 (0.56 to 0.66) Custer-eve SD e 0.10 GP face-to-face and teephone within-practice contacts on index day ony Tota contacts, mean (SD) 5163, 0.93 (0.30) 7076, 1.37 (0.53) 3717, 0.68 (0.49) GPT vs. UC NT vs. UC NT vs. GPT Poisson regression mode with ordinary residuas; adjusted d (n = 16,095), RR (95% CI) 1.49 (1.32 to 1.69) 0.72 (0.63 to 0.82) 0.48 (0.43 to 0.55) Custer-eve SD e 0.15 GP face-to-face and teephone within-practice contacts on days 2 28 ony Tota contacts, mean (SD) 3555, 0.64 (0.97) 4248, 0.82 (1.18) 3624, 0.66 (0.99) GPT vs. UC NT vs. UC NT vs. GPT Poisson GLLAMM with robust residuas; adjusted d (n = 16,095), RR (95% CI) 1.19 (1.13 to 1.24) 1.00 (0.94 to 1.07) 0.84 (0.80; 0.89) Custer-eve SD e 0.14 GLLAMM, Generaized Linear and Latent Mixed Modes. a Patients who died within 28 days of index day excuded. b A modes incude random effect on practice. c Incudes a 20 contact types constituting the primary outcome. d Adjusted modes adjust for practice variabes: ocation (Devon, reference; Bristo; Warwick; Norwich), practice ist size [arge (> 8000 patients), reference; medium ( patients); sma (< 3500 patients)], practice deprivation [non-deprived (at or beow average deprivation for Engand by APHO ratings), reference; deprived (above average for Engand by APHO ratings)] and patient variabes: age (categorised, 0 4; 5 11; 16 24; 25 59, reference; 60 74; 75 years), gender (reference: femae), IMD deprivation based on residentia postcode categorised into quinties by rank (reference: east deprived, quintie 5). e Approximate coefficient of variation for Poisson mode with og ink. 44 NIHR Journas Library

75 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 (a) 3.0 Mean number of contacts Nurse teephone Nurse face to face GP home visit GP teephone GP face to face 0.0 UC GPT Tria arm NT (b) Mean number of contacts Nurse teephone Nurse face to face GP home visit GP teephone GP face to face 0.0 UC GPT Tria arm NT FIGURE 8 Redistribution of within-practice workoad over (a) the 28-day workoad (primary outcome measure) and (b) the index day. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 45

76 RESULTS TABLE 14 Comparison of safety outcomes based on contacts across tria arms Deaths with 7 days of index day (a patients) UC (tota N = 7283; case note review N = 5572) GPT (tota N = 6695; case note review N = 5171) NT (tota N = 7012; case note review N = 5468) Tota deaths (n/1000 patients) 1 (0.1) 5 (0.7) 2 (0.3) Emergency hospita admissions (patients with case note review ony) a Did the patient have at east one emergency hospita admission within 7 days of index day b? n (%) Did the patient have at east one emergency hospita admission, panned in advance by heath professiona within 7 days of index day? n (%) Did the patient have at east one emergency hospita admission, unpanned by heath professiona within 7 days of index day? n (%) 52 (0.93) 59 (1.14) 69 (1.26) 38 (0.68) 38 (0.73) 36 (0.66) 14 (0.25) 21 (0.41) 33 (0.60) At east one emergency hospita admission within 7 days of index day: a c OR (95% CI) GPT vs. UC NT vs. UC NT vs. GPT Logistic regression, adjusted d (n = 16,095) 1.17 (0.75 to 1.85) 1.31 (0.83 to 2.07) 1.12 (0.73 to 1.72) ICC e (95% CI) (0.004 to 0.122) Number of bed-days for patients who had an admission; mean (SD), n 3.8 (6.4), (3.7), (5.7), 68 A&E contacts within 28 days of index day (patients with case note review ony) a Did the patient have at east one A&E attendance within 28 days of index day? n (%) 166 (3.0) 171 (3.3) 156 (2.9) At east one A&E contact within 28 days of index day: a,c OR (95% CI) GPT vs. UC NT vs. UC NT vs. GPT Logistic regression; adjusted d (n = 16,095) 1.18 (0.87 to 1.61) 1.09 (0.80 to 1.49) 0.92 (0.67 to 1.26) ICC (95% CI) (0.007 to 0.063) Number of A&E attendances within 28 days of index day, tota, mean (SD) 187, 0.03 (0.21) 180, 0.03 (0.19) 183, 0.03 (0.22) a Patients who died within 28 days of index day excuded. b A tota of 180 patients (1.1%) had one admission; no patients had more than one admission. c A modes incude random effect on practice. d Adjusted modes adjust for practice variabes: ocation (Devon, reference; Bristo; Warwick; Norwich), practice ist size [arge (> 8000 patients), reference; medium ( patients); sma (< 3500 patients)], practice deprivation [non-deprived (at or beow average deprivation for Engand by APHO ratings), reference; deprived (above average for Engand by APHO ratings)] and patient variabes: age (categorised, 0 4; 5 11; 16 24; 25 59, reference; 60 74; 75 years), gender (reference: femae), IMD deprivation based on residentia postcode categorised into quinties by rank (reference: east deprived, quintie 5). 46 NIHR Journas Library

77 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Patient management on the index day The disposition of a patients on the index day (covering the first two contacts on the index day if the first contact was within practice) is set out in Tabe 15. For patients who had their first contact outside the practice (ony three patients in UC, one of whom had an emergency hospita admission on the index day), no further patient disposition was covered. Within each arm, ony a sma number of patient management pathways covered the overwheming majority (> 90%) of patients. Within UC, ony three management pathways individuay accounted for > 1% of the tota patients with case note review data; in combination, these pathways represented > 90% of patients. Within GPT and NT there were five patient management pathways that individuay accounted for > 1% of patients with case note review data, and in combination represented > 95% of patients within each arm. The proportion of patients who received no within-practice management on the index day was higher in UC (5%; 279/5572) than in the triage arms (approximatey 1% in both triage arms). These patient management pathways on the index day are summarised in Figure 9. InGPTof4796 patients who were treated per protoco, 1017 (21.2%) were definitivey managed (i.e. had no further recorded contact with primary care or A&E within 28 days or hospita admissions within 7 days). In NT, of 4860 patients treated per protoco, 394 (8.1%) were definitivey managed. TABLE 15 Patient management on index day Contact a UC (N = 5572): n (%) No contact (primary care or hospita admission) on index day GPT (N = 5171): n (%) NT (N = 5468): n (%) 279 (5.0) 51 (1.0) 52 (1.0) First contact on index day: consutation request outside practice (incuding emergency hospita admissions) b 3 (0.1) 0 (0) 0 (0) First contact on index day: GP face to face Tota patients with first contact on index day: GP face to face 5034 (90.3) 293 (5.7) 480 (8.8) Second contact on index day No second contact on index day 4874 (87.5) 269 (5.2) 461 (8.4) GP face to face 19 (0.3) 11 (0.2) 2 (< 0.1) GP teephone 14 (0.3) 4 (0.1) 4 (0.1) Nurse face to face 84 (1.5) 7 (0.1) 7 (0.1) Nurse teephone 3 (0.1) 0 (0) 1 (< 0.1) Other practice 0 (0) 0 (0) 0 (0) Outside practice c 40 (0.7) 2 (< 0.1) 5 (0.1) First contact on index day: GP teephone Tota patients with first contact on index day: GP teephone 54 (1.0) 4796 (92.7) 22 (0.4) continued Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 47

78 RESULTS TABLE 15 Patient management on index day (continued) Contact a UC (N = 5572): n (%) Second contact on index day GPT (N = 5171): n (%) NT (N = 5468): n (%) No second contact on index day 40 (0.7) 2357 (45.6; 49.1 d ) 12 (0.2) GP face to face 14 (0.3) 1850 (35.8; 38.6 d ) 8 (0.2) GP teephone 0 (0) 60 (1.2; 1.3 d ) 1 (< 0.1) Nurse face to face 0 (0) 465 (9.0; 9.7 d ) 0 (0) Nurse teephone 0 (0) 1 (< 0.1; < 0.1 d ) 0 (0) Other practice 0 (0) 36 (0.7; 0.8 d ) 1 (< 0.1) Outside practice c 0 (0) 27 (0.5; 0.6 d ) 0 (0) First contact on index day: nurse face to face Tota patients with first contact on day of index consutation request: nurse face to face 197 (3.5) 25 (0.5) 50 (0.9) Second contact on index day No second contact on day of index consutation request 180 (3.2) 23 (0.4) 33 (0.6) GP face to face 15 (0.3) 1 (< 0.1) 13 (0.2) GP teephone 0 (0) 0 (0) 1 (0.02) Nurse face to face 0 (0) 1 (< 0.1) 3 (0.1) Nurse teephone 2 (< 0.1) 0 (0) 0 (0) Other practice 0 (0) 0 (0) 0 (0) Outside practice c 0 (0) 0 (0) 0 (0) First contact on index day: nurse teephone Tota patients with first contact on day of index consutation request: nurse teephone 3 (0.1) 1 (< 0.1) 4860 (88.9) Second contact on index day No second contact on day of index consutation request 1 (< 0.1) 0 (0) 1220 (22.3; 25.1 e ) GP face to face 1 (< 0.1) 0 (0) 3034 (55.5; 62.4 e ) GP teephone 0 (0) 0 (0) 73 (1.3; 1.5 e ) Nurse face to face 0 (0) 0 (0) 467 (8.5; 9.6 e ) Nurse teephone 1 (< 0.1) 0 (0) 35 (0.6; 0.7 e ) Other practice 0 (0) 0 (0) 15 (0.3; 0.3 e ) Outside practice c 0 (0) 1 (< 0.1) 16 (0.3; 0.3 e ) First contact on index day: other practice contact 2(< 0.1) 5 (0.1) 4 (0.1) a Incudes first and second contacts ony on index day. b Patients whose first contact was outside the practice are not considered further. c Incudes primary care out-of-hours contacts, wak-in centre contacts, A&E contacts and unreated/emergency hospita admissions. d Second percentage is based on denominator of a patients who had a GP teephone contact as their first contact on the index day (n = 4796). e Second percentage is based on denominator of a patients who had a nurse teephone contact as their first contact on the index day (n = 4860). 48 NIHR Journas Library

79 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Care on the index day No further contact UC: 5% GPT: 1% NT: 1% Same-day request GP face to face Triage UC: 90% GPT: 6% NT: 9% GPT: 93% NT: 89% Of patients triaged... No further contact GP face to face Nurse face to face GPT: 49% NT: 25% GPT: 39% NT: 62% GPT: 10% NT: 10% Other pathway UC: 5% GPT: 1% NT: 1% Other GPT: 3% NT: 3% FIGURE 9 Summary, by arm, of within-practice patient management on the index day. Resource use Components of the primary outcome Descriptive data regarding the 20 individua contact types that comprise the primary outcome are shown in Tabe 16; aso shown are descriptive data for a GP within-practice contacts, a nurse within-practice contacts, a within-practice contacts (incuding GP, nurse and unspecified), a wak-in centre contacts (numbers are simiar across a arms, indicating that any differences in recording of wak-in centre contacts across the ocations has not resuted in a recording bias across the arms) and a out-of-hours primary care contacts. Did not attend contacts Tota DNAs had simiar means across a treatment arms, and were a sighty higher proportion of the tota number of contacts (incuding attended and non-attended contacts) for the UC arm [2.5% (265/10,447); see Tabe 16] compared with the triage arms. The proportions of DNAs for teephone contacts made by the triaging cinician type in the two triage arms were simiar [1.7% (113/6671) for GP teephone contacts in the GPT arm, and 1.1% (61/5560) for nurse teephone contacts in the NT arm]. Patient sef-reported resource use The proportions of patients who contacted NHS Direct were simiar across the three arms, as were the mean numbers of contacts (see Tabe 16). Duration of consutations within tria Based on Cinician Form data, the mean contact duration in UC was onger than for either of the triage arms [9.5 (SD 5.0) minutes compared with 4.0 (SD 2.8) minutes in GPT and 6.6 (SD 3.8) minutes in NT; Tabe 17, incuding first management/triage contacts ony]. The duration of GP and nurse face-to-face consutations for ESTEEM patients (taken for a sampe of patients on 2 days) indicated that the mean duration of GP face-to-face consutations in the GPT arm was sighty onger than in NT [12.4 (SD 7.1) minutes compared with 11.5 (SD 6.4) minutes; see Tabe 17] and onger than in UC [9.8 (SD 5.1) minutes] when incuding consutations on the index day or subsequent day ony. Nurse face-to-face consutations had a mean duration of 13.9 (SD 8.8) minutes in GPT, 11.0 (SD 6.6) minutes in UC and 11.0 (SD 8.1) minutes in NT, again incuding ony patients being seen on the index day or the foowing day. Estimated patient cinician contact times indicated itte difference in overa contact time comparing GPT with UC, athough a sighty increased amount of the tota cinician time was fufied by nurses in the GPT arm. The overa patient cinician contact time was noticeaby onger in the NT arm, with a much Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 49

80 RESULTS TABLE 16 Descriptive summary of individua contact types comprising the primary outcome and patient-reported use of NHS Direct Contact type UC (N = 5572) GPT (N = 5171) NT (N = 5468) Within-hours surgery contacts: tota contacts, mean (SD) GP face to face a,b 8113, 1.46 (0.85) 4766, 0.92 (0.91) 6496, 1.19 (0.89) GP teephone a,b 605, 0.11 (0.39) 6558, 1.27 (0.77) 845, 0.15 (0.50) GP home visit a,b 48, 0.01 (0.12) 143, 0.03 (0.21) 107, 0.02 (0.19) GP unspecified a,b 2, < 0.01 (0.02) 12, < 0.01 (0.05) 10, < 0.01 (0.04) Nurse face to face a,b 1361, 0.24 (0.71) 1673, 0.32 (0.78) 1928, 0.35 (0.82) Nurse teephone a,b 50, 0.01 (0.11) 77, 0.01 (0.15) 5499, 1.01 (0.49) Nurse home visit a,b 1, < 0.01 (0.01) 0, 0 (0) 4, < 0.01 (0.03) Nurse unspecified a,b 2, < 0.01 (0.02) 6, < 0.01 (0.03) 7, < 0.01 (0.04) Genera unspecified a,b 0, 0 (0) 3, < 0.01 (0.02) 3, < 0.01 (0.02) Wak-in centre contacts: tota contacts, mean (SD) a Wak-in centre: doctor 3, < 0.01 (0.02) 10, < 0.01 (0.05) 5, < 0.01 (0.03) a Wak-in centre: nurse 24, < 0.01 (0.07) 13, < 0.01 (0.05) 21, < 0.01 (0.07) a Wak-in centre: unspecified 5, < 0.01 (0.03) 9, < 0.01 (0.04) 5, < 0.01 (0.03) Primary care out-of-hours contacts: tota contacts, mean (SD) Out-of-hours GP face to face a 61, 0.01 (0.11) 63, 0.01 (0.12) 52, 0.01 (0.10) Out-of-hours GP teephone a 66, 0.01 (0.12) 111, 0.02 (0.18) 94, 0.02 (0.16) Out-of-hours GP home visit a 3, < 0.01 (0.03) 31, 0.01 (0.10) 16, < 0.01 (0.06) Out-of-hours nurse face to face a 3, < 0.01 (0.02) 4, < 0.01 (0.03) 9, < 0.01 (0.04) Out-of-hours nurse teephone a 12, < 0.01 (0.05) 28, 0.01 (0.09) 12, < 0.01 (0.05) Out-of-hours nurse home visit a 1, < 0.01 (0.01) 0, 0 (0) 0, 0 (0) Out-of-hours unspecified a 69, 0.01 (0.14) 33, 0.01 (0.08) 104, 0.02 (0.18) A&E attendances: tota contacts, mean (SD) A&E attendances a 187, 0.03 (0.21) 180, 0.03 (0.19) 183, 0.03 (0.22) Subtotas: tota contacts, mean (SD) Tota within-practice GP contacts a,b 8768, 1.57 (1.02) 11,479, 2.22 (1.33) 7458, 1.36 (1.13) Tota within-practice nurse contacts a,b 1414, 0.25 (0.73) 1756, 0.34 (0.82) 7438, 1.36 (1.00) Tota within-practice contacts a,b 10,182, 1.83 (1.31) 13,238, 2.56 (1.60) 14,899, 2.72 (1.55) Tota wak-in centre contacts a 32, 0.01 (0.08) 32, 0.01 (0.08) 31, 0.01 (0.09) Tota primary care out-of-hours contacts a 215, 0.04 (0.27) 270, 0.05 (0.35) 287, 0.05 (0.33) DNAs: tota DNAs, mean (SD); % n tota contacts (attended and DNA) by type Faied consutations (patient DNA/was not contacted by teephone DNA) a within-hours practice consutations ony; GP face-to-face consutation 110, 0.02 (0.14); , 0.01 (0.12); , 0.01 (0.12); 1.1 GP teephone consutation 82, 0.01 (0.13); , 0.02 (0.15); , 0.01 (0.12); 8.0 Nurse face-to-face consutation 62, 0.01 (0.11); , 0.01 (0.10); , 0.01 (0.10); 2.8 Nurse teephone consutation 9, < 0.01 (0.04); , < 0.01 (0.07); , 0.01 (0.11); 1.1 Tota DNAs (a within-hours practice contact types) 265, 0.05 (0.24); , 0.05 (0.23); , 0.05 (0.23); NIHR Journas Library

81 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 TABLE 16 Descriptive summary of individua contact types comprising the primary outcome and patient-reported use of NHS Direct (continued) Contact type UC (N = 5572) GPT (N = 5171) NT (N = 5468) Patient-reported heath-care contacts c NHS direct contacts within 4 weeks of index day, excuding index day: n (%) (95.7) 3474 (94.5) 3194 (94.1) (3.1) 141 (3.8) 150 (4.4) 2 27 (0.7) 44 (1.2) 28 (0.8) 3 13 (0.3) 11 (0.3) 10 (0.3) 4 3 (0.1) 1 (0.03) 4 (0.1) 5 0 (0) 1 (0.03) 4 (0.1) 6 5 (0.1) 5 (0.1) 3 (0.1) Tota n Mean (SD), n (excuding 6) 0.06 (0.31), (0.35), (0.38), 3390 a Derived from case note review; patients who died within 28 days of index day excuded. b Excudes non-attended contacts or faiure to contact by teephone (DNAs). c Derived from questionnaire. TABLE 17 Duration of consutations during tria UC (N = 7283) GPT (N = 6695) NT (N = 7012) Duration of consutation (minutes) based on Cinician Form a,b data: mean (SD), n First management/triage contacts ony c 9.5 (5.0), (2.8), (3.8), 5510 Duration of face-to-face consutations (minutes) for ESTEEM patients: d mean (SD), n GP face-to-face consutations on day of index consutation request or day after index consutation request Nurse/nurse practitioner face-to-face consutations on day of index consutation request or day after index consutation request 9.8 (5.1), (7.1), (6.4), (6.6), (8.8), (8.1), 57 Estimated composite patient cinician contact time (minutes) on the index day e Overa estimated patient cinician contact time Estimated patient GP contact time Estimated patient nurse contact time a Cinician Form data incuded ony if Cinician Form is dated within 7 days of index consutation request (UC) or dated same date as index consutation request (GPT/NT). b Incudes 98 Cinician Forms that were recorded as patient DNA, but did incude duration data. c A triage contact is defined as a GP teephone contact on the date of the index consutation request in GPT, or a nurse teephone or GP teephone contact on the date of the index consutation request in NT. A first management contact in UC is defined as any contact within 7 days of the index consutation request. d Durations of face-to-face consutations recorded by form competion or eectronicay from practice computer system; practices were requested to record data on two separate days one Monday/Friday and one Tuesday/Wednesday/ Thursday; one practice coected data ony on a Tuesday and Wednesday, two practices coected data on one day ony. e Patients who DNA any within-practice contacts on the index day, or who were first managed outside the practice, were excuded. Patient management pathways that represented < 1% of patients in each arm were excuded (overa 92% of patients in UC were incuded, 97% in GPT and 96% in NT). Average durations for each contact type were derived from the Cinician Forms, audit of face-to-face GP/nurse contacts and standard unit timings (Curtis 38 ) as appropriate. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 51

82 RESULTS greater proportion of the time being covered by nurses, argey due to the nurse teephone triage aspect of patient management in this arm (see Tabe 17). Patient sef-reported heath status (European Quaity of Life-5 Dimensions) Intention-to-treat anayses using Tobit regression modes (Tabe 18) showed no evidence of a significant difference in EQ-5D score across the three tria arms (frequencies for the component questions of EQ-5D are shown in Appendix 21). Patient experience Patients evauated that it was easier to get through to the practice on the phone in the GPT vs. UC arm (mean difference 8.70, 95% CI to 0.89; Tabe 19). There was no evidence to indicate any difference between the UC and GPT arms in terms of ease of accessing prompt care, whereas the NT patients reported that it was significanty more difficut to access prompt care (mean difference 7.02, 95% CI 3.60 to 10.45, vs. UC; mean difference 6.63, 95% CI 3.23 to 10.03, vs. GPT). NT patients aso reported increased difficuty in seeing a GP or nurse compared with the UC arm, and increased difficuty in getting medica hep or advice compared with both the UC and GPT arms. Patients in the NT arm aso found their care to be ess convenient and reported ower overa satisfaction with their care compared with the other two arms; the mean difference in patient satisfaction was 3.94 (95% CI 1.88 to 5.99) compared with UC, and 2.60 (95% CI 0.58 to 4.63) compared with GPT. However, there was no evidence across the arms to indicate any differences in patient sef-assessment of the probem (the reason for the index consutation request) at the time of responding to the questionnaire. Frequencies for the individua responses to the patient experience questions are shown in Appendix 22). Case compexity One or more of the case compexity items was missing for a high proportion of patients and this proportion varied across the arms (see Appendix 23); 25% of the GPT patients did not have fu compexity data avaiabe compared with 18% of patients in UC and 16% in NT (excuding patients with a Cinician Form that indicated non-attendance). In a three arms it was evident that the degree of missingness for the socia, psychoogica and administrative compexity parameters was much greater than that for physica compexity. For those patients who had fu case compexity data avaiabe, there appeared to be some evidence for systematic variation across the arms; the distribution of the overa compexity score varied by arm. The highest mean case compexity score was seen in the UC arm [2.32 (SD 0.88) compared with 1.92 (SD 0.73) in GPT and 2.20 (SD 1.04) in NT]; athough the moda score was 2 in each arm, the distributions around the mode were different in each arm. TABLE 18 Patient-reported heath status (EQ-5D): ITT anaysis UC (N = 7283) GPT (N = 6695) NT (N = 7012) Tota EQ-5D: mean (SD), n (0.271), (0.274), (0.270), 2925 GPT vs. UC GPT vs. UC NT vs. GPT Tobit mode; a,b coefficient (95% CI); adjusted (n = 9404) 0.01 ( 0.02 to 0.04) 0.02 ( 0.01 to 0.06) 0.01 ( 0.02 to 0.05) ICC (95% CI) (0.003 to 0.014) a A modes incude random effect on practice. b Adjusted modes adjust for practice variabes: ocation (Devon, reference; Bristo; Warwick; Norwich), practice ist size [arge (> 8000 patients), reference; medium ( patients); sma (< 3500 patients)], practice deprivation [non-deprived (at or beow average deprivation for Engand by APHO ratings), reference; deprived (above average for Engand by APHO ratings)] and patient variabes: age (categorised, 0 4; 5 11; 16 24; 25 59, reference; 60 74; 75 years), gender (reference: femae), IMD deprivation based on residentia postcode categorised into quinties by rank (reference: east deprived, quintie 5). 52 NIHR Journas Library

83 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 TABLE 19 Patient experience of care: ITT anaysis UC (N = 7283) GPT (N = 6695) NT (N = 7012) GPT vs. UC NT vs. UC NT vs. GPT How easy or difficut was it to get through to the practice on the phone? [Mean difference (95% CI)] Linearised on scae of (ascending difficuty); a,b adjusted (n = 11,719) 8.70 ( to 0.89) 2.21 ( to 5.59) 6.49 ( 1.26 to 14.25) ICC (95% CI) (0.090 to 0.192) How easy or difficut was it to receive prompt care? [Mean difference (95% CI)] Linearised on scae of (ascending difficuty); a,b adjusted (n = 10,945) 0.39 ( 3.01 to 3.80) 7.02 (3.60 to 10.45) 6.63 (3.23 to 10.03) ICC (95% CI) (0.021 to 0.053) How easy or difficut was it to see a GP or nurse if you wanted to? [Mean difference (95% CI)] Linearised on scae of (ascending difficuty); a,b adjusted (n = 10,945) 3.63 ( 0.42 to 7.68) 7.30 (3.23 to 11.37) 3.67 ( 0.37 to 7.71) ICC (95% CI) (0.022 to 0.057) How easy or difficut was it to get medica hep or advice for the probem you presented? [Mean difference (95% CI)] Linearised on scae of (ascending difficuty); a,b adjusted (n = 11,062) 0.28 ( 2.68 to 2.12) 4.82 (2.38 to 7.25) 5.09 (2.69 to 7.50) ICC (95% CI) (0.009 to 0.027) How convenient was the care provided by your GP surgery on that day? [Mean difference (95% CI)] Linearised on scae of (ascending inconvenience); a,b adjusted (n = 11,721) 1.86 ( 0.70 to 4.42) 5.54 (2.96 to 8.13) 3.68 (1.13 to 6.24) ICC (95% CI) (0.011 to 0.032) Thinking about the reason why you contacted the GP surgery or heath centre that day, is the probem now...? [Mean difference (95% CI)] Linearised on scae of (ascending worsening; excuding Don t know ); a,b adjusted (n = 11,630) 2.15 ( 4.41 to 0.10) 1.74 ( 4.04 to 0.55) 0.41 ( 1.86 to 2.67) ICC (95% CI) (0.008 to 0.023) Overa, how satisfied or dissatisfied were you with the care received on that day? [Mean difference (95% CI)] Linearised on scae of (ascending dissatisfaction); a,b adjusted (n = 11,767) 1.33 ( 0.69 to 3.35) 3.94 (1.88 to 5.99) 2.60 (0.58 to 4.63) ICC (95% CI) (0.007 to 0.023) a A modes incude random effect on practice. b Adjusted modes adjust for practice variabes: ocation (Devon, reference; Bristo; Warwick; Norwich), practice ist size [arge (> 8000 patients), reference; medium ( patients); sma (< 3500 patients)], practice deprivation [non-deprived (at or beow average deprivation for Engand by APHO ratings), reference; deprived (above average for Engand by APHO ratings)] and patient variabes: age (categorised, 0 4; 5 11; 16 24; 25 59, reference; 60 74; 75 years), gender (reference: femae), IMD deprivation based on residentia postcode categorised into quinties by rank (reference: east deprived, quintie 5). Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 53

84 RESULTS In the ight of these observations, we fet that this measure of case compexity was not sufficienty robust to be used as a potentia case-mix adjuster. In its pace we eected to adopt patient age as a rough proxy for case compexity, acknowedging and accepting the imitations of that approach. Economic evauation Estimate of the intervention cost (unit cost) for the triage interventions Resources required for triage interventions The GPT and NT interventions invove resource use and cost for (1) training and set-up and (2) the GP or nurse contact time for the triage event. In addition, for the NT intervention there is a requirement for CDSS to support the intervention. Resource-use estimates presented here are the incrementa (additiona) resources required for deivery of triage interventions, in addition to the UC (contro). Training and set-up The training schedue required to estabish and set up the triage interventions consists of two components, first training on organisationa pans and arrangements for triage, regardess of type, and, second, specific skis training for either GP or nurse triage skis (Tabe 20). This tabe aso highights the need for CDSS to support the NT intervention. Tabes 21 and 22 report data coected within tria on the staff time, by staff grade, associated with training events for triage interventions. A active intervention practices in the tria (n = 28) had training on an organisationa approach to triage; this training was provided by Productive Primary Care. Organisationa training sessions were group training events, open to practice staff invoved in management of same-day requests by patients. These sessions were reativey short, with a mean (SD) duration of 84 (27) minutes for GPT, and 107 (31) minutes for NT. TABLE 20 Training and set up schedue for triage interventions Components (format) GPT NT Organisationa training (group) GP triage skis training (group) N/A Nurse triage skis training (group) N/A Nurse triage skis (individua/remote) N/A CDSS requirement (icence/set-up, impementation) N/A N/A, not appicabe. TABLE 21 Organisationa training for triage interventions, staff attending (mean count by grade) and mean duration of training Triage (n) Staff input (number) by staff grade: mean (SD) [range] GP Nurse Nurse practitioner Practice manager Cerica/ administrative Training duration (minutes): mean (SD) [range] GPT (13) 2.5 (1.1) [1 4] 0.08 (N/A) [0 1] 0.08 (N/A) [0 1] 1.2 (0.4) [1 2] 2.7 (1.6) [1 7] 84.2 (27.1) [60 120] NT (15) 0.7 (0.6) [0 2] 1.4 (1.0) [0 3] 0.13 (0.5) [0 2] 1.1 (0.6) [0 2] 2.1 (1.7) [0 7] (30.5) [60 180] N/A, not appicabe. 54 NIHR Journas Library

85 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 TABLE 22 Triage-specific training for triage interventions, staff attending (mean count by grade) and mean duration of training Staff input (number) by staff grade: mean (SD) [range] Triage (n) GP Nurse Nurse practitioner Practice manager Cerica/ administrative Training duration (minutes): mean (SD) [range] GPT (13) 2.5 (1.0) [1 4] 0.15 (0.6) [0 2] 0.08 (N/A) [0 1] 1.1 (0.6) [0 2] 2.1 (1.8) [0 7] 76.2 (27.0) [45 120] NT (15) 0.1 (0.3) [0 1] 2.7 (1.0) [1 5] 0.2 (0.6) [0 2] 0.3 (0.6) [0 2] 0.1 (N/A) [0 1] (110.7) [90 450] N/A, not appicabe. Genera practitioner triage skis training, provided by Productive Primary Care, was deivered in one session with a mean time of 76 (SD 27) minutes. Nurse triage skis training, provided by Pain Heathcare, was deivered using two sessions spread over one fu day, with a combined mean duration of 335 (SD 111) minutes. Nurse triage training aso incuded individua training sessions, deivered remotey as one-to-one training with individua staff (nurses ony) deivering the intervention; training was deivered over 3 hours (180 minutes) per person (1 2-hour session, foowed by 1 1-hour session). The number of nurses per practice attending these individua training sessions ranged from one to seven, with the mean number being 3.0 (SD 1.4). Data coected within the tria indicated that the provider of training for GPT practices (Productive Primary Care) deivered both the organisationa training and the GP triage skis training, with the time for each component being 84 minutes and 76 minutes, respectivey (as above), i.e. a tota training time at a mean of 160 (SD 49, range ) minutes. Further information on data coected to inform the estimate of training costs is presented in Appendix 24. Software (nurse triage) Software is used to support the NT intervention. The CDSS provided by Pain Heathcare is renewed annuay (see cost beow in Tabe 24), with annua icence renewa, and is incusive of 24-hour support. A minimum of one software icence is required per NT practice, on a singe-user basis; if a practice has mutipe users then each user requires a separate icence or software purchase. Impementation (set-up) of the software requires support from Pain Heathcare, with a 2-day impementation process (and associated fee, see Tabe 23). Cinician contact time (genera practitioner and nurse time) for triage contact Data on cinician contact time for the triage contact (index contact) are from within-tria data coection, and have been reported earier (see Tabe 17) by triage intervention. Summary data are presented in Tabe 23. To estimate cinician contact time for triage contacts, in cost anayses, participant-eve data were used when a GP deivered the GPT intervention (n = 5567) and when a nurse deivered the NT intervention (n = 5535); see Appendix 25 for detai. Mean contact time for GP and nurse triage contacts are 4.00 (SD 2.83) minutes and 6.56 (SD 3.83) minutes, respectivey. Appendix 26 reports mean triage contact time by practice. TABLE 23 Cinician contact time for triage intervention, by intervention type a Intervention n Mean (SD), minutes Percentie range, 5 95 observed (minutes) Range (minutes) GPT contact (2.83) NT contact (3.83) a See Appendix 26 for further detai. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 55

86 RESULTS Estimate of costs for deivery of triage interventions Resource use data reported above are combined with unit cost data and market prices to estimate the mean patient-eve cost per triage contact for GPT and NT. Tabe 24 presents the unit cost data used to estimate intervention costs. Unit costs for GP and nurse time incude cost eements associated with staff costs, practice expenses (associated staff costs, office and genera business expenses incuding teephone, overheads and capita costs) and an aowance for non-contact time (ratio of direct contact to non-contact time with patients); see the method set out by Curtis. 38 In the base-case unit cost estimate the cost per minute for GPs incudes cost aowance for quaifications, 38 and excudes costs associated with other direct staff costs (reference to direct staff costs appicabe ony to GP costs). This atter assumption is based on the nature of the teephone triage contact, as a singe teephone consutation contact. These unit costs are subject to other assumptions on cost components in sensitivity anayses. TABLE 24 Unit costs used in estimate of triage intervention costs Resource use unit Unit cost ( ) Source GP time (minutes) 3.40 Per-minute unit cost from Curtis, 38 tabe 10.8b, p. 183 (incudes quaifications, excudes direct care costs) Nurse time (minutes) 0.88 Cacuated from the cost per hour of face-to-face contact time reported in Curtis, 38 tabe 10.6, p. 180 (incudes quaifications) Nurse practitioner time (minutes) 1.52 Cacuated from the cost per hour of cient contact reported in Curtis, 38 tabe 10.7, p. 181 (incudes quaifications) GP management staff, e.g. practice manager, IT manager (minutes) 0.59 Saary based on median FTE band 6 Agenda for Change heath staff 2011, working 37.5 hours week/42 weeks per annum [source: (accessed 10 November 2014)] Cacuation of unit cost per minute based on cost structure reported in Curtis 38 for simiar saary grade worker (see Appendix 24 for detai) Receptionist/cerica worker/ secretary (minutes) 0.41 Saary based on median FTE band 4 Agenda for Change heath staff 2011 working 37.5 hours week/42 weeks per annum (source: Cacuation of unit cost per minute based on cost structure reported in Curtis 38 for simiar saary grade worker (see Appendix 24 for estimates for detai) Training fee: organisationa training for triage interventions (group) Training fee: organisationa training for triage, and/or GP triage skis (group) 1350 GP consutant, fee charged. Productive Primary Care, fat fee for 1-day training 1350 GP Consutant, fee charged. Productive Primary Care, fat fee for 1-day training Nurse triage skis training (group) 1129 Pain Heathcare, fat fee for 1-day training event Nurse triage skis training; individua one-to-one training (remote), per nurse Pain Heathcare, 3 hours per nurse at per hour Software (annua icence) 5709 Pain Heathcare, 2012 price for CDSS. Price is annua cost, incuding 24-hour support (24/7), for singe-user icence Software (project management and impementation fee) FTE, fu-time equivaent Pain Heathcare, 2012 price for project management and impementation of software Price is standard 2-day cost of impementation per practice 56 NIHR Journas Library

87 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Tabe 25 presents the estimated cost associated with the training and CDSS required to set up and support the triage interventions. These costs represent upfront training and CDSS costs, in the first instance, but aso refect an expected ongoing cost to support triage intervention in a practice setting over time (i.e. annua software cost, recurrent training need). In order to estimate a mean patient eve unit cost for triage interventions, estimated training, set-up and software costs are distributed over an estimate of the number of patients per practice expected to request a same-day appointment (with a GP), and therefore refect the expected estimated workoad against triage interventions (where avaiabe). Estimates of triage workoad (per practice per year) are based on data reported indicating a rate of 3.8 same-day requests per 1000 patients (on GP ist) per day; 4 this rate is appied using an average assumed practice size of 7000 patients to estimate the mean patient eve unit costs, resuting in a projected number of same-day requests per annum at approximatey 7000 (estimate 6916 requests). In sensitivity anayses we appy data coected across 1 week in each of the ESTEEM practices, where data indicate a rate of 5.53 (SD = 2.16) same-day appointment requests per 1000 patients on the practice ist; this eads to an estimate of 10,000 same-day requests per year with a practice size of Triage costs associated with training and software are estimated assuming that costs are distributed over a 12-month period (for base-case cost estimates). In sensitivity anayses the cost is estimated when training and software costs are distributed over a 24-month period (expected 2-year workoad) and over a 60-month time frame (athough in this atter scenario there are ikey to be additiona costs associated with the retraining associated with staff turnover). Using the above assumptions, the estimated mean cost for the training component in the GPT intervention is 0.43 per triage contact, and 1.84 per nurse triage contact for training and software cost components. Sensitivity anayses expore the impact of assumptions on number of same-day requests per year per practice (7000 patients, base case), assuming that the workoad for same-day requests per practice is 10,000 requests per year (based on data from ESTEEM), the training component in the GPT intervention costs 0.30 per triage contact, and training and software components cost 1.29 per nurse triage contact. The higher cost on NT refects the costs associated with software requirements, athough training costs for NT are aso twice those estimated in the GPT intervention. These cost estimates invove a number of assumptions; however, this component of cost is a reativey sma part of the tota mean estimated cost for the triage interventions (see Tabe 26), and adjustments to these assumptions make a negigibe difference to the estimated 12-, 24- or 60-month triage intervention costs [see Sensitivity anayses (primary cost anayses)]. Tabe 26 presents estimates of the cost for the triage interventions. GPT is estimated to cost per triage contact, and NT has an estimated mean cost of 7.62 per contact. The difference in cost estimates refects the reativey ow cost per minute for nurse time compared with GP time. The training cost component for GPT is a reativey sma proportion of triage cost (3.2%) compared with NT, for which the training and software costs represent 24% of the estimated cost for the NT intervention. Tabe 27 presents summary resuts for sensitivity anayses. Sensitivity anayses indicate mean triage costs per contact for GPT and NT interventions of and 7.11, respectivey, for which cost estimates are based on cacuation of costs over 24 months and distribution of costs over a 24-month workoad (same-day requests). The corresponding figures when costs are distributed over a 60-month (5-year) workoad are and 6.81, respectivey. Sensitivity anayses indicate that the estimate of the mean NT cost is 8.44 per contact when the practice needs to purchase two icences for the software used. Where an aternative estimate is used on expected number of same-day requests per practice (assuming a practice size of 7000), appying data from ESTEEM indicating 5.53 requests per day per 1000 patients, we estimate unit costs of and 7.07, respectivey, for GP and nurse triage contacts. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 57

88 RESULTS TABLE 25 Mean cost of per-practice training and software required for triage interventions, and estimate of mean cost per same-day appointment request Training component GPT (mean, ) NT (mean, ) Trainer s fees: a Trainer s fee for organisationa training session GP triage skis training Nurse triage skis N/A 1129 Nurse triage skis 1 : 1 training ( per nurse attending) N/A 878 } N/A Practice staff training cost: based on mean count and duration data reported (see Tabes 21 and 22, pus detai on 1 : 1 training) and unit costs (see Tabe 24, detai provided beow) Training subtota Software b N/A 7499 Training and set-up subtota ,912 Estimated training/software cost per triage contact (assuming 7000 requests per year per practice for a same-day appointment 4 ) N/A, not appicabe. a In the ESTEEM tria, the same provider was used for the organisationa training and GP triage skis; this provider had a fat rate fee of 1350 per day. This rate was not reduced if organisationa training was provided aone, as in the NT intervention training sessions. b Incudes software cost of 5709 pus impementation fee at Assumes that the average practice can maintain triage with a icence on a singe PC/singe-user system/basis. For additiona users additiona icences cost TABLE 26 Estimated unit cost for triage interventions per same-day appointment request Cost component for triage per same-day appointment request GPT (mean, ) NT (mean, ) Training and software cost Cinician time for deivery of triage intervention (SD 9.61) 5.78 (SD 3.37) Estimated mean triage cost (per same-day appointment request) [assuming 7000 requests for a same-day appointment per year per practice] TABLE 27 Summary tabe of sensitivity anayses and impact of different assumptions on estimate of unit cost for triage intervention (for same-day request) Scenario for unit cost estimates; sensitivity anayses refect base-case methods with stated difference GPT (mean, ) NT (mean, ) Base case Intervention costs for set-up/training/equipment spread over 24 months Intervention costs for set-up/training/equipment spread over 60 months Where mutipe icences (two icences) needed for NT Within-tria estimation of same-day appointment requests (n = 10,000 p.a. vs pain base case) Staff unit costs without quaifications (GP cost at 2.80 minutes vs minutes; nurse cost at 0.75 minutes vs minutes) Staff unit costs incuding GP cost per minute where direct care costs are incuded in estimation of GP cost ( 3.70 minutes vs minutes) NIHR Journas Library

89 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 In base-case estimates we appy unit costs of 3.40 and 0.88 per minute for GP and nurse time; however, when making different assumptions (see Appendix 27) and appying the pubished unit cost (per minute or hour) excuding cost aowance for quaifications, these being 2.80 and 0.72, respectivey, for GP and nurse time, we estimate unit costs of triage contact at and 6.90 for GP and nurse triage, respectivey. When using staff unit costs where there is an aowance for costs associated with other direct care staff costs, aigned to the GP roe, we appy unit costs of 3.70 and 0.88 (same as base case for nurse) and estimate unit costs of triage at and 7.62 for GP and nurse triage contacts, respectivey. Economic anayses In this section we present the anayses of tria outcome data, in combination with unit costs to estimate the costs associated with the primary outcome, and reated secondary and exporatory cost anayses. Estimate of resource use and cost: primary anayses Tabe 28 presents data on resource use by type of contact and the associated estimate of unit cost by type, for each arm of the tria, and the mean tota 28-day cost by intervention arm. Data are presented for the participants with compete data coection in the primary outcome (n = 16,211). The resource-use (contact) data are consistent with the effectiveness data presented earier, and are presented here for consideration aongside cost estimates. The mean 28-day estimates of cost are simiar for a three interventions, with a sighty above a mean of 75. Tabe 29 presents cost estimates by summary category, with cost estimates for GP contacts (in practice, via teephone and home visits) combined, and the same for nurse contacts. Data presented show that GP costs account for 88% of the mean 28-day cost in UC, and for 69% and 68%, respectivey, in GPT and NT. When the triage cost is combined with the GP cost in the active triage intervention arms, the combined triage and GP costs account for 85% of the mean tota 28-day cost. When nurse contacts are added to triage and GP contacts this subtota of primary care in-practice contacts account for 92 93% of the mean tota 28-day costs, with remaining contacts in the categories for out-of-hours primary care, wak-in centres and A&E, representing a sma proportion of the mean tota 28-day cost (7.5 8%). The mean costs associated with triage consutations are for GPT and 6.83 for NT. The resuts are presented in Tabe 29 for unadjusted cost differences across the three interventions, for mean tota 28-day costs, and aso for regression-based statistica anayses, using a mutieve mode (see Data anaysis), with adjustment for practice-eve custers and other prespecified covariates. Primary cost anayses, using these base-case estimates (regression estimates), show no statisticay significant differences in 28-day mean tota cost by intervention arm. Estimate of resource use and cost: secondary anayses Secondary anayses, using base-case assumptions on a per-protoco participant basis (see Gossary), shows the 28-day mean costs for GPT and NT sighty reduced compared with ITT anayses, but with no difference reported across a comparisons of cost data in base-case adjusted anayses (see Tabe 30). Sensitivity anayses (primary cost anayses) Tabe 30 presents sensitivity anayses, against the primary anayses on tota 28-day cost estimates, using different assumptions on anayses. Sensitivity anaysis presents resuts using base-case assumptions but with imputation methods used to repace missing data (imputation methods are consistent with those used in mutipe imputation anayses for the primary outcome; see Tabe 12). Here we see no difference in cost estimates or adjusted differences across the three tria arms. Tabe 30 reports sensitivity anayses. Sensitivity anaysis considers use of different regression methods, using a GLM regression approach with two different combinations of famiy and ink (using Gaussian and gamma distributions; the former being equivaent to assuming normay distributed data). These differing specifications of GLM regression modes show no difference in resuts, and indicate that it is appropriate to assume normay distributed tota cost data; therefore, base-case resuts are from a mutieve mode, with random effects, assuming normay distributed data and aowing for custer data. Consistent with the base-case anayses, GLM modes show no difference in 28-day tota cost across comparisons by intervention type. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 59

90 RESULTS TABLE 28 Description of primary cost data (summary view) and primary anayses with comparison by intervention Number of contacts over 28 days: mean (SD) [95th percentie range] Costs for care ( ) (primary outcome contacts) over 28 days: mean (SD) [95th percentie range] Contacts by type UC (n = 5572) GPT (n = 5171) NT (n = 5468) UC (n = 5572) GPT (n = 5171) NT (n = 5468) Triage (GP/nurse) 0.00 (0.00) [0 to 0] 0.93 (0.26) [0 to 1] 0.89 (0.31) [0 to 1] 0.00 (0.00) [0 to 0] (3.64) [0 to 14.03] 6.83 (2.40) [0 to 7.62] GP in surgery/practice 1.46 (0.85) [1 to 3] 0.92 (0.91) [0 to 3] 1.19 (0.89) [0 to 3] (36.62) [43 to 129] (39.15) [0 to 129] (38.41) [0 to 129] GP teephone (excudes triage) 0.11 (0.39) [0 to 1] 0.34 (0.72) [0 to 2] 0.15 (0.50) [0 to 1] 2.82 (10.10) [0 to 26] 8.86 (18.59) [0 to 52] 3.91 (12.94) [0 to 26] GP home visit 0.01 (0.12) [0 to 0] 0.03 (0.21) [0 to 0] 0.02 (0.19) [0 to 0] 0.95 (12.98) [0 to 0] 3.04 (22.94) [0 to 0] 2.15 (20.77) [0 to 0] Nurse in surgery/practice 0.24 (0.71) [0 to 1] 0.32 (0.78) [0 to 2] 0.35 (0.83) [0 to 2] 3.34 (9.67) [0 to 13.64] 4.43 (10.70) [0 to 27.28] 4.83 (11.26) [0 to 27.28] Nurse teephone (excudes triage) 0.01 (0.11) [0 to 0] 0.01 (0.15) [0 to 0] 0.12 (0.38) [0 to 1] 0.05 (0.56) [0 to 0] 0.08 (0.82) [0 to 0] 0.62 (1.99) [0 to 5.28] Nurse home visit 0.00 (0.01) [0 to 0] 0.00 (0) [0 to 0] 0.00 (0.03) [0 to 0] 0.00 (0.29) [0 to 0] 0.00 (0) [0 to 0] 0.02 (0.59) [0 to 0] Out of hours (tota) 0.04 (0.27) [0 to 0] 0.05 (0.35) [0 to 0] 0.05 (0.33) [0 to 0] 1.63 (12.40) [0 to 0] 1.88 (13.66) [0 to 0] 2.16 (14.70) [0 to 0] Wak-in centre 0.01 (0.08) [0 to 0] 0.01 (0.08) [0 to 0] 0.01 (0.86) [0 to 0] 0.24 (3.10) [0 to 0] 0.25 (3.32) [0 to 0] 0.23 (3.54) [0 to 0] A&E 0.03 (0.21) [0 to 0] 0.03 (0.19) [0 to 0] 0.03 (0.22) [0 to 0] 3.76 (23.95) [0 to 0] 3.90 (21.57) [0 to 0] 3.75 (24.28) [0 to 0] Tota 1.91 (1.43) [1 to 5] 2.65 (1.74) [1 to 6] 2.82 (1.68) [1 to 6] (57.19) [43 to 172] (65.45) [14.03 to ] (63.09) [7.62 to 184.9] 60 NIHR Journas Library

91 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 TABLE 29 Description of primary cost data (summary view) primary cost anayses with comparison by intervention type Number of contacts over 28 days: mean (SD) [95th percentie range] Costs for care ( ), (primary outcome contacts) over 28 days: mean (SD) [95th percentie range] Contact type UC (n = 5572) GPT (n = 5171) NT (n = 5468) UC (n = 5572) GPT (n = 5171) NT (n = 5468) Triage (GP/nurse) 0 (0) [0 to 0] 0.93 (0.26) [0 to 1] 0.89 (0.31) [0 to 1] 0 (0) [0 to 0] (3.64) [0 to 14.03] 6.83 (2.40) [0 to 7.62] GP a (excudes triage) 1.57 (1.02) [1 to 4] 1.29 (1.33) [0 to 4] 1.36 (1.13) [0 to 3] (43.02) [43 to 138] (53.75) [0 to 155] (48.79) [0 to 138] Nurse a 0.25 (0.73) [0 to 1] 0.34 (0.82) [0 to 2] 0.47 (0.95) [0 to 2] 3.39 (9.73) [0 to 13.64] 4.51 (10.84) [0 to 27.28] 5.46 (11.67) [0 to 27.28] Subtota: primary care (items above) 1.83 (1.31) [1 to 4] 2.56 (1.60) [1 to 6] 2.72 (1.54) [1 to 6] (44.97) [43 to 155] (55.02) [14.03 to 169] (50.33) [7.62 to ] Out of hours (tota) 0.04 (0.27) [0 to 0] 0.05 (0.35) [0 to 0] 0.05 (0.33) [0 to 0] 1.63 (12.40) [0 to 0] 1.88 (13.66) [0 to 0] 2.16 (14.70) [0 to 0] Wak-in centre 0.01 (0.08) [0 to 0] 0.01 (0.08) [0 to 0] 0.01 (0.09) [0 to 0] 0.24 (3.10) [0 to 0] 0.25 (3.31) [0 to 0] 0.23 (3.54) [0 to 0] A&E 0.03 (0.21) [0 to 0] 0.03 (0.19) [0 to 0] 0.03 (0.22) [0 to 6]/ [0 to 0] 3.76 (23.95) [0 to 0] 3.90 (21.57) [0 to 0] 3.75 (24.28) [0 to 0] Tota (28-day care/cost) 1.91 (1.43) [1 to 5] 2.65 (1.74) [1 to 6] 2.82 (1.68) [0 to 19]/ [1 to 6] (57.19) [43 to 172] (65.45) [14.03 to ] (63.09) [7.62 to 184.9] Comparison by intervention type GPT vs. UC (n = 10,743) NT vs. UC (n = 11,040) GPT vs. NT (n = 10,639) Comparison of tota 28-day cost by type, no adjustment for covariates 0.20 (2.75) [5.59 to 5.18] 0.27 (2.33) [ 4.29 to 4.83] 0.47 (3.00) [ 6.34 to 5.41] Primary anayses, tota 28-day cost; base-case estimates with adjustment for covariates a c 1.76 (2.52) [ 6.70 to 3.18] 0.19 (2.53) [ 5.15 to 4.78] 1.57 (2.51) [ 6.49 to 3.36] a Regression covariates: age (by category), sex, site, practice size, deprivation (IMD ) by quintie, practice deprivation, custer by practice ID. b Regression methods for base-case anayses use mutieve random effects mode (xtmixed), using Stata 12.0 (StataCorp LP, Coege Station, TX, USA) (see sensitivity anayses for GLM approach). c ICC = Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 61

92 RESULTS TABLE 30 Secondary anayses and sensitivity anayses against primary cost anayses, tota 28-day cost Costs for care ( ), (primary outcome contacts) over 28 days: mean (SD) Comparison of mean costs ( ), by type of care (for same-day request): mean (standard error a ) [95% CI] [regression methods b ] UC GPT NT GPT vs. UC (n = 10,743) NT vs. UC (n = 11,040) GPT vs. NT (n = 10,639) Base case: ITT, adjustment for covariates; xtmixed (mutieve mode) (57.19) (n = 5574) (65.45) (n = 5177) (63.09) (n = 5468) 1.76 (2.52) [ 6.70 to 3.18] 0.19 (2.53) [ 5.15 to 4.78] 1.57 (2.51) [ 6.49 to 3.36] Secondary anayses: per protoco adjustment for covariates; as per base case (xtmixed) (57.19) (n = 5574) (65.22) (n = 4796) (63.44) (n = 4860) 2.08 (2.62) [ 7.22 to 3.06] 0.02 (2.64) [ 5.19 to 5.15] 2.06 (2.64) [ 7.23 to 3.11] Sensitivity anayses As base case, with imputation of missing data, using predictive mean matching, with covariate adjustment, as used in main anayses for effectiveness data (56.65) (n = 7281) (65.04) (n = 6689) (63.35) (n = 7012) 1.36 (2.57) [ 6.40 to 3.68] 0.22 (2.60) [ 4.88 to 5.33] 1.58 (2.57) [ 3.47 to 6.63] Using GLM regression mode; with adjustment for covariates; regression mode = GLM, Gaussian/identity (famiy, ink) As per base case 2.03 (2.75) [ 7.41 to 3.36] 0.79 (2.22) [ 5.14 to 3.56] 1.23 (2.93) [ 6.97 to 4.50] Using GLM regression mode; with adjustment for covariates; regression mode = GLM, gamma/identity (famiy, ink) As per base case 2.14 (2.83) [ 7.69 to 3.41] 1.13 (2.12) [ 5.27 to 3.02] 1.00 (3.01) [ 7.00 to 4.99] As per base case except for change to unit costs for out-of-hours contacts (54.71) (64.24) (60.89) 1.98 (2.61) [ 7.09 to 3.14] 0.20 (2.63) [ 5.35 to 4.95] 1.78 (2.63) [ 6.92 to 3.37] Low unit cost scenario for GP and nurse time (per-minute costs) from PSSRU (Curtis 38 ), excuding costs for quaifications and excuding costs for direct care (48.52) (54.08) (53.23) (2.00) [ 4.27 to 3.59] 1.21 (2.01) [ 2.74 to 5.16] 1.55 (2.00) [ 5.47 to 2.37] Assuming GP consutation foowing same-day request has shorter duration than that estimated for a broad category of GP consutations: assuming unit cost of (vs. 43 base case) for the same-day GP consutation in UC arm ony, based on mean reported contact time of 9.5 minutes (vs minutes in base case) (54.04) (65.45) (63.09) 5.43 (2.50) c [0.52 to 10.33] 7.00 (2.51) c [2.07 to 11.92] 1.57 (2.51) [ 6.49 to 3.36] a Robust standard error in the case of GLM adjusted covariate. b Regression covariates: age (by category), sex, site, practice size, deprivation (IMD ) by quintie, practice deprivation, custer by practice ID. c Statisticay significant (95% CI). 62 NIHR Journas Library

93 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Sensitivity anaysis (see Tabe 30) addresses uncertainty in choice of unit cost data for out-of-hours primary care contacts: in the base case, one standard unit cost is appied for a out-of-hours primary care contacts compared with sensitivity anaysis, for which a different cost structure is used for unit cost data on out-of-hours primary care contacts (see Appendix 28). Sensitivity anaysis shows no difference in resuts when aternative data are used for out-of-hours unit costs. Sensitivity anaysis presents resuts for which the unit cost data reated to GP time cost, and nurse time cost, are based on the ower estimate of cost presented by Curtis. 38 In this scenario, anayses are based on a unit cost of 2.80 per minute for GP time (vs in base case), with no aowance for costs associated with GP quaifications or for costs associated with direct costs for reated staff. In this anaysis, unit cost data for nurse time, per minute, is 0.75 (vs in base case), with no aowance for costs associated with nurse quaification. The majority of the mean 28-day tota costs are reated to GP and nurse contacts, and, in this ow-cost sensitivity anaysis, the mean 28-day cost is ower for each of the intervention types, at 59.47, and for UC, GPT and NT, respectivey, but we see no difference in costs in comparisons across intervention type. In the tria resuts (see Tabe 17) we see a mean contact time for the UC index contact (contact resuting from same-day request), which is a ower duration of contact than that seen in the standard costing used here for a GP consutation; in the unit cost estimate appied in base-case cost anayses an aowance is made for 11.7 minutes per GP contact (in surgery). This expected consutation duration is based on anayses of data coected in a UK Genera Practice Workoad Survey (Leeds Information Centre, 2007), which invoved a representative sampe of 329 practices across the UK, with staff competing diary sheets over a 1-week period (as cited by Curtis 38 ). The estimates of contact duration, and subsequent cost, are based on an aocation of GP time across a recorded number of consutations, and not on a duration recorded per contact. The data coected in ESTEEM, in the UC arm, suggests that GP consutations resuting from requests for appointments on the same-day have a mean (SD) duration of 9.5 (5.0) minutes (see Tabe 17), which is shorter than the estimated 11.7-minute duration reported by Curtis 38 in reation to the broader category of GP consutations (in practice), of which same-day requests are one component. Data from ESTEEM, athough based on over 5600 observations, are from 14 GP practices, and are coected in a tria setting, and are estimates of mean contact time, therefore some caution is needed on the generaisabiity of these data, and on any direct comparison with data estimated using a top-down approach. Athough we succeeded in recruiting arge numbers of practices and patients from diverse geographica ocations, enhancing the generaisabiity of our resuts, our findings may be ess appicabe to practices serving popuations with greater ethnic diversity or those ocated in inner-city areas with very high eves of deprivation. However, if this difference were to be supported by future research, a difference of 2.2 minutes in consutation duration woud refect a reduction in cost for a GP consutation (in practice) from 43 to for a consutation resuting from a same-day request. Therefore, in sensitivity anaysis, we use a unit cost of (9.5 minutes 3.70/minute) for the UC index contact (GP consutation). In this scenario we see that the 28-day cost for UC is ess than those estimated for both GPT and NT intervention arms. This difference in 28-day cost is statisticay significant, with regression anayses predicting a mean cost difference of 5.43 (ower) for UC compared with GPT, and 7.00 (ower) for UC compared with NT. Exporatory cost anayses: same-day contacts and cost Tabe 31 presents cost estimates for contacts that are reported on the same-day as the same-day request, comprising the index contact and subsequent contacts that constitute the POM, on the same day. The tota number of recorded contacts over 28 days, aggregating over a participants (n = 16,211 in compete case anayses) is 39,736, and 56% of these (n = 22,471) are recorded as contacts on the same day as the initia request (see Appendix 29). The mean tota number of contacts on the same day are 0.99, 1.49 and Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 63

94 RESULTS TABLE 31 Same-day contacts: description of data, base case (detaied view) and exporatory economic anayses on the same day as appointment request Number of contacts on same-day : mean (SD) Costs for care ( ) (primary outcome contacts) on same day : mean (SD) Contacts by type UC (n = 5572) GPT (n = 5171) NT (n = 5468) UC (n = 5572) GPT (n = 5171) NT (n = 5468) Triage (GP/nurse) N/A 0.93 (0.26) 0.89 (0.31) N/A (3.64) 6.83 (2.40) GP in surgery/practice 0.91 (0.30) 0.43 (0.50) 0.66 (0.48) (12.75) (21.66) (20.71) GP teephone 0.13 (0.11) 0.02 (0.13) 0.18 (0.13) 0.34 (2.98) 0.43 (3.38) 0.46 (3.47) GP home visit 0.00 (0.02) 0.01 (0.09) 0.00 (0.06) 0.04 (2.08) 0.83 (9.52) 0.36 (6.30) Nurse in surgery/practice 0.05 (0.22) 0.10 (0.31) 0.11 (0.31) 0.69 (3.01) 1.40 (3.01) 1.46 (4.28) Nurse teephone 0.00 (0.04) 0.00 (0.02) 0.01 (0.09) 0.01 (0.23) 0.00 (0.13) 0.04 (0.46) Nurse home visit 0.00 (0.00) 0.00 (0.00) 0.00 (0.01) 0.00 (0.00) 0.00 (0.00) 0.00 (0.30) Out of hours (tota) 0.00 (0.04) 0.00 (0.06) 0.00 (0.07) 0.04 (2.01) 0.12 (2.94) 0.11 (3.09) Wak-in centre 0.00 (0.01) 0.00 (0.02) 0.00 (0.02) 0.01 (0.55) 0.02 (0.81) 0.01 (0.78) A&E 0.00 (0.07) 0.01 (0.09) 0.01 (0.08) 0.54 (7.78) 0.82 (9.57) 0.66 (8.54) Tota 0.99 (0.31) 1.49 (0.57) 1.70 (0.55) (14.49) (24.50) (21.93) N/A, not appicabe. 64 NIHR Journas Library

95 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO , for UC, GPT and NT, respectivey. Estimates of mean cost for these same-day contacts are 40.96, and 38.25, for UC, GPT and NT, respectivey. Comparison of these cost data indicates a statisticay significant difference in cost when comparing GPT and UC, and NT compared with UC, but no difference between GPT and NT (athough here the reported difference in cost of 3.17 is cose to being statisticay significant, with a 95% CI of 6.31 to 0.03). However, cost estimates are presented on same-day contacts for competeness, and it is difficut to interpret the same-day costs in the context of an intervention (triage) that is expected to have an immediate impact on number of contacts, with the effect of the intervention expected to be seen over the onger term (in this instance 28 days). As discussed above (primary cost anayses) the estimated duration (time aowance) for a GP consutation, in the practice, is 11.7 minutes, using the data reported aongside the unit cost estimated by the PSSRU (Curtis 38 ); a unit cost of 43 is used in base-case anayses (based on 11.7-minute duration). The data in Tabe 32 with an adjusted cost for a same-day GP consutation (at 35.15), in the UC intervention arm ony, woud suggest a UC same-day cost of compared with the estimated using base-case unit costs, a mean difference (reduction) of 7.17 in the estimated mean same-day costs, with the other triage intervention estimates remaining unchanged. In this scenario (sensitivity anaysis; see Tabe 32) we see a difference in costs, same-day cost estimates, across the three intervention arms, with UC having a esser estimate of same-day costs, with this being statisticay significant in comparisons of UC with NT (UC 4.57 ess than NT). However, as discussed above, in sensitivity anayses, it is important to be cautious when comparing data on mean duration of a GP contact from prior workoad anayses 38 and the participant-eve mean contact time reported in ESTEEM. TABLE 32 Same-day contacts: exporatory anaysis and costs for primary outcome contacts on the same day as appointment request Costs for care ( ), (primary outcome contacts) on same day: mean (SD) Comparison of mean costs ( ), by type of care: mean (robust se) [95% CI] UC (n = 5572) GPT (n = 5171) NT (n = 5468) GPT vs. UC (n = 10,743) NT vs. UC (n = 11,040) GPT vs. NT (n = 10,639) Same-day cost; no adjustment for covariates Same-day cost; tota 28-day cost; base-case estimates [with adjustment for covariates a ] b,c (14.49) (24.50) (21.93) 6.05 (1.73) 2.70 (1.72) [ 9.43 to 2.67] [ 6.08 to 0.67] 5.75 (1.61) 2.58 (1.61) [ 8.91 to 2.59] [ 5.73 to 0.58] 3.3 (2.23) [ 7.71 to 1.02] 3.17 (1.61) [ 6.31 to 0.03] Sensitivity anayses: Assuming GP consutation in UC is shorter (9.5 minutes) than estimated in the base case (11.7 minutes), with unit cost at 35.15, compared with base case of (12.79) (24.50) (21.93) 1.38 (1.59) [ 1.73 to 4.49] 4.57 (1.59) [1.46 to 7.68] 3.17 (1.61) [ 6.31 to 0.03] a Regression covariates: age (by category), sex, site, practice size, deprivation (IMD ) by quintie, practice deprivation, custer by practice. b Regression methods incude anayses using GLM modes (see sensitivity anaysis). GLM mode with/without assumption of normay distributed data are simiar and simiar to those in base-case mutieve mode. c ICC (base-case mutieve mode) = Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 65

96 RESULTS Exporatory anayses: contact duration for genera practitioner contact (in practice) foowing triage intervention Tabe 33 presents data from a sampe of participants from whom data on duration of GP contact was coected foowing a triage intervention. These data were coected as part of a sampe of data gathered in each of the participating GP practices, over approximatey 2 days in each practice. The research question of interest was whether there is a difference in GP contact duration (in surgery), foowing a triage contact, compared with the estimated duration of 11.7 minutes, as used in the base-case cost data. 38 Sampe data, coected over 2 days in each of the GP practices, indicate that consutation duration for GP contacts (in practice), on the same day as either GP- or nurse-ed triage contacts, and are not significanty different from the estimated duration of 11.7 minutes. Data on these contacts (foowing triage), presented in Tabe 33, are from a reativey sma sampe size (n = 244; n = 415), therefore data are iustrative and are not abe to support any concusions. Estimated tota contact time for practice staff over 28 days Based on the mean time estimated for the deivery of the triage interventions, and the assumptions over duration of primary care contacts (see Tabe 33), it is estimated, based on mean number of contacts (reported in Tabe 12), that tota contact time for primary care (practice deivered) contacts, over 28 days, is minutes, minutes and 27.5 minutes, for UC, GPT and NT, respectivey. GP contact time over 28 days is minutes, 17.6 minutes and minutes, for UC, GPT and NT, respectivey. If in UC contacts (ony), the same-day request contact (GP in practice) is assumed to have a duration of 9.5 minutes (vs. base-case assumption of 11.7 minutes), the estimate for tota contact time (minutes) for practice deivered contacts reduces to minutes (vs minutes) and the estimate for the GP contact time, over 28 days, reduces to minutes (vs minutes). Economic anayses: cost consequence anayses The cost estimates indicate that there are no differences between interventions by type in the mean 28-day cost estimates (base case). These findings are robust to sensitivity anayses presented, with the exception of the assumption on the unit cost for same-day GP consuts. Tabe 34 presents cost estimates aongside the other potentiay reevant outcomes reported in this tria, in the form of a cost consequence anaysis, in which disaggregated cost data are presented aongside outcome data. Athough triage interventions incude a higher number of contacts, incuding the index contact, the 28-day costs, and those costs more directy faing on practice-eve budgets, are the same across a three interventions. There is no difference in safety outcomes, DNAs or sef-report heath status at 28-day foow-up. There is evidence of differences in sef-reported patient satisfaction, with NT indicated as not performing as we as UC or GPT in sef-reported patient satisfaction outcomes. TABLE 33 Duration of face-to-face consutations foowing triage for ESTEEM patients Duration of face-to-face consutations for ESTEEM patients UC GPT NT First management (index) GP consutation in practice: a mean (SD), n 9.5 (5.00), 5693 N/A N/A Duration of GP consutations after triage contact on day of after index consutation request: b mean (SD), n N/A 12.4 (7.12), (6.43), 415 Estimate of mean tota contact time (minutes), over 28 days, c for: Practice-deivered contacts GP contacts N/A, not appicabe. a Data coected via Cinician Form within tria, a contacts/a practices. b Data coected as part of sampe data coection, over 2 days in each practice. c Based on data for mean contacts presented in Tabe 12, and assumptions on duration of contacts reported in Tabe NIHR Journas Library

97 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 TABLE 34 Cost consequences matrix, UC costs and consequences compared with triage interventions Costs and consequences UC GPT NT Summary comparison Number of contacts over 28 days (primary outcome) 1.91 (1.43) 2.66 (1.75) 2.82 (1.68) UC fewer contacts (see Tabe 12) Practice-eve costs ( ) (triage, GP, nurse), over 28 days (44.97) (55.02) (50.33) NS Tota 28-day costs ( ) (primary outcome) (57.19) (65.45) (63.09) NS Safety outcomes: Emergency unpanned hospita admission, %(n) with at east one event, within 7 days of index contact 0.25% (14) 0.41% (21) 0.60% (33) Few events; NS (see Tabe 14) Safety outcomes: A&E attendance, % with at east one attendance within 28 days of index contact 3% (166) 3.3% (171) 2.9% (156) Few events; NS (see Tabe 14) EQ-5D heath-state vaue (at foow-up) 0.77 (0.27) 0.77 (0.27) 0.78 (0.27) NS (see Tabe 19) n = 3281 n = 3246 n = 2925 Non-attended contacts (DNAs), % of tota contacts (n) 2.5 (265) 1.8 (241) 1.8 (267) Patient experience outcomes: See Tabe 19 Reporting difficuty (fairy or very difficut) in receiving prompt care, % 5.1 (n = 3816) 5.8 (n = 3727) 10.2 (n = 3461) NT reporting greater eve of difficuty vs. UC and vs. GPT Reporting difficuty (fairy or very difficut) in seeing a GP or nurse, % 9.5% (n = 3883) 11.7 (n = 3620) 15 (n = 3501) NT reporting greater eve of difficuty vs. UC and vs. GPT Reporting difficuty (fairy or very difficut) getting medica hep or advice for the probem they presented, % 4.6 (n = 3819) 4.7 (n = 3798) 7.4 (n = 3502) NT reporting greater eve of difficuty vs. UC and vs. GPT Reporting care provided by GP surgery on that day was not very or not at a convenient, % 4.7 (n = 4072) 7 (n = 4022) 9.5 (n = 3689) NT reporting greater eve of difficuty vs. UC and vs. GPT Reporting fairy or very dissatisfied with care received on that day, % 3.2 (n = 4093) 4.4 (n = 4034) 5.5 (n = 3704) NT reporting greater eve of difficuty vs. UC and vs. GPT NS, not significant. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 67

98 RESULTS Process evauation The participating practices were diverse in size, ocation, staffing and their experience of triage. Tabe 35 provides a matrix for the process evauation anaysis. Some of the data in this tabe are based on the subjective accounts of the sma number of participants in each practice and shoud be viewed accordingy. The category communication is based on whether staff participants perceived that they had been incuded in decision-making about entering the tria and other aspects of practice organisation that affected them. This aso incuded whether they reported the presence or absence of effective forma and informa structures for communication, such as incusive practice meetings, communa coffee breaks or other occasions in which practice issues coud be discussed freey. Poor cinica staff stabiity indicates recent oss of core medica staff members who had not been repaced and a greater than norma reiance on ocum doctors. Previous appointment system summarises staff members accounts of the appointment system before the tria. Overview of triage experience is based on the summary of participant responses for each practice produced as part of the framework anaysis by the process evauation researchers. Structure of the resuts The resuts are based on participants accounts of their perceptions and experiences, and no attempt was made to vaidate what peope said. This was because the aims of the process evauation were to describe peope s experiences and views. Other sections of this report describe quantitative data about the impementation of triage. The resuts section is structured as foows: how the tria was experienced how teephone triage interventions were impemented how triage was experienced acceptabiity of triage the extent to which the teephone triage interventions are seen to work or not work. How the tria was experienced One of the chaenges of the process evauation was disentanging experiences and opinions reating to taking part in the tria from those reating to the intervention itsef. The former wi infuence the atter and vice versa in an iterative cyce. This became increasingy apparent during the data anaysis. For this reason, experience of the tria wi be discussed in some detai. The key themes expored are motivation to participate in the tria and communicative practices in reation to participation, attitudes towards the arm into which practices were randomised, impementation of the study protoco, and perceptions of tria-reated workoad. Motivation to take part in the tria Motivations for entering the tria, and whether these were consensua and effectivey communicated within practices, affected how the tria intervention was experienced by practice staff. There was a range of motivations reported for entering the tria. Research-reated motives cited by informants incuded the fact that theirs was a research practice or a practice that vaued research, wanting to know the answer to the research question, and the fact that this was seen to be a good tria. Chaenges in current practice were aso cataysts; practices were aware when their appointments system was not working we; they were not coping with increased demand, and increasing workoads and triage was seen as a way of addressing this. I m not sure reay, just a sort of sense that it woud be something that woud invove everybody and there s been a ot of tak about, you know, some peope finding it difficut to offer enough appointments and you know, that sort of workoad issues and the staff having difficuty with the phones and it just seemed ike it was an opportunity to try something out, which sort of shows wiing to make some changes if changes proved you know, proved to work I suppose. GP 05, practice 4 (NT) 68 NIHR Journas Library

99 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 TABLE 35 Overview of the process evauation practices Practice Triage type Size of practice and ocation Communication Persona ists Cinica staff stabiity Previous appointment system effectiveness Triage practised before the tria Perceived appointment reduction from triage (conversion) Overview of reported triage experience Practice 1 Nurse Large practice, suburban Very poor Yes Poor Poor No No Very negative Practice 2 Nurse Large practice, suburban Good No Good Good Yes No Quite positive Practice 3 Nurse Sma practice, rura Good No Good Good Yes Yes Positive Practice 4 Nurse Large practice, suburban Good Yes Good Reasonabe Yes Yes Positive Practice 5 GP Sma practice, rura Mixed Yes Good Poor No Mixed; variabe Mixed Practice 6 GP Large practice, urban Poor No Good Good Yes No Mixed Practice 7 GP Large practice, suburban Good Yes Good Reasonabe Yes Divided opinions Mainy positive Practice 8 GP Sma practice, urban Very poor Yes Poor Good Yes Yes Positive Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 69

100 RESULTS Introducing triage had often been mooted in practices but abandoned because of ack of consensus on how to estabish and organise it, and the daunting enormity of the task. The ESTEEM tria was seen as a good way of trying out triage without interna disagreements on how to do it, and with a cear and immutabe mode and support from outside the practice. Part of the training for the ESTEEM tria was a presentation by the organisation Productive Primary Care, taking pace after randomisation, and offering a cear mode of efficacy of triage, which some practices found inspirationa, athough this presentation created some probems, which wi be discussed beow. Three of the practices were excited by the fact that the tria was something that the whoe practice coud be engaged in, and its cohesive effect for practice staff was highy vaued. However, staff in four of the practices reported ack of consensus about taking part, with some staff feeing that they had not been consuted or that their reservations had been gossed over. In two of the practices, both in which there was no consensus, the motivation to take part was reported to be financia. Often it was a combination of severa of these factors, as we as the persuasive powers of one or two highy motivated individuas which drove the decision to take part. Haf of the practices reported that the decision had been made with fu consutation and consensus from a of the staff. The range of pathways to entering the tria was echoed in a variety of communication styes about doing so, from top-down authoritarian to democratic and consutative. In practice 1, a NT practice, as an exampe of the former, one of the partners was reported to have been motivated by the remuneration attached to the tria and himsef referred to the status and resources associated with being a research practice. Doctors here had been discussing the possibiity of triage for some time without reaching agreement, and fet that this coud be a way of introducing it without dissent because the protoco coud not be argued with or changed. They discussed it among themseves but not with other practice staff, whose opinion was not a concern to them. Yes and aso peope understand that we are a research practice and we are obiged to do so much research. And if we don t do so much research then our status and our resources that come with that wi get chopped and if they don t pay the game then that wi get chopped so even if they don t necessariy think this is best thing since siced bread, that s going to happen and if they are not on board then... GP 04, practice 1 (NT) Consequenty, other staff, incuding the nurses who ended up performing the triaging, reported that they had not been incuded in the decision-making and fet that the decision had been imposed on them by the peope east affected by its consequences. The consequent ack of ownership of the tria impacted negativey on how triage was experienced and on its acceptabiity, and appeared to have a divisive effect on the practice as a whoe. No, I think the peope who decided they wanted to do it they just thought the money was an attraction and they didn t think of the consequences. They didn t ask me who woud do the appointments and how it woud be, and it was just this is what we re doing. And they don t isten to the peope that know. It is a shame reay. Manager, practice 1 (NT) In contrast, incusive and democratic decision-making appeared to have a cohesive effect on practices and impacted positivey on how the tria was experienced. Participating in the tria aso provided an opportunity for good communication and deveoping good staff reationships. In practice 2, aso a NT practice, the practice manager drove the decision to take part but the decision was taken after discussion with cinica and a other staff, and everyone in the practice was invoved and motivated. Staff members were peased to have a protoco to guide the introduction of triage and anticipated that the effect of the tria woud be to bring the practice together. The nurses were very peased to be randomised to NT and the earning opportunity this presented. 70 NIHR Journas Library

101 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 And the other point I d make is, I think it brought the practice together in a number of facets, in a number of different ways because we engaged amost the whoe practice. Everybody knew it was going on; everybody was invoved in it in some shape or form. And with two new nurses, again, I think the rigours of constraints of the ESTEEM tria heped to bring the process into pace, that they woudn t have earnt in any other way. And I think massaging the information and coecting it, was a ot of fun... Practice manager, practice 2 (NT) Staff at another NT practice, practice 3, reported mixed reactions to entering the tria. Doctors reported the motivation as responding to a perceived need for a better way of handing same-day appointment requests, and were aso attracted by the remuneration. Administrative staff, however, fet that they had been sod the idea rather than fuy consuted. It is worth noting that neither of the two NT practices that reported being financiay motivated to enter the tria provided any additiona nurse capacity to absorb the increased workoad caused to nurses, as recommended by the research team. Triaging was added to nurses aready busy working day. Despite not feeing fuy incuded in the decision-making, staff at practice 3 aso reported a cohesive effect on the practice, and excitement about trying something new, perhaps refecting the exceent reationships in this practice, as observed by the researcher. The practice manager described a mood of excitement at taking part and peasure that patients might benefit: But you know, aso we were peased that we were doing it in one sense because it s interesting to do something new and to have a different chaenge. And potentiay if it was going to have a good effect on the patient and doctor appointments then yeah, we were quite excited I suppose from that point of view. Practice manager, practice 3 (NT) In some of the practices there was no consensus among medica staff about entering the tria. Practice 6, a GPT practice, was one of these. Some doctors reported that pressure had been appied by one of the partners to agree to it. Communication in this practice did not appear to be optima and informants accounts of a number of issues, incuding tria participation, were at variance with each other. I think that we were pressurised by one partner, yes... And when I raised concerns it was read as objections and I was pushed down. GP 03, practice 6 (GPT) Motivation to enter the tria: key themes Reasons for entering the tria, how the decision was made and by whom, and how the decision was communicated a affected how it was experienced. Motivations reported were an interest in research, the high quaity of the tria, a perceived need to change the appointment system, an interest in trying triage to manage demand, the fact that a cear mode of how to introduce triage was provided, and the financia incentive. There was a range of communication styes in reation to the decision to take part, from top-down to democratic or consutative. There was not aways consensus within practices about taking part, even among doctors. A number of practices regarded participating in the tria as an exciting chaenge. Severa practices described tria participation as having a cohesive effect on the practice and improving communication. Attitudes to randomisation How the tria was experienced by staff was reated, at east in part, to their attitudes to the arm into which they were randomised. In two of the NT practices practices 2 and 3 staff said that they were peased to have been randomised to NT and were excited about the earning opportunities NT presented. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 71

102 RESULTS We we were very, you know peased to be part of research and we iked getting the nurse-ed triage because we fet it d be a good earning curve. Nurse, practice 2 (NT) In practice 3, one of the GPs reported being reieved that they had not been randomised to GPT, which was perceived as being more effort for doctors. In practice 1, on the other hand, none of the cinica staff, nurses or doctors was peased with being aocated to NT. Both of the practice 4 doctors interviewed aso had reservations about NT, beieving that triage shoud be done by the most highy trained cinicians, i.e. doctors, especiay if they coud be triaging their own patients. I think most peope recognise that triage is probaby the most difficut thing to do and that it often works best when the most highy trained peope do it. And that probaby is why personay I fee as though it might be better if I did it for my own patients, which is what I effectivey thought I was doing before. And I think that aso, you know, that agorithmic medicine is pretty awfu and not reay as cever as it ooks. GP 05, practice 4 (NT) In two of the GPT practices (practices 7 and 8) a few staff, but not a, expressed disappointment that they had not been randomised to NT, but most were peased with GPT, as were those in practice 5. Signing up for the ESTEEM tria was a gambe reay. Because we may have been randomised into one of the other two eves and I te you what, if we d been randomised into NT, we d have been up the creek without a padde [aughter]. GP 04, practice 5 (GPT) Reasons given incuded nurses not being keen and the perception that GPs coud do more on the phone. In contrast, practice 6 had a very experienced nurse practitioner who for the ast 10 years had triaged, face to face, a patients requesting a same-day doctor s appointment when the appointments had run out. Randomisation to GPT represented a ack of fit with existing experience and expertise within the practice, wasting the resource of an experienced triaging nurse practitioner. Their duty doctor system aso caused additiona workoad for GPs through ost consutation time. Attitudes to randomisation: key themes Of NT practices, two were peased with their randomisation arm and two woud have preferred GPT. There was a perception that GPT was a more appropriate mode than NT. There was a distrust of CDSS. Of GPT practices, three were peased with their randomisation arm and one was not. The GPT practice that was dispeased had had a highy effective nurse face-to-face triage system in pace prior to the tria and no probems managing same-day appointment requests. Impementation of the study protoco Athough most of the practices had no probem with understanding and adhering to the tria protoco, the process evauation did revea some areas of confusion and misinterpretation and infideities to the protoco. Practice 2, for exampe, discosed that they forgot to record a of the phone cas during the run-in audit and remained unsure of the suitabiity of some cas for the tria, for exampe chidren with breathing difficuties. The few gitches are basicay maybe not understanding possiby how it ran 100% from our point of view and being tod different things, ike if we booked in for tomorrow it wasn t ESTEEM and then it was suddeny ESTEEM and why did it change?... The other probem I have and I m sti not quite sure how we get round that is that if peope... peope are more incined now to come down to 72 NIHR Journas Library

103 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 the desk at nine o cock in the morning. Now the triage appointments don t start unti... I can t remember, at east ten. Now I have a probem saying Can you go home, doctor wi phone you and then you might be coming back so it just seems a itte bit... Receptionist, practice 2 (NT) Doctors reported forgetting to ask for consent for notes review. The main probem is remembering to ask their permission, so the ast question at the end. And no matter how ong you do it for, it does hep if it s the second day in a row but we a work part time, and you often find that you ve forgotten to ask them the question. Caing them back a second time to say by the way I forgot to ask that is quite difficut. I did that a few times but it gets to the point that we re so busy that a of us stopped doing it. GP 05, practice 5 (GPT) Simiary, a patient interviewee from practice 6 reported that she had presented at the desk and been tod to go home and wait for a phone ca; this same patient denied having requested a same-day appointment in the first pace. Practice 8 aso sent home patients who presented at the desk requesting same-day appointments, mainy because receptionists had no appointments to give. I: So, okay, can I just make sure I ve got this straight? So if somebody turns up and wants an appointment that day they can t have it, they have to go home and make the phone ca? R: Yeah, they do reay. And we don t ike doing that, but it s not a drop-in centre. Practice manager, practice 8 (GPT) Practice 5 had previousy attempted and abandoned the Advanced Access mode promoted by this organisation. There was simiar confusion in practice 7. Because we were doing a triage system and as cinicians we re a trained in triage and using our skis effectivey to triage patients. But then the tria was saying no, do today s work today and if they need to be seen book them in. So are we just a booking service then? GP 01, practice 7 (GPT) Impementation of the study protoco: key themes Most practices had no probem understanding and impementing the study protoco. One practice was mistakeny incuding patients who presented in person to the reception desk to request a same-day appointment. They woud have been informed during training that such patients were ineigibe for ESTEEM, and it was ikey that this had subsequenty been forgotten and then not communicated among practice staff. Practices at which receptionists had no appointments to give were aso sending home peope wanting a same-day appointment, teing them to phone in. One practice forgot to record a of the phone cas during run-in. There was some uncertainty in two practices about patient incusion and excusion criteria. Doctors forgot to ask for consent for notes review. One practice abandoned the tria on busy days. One practice beieved the ESTEEM protoco required practices do today s work on the day, i.e. confused it with the Advanced Access principes promoted by Productive Primary Care. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 73

104 RESULTS Perceptions of tria-reated workoad Staff in some practices reported that the tria added consideraby to their workoad, others that workoad had decreased, with disagreement about this both within practices and within occupationa groups, especiay doctors. In practice 1, a practice characterised by poor communication and no prior triage experience, perceptions of the tria were amost universay negative. A receptionist described competing the tria og sheets for the integrity checks as just more work and time-consuming, and reported that receptionists hated doing it. One of the nurses compained that the Cinician Forms were bady designed, obviousy not designed by a nurse.... and there s not much space between that bit and that bit, between the patient computer ID and that bue ine at the top there, yeah. Yeah it s trying to get quarts into pint pots and get everything on to the sheet. Nurse, practice 1 (NT) A GP found the forms straightforward and simpe but confessed to often forgetting to compete them. Doing so added to the consutation time. Obviousy it adds ength to the consutation which is a bit of a nuisance. And we haven t made any provision for that, so the days that I ve had a ot of patients going through the study, I ve run ate then I woud redo them [aughter]. GP 02, practice 1 (NT) Coating the data was aso described by the deputy practice manager as time-consuming; the system of their practice did not aow for buk Read Coding of patient records to fag that they were part of the tria. Even the visits by the ESTEEM researcher were perceived as something of a time burden. Doctors were in disagreement about whether the tria had increased or decreased their workoad, perhaps refecting rea workoad disparities and different eves of triaging effectiveness among doctors. Practice 2 had extensive prior teephone triage experience but had underestimated the requirements of the tria, which had greaty increased their workoad. They reported a great dea of tension whie it was taking pace, particuary in reception. The paperwork was hard to keep up with, athough the practice manager found coecting and coating the data a ot of fun. However, doctors reported feeing that they had more contro over their workoad. They fet that the training had been good and the run-in exceent, and appreciated the approachabiity of the ESTEEM researcher. Practice 3 aso had prior triage experience and reported no probems with the protoco, which they found cear, or the paperwork, and reported that the training had made everything easy. Nurses workoad was reported to have increased enormousy, however. One informant fet strongy that the initia training was much too compicated and had fostered anxiety and confusion: Yeah, I just thought, Oh I reay can t be bothered with this, you know, we ve got enough to do without this. And I te you why, it s because the chap who came to show us it, he went into too much detai for us. We didn t need to know a that stuff that he went on about and it made us a as receptionists think, Oh my crikey. Whereas a they reay had to say to us was We re trying out a new system, this is what you ve got to say to patients... and it woud have been fine. But he made it so compicated. He went through stuff that we didn t need to know about, about what the doctors have got to do, what... so to me that first tak that we had, we shoudn t have sat through that, because we a fet the same. Receptionist, practice 4 (NT) 74 NIHR Journas Library

105 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 The practice manager in practice 5, which had no prior triage experience, found that the Productive Primary Care session was pivota in making the tria a success and in getting more sceptica staff enthused about it. However, the GPs in this practice reported extremey high stress eves as triage cas were added to their ever-increasing workoad, athough two of the doctors conceded that they did fee more in contro of their workoad despite perceptions of its increased voume. I think it made ife quite difficut, and it is sti quite difficut at the moment. It has made it more stressfu, it s made it busier, it s made the work more intense, erm, so yes, I mean ooking at the benefits from it, there must be some and I m sure we get to that ater. But yes, being honest it has been quite difficut. GP 05, practice 5 (GPT) Femae doctors were under particuar pressure, with more patients wanting to consut them. Practice 5 was characterised by poor communication between management and doctors and between management and the reception team. Doctors workoad in practice 6, a practice with prior triage experience, decreased under the tria or it increased, depending on doctors accounts. One GP reported having to work extra sessions because of the surgery time ost to the duty triage system, whereas two other doctors caimed a decreased workoad as patients were deat with on the phone. On the mornings when we re doing the triage I think we are working ess intensey than we used to. GP 02, practice 6 (GPT) Paperwork was checked and competed every evening by the reception manager, which increased her workoad but took the pressure off other staff. Training and support from the ESTEEM researcher was reported to have been good. In practice 7, which had prior triage experience, response to the tria was mainy positive, athough there had been dissent among doctors about it, and some GPs fet that the quaity of triage was poor because of the voume of cas. The tria paperwork had not been experienced as a probem. The ESTEEM researcher was regarded as approachabe and easy to contact, and the run-in period was considered invauabe. However, a perverse effect of the tria protoco was noted. Patients who woud previousy have been given an appointment to see a nurse were now put on the triage ist if they had asked to be seen that day, thus increasing triaging doctors workoad. Because we woud aocate, you know, if somebody said they had an earache or a sore throat or a chesty cough we woud have just automaticay aocated them to the practice nurse, nurse practitioner but because of the tria everything had to go on the doctor, you know, the doctor s triage. So peope were phoning up who woud normay have spoken to, say, T [nurse] or L [nurse] and we were saying to them, no, you ve got to go on to the doctor s ist so they didn t quite understand that because I think they fet that they were wasting the doctor s time when they coud have been seen that, you know, the nurse because it was something that they ve seen in the past. Receptionist, practice 7 (GPT) Finay, practice 8 had a mixed experience of the tria, with considerabe disparity of opinion on how it was experienced and quite a ot of negativity. This practice had previous triage experience but poor communication, with an absence of both forma and informa opportunities for sharing knowedge, information and concerns, and there were major staff continuity probems. The extra paperwork was described as an additiona burden that hindered the abiity to manage the workoad. This practice did not consider it had received enough support from the ESTEEM team, and had not received the promised feedback from the audit of demand. Setting up had been probematic owing to Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 75

106 RESULTS the practice starting the tria using a software soution deveoped during the tria and ater abandoned (for detais, see Strengths and imitations). The software was designed to automaticay appy a study-specific Read Code to the eectronic records of ESTEEM patients by way of a short keyboard command typed by the receptionist receiving the patient s teephone ca. The software required a macro to be set up on a of the computers used by the doctors, which, at this practice, was probematic due to the number of doctors working in different rooms. Staff members were consequenty not confident that everything had been set up propery. Tria-reated workoad: key themes Tria paperwork was perceived as burdensome in most of the practices. Tria paperwork and data coation was time-consuming and hard to keep up with. The paperwork added to consutation times, sometimes causing GPs to run ate. Staff sometimes forgot to compete the paperwork. Deciding how patients were to be managed under the tria needed to be discussed. Visits from the ESTEEM researcher were sometimes considered time-consuming. IT issues had caused additiona work. How the teephone triage interventions are impemented in different practice settings Practices were advised by the study team to triage in set bocks of time (i.e. whoe sessions) each day, and a process evauation practices did so. Most practices triaged in the mornings, athough some had both morning and afternoon triage sessions. The variation in arrangements for triage is notabe, athough this was mosty within the imits permitted by the manua. Nurse-ed teephone triage practices Two of the NT practices triaged ony in the mornings, whereas the other two aso incuded afternoon triage sessions. In practice 1, a arge practice in a sma town with a persona ist system but poor staff continuity, two nurses performed triage in the mornings, athough usuay ony one at a time. There were about 30 triage cas per day. No extra nursing capacity was provided, and nurses had to work extra hours unpaid. Practice 2, a arge practice in a sma town with no persona ist system had three nurses triaging each morning, a number that had needed to be increased from two due to demand. Two extra nurses were appointed and existing nurse hours were aso increased. Practice 3, a sma rura practice with no persona ist system, did not provide additiona nursing capacity and nurses worked extra hours unpaid to cope with the additiona workoad. Nurses triaged aone for an hour and a haf in the morning, and the same in the afternoon. Because one of the doctors finished at 4 pm the nurse woud come in eary in her own time on that doctor s duty day to fit triaged patients into his surgery. Timing was an issue in this practice, as in others, because doctors cinics coincided with triage times, which sometimes did not eave patients enough time to get in to see the doctor, especiay those dependent on the bus. This coud eave appointment sots unfied and doctors having to work ate. The system was reported as not working we on Friday afternoons when peope had to be seen before the weekend. The fourth NT practice, a arge urban practice with a persona ist system, appointed two additiona nurses for the tria. Initiay, triaging took pace a day, athough demand was found to be ower in the afternoons, and some days triaging took pace ony in the mornings. GPs changed their surgery times to ater in the day to aow triaged patients who needed an appointment time to get to the surgery. 76 NIHR Journas Library

107 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Nurse-ed teephone triage was a chaenge for practices with a persona ist system at which patients needing an appointment were triaged into the next avaiabe sot. I think the things which we ve found awkward are that we are a practice where we encourage in favour of persona ists and we agreed to this, we understand it s part of the tria, so it s not a surprise, but nevertheess, you know, we ve had days when say four of us are here and each of us are seeing each other s patients. Now I m seeing someone of [GP name] and he s seen some of mine, which is not very hepfu... And that s annoying. I think we woudn t want that to continue. That woud tend to grate and woud break down a good doctor patient reationship after a whie. GP 04, practice 4 (NT) Impementation of nurse-ed triage: key themes Two practices triaged ony in the morning, whereas the other two aso did so in the afternoons. The number of nurses triaging at the same time varied from one to three. Two of the practices provided extra nursing capacity for the tria. Two did not. Timing of triage in reation to surgery time coud be a difficuty, and doctors and nurses had to adjust their hours to accommodate this. Triaging into next avaiabe sot is a chaenge for persona ists. Genera practitioner-ed triage practices Athough the ESTEEM intervention manua prescribed many aspects of how GPT was to be impemented, it was essentia that practices had some fexibiity to design the intervention based on their oca context. Consequenty, there was marked variation in the ways in which GPT was impemented between the practices. There were two main organisationa structures for GPT: a duty triage doctor system, for which GPs took it in turns to provide a dedicated triage service for patients, and an integrated system, for which triage (usuay of doctors own patients) was combined with norma surgery. However, practices seemed to have evoved systems that suited them, often invoving a combination of the two organisationa structures, especiay in practices operating persona ists. Two practices (practices 5 and 7) combined a duty doctor system with presurgery triage, in which non-duty doctors picked their own patients off the triage ist and caed them back before the duty doctor took over triaging. When an appointment was necessary, patients were triaged to an appointment with the triaging duty doctor, to their own doctor or to the first avaiabe appointment, depending on the practice. This is arguaby a third organisationa pathway for GPT, being neither purey a duty system (because a doctors triaged their own patients rather than just one doctor triaging a patients on the ist) nor a combined system (being performed outside surgery time). Both systems presented issues of workoad disparity: The duty triage system meant that doctors on triage duty on busy mornings, especiay Mondays, strugged to get through the triage ist. Disparities in the integrated system resuted from extra demand faing on certain doctors. For exampe, some femae doctors faced extra demand from femae patients with certain conditions; some doctors were more popuar than others, whereas other GPs had a more chaenging or demanding patient ist. Our difference in terms of numbers of phone cas is immense. If you work it out per session per year, some doctors are... the number of phone cas you do per session per year may be in the region of 250 phone cas per year. Some of us do 2000 phone cas per session per year... I was taken on as an extra partner here so naturay anybody who wasn t happy with their own GP immediatey came to me. So by definition I had a motey crew. GP 06, practice 7 (GPT) Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 77

108 RESULTS In practice 5, a sma rura practice with a persona ist system, a the doctors triaged their own patients for an hour and a haf in the mornings foowed by their surgery, at which point a duty triage doctor system came into operation. Athough this worked we, the four minutes per patient schedued for triage in this practice caused some doctors to get behind with the triage ist and, consequenty, their surgery. Receptionists in this practice aso had extra appointments to offer, so that wak-in patients and those unsuitabe for triage coud be given appointments; this was not the case in a practices. Advance appointments were aso avaiabe (foowing a faied attempt to foow the infuentia Productive Primary Care Advanced Access -based mode of do today s work today, which had impressed many of the practices). Surgery times were adjusted to accommodate triaged patients. There was sti a faw in the system, in that peope phoning in the ate end of the morning session coud not be phoned back unti 3 pm because triage sessions were immediatey foowed by surgeries, then unch, then home visits. In addition to the ong wait for ring-back that this created, if peope were phoned at 3 pm there was sometimes no time for them to get to the surgery for their appointment, especiay if they traveed by bus. Doctors did stay ater in the evening to accommodate this but sti found that their appointment sots were a gone before afternoon triage was finished. Practice 6, a arge urban practice with no persona ist system, operated a duty triage system with one doctor on duty at a time on a rota system. The duty doctor ater saw, where possibe, the patients he/she had triaged, which operationaised one of the main perceived advantages of GPT that the consutation has been started on the phone, which shortens the appointment time, and the patient s history does not have to be taken more than once. It aso addressed the continuity probems that can be inherent in triage. However, practice 6 had used a uniquey effective system for same-day appointment requests prior to the tria, in which patients were triaged face to face by the nurse practitioner. GPT, according to the ESTEEM protoco, did not aow for patients wanting urgent appointments to be seen by the nurse practitioner. This increased the doctors workoad at this practice so that extra hours had to be worked and ocums empoyed. Practice 8 is a sma urban practice with persona ists but very poor staff continuity. Their triage system was a duty doctor one, athough this practice chose to incude other things on the ring-back ist, such as queries and requests for resuts, resuting in pages and pages of ring-backs to do, described as a particuar chaenge for the doctor on duty on Mondays. Impementation of genera practitioner-ed triage: key themes There were two main systems: a duty triage doctor system, in which doctors triaged patients but did not see them, and an integrated system, in which triage was combined with surgery. Two persona ist practices adapted the duty doctor system so that a doctors triaged their own patients from the ist before surgery and the duty system took over. Both duty doctor and integrated systems produced workoad disparities between GPs. The duty doctor system meant that surgery appointments were ost whie doctors were triaging, increasing workoad for the other doctors. The integrated system in which triage was combined with surgery produced work overoad and the risk of running ate. Timing of triage in reation to appointments required adjustment of working hours. Some practices triaged patients needing appointments to the triage duty doctor, some to the patient s own doctor and some to the next avaiabe appointment sot. The system where triaged patients were not given appointments with their own doctor was a chaenge for some doctors in practices with persona ists. 78 NIHR Journas Library

109 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 How triage was experienced It is evident from the previous sections that how triage was experienced in the tria practices was mediated by how the tria itsef was experienced, and those factors combined to infuence how acceptabe teephone triage was fet to be. Introducing a major organisationa change under tria conditions is not the same as introducing it in rea ife, when a mutipicity of factors such as the organisationa structure and cuture, staff skis and experience, perceived need, staffing issues, geographica ocation, patient demographics and so on wi infuence the mode of triage chosen, how it wi be organised, who wi perform it and under what circumstances and so on. Being randomised into an intervention arm, which may not necessariy be viewed as the most appropriate one for that particuar practice, and working to an immutabe tria protoco under externay given time constraints, are not necessariy the optima conditions for effective change management to take pace. The foowing report of how triage was experienced shoud be read in ight of this. No cear patterns emerged from the data on staff experience of triage, either within practices, within occupationa groups or in reation to whether GPT or NT was being discussed. There were many positive and negative experiences reported and a compexity of opinions on what worked, why and for whom, sometimes from the same informant. However, there were severa key themes to emerge of factors that affected the experience of triage and whether it was found to be acceptabe. Many of these echoed the experience of the tria itsef and were underpinned by the same aspects of practice organisation, interreationships and cuture. These incuded: the quaity of reationships, consutation and communication within the practice whether a stakehoders fet fuy consuted and informed about the introduction and organisation of triage how it was decided to enter the tria and by whom how triage was organised whether there was adequate staff capacity to perform triage effectivey whether triage was perceived to adversey affect continuity and cinician patient reationships how busy the practice was doctors workoad voume whether staff fet in contro of their workoad how staff perceived their roe, ideas about what constitutes good cinica practice and good care, safety and risk the organisation and effectiveness of the appointment system before the tria perceptions of the intervention arm into which they were randomised. In addition, a staff members interviewed were individuas, whose personaities, reationships, attitude to change, perception of their roe and a host of other factors are a different. For triage itsef, how it was organised, the mode of the patient within the practice cuture, patient expectations and behaviour, perceived need for change, and many other soft factors that were difficut to evauate underpinned attitudes and experiences. What worked we for whom? The benefits and advantages of triage The positive aspects of triage reported by practice staff have been organised into the four main themes: (1) effects on work and workoad, (2) benefits for individuas, (3) advantages for the practice as a whoe and (4) benefits for patients as perceived by staff. A further theme is patients positive experiences and views (5). Effects on work and workoads Practice staff often reported that they had previousy fet that they were batting against a tida wave of increased and increasing workoad demand, and beneficia effects of triage on workoad featured strongy in their accounts. GPs in both intervention arms reported experiencing ess pressure and stress, and having more fexibiity and contro over their work. When NT was working we, appointments were more appropriate, as fewer minor conditions were referred and work pressure was dramaticay reduced. A GP informant in practice 3, for exampe, reported that appointments had been reduced by haf. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 79

110 RESULTS In GPT, face-to-face consutation time was fet to have been reduced as doctors had been abe to begin the consutation on the phone. This was particuary fet to be the case in practices where doctors combined triage and surgery time, usuay those with a persona ist system such as practices 5 and 7, where triaging doctors booked patients into appointments with themseves. Some doctors found teephone work ess taxing than face-to-face consutation. Effects on workoads: key themes Appointments were reduced by up to haf (ony one practice reported this). NT referred appointments were appropriate. GPT was fet to reduce face-to-face consutation times, especiay in practices with a combined GPT system with persona ists. There were fewer interruptions to GPs from administrative staff. There was no more morning rush of phone cas. Receptionists were more in contro of their workoad. Benefits for members of staff Doctors reported the benefits of triage as reating to aspects of their workoad, as described above, and as partners, those for the practice as a whoe, which are described beow. For triaging nurses there was the bonus of earning new skis, having new responsibiities and the opportunity to form reationships with a wider range of patients, a of which increased their job satisfaction. The CDSS used by nurses for triaging was described positivey by a number of them. It was found to be exceent for triage training and for enhancing earning and skis in genera, even for those with prior triage experience. It was usefu for triggering what questions to ask, and to hep understand how patients present, thus increasing knowedge and confidence. I woud say it s good... it increases... the person who s triaging, it increases their knowedge base definitey I woud say and increases their overa nursing experience reay, you know, it gives you knowedge. So it gives you... you re a bit more empowered when you see patients then face to face. It gives you confidence as we chatting to the patients because after you ve done it for a bit it gives you that itte bit more confidence to tak to peope and... yeah, and the ayout was very good. It was very quick and, you know, you coud get on to things very quicky. Nurse, practice 3 (NT) Some nurses commented that the ayout of the CDSS was good; reevant sections coud be accessed quicky, it was comprehensive, and nothing that they were presented with was not covered. It was particuary good on sef care, an invauabe too for teaching patients the sef-management skis and the judgement to know if their condition reay was urgent, that was hed to be key to managing demand. I personay think it s been a ot easier, so I think they re getting, they re getting more used to it and trying home care a bit more now than perhaps they were. Because I m sure a surgeries are the same, you have peope whose name crops up and up and up. And at the beginning of the triage system the same names were coming, aways being booked in. And I m guessing that perhaps they weren t getting the doctor s appointments they were after, and have earnt now actuay is this reay urgent, can it wait, can I do this and this first, and book an appointment in two days time? Receptionist, practice 3 (NT) 80 NIHR Journas Library

111 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Receptionists were deighted with the triage system, which made their job easier and ess stressfu, as they no onger had to attempt to find rare appointments, make judgements about urgency or dea with irate or distraught patients; they just had to put them on the triage ist. It is briiant, it s absoutey briiant. Because I think the worst part of my job and any of the receptionists that you ask, has aways been that there s never enough appointments to go around for the peope that want them. Receptionist, practice 5 (GPT) Benefits for members of staff: key themes Nurses earned new skis and had more responsibiity. Nurses generay enjoyed triaging. There were enough appointments. Patient demand for appointments was under contro. CDSS was a good training too for nurses. Receptionists job was much easier. Doctors were more in contro of their workoad. Doctors workoad was reduced. Benefits for the practice Triage was seen as benefiting practices by providing a more rationa and efficient way of aocating appointments, in contrast with the informa triage by receptionists that had been happening before, and appointments were more appropriate, with fewer patients with trivia conditions taking appointments they did not need. I think it s just been reay good for the surgery to see a different way of maybe booking appointments. A much more efficient way of doing the appointments instead of eaving it to us. Because you know, we have no medica knowedge. We re not abe to say to someone that s urgent and that s not, whereas they are. So I think from that point of view the patient wi get what they need much quicker than they woud have done. Receptionist, practice 7 (GPT) Triage was appreciated as being an equitabe and fair system, with appointments aocated according to need, in which there were aways appointments avaiabe. It was regarded as the best possibe use of resources and was cost-effective because the number of appointments appeared to have been reduced. In practices in both intervention arms staff expressed surprise at how few patients actuay needed to be seen. Access was improved in practices where it had previousy been poor, and, in those where accessibiity had been good before, appointments were reduced by the abiity of triaging nurses to provide reassurance and teach sef-management skis and the abiity of triaging doctors to dea with conditions over the phone. There was a suggestion that good access can perpetuate rising demand by creating its own expectations, and that teaching sef-management was a sustainabe way of reducing demand as patients began to earn to manage their conditions themseves and to know when an appointment was not necessary, and so woud become ess ikey to request one in the future. Practices therefore had more contro over demand. Using CDSS was considered exceent for promoting sef-management. Yeah, it was interesting actuay. Some were... yeah, some... after taking at ength to them on the phone they actuay then reaised actuay they re not quite as bad as what they think they are and that they are... a few measures they coud do themseves and then you make an appointment for ater in the week and then say, Actuay if you just try these different measures, ike steam inhaation and things ike that, you coud actuay make yoursef fee a ot better. And then a ot of peope don t, they don t take any painkiers. So they say, Oh I ve gone over on my foot today, something s Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 81

112 RESULTS cracked. Have you tried... have you taken any paracetamo? Rested it? No, just want to see a doctor. You know, We try these measures first. Nurse, practice 3 (NT) Staff iked the triage system, against managers expectations in some cases, which made for a happier atmosphere in surgeries. Severa practices reported how much they had enjoyed the sense of cohesion and common purpose the triage tria had produced, and the improved communication and better reationships that came with it. It was an opportunity to be innovative and dynamic, which had a positive impact on practice cutures. Reationships with patients were aso widey reported to have been improved, not ony by triaging nurses, who were given the opportunity to get to know patients better and interact with a wider range of patients than before, but aso by reception teams, who reported that a ot of the interactiona tension and stress had vanished from their encounters with patients. Benefits for practices: key themes Practices coud provide a more rationa and equitabe appointment system. There was more contro over demand. It appeared to be cost-effective and a good use of resources. Appointments were fet to have been reduced. There is potentia for a sustainabe reduction in demand as patients are taught sef-care skis. Introducing triage gave practices a sense of cohesion and common purpose. An opportunity to be dynamic and introduce change. Reationships with patients were improved. Staff perceptions of benefits for patients Staff reported that patients iked the triage system once they, and staff, had had the chance to get used to it. Like a change, it took time to become embedded. There were a number of perceived benefits for patients, not east of which was the perception that they were receiving a better quaity of care and faster and more equitabe access to cinica expertise when the sickest peope get seen soonest and everybody gets the right hep at the right time. I think most peope were reassured by the, that thing about whoever gets, this is reay about making sure that the peope are sickest get seen soonest. Receptionist, practice 7 (GPT) Patients no onger had to exaggerate symptoms or pitch in order to get an appointment because there were more appointments avaiabe and they were therefore more reaxed and ess stressed and anxious, which improved their reationship with the practice. We, appointment basis, there are, we can aways give something now, whereas before we were having to say, even a our emergencies have gone. Because they got the patients, sorry, the patients got used to, if they coudn t get an appointment they d say, it was an emergency, because they knew we had to give them one. Receptionist, practice 3 (NT) In practices whose appointment system had not worked we for patients before the tria, the very fact of being abe to speak to a cinician was a huge benefit of the triage system. Patients coud obtain immediate reassurance, and it was fet that the opportunity to earn sef-management skis was empowering for patients, enabing them to take contro of their own heath. 82 NIHR Journas Library

113 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Teaching sef-management seemed to be a particuar bonus of NT, supported by the CDSS; none of the GPT practices mentioned it. Such knowedge was confidence buiding for patients, who aso got to earn what was and was not an emergency, which coud pre-empt anxiety as we as unnecessary appointment requests. Patients were reported to be happy to tak to a nurse and some preferred it, being aware that doctors time was vauabe and being reuctant to waste it. And I think they re just gratefu. And they often say I think they, sometimes they quite ike the nurse ringing them back. I think because they think it s perhaps a bit siy speaking to the doctor about it but actuay they don t fee quite so uncomfortabe speaking to the nurse, you know I think that s been a positive side of it. Nurse, practice 4 (NT) Some peope sometimes were reported to have found it easier and ess embarrassing to expain things to nurses. The triage ca with a nurse coud itsef be therapeutic. Patients were frequenty gratefu not to have to come to the surgery; it saved them time and effort. For some peope, being caed back by a doctor or nurse represented individuaised care, which made them fee cherished. Staff perceptions of benefits for patients: key themes Patients iked the system after a period of adjustment. Fairer. The sickest patients got seen soonest. Patients no onger had to caim it was an emergency to get an appointment. Less stressfu for patients so they were more reaxed with staff. Patients sometimes actuay preferred taking to a nurse, which coud be ess embarrassing than taking to a doctor. The ring-back from the nurse coud itsef be therapeutic. Being caed back made patients fee cared for. Reationships with patients were improved. Patients accounts of their experience of triage Patients at NT practices reported being peased and surprised at getting a same-day appointment. Others were deighted not to have to go to the surgery and to have their probems resoved with a phone ca. The wait to be caed back was of an acceptabe ength; from 10 minutes to 1 hour were the times cited. Patients were aware that other peope were getting appointments that were not reay necessary, and were gad that a system had been introduced to weed out the time-wasters, making appointments avaiabe for those who needed them most. However, they aso fet that triage coud ensure that they themseves were not going to see the doctor for something siy. Patients appreciated getting to speak to someone with medica knowedge straight away. So being abe to get medica trained advice straight away, I think is a reay good idea and pus if anybody s got something persona they may not want to discuss with a receptionist, then, you know. Patient 03, practice 3 (NT) There were references to reief at not having to discose persona probems to receptionists, particuary from patients of practice 3, whose receptionists were not permitted to eicit the reason for the appointment request. Patients fet vaued, supported and reassured by the fact that someone woud ca them back. Nurses were fet to offer the benefits of a combination of common sense, sympathy and medica expertise, enabing patients to tak about how they were feeing. Patients at GPT practices aso fet that the system was good for weeding out time-wasters, and that it made good sense for the doctor, rather than the patient, to make the decision on whether an appointment was necessary. Many were gad not to have to come in to see the doctor in person if they did not have to, feeing that it prevented them from wasting the doctor s time and aso saved them time themseves, whereas others described the system as good as it enabed them to get an appointment. One patient said that the system worked but fet he woud rather have showed his condition to the doctor. Another patient described triage as a responsive Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 83

114 RESULTS system, which was efficient and convenient. Patients had not had to wait too ong to be rung back. They said they were happy to discuss their probem with a doctor on the phone, and that it was reassuring and gave them confidence. Patients accounts of their experience of triage: key themes Patients were peased and surprised to get a doctor s appointment. Patients were gad not to have to go into the surgery. The system was fairer and eiminated time-wasters. Triage ensured that they woud not see the doctor for something trivia. Patients iked the fast access to cinica advice. Being rung back made patients fee vaued. The wait to be rung back was of an acceptabe ength. Patients appreciated nurses combination of common sense and cinica knowedge. Patients were gad that doctors, rather than receptionists, made the decision about whether they needed an appointment. Patients were gad not to have to te receptionists their symptoms. What worked ess we and for whom: chaenges of triage Effects on work and workoads GPT In contrast with the previous section, staff whose experiences are reported here found that the triage systems had an adverse effect on their work and workoads. GPT, as we have discussed, was broady organised in three different ways: (1) the duty triage doctor system, in which triage is performed by one or more doctors on a rota system for a period during which this is their soe duty; (2) the integrated system, in which doctors triaged in some combination with their surgeries; and the hybrid of the two, adopted by two of the GPT practices, in which doctors triaged their own patients before surgery, after which a duty triage doctor took over. Each system had an impact on workoad. In a duty GPT system, there are particuary busy days, especiay Mondays, and doctors triaging then can be overoaded. In addition, there are different doctor styes, preferences and eves of experience of triage and teephone consutations. GPT, as with any activity invoving human behaviour, is not a consistent phenomenon; as this practice manager puts it, a doctors are different: A doctors are different, aren t they, and it s trying to get them a regimented doing the same thing. So some wi do it one way and then some don t reay want to do it that way, and it s just knowing their itte ways reay. Practice manager, practice 8 (GPT) Some doctors are perfecty comfortabe working on the phone, but for those who are not, perhaps through a ack of confidence and experience of triaging and of teephone consutations in genera, triaging can be experienced as aborious, stressfu and sow. This coud resut in subversion of the triage process. We ve got one GP cose to retirement and he just didn t reay triage at a. He was just a gorified receptionist which was very frustrating. He d just speak to them and book them in anyway, you know, so [sigh]... Practice manager, practice 7 (GPT) Some doctors just coud not keep up the requisite pace. One practice manager taked despairingy of such a doctor, whose attempts at triaging were as ong as fu teephone consutations, eaving the whoe practice behind with work. 84 NIHR Journas Library

115 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 In practices in which there is not the fu quota of doctors such as practice 8, which had a duty doctor system doctors had a particuar burden, exacerbated by a rota system that had the same doctor coping with busy Mondays each week. The duty triage system had the additiona chaenge that the period when the duty doctor was triaging meant time ost to the surgery, thus reducing the number of appointments avaiabe. This paced an additiona burden on the doctors not on triage duty, who then had to work extra sessions or take paperwork home. There was aso the probem that demand is uneven and unpredictabe with attendant workoad effects. Because some days there s about two pages, on a Monday or whatever, some days. You never know, do you, what days are going to be busy in genera practice. And some days you think, Oh ook at that ist, it s just pages! Practice manager, practice 8 (GPT) The integrated triage system, in which triage is combined with surgery appointments, coud aso cause excessive workoad. So I think the work oad is reay quite high with this system and I think the workoad has got to the extent where actuay it fees, for some GPs, it fet impossibe to get the work done in a working day. There are so many cas coming in, and seeing patients, and doing visits, none of the paperwork was getting done. So the working day became onger, more stressfu... GP 05, practice 5 (GPT) NT For triaging nurses there was aso a reported probem of work overoad. Those in the two NT practices that did not appoint extra nurses for the tria were particuary overworked, somehow having to graft hours of triage a day onto their aready busy schedue. The nurse in one of these practices, practice 1, described the pressure and stress of having a high workoad to be undertaken in conditions of extreme time urgency, in which she coud have a ist of 26 patients to ring back before her cinic started immediatey afterwards. There was concern that the pressure woud ead to mistakes. Even in practices where additiona nurses had been appointed for the tria, nurses described being daunted by the coumns of triage cas to be made. Getting hod of peope was a frustration for nurses as it was for doctors. Another frustration was not being abe to hear what patients were saying when there was a ot of background noise, either from the patient s side or in the practice. Nurses faced the additiona chaenge of having to perform triage using CDSS. This was found to cause a number of probems, and some nurses, especiay those experienced in triage, chose not to use it, or to ignore what it said. There s some of it that I just think is, the thing with [the CDSS] is that sometimes you can ask something, for instance someone s got a headache and next minute you know it s practicay going into them having a brain tumour and that was the part of it that used to annoy a of us. Because it woud bring up boxes that were competey unnecessary. And quite franky I just fip out of them. Nurse, practice 2 (NT) Others used it reuctanty under the impression that doing so was mandatory for the tria, rather than just as a support too. One of the issues with the CDSS was that it appeared not to be sensitive to a of the probems with which patients present when requesting a same-day consutation. It had been designed for acute primary care/urgent needs (such as in out-of-hours primary care) and was not viewed by nurses as whoy comprehensive for in-hours teephone triage in a number of ways. Some nurses fet that it Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 85

116 RESULTS acked information on commony presented probems in primary care such as contraception, medication interactions, withdrawa of medications, medication reviews, moes, for queries from patients with diabetes about whether they shoud increase their insuin, or on medication for anxiety. I think the software isn t necessariy the best for primary care. It s very good for out-of-hours but primary care it s not. There s gaps in it. For instance if someone rings up to have a moe ooked at, as one of the exampes, there s nothing on the [CDSS] system that deas with moes. At a. Not at a. Because in out-of-hours it doesn t occur, and [the software] I understand was buit up for out-of-hours... Nurse, practice 1 (NT) Some of the nurse users fet that triaging with CDSS was aborious and ong-winded, sometimes taking minutes for each patient, in comparison with GPT for which, at east in one practice, ony four minutes per patient was aotted. The CDSS sometimes directed nurses to ask what were described by some as ridicuousy inappropriate questions that coud not be bypassed, for exampe about sexuay transmitted diseases and pregnancy for a patient in her eighties. Some nurses fet the CDSS forced them to give appointments against their own judgement, resuting in them having to fied compaints about inappropriate appointments from doctors. Some doctors reported finding the CDSS summaries to be unhepfu, even miseading, and did not use them, preferring instead to start the consutation from scratch. Consequenty, no time seemed to be saved in consutations. When somebody ese has taken the history, I think I woud fee very uncomfortabe about not going over the questions mysef. GP 02, practice 1 (NT) Nurses had a number of suggestions for improving the conversion rate under NT. One suggested that if they coud prescribe certain things under the Patient Group Directive, such as antibiotics for urinary tract infections, a great many GP appointments coud be saved. Chaenges of genera practitioner-ed teephone triage, workoad issues: key themes GPT, whichever way it is organised, can increase doctors workoad. Not a doctors are comfortabe or confident with teephone work. It can be stressfu for doctors trying to keep up with the voume of work. There are workoad disparities between doctors. Demand is unpredictabe and therefore hard to manage, which can ead to wasted appointments or, conversey, the need to work ate. Ring-back can be a source of frustration and wasted time when peope cannot be reached. Chaenges of nurse-ed teephone triage, workoad issues: key themes There is work overoad, especiay when triage is added to usua workoad with no extra nursing capacity. Ring-back is a source of frustration and wasted time when peope cannot be reached. Doctors prefer to take the patient s history again rather than using the nurse s summaries; essentiay, the patient has his/her history taken twice. There are issues associated with using CDSS. The CDSS was perceived as going into too much irreevant detai and generating inappropriate questions. It is aborious and ong-winded and takes too much time. It is not sensitive because it was not deveoped for primary care. Its advice can confict with nurses judgement. It advises appointments against nurses judgement, which can cause probems with doctors. 86 NIHR Journas Library

117 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Its summaries are unhepfu to doctors, who tend to ignore them. Not being abe to prescribe imits the number of GP appointments nurses can prevent. Doctors have different regime preferences for certain conditions, making the judgement about whether an appointment is necessary difficut. Chaenges for members of staff Triage can represent a particuar chaenge in practices with a persona ist system. In some practices with a duty doctor system, the triaging doctor woud triage patients who needed an appointment into the next avaiabe sot, rather than into the patient s own or usua doctor s surgery. In practices with a persona doctor system, the cose doctor patient reationship was considered to represent good care and was highy prized. Restricted access to the doctor by his/her own patients was one of the consequences of triage that doctors fet coud cause good doctor patient reationships to disintegrate over time; they found it personay distressing to observe their patients going in to see another doctor whie they were on the premises. The oss of routine and advance appointments that resuted from triage aso impeded patients access to their own doctor, and was aso seen as detrimenta to the persona ist system and continuity of care and dysfunctiona for accessibiity in genera. In the GPT practice with an integrated triage system there were accounts of how incrediby stressfu doctors found the combination of triaging and seeing patients in routine appointments. In the duty system, too, the intensity of the workoad coud have an adverse effect on the quaity of consutations. But aso the voume of cas, if there was one triager and there were huge numbers of cas, you know, the quaity went down of the triage and peope were just trying to book, book, book, book and we re not going into the depth of the consutation because they just coudn t. GP 01, practice 7 (GPT) Safety and risk were saient for both doctors and nurses. One doctor who had experienced a serious adverse event in a previous cinica setting had an ever-present fear of a catastrophic mistake, and another fet strongy that if you re practising safe medicine peope shoud be seen. Nurses woud err on the side of caution to avoid risk, one arguing that she was risking her registration if she did otherwise. This caution reduced the number of appointments that coud be prevented. In practices in which appointments had aways been scarce, patients had acquired the habit of exaggerating their symptoms in order to get an appointment, and this made it difficut to make a judgement about whether an appointment was appropriate. In one practice, for which anxiety about doctors getting cross was a recurrent theme, a nurse informant cited this as an underying anxiety when deciding whether or not an appointment was necessary, in case she got it wrong. I suppose you fee the pressure a itte bit because you worry about those peope who you didn t bring in or you worry that the GP might be annoyed with you about peope that you did bring in. Nurse, practice 4 (NT) Both doctors and nurses found triaging tiring. Some nurses found triage boring and not a good use of their skis, making them fee ike teephone operators, and others found spending hours at a computer screen hard going: For me personay I suppose I don t want to be sitting on the phone a morning. I m a nurse practitioner and actuay I d rather be seeing patients. Nurse, practice 2 (NT) Receptionists sti had to conduct what coud be described as informa pretriage triaging, by asking patients what the probem was so that the triaging cinician coud prioritise the ring-back ist. This was an Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 87

118 RESULTS optiona part of the ESTEEM receptionists protoco, athough receptionists were advised that if patients were happy to provide information, it woud assist with prioritisation. Some hated doing this, feeing that it was not their pace because they had no cinica expertise, and that it was intrusive and inappropriate. Ony one of the practices decided that the receptionists shoud not ask patients why they wanted an appointment, and they were gad not to have to. Patients coud become abusive when asked, which was unpeasant for staff. It s funny I think [aughter] it s reay funny what some peope wi share without batting an eyeid. I ve had two where I ve just sort of recenty picked up the phone and someone s sort of shared with me, oh I ve got this awfu pain in my backside, right up my bottom. And then other peope who wi just not and are very angry that you re asking for information, so again the girs have had to expain that it heps the doctors to prioritise the cas if they have an idea of what the probem is. Practice manager, practice 5 (GPT) In practices in which there was not the cinica capacity to provide enough appointments to meet demand, such as in practice 1, or where they were attempting to embrace the Advanced Access mode of triage advocated by the Productive Primary Care presentation, no appointments or fewer appointments were avaiabe outside the triage system. Prebookabe appointments were reduced in most of the GPT practices, because doctors were tied up doing triage duty and this caused difficuties for receptionists, who had no appointments to offer, as some patients coud get angry and abusive. The fact that in some practices patients did not ike the triage system caused difficuties for staff, especiay the reception team and sometimes for triaging nurses. Patients coud see triage as a barrier to access, an unnecessary extra step to getting an appointment, and they made their opinions known. Reception represented the front ine in this strugge and staff found it hurtfu. Patients threatened to take their custom to A&E departments and were sometimes abusive to staff. This was a particuar issue in practice 1, where staff said they woud eave if triage continued after the tria, but other practices aso experienced verba abuse or at east bad temper from patients. Patients made forma and informa compaints. Chaenges for members of staff: key themes Not seeing their own patients was distressing for doctors who vaued having a persona ist. The high voume of cas caused distress to triaging cinicians. There were concerns about risk and safety because patients were not seen face to face. Nurses were anxious about doctors reactions if they booked an appointment inappropriatey. Doing so was often a resut of CDSS-generated instructions. Receptionists disiked having to ask peope why they wanted an appointment. Doing so coud resut in abuse from patients. The oss of or reduction in prebookabe appointments was a reduction in both access and continuity, and caused probems for receptionists when patients were annoyed at not being abe to book in advance. Patients did not ike the system and coud be abusive. Chaenges for the practice Despite the fact that practices were remunerated for participation in the tria, there was concern expressed that performing triage was costing them money. Apart from the increase in teephone bis, one doctor argued that in NT patients appeared to be effectivey consuting twice for the same condition and had their history taken by two different peope, and that this had cost impications for the practice. (It is worth noting that this doctor was from a NT practice whose motivation to participate in the tria was described by staff as financia, and which chose not to use the tria payment to provide any additiona nursing capacity.) 88 NIHR Journas Library

119 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Um, we, my incination is to fee that we have frequenty ended up with a patient in touch twice for the same probem, once by the nurse and once by me. And there isn t vaue added to my consutation by the nurse having spoken to the patient aready beforehand and therefore we pay for two consutations rather than one. GP 03, practice 1 (NT) Another doctor, from a GPT practice, cited the 51 appointment sots ost per week to duty triage as a cost to the practice, as ocums had to be empoyed and extra sessions worked to fi the gap. The increase in the teephone bi was another cost cited. Staff in five of the eight intervention practices caimed that there had been no reduction in appointments (practices 1, 2, 5, 6 and 7). Triage was fet to be a ot of effort for very itte gain, causing organisationa probems and, in one practice in particuar, dissent and dissatisfaction among staff and patients aike. Routine advance appointments disappeared as appointments were bocked out for triaged patients and this was a recurring dissatisfaction, reported to be a major cause of patients compaints. In the two NT practices that had not provided additiona nurse time (practices 1 and 3) the knock-on effects of ost nurse appointments were causing difficuties for both practice and patients. Chronic disease management in particuar had been negected, which woud impact on practices achieving their Quaity and Outcomes Framework targets (NHS Empoyers ), with potentia oss of income and reputation. Practice 3 had to have a gap from ESTEEM whie nurses caught up. Another difficuty was the issue of negotiating the timing of surgeries in reation to the timing of triage. Often these took pace at the same time of day, and triage appointments were wasted when there was no time for patients to get into the practice in time, and then surgeries ran over time in an attempt to fit peope in. In two of the NT practices some of the doctors were not happy with NT as a mode. The CDSS was fet by some to be too tick boxy, and agorithmic medicine was in any case considered an affront to good patient care. It was fet that nurses did not have diagnostic or triaging skis and that taking a patient s history was a highy skied proficiency acquired over years of training and practice by doctors but not nurses. Finay, somewhat perversey there was a concern that if triage was successfu the improved access it provided woud not so much manage demand as increase it. And there is a concern in peope that firsty, by triage, you re going to increase demand because the more you answer demand the more peope wi demand. So that s one concern. GP 06, practice 7 (GPT) Chaenges for the practice: key themes There was concern about the cost of triage. In NT practices patients were assessed twice. In some GPT practices ocums had to be empoyed to cover surgery time ost to duty doctor triage. The teephone bi increased. There was no perceived reduction in appointments in some practices. The triage system caused dissent and dissatisfaction among staff. Patients perceived triage as a barrier to access. Some doctors in NT practices were not happy with NT as a mode. CDSS was seen by some as inappropriate care. There was concern that demand woud increase as access improved. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 89

120 RESULTS Staff perceptions of chaenges for patients Some staff members reported that triage was making patients anxious. The fact that patients who went on to have appointments with someone other than the person who had triaged them and had to have their history taken twice was fet to be a disadvantage. Patients were sometimes uncomfortabe about discussing medica probems with nurses, which was an unfamiiar thing to do. If they had ongoing probems with which their own doctor was famiiar, triage meant having to te their story a over again from the beginning to someone new. Waiting for ring-back coud be inconvenient, and even stressfu, for patients, especiay if they were in a state of pain or anxiety or if they had things to do get to work, get chidren to schoo and so on. Patients were often not in a situation where they coud be caed back; they were driving, or at work, somewhere noisy or not in a situation of privacy. Mobie phone signas were often poor. Some patients were frustrated by triaging with CDSS. They just said this is absoutey a waste of time, why are you asking me a these ridicuous questions. Yes and a few of them got a bit abusive. Nurse, practice 2 (NT) Some patients hated being asked to te receptionists what was wrong. Staff aso identified patients for whom triage was not and never woud be appropriate. These incuded peope with a previousy unreported anxiety or depression who may have spent days pucking up the courage to ring, and where the patient has mutipe compex comorbidities, as we as patients with hearing disabiities or earning disabiities or dementia or who were very od or confused, who were excuded as per protoco. Some patients just did not ike the triage system in genera. Staff perceptions of chaenges for patients: key themes Some patients were made anxious by triage. Patients had their history taken twice. Some patients were uncomfortabe taking about persona medica matters to a nurse. Waiting for ring-back coud be inconvenient and stressfu for patients. Patients were not aways in a situation where they coud receive a phone ca. Patients did not aways ike to te receptionists why they wanted an appointment. Some patients found the CDSS questions to be inappropriate. Triage may never be appropriate for some patient groups. Patients accounts of their experience of triage There was evidence from patients accounts of a certain probem in reation to tria recruitment in one of the practices where triage meant that there were no advance appointments avaiabe. Patients phoning for an appointment and being tod they coud not book ahead were asked if they wanted an appointment that day. If they said they did, they were incuded in ESTEEM, even athough their reason for wanting a same-day appointment was that no other sort of appointment was avaiabe. One patient to whom this had happened reported being confused about how the appointment system worked, because she had been permitted to book an advance appointment in the past. She had searched for, and faied to find, an expanation of the appointment system in her practice. She found the triage system a very ong-winded way of getting an appointment, particuary as she knew her condition a changed moe woud need to be ooked at. Another patient from this practice practice 1 found the triage system unnecessary, inconvenient and time-wasting, and reported that staff had tod her they did not ike it either. Yeah, we it was siy. It was unnecessary. It was just unnecessary. You know, it was a waste of everyone s time, a waste of my time, a waste of her time. You know, a it did was push the appointment back you know, and, and, it was very inconvenient. Patient 06, practice 1 (NT) 90 NIHR Journas Library

121 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Another patient who aso found triage using CDSS ong-winded expressed concern about whether her edery mother coud have coped with answering so many questions. There were doubts whether GPT adequatey discharged a doctor s duty of care, and the concern that the quaity of diagnosis must be adversey affected because triage reies on the patient being abe to hear and understand what the doctor says on the phone. But I think perhaps, it depends whether the fraternity, and by that I mean the doctors fraternity, fees that they ve discharged the duty of care, I think. Without getting ega, I mean duty of care in so far as he s happy that what he s said is understood and is OK, cos it seems to me that there is some kind of differentiation in the eve of, not ony the eve of service but in the eve of the quaity of the diagnosis if you can t see what it is you are ooking at. Patient 03, practice 6, GPT Patients aso reported the difficuties of waiting for the ring-back when they were unikey to be in a situation where they coud receive a ca. They described having to sit around waiting, not knowing if they had an appointment or not, not being abe to pan their day, not knowing whether they shoud go to work and, if so, whether they needed to request time off for a doctor s appointment. Teing the receptionist what was wrong was a common reason for disiking the system, with the view that such things are too persona to discuss with strangers over the phone with fears about confidentiaity and who ese might hear. There were concerns about how triage woud work for patients with poor communication. Patients accounts of triage: key themes Patients had sometimes not initiay requested a same-day appointment. Some patients were confused about how the new appointment system worked. Some patients considered triage to be unnecessary and inconvenient. There were concerns that certain peope woud not cope with the number of questions emerging from the NT. Some patients found it difficut and inconvenient waiting to be rung back. Patients disiked teing the receptionist the reason for their appointment request. Some patients were concerned about the confidentiaity of the teephone conversation. Acceptabiity of triage There was considerabe variation in the degree to which triage was acceptabe, both between practices and within practices. Accounts of positive experiences of both NT and GPT suggest that triage can, under certain circumstances, be an acceptabe mode of service deivery. However, the extent of accounts of negative experiences and chaenging aspects of triage strongy suggest that triage was not acceptabe for a number of staff and patients across the practices. When the criteria of good communication within the practice, supportive staff reations and fu consutation with staff regarding tria participation were not present, and especiay if there was a cuture of resistance to change, triage was ess ikey to be found acceptabe by staff. For some staff and patients, teephone triage chaenged beiefs about what constitutes good patient care, in particuar the axiom that seeing patients is an essentia component of good, safe care. There was anxiety among staff about adverse events and a fear of itigation. In addition, some staff fet performing triage was not an appropriate use of their skis, and it was aso experienced as both boring and stressfu. Both nursing and reception staff found aspects of teephone triage forced them into a roe with which they were uncomfortabe. Some doctors and nurses were not comfortabe with teephone work. Staff beieved there were patient groups for which teephone triage was inappropriate. Triage was fet to compromise continuity of care, more so where advance appointments were no onger avaiabe, and GPT coud be irreconciabe with a persona ist system. A of these factors impacted on the Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 91

122 RESULTS acceptabiity of teephone triage. Many patients found triage unacceptabe; in particuar, it coud be experienced as an unnecessary barrier to access to the doctor. What infuences the extent to which the teephone triage interventions are seen to work or not work? In this section we summarise materia from the previous sections to consider the issue of acceptabiity and perceived effectiveness. There is no doubt whatsoever that staff perceived that a teephone triage system can work to manage demand, increase patient access and reduce cinica workoads. It is aso cear that this is not inevitaby the case. Athough the data are compex, and do not revea any consistent and predictabe patterns of what might predispose triage to success or faiure, it is possibe to identify factors that are ikey to have an impact given certain conditions. This may be at east hepfuy suggestive, if not definitivey predictive, of what can make the introduction of teephone triage successfu or not. What is seen to work in teephone triage? Teephone triage deivered by GPs or nurses was seen to be effective for reducing both appointments and consutation times, and therefore for rationaising resources. Teephone triage deivered by nurses can work we. It can be perfecty effective and acceptabe to both practice staff and patients, and can be fet to reduce doctors appointments. In this process evauation of the eight intervention practices studied, the practice whose teephone triage was perceived to significanty reduce doctors appointments more than any of the others was a NT practice. NT was fet to potentiay create a sustainabe ong-term reduction in doctors appointments by teaching sef-management skis to patients and the conditions in which a doctor s appointment is and is not necessary. NT is ikey to be more effective when performed by a nurse practitioner or a primary care nurse, i.e. a nurse with experience of history-taking and diagnosis, and who can prescribe, but these are not necessary preconditions for success. It may be more ikey to effectivey reduce doctors appointments when nurses can prescribe certain medications for commony presented conditions. Using CDSS can be a usefu training too for nurses earning triage, athough some nurses are aready experienced at taking histories, diagnosis and assessing appropriate treatment options, and some nurses fee it increases the amount of time taken to triage. Triage training and experience can encourage more effective triage by both nurses and doctors. Nurses are ess ikey to have experience in history-taking, diagnosis and treatment, and are ikey to require a onger induction period. If there is aready a cuture of triage and teephone consutation in the practice the effectiveness of the introduction of a more widey appied triage system wi be increased and expedited, as both cinicians and patients are accustomed to a remotey deivered service. Initia resources need to be devoted to setting up an effective triage system that wi produce resource savings when estabished. Staffing needs to be appropriate. Good effective communication within the practice, with forma and informa opportunities for airing and sharing probems and successes, predisposes the practice to more successfu introduction of triage. A preparatory assessment of needs and capacity predisposes practices to more successfu introduction of triage, and a run-in period to practise the intervention is ikey to be hepfu, as it was during the ESTEEM tria. Commitment to the change by a stakehoders is essentia, and shoud be estabished before triage is set up, and a champion to drive it forward and address probems is hepfu. Triage can improve access, and may be more acceptabe at times when access has been poor in the past, athough previous good access may create easier and more trusting reations between staff and patients. The opportunity to hear the experience of others who have successfuy introduced the innovation can inspire and motivate staff to drive the change, but it is probaby important that this is deivered by peope without an overt commitment to a particuar mode that may not be the most appropriate for the practice. 92 NIHR Journas Library

123 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 What is seen not to work in teephone triage? Many of the aspects of triage which were perceived not to work were a function of the tria rather than triage per se. An exampe is randomisation to an arm that is perceived as ess effective than the system in pace before the tria, for exampe when effective face-to-face NT was repaced by ess effective GPT. Randomisation is not ikey to be an effective way of introducing a major organisationa change. Another exampe is when practices were aready effectivey managing their demand for same-day appointments so that there was no perceived need for change. It is axiomatic that modes of triage that do not evove from the organisationa cuture and framework of a particuar practice, and particuary from a perceived need for change, are ess ikey to work, at east in the short term. In some practices the timing demands of the tria enforced the introduction of triage at times that were fet to be inappropriate, such as during a fu epidemic. Impementing triage is aso more probematic in practices with authoritarian hierarchies and poor staff communication, in which there are no conditions for formay or informay discussing concerns and experiences. In reation to whether or not NT is seen to work, it is worth noting that some nurses are frequenty nervous about triage, which may ie outside their comfort zone of experience and training, especiay if they have no diagnostic experience and rey on CDSS. Nurses embark on teephone triage from a very different starting point from doctors, and embedding NT into practice is ikey to take onger than perhaps a GPT system might, and require additiona preparation, support and resourcing. Launching straight into a NT system in the absence of this preparation is unikey to be as successfu as a carefuy managed change. NT is ess ikey to be successfu if practices do not support nurses by ensuring that there is sufficient nurse capacity to undertake this new roe, as we as performing their usua roes (such as chronic disease management, vaccination programmes and management of dressings). Nurses need to be proficient in IT, which was not aways the case in this tria. Athough it was regarded as a usefu training too by some nurses who had no triage experience, the CDSS that was used (which was designed for the different demands of out-of-hours primary care services) was judged by some nurses to be unsuitabe for use in daytime primary care triage in a number of ways. Using CDSS may ead some nurses to fee that their judgement is undermined with regard to when an appointment is necessary. This may increase the numbers of patients advised to attend for a face-to-face appointment, as CDSS errs on the side of caution and may aso cause a strain on practice reationships. It was seen by some to be ong-winded and seen as increasing the time taken to triage. Doctors without experience and training in triage can ack confidence and take too much time to triage, and doctors with no experience in or desire to practice teephone consutation are ess ikey to be efficient at triage or to have a commitment to its success. For a few doctors, teephone triage is an insut to their mode of good and safe care, and as such can represent an existentia chaenge to their identity as good doctors. In addition, in the ESTEEM tria, the triage mode initiay promoted by Productive Primary Care (and sometimes confused with the ESTEEM protoco) was not whoy compatibe with triage within ESTEEM, and this may have confounded and confused the deivery of triage in the tria. Superimposing a predesigned (and quite extreme) mode of triage such as the Advanced Access mode onto an organisation without carefu consideration of specific aspects of that organisation is unikey to work, and practices that tried to do this were forced to abandon the attempt. In the Productive Primary Care mode receptionists do not have appointments to offer, and advance appointments cannot be made. Both staff and patients at practices who tried to impement this, as we as those in which oss of advance appointments was an adverse effect of the need to prioritise appointments for the triage system, found this uncomfortabe. It can spoi good practice or patient reationships and can be embarrassing and disempowering for receptionists. It excudes patients who, for whatever reason, find teephones inaccessibe. It is a barrier to continuity, as patients can no onger book ahead with their own doctor who knows their history, thus undermining the reationship of trust between doctor and patient. The Advanced Access system was advocated as a response to the Department of Heath target that patients shoud be abe to see a doctor within 48 hours 66 but was modified, as recommended by Department of Heath poicy in 2004, foowing a pubic confrontation with the Prime Minister by a disgrunted patient in Pope et a. 67 highight a number of pubications which identify probems with the acceptabiity of Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 93

124 RESULTS Advanced Access to both patients and cinicians, reated to its adverse impact on doctors workoad, continuity of care and patients. There is no one approach to performing GPT which is appropriate for a or most practices. Effectiveness depends on a number of contingencies, for exampe whether there is a usua doctor system, individua doctors confidence and competence, whether doctors have prior triage experience and/or experience of teephone consutations, and whether their mode of good patient care and safe practice encompasses a commitment to seeing patients face to face. Teephone triage is more acceptabe to some types of patients than others, so practice catchment and popuation wi infuence effectiveness. Success wi aso reate to how effective a system for providing same-day appointments exists before a triage system is introduced. GPT can increase workoad and exacerbate workoad disparities between doctors in a practice. Ring-back issues are a constant chaenge in teephone triage. Deay in ring-back, patients not being abe to receive cas, patients who cannot be reached and poor mobie phone signas can make teephone triage ess effective, frustrate both staff and patients, and waste time. Practices in which triage was organised in a way that meant patients were waiting for ong periods, sometimes hours, for ring-back were doing their patients a disservice and undermining the success of teephone triage as an efficient system. Strategies for timey ring-back shoud be prioritised. Simiary, noise, either in the practice or in the patient s home, can compromise the safety and effectiveness of teephone triage. Practices need to provide a quiet space for triaging, and receptionists coud advise patients waiting for ring-back that they wi need to be in a pace of privacy and quiet. There are safety and confidentiaity issues in ringing peope when these conditions cannot be fufied. Patients shoud be assured of privacy and confidentiaity when they are rung back; there was concern among patients about who might be istening in. There are certain patient groups for whom triage is inappropriate, for exampe some peope with menta iness, substance abuse issues, earning disabiities, dementia, hearing disabiities or who are very od, who were excuded from ESTEEM. Poicies need to be in pace to exempt such patients from a triage system. 94 NIHR Journas Library

125 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Chapter 4 Discussion and concusions Summary of findings Our findings show that introducing either GPT or NT resuted in an increase in the number of primary care contacts in the 28 days foowing a patient s request for a same-day consutation with a doctor when compared with practices usua processes for handing such requests. Triage was associated with an overa reduction in GP face-to-face contacts. However, GPT resuted in an increase in GP teephone and face-to-face contacts when combined together, whereas NT resuted in a reduction in this measure. These changes refect redistribution of GP workoad from face-to-face to teephone consutations foowing the introduction of GPT, and redistribution of workoad from GPs to nurses foowing the introduction of NT. However, when considering differing patterns and duration of patient care across a three arms, there was no difference in 28-day heath-care costs of patients. Triage appeared safe, and no differences in patient heath status were observed across the arms. NT was associated with a sma reduction in patient satisfaction compared with GPT or UC. Primary care contacts in the 28 days foowing a same-day consutation request Our primary outcome was the number of contacts undertaken in primary care settings (GP practices, out-of-hours primary care services, wak-in centres and A&E departments) in the 28 days foowing a same-day face-to-face GP consutation request. Our findings in reation to the primary outcome are cear. Impementing triage, whether GP- or nurse-ed, resuted in additiona, not substituted, workoad. In UC, 51% of patients received ony one contact across the 28-day foow-up period foowing a same-day consutation request. In contrast, this proportion was reduced to 23% in GPT and 12% in NT. In this study, 16,211 patients generated 39,736 consutations with a cinician in the 28 days foowing a same-day consutation request. Of these consutations, 22,471 (57%) took pace on the index day. Across the 28-day period, ony very sma numbers of patients (550 in tota) were seen in A&E and there was no evidence that either form of triage increased or reduced attendances at A&E. Our main ITT anaysis demonstrated that GPT resuted in a reative increase in the rate of primary care contacts across the 28-day period of 33% when compared with UC; the equivaent increase foowing NT was 48%. A sma increase (4%) in the rate of contacts was observed when NT was compared with GPT. As to be expected, per-protoco anaysis of the primary outcome demonstrated an intensification of the treatment effects of both GPT and NT. The findings of the main ITT anaysis were robust to the use of imputed data for cases where primary outcome data were not avaiabe. In recognition of the inevitabe need for a proportion of patients to be seen face to face foowing triage, we undertook a sensitivity anaysis in which we combined a within-practice contacts on the index day as just one contact. This anaysis demonstrated an increased rate of contacts in GPT of 10%, and in NT of 12%, when compared with UC. Interpreting the ITT anaysis of the primary outcome in conjunction with the above sensitivity anaysis, it is evident that there was an increased rate of contacts over the 28-day period in both triage arms compared with UC. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 95

126 DISCUSSION AND CONCLUSIONS Additiona sensitivity anayses were undertaken, based on dividing the primary outcome chronoogicay into contacts taking pace on the index day ony ( day 1 ) and those taking pace during the remaining 27 days of the 28-day foow-up period (days 2 28). Considering ony contacts taking pace on the index day, the rate of contacts in GPT compared with UC was increased by 51% and in NT by 72%, argey attributabe to the teephone ca undertaken in triage. There was an increased rate of contacts in NT compared with GPT of 14% again, argey attributabe to the teephone ca undertaken in triage. The increased rate of contacts in the triage arms was greater on the index day than on the subsequent foow-up days. Care on the index day Numbers of contacts and disposition of patients In UC, a substantia majority of patients (87%) received a GP face-to-face contact, with no further primary outcome contacts on the index day. Other patient pathways were represented by substantiay smaer proportions of patients. In contrast, patients in the triage arms had more diverse patterns of patient management when compared with UC, possiby refecting a more fexibe approach to patient assessment and management in the triage arms. Where GPT was impemented, 46% of patients received ony the GP triage contact on the index day, and 36% received a GP face-to-face consutation on the index day foowing the initia GP triage contact; a smaer proportion (9%) of patients had a nurse face-to-face consutation on the index day foowing their initia GP triage contact. A sma proportion (6%) of patients received a GP face-to-face consutation on the index day instead of being managed under the triage system. Where NT was impemented, 22% of patients received ony the nurse triage contact on the index day, and 56% of patients received a GP face-to-face consutation foowing the initia nurse triage contact; 9% of patients received a nurse face-to-face contact foowing nurse triage. For a sma proportion of patients (9%), their first contact on the index day was a GP face-to-face consutation instead of being managed under the triage system. Distribution of cinician time in practice Considering workoad from the perspective of the GP, the introduction of GPT was associated with a 55% reduction in the rate of GP face-to-face contacts on the index day (compared with UC), athough there was a 49% increase in the rate of GP overa teephone and face-to-face contacts combined. Across the whoe 28-day foow-up period, the reduction in the rate of GP face-to-face consutations was smaer (39%), refecting the deferra of some workoad from the index day to the foow-up period, whereas there was a 38% increase in the rate of GP teephone and face-to-face contacts combined. Introduction of NT was associated with a 31% reduction in the rate of GP face-to-face consutations on the index day (compared with UC), and a reduction of 28% in the rate of GP teephone and face-to-face contacts combined. Across the whoe 28-day foow-up period, there was a 20% reduction in the rate of GP face-to-face consutations, with a 16% reduction in the rate of GP teephone and face-to-face contacts combined. In considering these observations regarding GP contacts, it is worth noting that the introduction of GPT invoved GPs undertaking the work of triage in addition to the (reduced) number of face-to-face contacts; in contrast, where NT was introduced, nurses, not GPs, deivered the triage eement deivering the resuting gain to GPs in reduced numbers of GP face-to-face consutations. 96 NIHR Journas Library

127 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Taking account of both the number and type of contacts in the practice (whether with a GP or nurse, face to face or teephone) and their associated durations provided additiona insight into cinician workoad, which were somewhat different from those based soey on the number of contacts. In UC, the estimated duration of patient cinician contact time within the practice on the index day (based on the first two contacts within the practice) was 9.6 minutes, compared with 10.3 minutes when GPT was impemented, and 14.8 minutes where NT was impemented. The distribution of the time by cinician was, however, markedy different across the three arms. In UC, the substantia majority (9.1 minutes) of the 9.6 minutes estimated contact time on the index day comprised contact with the GP. Where GPT had been impemented, the estimated GP contribution to the overa workoad was 9.0 out of the 10.3 minutes, with nurse contact accounting for 1.3 minutes, the atter usuay resuting from nurse face-to-face contact. Where NT had been impemented, of the tota 14.8 minutes of estimated contact time provided on the index day, overa, around haf of this was with GPs (7.7 minutes), with the remainder (7.1 minutes) with nurses. Resource use Overa, the substantia majority of service use in a three arms took pace in GP practice settings. Ony very sma numbers of contacts occurred in A&E, wak-in centres or out-of-hours primary care services over 28 days. The simiarity of the rates of contact with non-practice-based services across arms provides no evidence to suggest that triage encouraged patients to seek care outside the practice. Overa, we observed ow rates of patient non-attendance compared with that reported in other studies, 2 thus there was no evidence that triage may advantage practices by reduced non-attendance rates. There was aso no evidence of differences in patient sef-reported use of NHS Direct between tria arms around 4 6% of participants reported using NHS Direct in the 28-day foow-up period. Economic anaysis Overa, costs incurred were very simiar across a three arms across the 28-day foow-up period. Costs incurred on the index day were observed to be ower in both triage arms; the projected cost saving reating to care on the index day (compared with UC) was approximatey twice as much in GPT compared with NT, athough overa the absoute differences were modest ( 5.75 vs per patient). Athough there is a difference in the number of contacts by triage arm over the 28-day foow-up period, mean costs for primary care contacts (primary outcome) are simiar for UC, GPT and NT. This indicates that the costs associated with triage are offset over the 28-day foow-up period, with the added cost of triage resuting in fewer GP face-to-face consutations in practice when patients initiay request a same-day GP appointment. The main area of contact, and cost, is associated with GP consutations in the practice, representing amost 90% of the costs in the UC arm, and data indicate few contacts with out-of-hours primary care, wak-in centre or A&E compared with the eve of contact with the GP in a primary care setting. Estimates of intervention cost for the triage interventions are an important factor in the cost anayses. Estimates of the intervention cost for triage contacts are presented in a transparent manner, and foow the methodoogy for unit costs in heath and socia care used by the PSSRU, 38 which are widey accepted and used within heath service research. The main cost component for triage contacts is the GP or nurse time associated with the triage contact, and this has been coected at a participant eve across the ESTEEM tria, providing good quaity data on which to estimate unit costs. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 97

128 DISCUSSION AND CONCLUSIONS Base-case cost estimates, over the 28-day foow-up period, used pubished unit costs for primary care contacts and A&E contacts. These costs refect the opportunity cost associated with heath-care resources; however, these cost estimates may be open to some imitations. Therefore, sensitivity anayses have been presented, in estimates for the triage contact unit costs and for the estimates of 28-day foow-up cost, to address uncertainty in the use of pubished unit costs. Where different assumptions have been used for unit costs for heath-care contacts, we specificay used ower unit cost estimates that did not incude components of cost associated with the quaification costs for GP or nurse contacts, and did not incude costs associated with the direct support staff for GPs. Athough we saw much ower overa estimates of 28-day costs, we did not observe differences between intervention arms. In base-case anayses, as per the prespecified anaysis pan, we used pubished unit costs for GP contacts. In so doing, we used an assumption that within-practice GP consutations have a mean duration of 11.7 minutes, 38 this being based on an aocation of GP consutation time over an estimated number of GP in-surgery consutations. However, the ESTEEM tria has considered use of triage in the subpopuation of primary care patients for whom a same-day request is made to see a GP. Tria findings indicate that the mean time for a within-practice GP consutation is shorter (estimated 9.5 minutes) than that derived for cost estimates across the broader GP patient popuation. The difference in the ESTEEM tria popuation and the data reported for the broader primary care patient group (a difference of 2.2 minutes, a difference of 8.14 at base-case cost assumptions), if supported by future research, may infuence the interpretation of findings in the cost anayses presented, showing UC to be ess costy than the GPT and NT intervention arms. This woud be the case even where the difference in time/cost is smaer, given the estimate for the cost per minute of GP time and the reativey sma tota costs being compared. It is difficut to know from the tria if this estimated mean GP contact duration (UC) of 9.5 minutes wi hod in a arger sampe or if GP consutations in the two triage arms woud have been equay different (from expected mean of 11.7 minutes). New data are required to inform this issue, as these research questions were not specified in the ESTEEM tria. One hypothesis may be that a proportion of contacts among the requests for same-day GP consutations are indeed reativey short and coud be managed via a triage teephone contact, and it is these contacts (when presenting for a face-to-face GP consutation) that drive a mean duration of 9.5 minutes that is ess than expected. What we may be seeing in the triage interventions is that ess compex presentations are being managed via teephone triage, and represent those contacts that are resoved at triage eve and do not need a further primary care contact. However, an aternative expanation may be that the data from ESTEEM on mean contact time are not comparabe to those used to estimate the unit costs for GP consutations reported by PSSRU, 38 and the timings recorded in the ESTEEM trias may need to be adjusted to incude an additiona aowance for non-contact activities during the consutation time aocated for GPs. Safety Triage appeared safe. There was no evidence of excess hospita admissions within 7 days or A&E attendance within 28-day foow-up period in either triage arm when compared with UC. There was aso no evidence of excess deaths within 7 days of the index consutation request. Patient heath status At the point of competing a questionnaire (around 28 days foowing the index consutation request), patients reported simiar eves of resoution of the origina probem in a three arms. Interestingy, around 53 59% of patients across the three arms reported being much better by the time they returned a questionnaire. Thus, around 45% of participants reported significant residua issues reated to their origina consutation request by the time they returned a questionnaire. Perhaps not surprisingy given these observations on probem resoution, there were no differences in sef-reported heath status by tria arm, as measured by the EQ-5D. Triage, whether GPT or NT, appeared to achieve simiar heath outcomes compared with UC. 98 NIHR Journas Library

129 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Patient experience of care Athough, overa, patients experience of care was met with high eves of satisfaction (in the order of 90% being very or fairy satisfied), patients in the NT arm were somewhat ess satisfied than those in the GPT or UC arm. Patients reported that it was easier to get through to the practice on the phone in practices impementing GPT in comparison with UC, and that it was harder to get prompt care in NT by comparison with both GPT and UC. Patients aso reported that it was more difficut to see a doctor or nurse if the patient wanted to do so in NT, but not GPT, compared with UC, and that it was harder to get medica advice in NT compared with both GPT and UC. NT was reported as being ess convenient than either UC or GPT. In summary, NT appeared to be somewhat ess acceptabe to patients than either GPT or UC. There was no significant difference in the acceptabiity to participants of GPT when compared with UC. GPT overa appeared as acceptabe to patients as UC. Research findings in context The ESTEEM tria was deveoped in response to the ack of substantia research evidence concerning the cinica effectiveness and cost-effectiveness of teephone triage in genera practice. No substantive new research examining the effects of teephone triage for acute care on in-hours primary care workoad, resource use, safety or patient experience has been reported during the course of this tria (athough new evidence on the safety of teephone triage in out-of-hours care may be pertinent 68 ). Much of this iterature is summarised in a systematic review by Bunn et a. 27 undertaken in Athough there was heterogeneity in reported outcomes, key findings of that review indicated that teephone triage or consutation with either a GP or a nurse reduced the number of face-to-face contacts with GPs. Teephone triage was safe, and patients found triage to be acceptabe overa. However, methodoogica faws were noted in many of the incuded primary studies, hence caution was urged when interpreting findings. Our findings contribute important data that extend the existing evidence base around patient experience and the safety of teephone triage and, we beieve, provides definitive answers to questions around primary care contacts, workoad and NHS resource use. Workoad Genera practitioner-ed teephone triage has been promoted by the NHS Institute for Innovation and Improvement as a mechanism for improving patient access and reducing GP workoad. There has been an increase in the uptake of teephone triage systems despite the ack of a robust evidence base especiay evidence on the impact on genera practice and wider primary care workoad. Our findings are nove and suggest that teephone triage of patients seeking a same-day appointment increases primary care workoad (in terms of number of contacts) within the 28 days foowing the patient s consutation request, but with no observed changes in overa cost when compared with UC. A key finding in this study, however, is that any change in workoad is restricted to genera practice: GPT or NT did not appear to resut in overfow into other primary or secondary care services. We found no difference between arms in the number of out-of-hours primary care contacts or A&E attendances. Inconsistent messages have emerged from past research. For exampe, McKinstry et a. 13 found no difference in out-of-hours primary care contacts foowing GP teephone and GP face-to-face consutations. However, other research suggested that NT might ead to significanty more out-of-hours primary care and A&E contacts than UC. 3 However, because the majority of contacts occurred in genera practice, our data provide an important proxy for workoad within this context. Past research has shown that teephone triage or consutation by a GP or a nurse was associated with a decrease in same-day GP workoad: the number of same-day GP appointments being reduced to between 39% and 44%. 3,11 However, reconsutation rates within the few weeks after teephone consutation have been shown to increase by a simiar magnitude, around 30 50%. 3,13 Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 99

130 DISCUSSION AND CONCLUSIONS Our findings confirm that a triage system for same-day requests reduces GP face-to-face appointments. Across the 28 days foowing the initia same-day request there were, on average, 1.46 GP face-to-face contacts per patient in UC. This reduced by 37% to ess than one GP appointment (0.92) under a GPT system and by 18% (1.19 average appointments) under a NT system. However, any time saving for GPs due to reduced face-to-face contacts over the 28-day foow-up period woud have been offset, to some extent, by the increased time spent performing triage in GPT. The majority of primary care contacts foowing a same-day consutation request occurred in the practice on the day of the consutation request. GPs undertaking phone care have been reported to definitivey manage around 29% of same-day requests 11 and nurses around 26% in this first pass. 3,69 In our study, in the GPT arm, 21% of patients managed per protoco had no further contacts foowing triage the equivaent figure for NT was 8%. A key issue to consider, however, is the patient s overa cinician contact time taking into account the number, type (defined by cinician and mode) and duration of contacts. We found that both triage systems increased cinician contact time reative to UC. Simiar findings have been reported with respect to NT; 3 even though there was an increase in overa cinician contact time, there was a reduction in GP contact time, reative to UC, a reduction which was offset by increased nurse contact time. Our findings mirrored this for NT but do not show any substantia time-saving for GPs foowing GPT. Rather, GP contact time foowing GPT was ony sighty ess than that estimated for UC, athough there was an important redistribution of patient contact time with increased use of nurse input. It might have been assumed that face-to-face consutations woud be shorter foowing triage, on the basis of the history having aready been documented during the triage ca. However, we observed that in both triage systems, the duration of subsequent GP face-to-face consutations were, in fact, onger than index consutations in UC. A triage system is designed such that the sickest patients are assessed and may be seen face to face in the practice subsequent to triage. This particuar subgroup of patients, i.e. those with data on a subsequent GP face-to-face consutation on the day of or day after the index triage consutation, are ikey to have more presenting probems and greater heath compexities. 12 A triage system might offer fexibiity to ensure that patients with compex heath needs are provided with the onger consutations necessary. Safety Bunn et a. 27 did not examine patient safety in detai there was imited evidence in that review to provide a comprehensive assessment on the safety of triage. To date, the message is mixed. For exampe, RCTs comparing teephone consutations with face-to-face consutations found no difference in the number of A&E attendances 13,70,71 or hospita admissions. 70 Two RCTs comparing triage by GPs and nurses in out-of-hours primary care settings found no difference in hospita admissions. 5,15 Richards et a., 3 however, found an increase in A&E admissions foowing nurse teephone triage in genera practice. 3 McKinstry et a. 72 suggested that teephone consutations may be ess ikey to incude sufficient information to excude important serious inesses and therefore were potentiay more ikey to compromise patient safety. Existing evidence specificay reating to NT suggests that it is safe. Richards et a. 20 conducted research to assess the quaity of information-gathering and accuracy of decision-making by nurses undertaking teephone triage. 20 Independent GP and nurse assessors rated audio-recordings of nurse teephone triage consutations. Out of 218 consutations ony 7 (around 3%) were rated as potentiay unsafe, suggesting that, overa, nurse teephone triage is safe. Research into nurse teephone triage in out-of-hours primary care settings aso suggests that triage is safe. A recent systematic review by Huibers et a. 68 noted that for observationa studies (where rea patient outcomes were assessed) triage was safe in 97% of a patients contacting out-of-hours primary care and in 89% of patients with high urgency. 100 NIHR Journas Library

131 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Like most of the studies noted above:... our tria was not powered to inferentiay test safety outcomes and we cannot concusivey rue out differences between study groups. In view of this factor and of other evidence [ 20,72 ] caution is needed before firm concusions can be drawn from these resuts, and further studies, possiby incuding different study methods such as significant event audit, might provide usefu additiona evidence about the safety of triage. Campbe et a. 64 Patient experience There is imited evidence on the acceptabiity of teephone triage in primary care. 27 Generay, patients have found teephone consutations with GPs and nurses to be acceptabe, 11,13,70,73 75 athough one systematic review incuded primary studies that, overa, showed patient dissatisfaction with teephone consutations. 76 Our findings are consistent with past research with respect to GPT. 11,13 NT, on the other hand, was somewhat ess acceptabe than both GPT and UC. Perhaps this shoud not be surprising given that patients originay caed the practice wanting to see a GP, but were in fact offered a teephone consutation with a nurse considered in that context, the sma reduction in satisfaction we observed seems predictabe. Process evauation A great dea of information that is interesting and usefu has emerged from the process evauation of the ESTEEM tria. However, as the previous sections have demonstrated, triage has not emerged as a phenomenon with predictabe patterns from which rues and guideines can be produced which woud be appicabe regardess of context. How triage was operationaised, how it was experienced, its acceptabiity, and whether it worked for practices, staff and patients emerged as highy contingent, idiosyncratic and variabe. The findings in this process evauation echo those of a quaitative study of the Advanced Access mode of teephone triage, 67 which aso found wide variance in interpretation and impementation of the mode, and that informa organisationa behaviour resuted in its adaptation to practice contexts, norms and vaues. Our process evauation was undertaken within the context of deivering a RCT; it was not triage aone that was being evauated, but rather triage introduced and deivered under tria conditions. This adds another compication to interpreting the findings. The demands of the tria introduced confounding factors, additiona pressures and strictures to be negotiated at the same time as introducing a major organisationa change. A further difficuty is that our quantitative data were coected at a particuar point in time. In amost a cases this was fairy soon after the tria began, when teephone triage was new and, for most practices, chaenging, depending on peope s individua, coegia and institutiona situations. Yet innovations take time to sette down. How a phenomenon is experienced is mediated by the dynamics of time and experience and by its gradua normaisation. Findings may have been different if data had been coected at a different stage in the process, perhaps when staff had had time to become more accustomed to teephone triage and fet more competent and confident. Indeed, by the time process evauation fiedwork was undertaken, participants were aready contrasting what they or others fet at first with how they fet at the time of the interview. Perhaps the most striking finding from this extensive quaitative study is that there is no strong and compeing narrative about what works and what does not work in reation to teephone triage in primary care. There are no predictabe patterns of causaity from which messages about effectiveness can be abstracted in reation to this particuar organisationa change. Athough GPT works to manage demand for some doctors in some practices, other doctors, sometimes in the same practices, see themseves transformed by triage into gorified booking cerks with an even higher workoad. Simiary, whie NT works in some practices it is not we regarded esewhere. One can interrogate the data in vain in search of causa expanations. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 101

132 DISCUSSION AND CONCLUSIONS Where NT was judged not to have been successfu, we witnessed evidence of poor communication practices. In one particuar setting, there was anxiety and uncertainty on account of recent substantia medica staff changes. In addition, difficuty with impementation was sometimes observed to be associated with poor perceptions of pre-existing appointment systems, probems in interna staffing and reationships (incuding poor reationships with patients), ack of consensus about tria participation among the practice team, and questions about the motivation to participate in the study. On occasions, the process evauation team observed that reationships between staff and patients appeared strained and that some patients had to pay phone charges to make appointments. In one setting where NT was judged not to be successfu, no extra resources had been provided to the nurses performing the triage, and nursing staff fet that the doctors had not discussed the tria with them during the course of impementation. However, many of the same factors a chaotic appointment system before the tria, poor communication within the practice, continuity issues and no additiona nursing resources prevaied in other NT practices, at which NT was judged to have been successfu, to have reduced doctors appointments by haf and to have been experienced positivey by most staff and patients. How can this be expained? One approach woud be from the theoretica framework of compexity theory. Here innovation within organisations is seen as an organic and adaptive process in a continuous state of fux and adaptation as the organisations adapt and innovations interact with each other in perpetua iteration. 77 Such an approach rejects the notion that reationships of cause and effect can be identified by breaking the organisation and the innovation down into component parts. 78 Instead, heath-care organisations are understood as compex adaptive systems, which can be made sense of ony by understanding the interpay of reationships within microcontexts that are not controabe by an externa agent. Methodoogies that enabe the capture of microcontextua factors in time and pace, such as ethnographic, ethnomethodoogica and inguistic approaches, are recommended to compement the more traditiona approach to gathering information of the meaning and experience of this intervention for participants. Process evauations and other research into organisations may be enriched by a more diverse methodoogy. The findings from the process evauation demonstrate the compexity of primary care organisations, and the significance of individua practice cutures. No widey appicabe causa concusions about what works and what is acceptabe have emerged. The process evauation has provided rich data on the experience of different modes of triage, and how these modes were adapted to oca circumstances. Our findings demonstrated that both modes of triage were sometimes experienced positivey by staff and patients. It has aso raised issues that any practice considering impementing triage might want to refect on before proceeding. Strengths and imitations The ESTEEM tria has a number of strengths. The tria was a rigorousy conducted muticentre custer RCT, using both quantitative and quaitative methods and in fu conformity with the CONSORT guideines. 57 It was a definitive study, incuding a arge number of both practices and patients with few excusion criteria and a spread of geographica ocations, increasing the externa vaidity of the tria and the generaisabiity of the findings. Athough we succeeded in recruiting arge numbers of practices and patients from diverse geographica ocations, enhancing the generaisabiity of our resuts, our findings may be ess appicabe to practices serving popuations with greater ethnic diversity, or those ocated in inner-city areas with very high eves of deprivation. Additionay, the organisation of the practices to run the interventions in three waves aowed us to undertake the tria over a spread of seasons over 2 years. The study used a remote web-based randomisation system for the aocation of practices and we conducted the anayses according to a predefined anaysis pan agreed with the Tria Steering Committee. The tria was fuy powered on a prespecified primary outcome and we achieved our recruitment target. The custer randomised design prevented contamination. 102 NIHR Journas Library

133 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Our assessment of feasibiity was based on a conservative estimate of just 20% of patients requesting same-day consutations. In the event, and athough based on rather crude data (see Tabe 8), around 40% of patients probaby presented same-day consutation requests, more in ine with reevant pubished iterature 4 (of which we were aware but had considered as an estimate ony). The design of the study to incude a piot and feasibiity phase with a buit-in observationa study prior to the main tria phase proved invauabe for deveoping the tria materias and procedures to be roed out across the four centres. PPI contributed to both the deveopment of patient recruitment materias and membership of the Tria Steering Committee. Our origina patient recruitment target in gaining access to the primary outcome data was exceeded owing to the introduction of a process, borne out of the piot study, invoving an initia verba consent foowed by a written communication giving patients the opportunity to opt out. We were gratefu for the understanding and support of the regiona REC in this regard, and see this understanding as the demonstration of criticay important insight for future simiar research initiatives. With this amendment, we were abe to document our primary outcome in 77% of tria participants, rather than the 53% we originay hoped to seek. Despite the chaenge of the coection of our primary outcome data being undertaken by different study personne across the four centres, we repeatedy demonstrated a high eve of agreement between the four tria researchers with respect to the coection of primary outcome-reated materia extracted when undertaking the case note reviews at different study time points. A further strength of the study is the parae process evauation undertaken to provide context to the main outcomes for the study and to assess the reasons why teephone triage is seen to work or not work. Practica chaenges were encountered in conducting ESTEEM. Recruitment of genera practices to this compex tria was a particuar chaenge. Even when recruited, practice retention notaby in NT proved probematic with 12 out of 54 practices withdrawing from the study, 10 of these having been randomised to deiver NT. Time and practica constraints of operating the tria imited our abiity to exacty match repacement practices on the stratification variabes of practices that chose to withdraw. We did not have a arge poo of substitutes from which to seect repacements (sometimes ony one repacement was avaiabe). It is important to note that no practices withdrew once the intervention had begun and ive data coection estabished; our sense was that practices fuy appreciated the size of the chaenge of deivering NT ony when forced to consider the practica reaities of staffing, nurse avaiabiity and system redesign foowing randomisation. However, there were considerabe time and cost impications of practices deciding to withdraw after having undertaken training and preparation work. Considerabe care was expended to ensure that repacement practices remained bind to their aocation right up to and incuding the point at which they agreed to participate, thus ensuring aocation conceament. Given a of these considerations, we thus fee that our methods of repacing practices, athough pragmatic, did not introduce bias. Our estimates of patient eigibiity at entry to the tria were based on data from Receptionist s Log Sheets. This proved a chaenging part of our data coection; reception staff members were observed to be extremey busy, and we have no reiabe objective measure of tota receptionist workoad (and thus can ony provisionay estimate the proportion of eigibe patients). The proportion of patients excuded from the tria was observed to be highest in NT, raising the possibiity that reception staff in some way provided an informa fiter with respect to recruitment to the tria. Being aert to this, we invested substantiay in this area, conducting a series of integrity checks prior to ive data coection to ensure that at east 75% of eigibe patients were receiving the reevant intervention. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 103

134 DISCUSSION AND CONCLUSIONS The tria paced a considerabe burden on practice staff, incuding reativey short but intense periods of competing data coection forms and iaising with the research team. The NT practices experienced the burden of incorporating CDSS into their daiy work. In considering the fideity of the tria a key issue to consider is what dose of triage was actuay deivered. We did attempt to carify the extent to which the CDSS had been used within the nurse teephone triage process (see Appendix 14). For exampe, we expored the number of questions that had been used and the duration for which the CDSS was open during the course of a consutation, and derived a measure of intensity of CDSS use combining the number of questions asked and the duration of the consutation. These are areas that woud benefit from further investigation an important consideration woud be the possibiity that NT was seen to work or not to work without having a fu or cear understanding of the extent to which CDSS was used or not used. In addition, the deiberate recruitment of practices with itte or no triage experience to undertake the tria in order to assess the impact of orchestrating a triage system may have meant that ack of confidence in triaging or organising the system, combined with the reativey short-term commitment to compete the tria, may have infuenced the way the interventions were impemented within practices. We identified a potentia issue regarding practices running triage for a reativey short data coection period (approximatey 5 weeks, or unti the patient recruitment target was reached) and our proposed 28-day foow-up period for these patients. Patients entering the tria in the initia weeks of the data coection period may have had subsequent care within 28 days that was managed under the triage system (i.e. subsequent care deat with by triage if further same-day consutations were requested, possiby for the same probem), athough patients entering towards the end of the data coection period woud do so ony if the practice opted to continue the triage arrangement beyond the tria. If a difference was observed between arms in the tendency to revert to the previous UC arrangement after the end of the data coection period, there woud be concern over whether this woud introduce bias to the primary outcome. It woud not have been possibe to fund the resutant NHS excess treatment costs of a triage practices to continue triaging for the whoe of the foow-up period. As such, we requested that a triage practices revert to their UC arrangements after the end of the data coection period, at east unti the fina outcome data were measured. A but two of the triage practices agreed to do this; however, we have taken this into account within our sensitivity anayses and found it to have no effect on our resuts. When compex interventions, such as we have outined, might be introduced across the wider heath-care system, it is of importance to recognise the need for a reasonabe ength of foow-up before providing a determination of success or faiure with respect to the intervention itsef. Taking a view of resource use just with respect to the index day woud have ed to very different concusions from the broader 28-day view that we took with respect to determining tria outcomes. On a reated matter, it woud have been of interest and potentia importance to undertake a onger period of foow-up within practices but we fet that this was not practica on account of the uncertain nature of any benefit for patients or practices, and of the very substantia organisation impications incurred by practices across a their systems when introducing triage or agreeing to participate in the research. Ideay, however, a foow-up period of 6 months or 1 year woud have been usefu and it is possibe that this may have ed to different concusions, either supporting or opposing the use of triage systems. In coection of our primary outcome data, it was not possibe for the tria researchers to be binded, as the tria researchers provided training and support to practice staff during the tria. However, a tria anayses were undertaken by an independent statistician who was bind to practice aocation and foowed a statistica anaysis pan that was eaborated a priori. 104 NIHR Journas Library

135 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Impications for future research The core design invoved a piot and feasibiity phase, with cear stopping rues eaborated a priori, before moving to the fu tria phase. The piot phase provided important earning, but a onger piot phase, and, in particuar, a onger gap between the piot phase and the main tria, woud have been hepfu. No substantia gap had been schedued between these two phases of the research. The two-phase approach taken represents, in our opinion, an exceent opportunity and mode for pubic support of trias but does create probems in reation to staffing and resourcing modes for the tria. We have previousy referred to the fexibe, pragmatic and hepfu support we obtained from the REC with whom we had reguar exchanges, especiay in the eary days of the study. Their agreement to a revised consenting process underpinned the success of this study. Without the approach they adopted, it is ikey that the proportion of patients for whom we were abe to perform case note review woud have been much smaer (and thus we woud not have secured our POM). The revised consenting process to which the REC agreed was possiby somewhat unusua in the UK, invoving, as it did, verba consent to case note review being recognised as consent. We suppemented this process with an opportunity to opt out using foow-up written consent, but the use of verba consent is important and underpinned the success and generaisabiity of the study. Ethics committees face chaenges when receiving requests such as this from researchers. But when research is carefuy designed and has had reevant PPI in the design and impementation of the research, and where the work is undertaken in cose coaboration with the REC, we beieve that this pragmatic approach supports pubicy funded research in a way that is commendabe. This is an area where guidance might usefuy be given to ethics committees. An aternative decision not to grant the amendment supporting the revised consent process woud have been potentiay very damaging to this study and incurred major staffing and academic disruption as we as significant oss of pubic finances. Our tria invoved a custer randomisation design, with custers being designated at the eve of genera practices. Athough the RCT design is seen as providing potentiay high-quaity evidence, when undertaking evauations of compex organisationa interventions in primary care settings (as accompished in this tria), aternative approaches are possibe possiby based on ethnographic or reaist approaches. Our parae process evauation incorporated important eements of ethnographic investigation in both the piot and main tria phases. Were we to undertake a simiar evauation in the future, we woud propose adopting a simiar research design to that which we adopted here, using a more diverse methodoogy. Our estimates of sampe size were based on a number of assumptions incuding the proportion of patients that might be anticipated to present same-day consutation requests. Our actua estimate (around 40% of patients presenting consutation requests) was based on reception staff competing a og sheet. In genera, we found that the abiity of receptionists to undertake such tasks, whie continuing to deiver an effective service in what we acknowedge is a pressured environment, was imited, as refected in the number of missing data encountered in such og sheets. In retrospect, however, a simiar future study might reasonaby estimate this proportion as around one-third (33%). In addition, athough we had reimbursed practices for receptionist time in contributing to the study, we might have more activey ensured that dedicated receptionist time was indeed made avaiabe to support the tria processes. The outcomes seected were based on previous research and in the event proved generay acceptabe, workabe, and, we beieve, appropriate to the investigation. Our primary outcome the number and rate of contacts incurred in primary care settings in the 28 days foowing an index consutation request for a same-day, face-to-face appointment with a GP was based on previous research undertaken by one of the co-appicants. 3 Extracting data informing the POM invoved undertaking a arge number of reviews of patient case notes. This invoved prior consent from the patient with agreement to review their case notes provided in around 77% of cases. A range of secondary outcomes were aso investigated incuding outcomes reating to patient safety (deaths, A&E attendances, hospita admissions) and patient experience of care. These secondary measures aso proved generay acceptabe and workabe. We identified important Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 105

136 DISCUSSION AND CONCLUSIONS inconsistencies with respect to the documentation of wak-in centre attendance this documentation varying between regions. Considering that there was no reason why this shoud be systematicay different across the three tria arms, we eected to use wak-in centre attendance based on case note review data. Wak-in centre attendance overa, however, was ow among tria participants. We were aso unabe to effectivey capture any contacts with the nationa nurse-ed teephone advice service NHS Direct, these contacts not being routiney recorded in the patient record. Once again, we did capture information on patient sef-reports of NHS Direct usage; further work is required to investigate the vaidity and reiabiity of these sef-reported measures of resource use. With regard to the inferentia anayses, the appropriate ICC or custer-eve SD (approximating the coefficient of variation) is reported, which may prove usefu for other researchers undertaking simiar work. Our process evauation identified that practices participated in this tria for a range of reasons. One important reason that may be of reevance to future research was that the joint engagement in a research project provided a degree of cohesion and common purpose with respect to the practice team. For participating practices at east, the research question was seen as reevant and of importance the research design appeared of interest, and the research process itsef seemed to offer specific advantages to potentia participant practices. We used an externa consutant to support practices in the eary stages of reviewing workoad, appointment systems and interna practice processes with respect to deivering care to patients making same-day appointment requests. The externa consutant who supported this study visited the practices, reviewed practice workoad in detai and provided advice on reconfiguring appointment systems to aow for the impementation of triage (where this was appropriate). In the eary stages of the tria, we did recognise that the externa consutant may have been at risk of promoting a sighty different mode of care from that we intended. The consutant s communication stye was persuasive and attractive to a number of the practices, and, for this reason, to ensure fideity with respect to the intervention, we fet it appropriate to discuss this specific issue with the consutant. That discussion was we received and we are confident that the changes introduced by practices were both reevant and appropriate to the design and deivery of the ESTEEM intervention. We encountered probems for receptionists in reation to the routine recording of information regarding patients who were attending. Asking extremey busy receptionists to undertake routine documentation of workoad is extremey chaenging, and possiby simpy non-viabe on anything but a reativey sma scae. It woud have been desirabe to define the study popuation in detai throughout the duration of data coection, but this woud invove routine documentation of a patients requesting appointments with a doctor, and then defining those who were requesting same-day appointments in a consistent way. Athough this may appear a straightforward task, given the busyness of the front end of primary care and the considerabe responsibiities that receptionists bear, sometimes under extremey stressfu and chaenging conditions, routine data capture invoving receptionists may be extremey difficut. To address this, our intention was to expore the possibiity of using eectronic computerised means of defining the sampe popuation using a singe key stroke entry attributed to a patient requesting a same-day appointment. Primary care computer systems are both diverse and compex and, despite our best efforts, we were unabe to achieve this within the time frame of the tria. We did expore eectronic capture of the above information and reated timings information with respect to individua consutations but found this approach not sufficienty reiabe across various IT patforms to resut in effective data capture. We were gratefu for the financia support of the Comprehensive Loca Research Network in deivering and exporing this. Foowing the investment of considerabe effort with a commercia academic partner, we beieve we amost achieved this but coud not deiver a sufficienty robust system within the time frame of the tria to support the tria processes. This woud be an important area for future deveopment, and one that coud be achieved, potentiay, at ony modest cost. The routine capture and identification of sampes of reevant patients by receptionists, and the capture and extraction 106 NIHR Journas Library

137 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 of timings data from routine eectronic patient records, are both important and reevant issues for future research. The ESTEEM tria has specificay addressed triage-based approaches to managing the consutation requests of individuas seeking same-day appointments. Two further reated areas are worthy of further research: 1. Extending the reach of the research to incorporate a individuas presenting consutation requests, such as is proposed in Doctor First modes of care (patient access: productive primary care: 2. Exporing the use of current aternatives to teephone-based consutation for such individuas, possiby expoiting the potentia of video (Skype ; Skype Ltd, Rives de Causen, Luxembourg), text (e-mai) or socia media-based approaches [such as Facebook (Cambridge, MA, USA), or Twitter (San Francisco, CA, USA)]. Athough ESTEEM has highighted the overa acceptabiity to patients and practitioners of triage-based approaches, further research might expore the advantages/disadvantages to specific popuation groups [e.g. on the basis of age, ethnicity, anguage, educationa attainment or setting (urban/rura)] of remote or IT-based approaches to managing heath-care requests. Impications for heath care The ESTEEM tria has, we beieve, provided important research evidence regarding the cinica effectiveness and cost-effectiveness of teephone triage for managing same-day consutation requests in genera practice. The specific context is important our target popuation was individuas, or their representatives, making requests for same-day face-to-face appointments with their GP. This particuar popuation represents around 30% of genera practice-based primary care workoad, 4 and, for the reasons we have outined, we beieve the findings are generaisabe to a arge section of the UK primary care popuation. However, they do not focus on other patients requesting appointments with a GP, such as those wishing to book appointments in advance, or those attending GP-ed chronic disease management cinics, etc. There may be important potentia to expore the use of teephone triage in consutation for this arger group of patients, athough the design and impementation of any such study woud potentiay be usefuy informed by the research findings we present. Two major modes of GP First care are currenty promoted in the UK by quasi-commercia organisations. 79,80 Interest in these modes has aso been expressed by the Productive Primary Care workstream of the NHS Institute for Innovation, and further investigation of the potentia for such modes has recenty been proposed by the NIHR Heath Services Deivery Research Board. 81 The summary of our research findings suggests that, overa, triage, whether performed by a GP or by a nurse using CDSS, shoud be introduced with fu awareness of the whoe-system impications arising from the decision to impement such a process. Athough GP face-to-face contacts on the index day were reduced, taking a broader, 28-day perspective, overa primary care contacts increased, even after accounting for the inevitabe additiona consutation incurred by the introduction of triage on the index day. Despite these changes in workoad and in work distribution overa, NHS costs per patient across the three arms of the tria were amost identica. Triage of both types appears safe and no differences in patient heath status were observed. Furthermore, NT appeared somewhat ess acceptabe to patients than either GPT or UC. We fee that the margina reduction in satisfaction in NT is understandabe given that patients requesting a face-to-face consutation with a GP actuay received a teephone consutation with a nurse. Having said that, it is important to recognise the potentia benefits that some nurses reported foowing their engagement in the nurse-triage arm of the tria. Some nurses reported positive professiona benefits, describing the use of skis that they fet were otherwise underexpoited. We did not incorporate a measure of job satisfaction in our study but this might be an important consideration for any simiar work. In some Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 107

138 DISCUSSION AND CONCLUSIONS settings, the adoption of GPT or NT processes may be seen to offer a soution to managing workoad pressures, athough carefu consideration needs to be given to address patients experience of care, and to address potentia knock-on effects that might arise esewhere in primary care foowing introduction of a triage system. When whoe-system redesign is anticipated, carefu consideration needs to be given to both the human resource management and the eectronic systems support required to effectivey deiver the intervention. Computer systems as they reate to primary care are compex and diverse; there are at east eight major primary care computer systems commony in use across the UK. Working with system deveopers to expoit the potentia of both the eectronic record and practice computer administrative system represents an important but underexpoited opportunity in the UK. The CDSS we adopted has been widey used across the UK in supporting out-of-hours primary care and NHS Direct. It has not previousy been widey adopted or researched within the context of in-hours primary care. Overa, the CDSS integrated reasonaby we with estabished practice computer systems but further work may need to be done in this regard and across a wider range of primary care computer systems than we were abe to expoit in this study. The data that we have reported reate to the origina research questions posed and the a priori statistica and heath-economic anayses defined. Many other areas of investigation might be addressed within a secondary anaysis of this data set, for exampe examining the potentia access issues or patient experience outcomes for subgroups of the sampe popuation. Triage systems appeared generay acceptabe and, in the broadest terms, simiar to UC with respect to patients reports of their experience of care; however, we have not yet had the opportunity to examine the experience of important subgroups of our sampe popuation for exampe younger or oder peope, parents or individuas who were unabe to attend the practice within reguar hours (or found it difficut to do so) owing to work commitments. A of these might be expected to have substantiay different views from the overa popuation with respect to their experience of care, and this represents just one of many further anayses that might be undertaken with respect to the current data set. Genera practitioners and their teams might wish to ask Shoud we introduce triage?. The evidence from ESTEEM is summarised in Tabe 36 for the main effects that might reasonaby be anticipated foowing the introduction of either GP- or nurse-ed triage. The findings from the process evauation suggest that the data presented in Tabe 36 shoud be carefuy considered in the ight of practice organisation and the particuar oca circumstances prevaiing. Introducing triage, whether GPT or NT, was associated with an increased primary care contact rate of 33% in GPT, and 48% in NT (10% and 12%, respectivey, after accounting for the inevitabe additiona contacts incurred as a resut of triage). However, the nature and professiona distribution of the work is ikey to change, athough overa costs to the NHS (over 28 days) shoud remain unchanged. In particuar, introducing GPT is ikey to reduce the number of GP face-to-face consutations by around 40% over a 28-day period, partiay offset by an increase in the number of teephone consutations. Introducing NT is ikey to resut in reduced numbers of GP face-to-face contacts by around 20% over 28 days but with a substantia increase in nurse contact time on the index day. These findings might have different reevance for different practices. For exampe, if the priority is to reduce demand on GPs then NT might be appropriate. If the priority is to contro demand for GP face-to-face appointments then GPT might be appropriate; athough it incurs a simiar amount of work in tota, the timing is more in the GPs contro. Unanticipated and whoe-system effects shoud be taken into account when considering the introduction of cinician triage. Cinician triage of patients seeking same-day consutations with a GP may offer substantia advantages in supporting the deivery of patient care, and potentiay offers a usefu approach in the armamentarium of toos faciitating the deivery of effective NHS primary care. 108 NIHR Journas Library

139 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 TABLE 36 Summarising evidence from ESTEEM considered in reference to UC GPT NT Primary care contact rates (%) Counting a contacts on index day as separate consutations Counting a within-practice contacts on index day as a singe consutation GP face-to-face contacts on index day ony GP face-to-face contacts over 28 days foowing index consutation request GP teephone and face-to-face contacts on index day ony GP teephone and face-to-face contacts over 28 days foowing index consutation request Patient cinician contact time (minutes) on index day (estimate) Overa GP and nurse contact time (for a patient making a same-day consutation request in UC, 9.6 minutes) Overa GP contact time (for a patient making a same-day consutation request in UC, 9.1 minutes) Overa nurse contact time (for a patient making a same-day consutation request in UC, 0.6 minutes) Safety A&E attendance in 28 days foowing index consutation request Emergency hospita admission rate in 7 days foowing index consutation request Mortaity rate in 7 days foowing index consutation request Patient experience of care Overa satisfaction ( ) a Heath economics Cost per patient, decreased;, increased;, unchanged. a Margina reduction ony. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 109

140

141 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Acknowedgements Contributions of authors Professor John L Campbe (Professor of Genera Practice and Primary Care; chief investigator) took overa responsibiity for the study; supervised tria conduct; was invoved in a stages of the ESTEEM tria, incuding conception, design, anaysis and interpretation of data; drafted and criticay revised the report for important inteectua content and approva of the fina version. Mrs Emiy Fetcher (tria manager) was responsibe for the day-to-day management of the study and acquisition of data; coordinated research staff across sites; contributed to interpretation of data; drafted and criticay revised the report for important inteectua content and approva of the fina version. Professor Nicky Britten (Professor of Appied Heathcare Research; process evauation ead) took overa responsibiity for the process evauation and was invoved in a stages of the ESTEEM tria, incuding conception, design, interpretation of data and criticay revising the report for important inteectua content and approva of the fina version. Professor Coin Green (Professor of Heath Economics; economic evauation ead) took overa responsibiity for the economic evauation; ed the economic anaysis; and was invoved in a stages of the ESTEEM tria, incuding conception, design, interpretation of data, and drafting and criticay revising the report for important inteectua content and approva of the fina version. Dr Tim Hot (NIHR academic cinica ecturer; principa investigator for genera practices near Warwick) supervised the tria and research staff in this region and was invoved in a stages of the ESTEEM tria, incuding conception, design, interpretation of data, and criticay revising the report for important inteectua content and approva of the fina version. Professor Vaerie Lattimer (Head of Schoo of Nursing Sciences, Professor of Heath Services Research; principa investigator for genera practices near Norwich) supervised the tria and research staff in this region and was invoved in a stages of the ESTEEM tria, incuding conception, design, interpretation of data, and criticay revising the report for important inteectua content and approva of the fina version. Professor David A Richards (Professor of Menta Heath Services Research; co-appicant) was invoved in a of the stages of the ESTEEM tria, incuding conception, design, interpretation of data and criticay revising the report for important inteectua content and approva of the fina version. Dr Suzanne H Richards (senior ecturer; co-appicant) was invoved in a stages of the ESTEEM tria, incuding conception, design, interpretation of data, and drafting and criticay revising the report for important inteectua content and approva of the fina version. Professor Chris Saisbury (Professor in Primary Heath Care; principa investigator for genera practices near Bristo) supervised the tria and research staff in this region and was invoved in a stages of the ESTEEM tria, incuding conception, design, interpretation of data and criticay revising the report for important inteectua content and approva of the fina version. Professor Rod S Tayor (Professor of Heath Services Research; statistica ead) took overa responsibiity for the main tria anaysis and was invoved in a stages of the ESTEEM tria, incuding conception, design, interpretation of data, and drafting and criticay revising the report for important inteectua content and approva of the fina version. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 111

142 ACKNOWLEDGEMENTS Dr Raff Caitri (research feow; tria researcher in Exeter) operationaised the protoco, was invoved in acquisition of data, contributed to tria management responsibiities and contributed to additiona anayses, interpretation of data, and drafting and criticay revising the report for important inteectua content and approva of the fina version. Ms Vicky Bowyer (research feow; tria researcher in Warwick) operationaised the protoco, was invoved in acquisition of data and contributed to criticay revising the report for important inteectua content and approva of the fina version. Dr Katherine Chapin (research feow; tria researcher in Bristo) operationaised the protoco, was invoved in acquisition of data and contributed to additiona anayses and criticay revising the report for important inteectua content and approva of the fina version. Miss Rebecca Kandiyai (research feow; heath economics researcher) was invoved in acquisition of data and contributed to the economic anaysis and the drafting/approva of the fina version of the report. Dr Jamie Murdoch (senior research feow; tria researcher in Norwich) operationaised the protoco, was invoved in acquisition of data and contributed to criticay revising the report for important inteectua content and approva of the fina version. Ms Linnie Price (associate research feow; process evauation researcher) operationaised the protoco, was invoved in acquisition of data, undertook the quaitative anaysis and interpretation of data and contributed to the drafting and approva of the fina version of the report. Ms Juia Roscoe (associate research feow; tria researcher in Warwick) operationaised the protoco, was invoved in acquisition of data and contributed to criticay revising the report for important inteectua content and approva of the fina version. Mrs Anna Varey (research feow; tria researcher in Norwich) operationaised the protoco, was invoved in acquisition of data and contributed to criticay revising the report for important inteectua content and approva of the fina version. Dr Fiona C Warren (ecturer; statistician) contributed to the deveopment of the statistica anaysis pan, undertook the main tria anaysis and contributed to drafting and criticay revising the report for important inteectua content and approva of the fina version. We woud ike to thank the externa members of the Tria Steering Committee for their advice and support for the project: Professor Brian McKinstry (University of Edinburgh), Professor Sandra Edridge (Queen Mary University of London), Professor Brendan Deaney (King s Coege London), Dr Louise Locock (University of Oxford), Mrs Barbara Tibury (Patient Representative) and Mr Geoff Barr (Patient Representative). Our thanks aso go to the Data Monitoring Committee comprising Professor Bruce Guthrie (Dundee University), Dr Ben Carter (Cardiff University) and Professor Pau Kinnersey (Cardiff University). We are aso gratefu to our Patient and Pubic Invovement group for their advice and input throughout the tria. We woud ike to thank both the staff and the patients of our participant genera practices who gave generousy of their time to support the study and compete data coection materias. We aso thank staff within South West 2 Research Ethics Committee, primary care trusts, Comprehensive Loca Research Networks and PCRNs across the four regions, who provided vauabe assistance to us throughout the study. For their administrative support during the tria, we woud ike to thank Donna Poade, Amy Gratton, Juia Carver, Giian Bunt, Vivienne Owen, Lynn Green and Lorna Burche. Likewise, we thank Caire Teford, Frances Carpenter and Sue Rugg for their contribution as researchers during the course of the study. 112 NIHR Journas Library

143 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 We extend our thanks aso to Pain Heathcare for providing the Odyssey PatientAssess CDSS to support the nurses triaging decisions, and for deivering the training in using this software and technica support to practices. Likewise, we thank Dr Steve Cay and Mr Dion Sykes from Productive Primary Care for providing training and advice to intervention practices in organising the triage systems and in GP triage skis. Thanks aso to Jane Vickery, Laura Cocking and Eiot Carter at the Peninsua Cinica Trias Unit for their guidance and support in buiding and maintaining our study database and to CFEP-UK for their support with doube data entry. Pubication Campbe JL, Fetcher E, Britten N, Green C, Hot TA, Lattimer V, et a. Teephone triage for management of same-day consutation requests in genera practice (the ESTEEM tria): a custer-randomised controed tria and cost-consequence anaysis. Lancet 2014;384: Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 113

144

145 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 References 1. Department of Heath. Deivering Investment in Genera Practice: Impementing the New GMS Contract. London: Department of Heath; Saisbury C, Montgomery AA, Simons L, Sampson F, Edwards S, Baxter H, et a. Impact of Advanced Access on access, workoad, and continuity: controed before-and-after and simuated-patient study. Br J Gen Pract 2007;57: Richards DA, Meakins J, Tawfik J, Godfrey L, Dutton E, Richardson G, et a. Nurse teephone triage for same day appointments in genera practice: mutipe interrupted time series tria of effect on workoad and costs. BMJ 2002;325: Stoddart H, Evans M, Peters TJ, Saisbury C. The provision of same-day care in genera practice: an observationa study. Fam Pract 2003;20: Lattimer V, George S, Thompson F, Thomas E, Muee M, Turnbu J, et a. Safety and effectiveness of nurse teephone consutation in out of hours primary care: randomised controed tria. The South Witshire Out of Hours Project (SWOOP) Group. BMJ 1998;317: /bmj Marsh GN, Horne RA, Channing DM. A study of teephone advice in managing out-of-hours cas. J R Co Gen Pract 1987;37: Markund B, Bengtsson C. Medica advice by teephone at Swedish heath centres: who cas and what are the probems? Fam Pract 1989;6: Dae J, Crouch R, Loyd D. Primary care: nurse-ed teephone triage and advice out-of-hours. Nurs Stand 1998;12: Office for Nationa Statistics. Living in Britain. London: Office for Nationa Statistics; NHS Information Centre. GP Workoad Survey URL: (accessed 12 November 2014). 11. Jiwa M, Mathers N, Campbe M. The effect of GP teephone triage on numbers seeking same-day appointments. Br J Gen Pract 2002;52: Richards DA, Meakins J, Godfrey L, Tawfik J, Dutton E. Survey of the impact of nurse teephone triage on genera practitioner activity. Br J Gen Pract 2004;54: McKinstry B, Waker J, Campbe C, Heaney D, Wyke S. Teephone consutations to manage requests for same-day appointments: a randomised controed tria in two practices. Br J Gen Pract 2002;52: Richards DA, Godfrey L, Tawfik J, Ryan M, Meakins J, Dutton E, et a. NHS Direct versus genera practice based triage for same day appointments in primary care: custer randomised controed tria. BMJ 2004;329: Thompson F, George S, Lattimer V, Smith H, Moore M, Turnbu J, et a. Overnight cas in primary care: randomised controed tria of management using nurse teephone consutation. BMJ 1999;319: Lattimer V, Sassi F, George S, Moore M, Turnbu J, Muee M, et a. Cost anaysis of nurse teephone consutation in out of hours primary care: evidence from a randomised controed tria. BMJ 2000;320: Brown A, Armstrong D. Teephone consutations in genera-practice: an additiona or aternative service. Br J Gen Pract 1995;45: Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 115

146 REFERENCES 18. Pooe SR, Schmitt BD, Carruth T, Peterson-Smith A, Susarski M. After-hours teephone coverage: the appication of an area-wide teephone triage and advice system for pediatric practices. Pediatrics 1993;92: Homstrom I. Decision aid software programs in teenursing: not used as intended? Experiences of Swedish teenurses. Nurs Heath Sci 2007;9: x 20. Richards DA, Meakins J, Tawfik J, Godfrey L, Dutton E, Heywood P. Quaity monitoring of nurse teephone triage: piot study. J Adv Nurs 2004;47: x 21. Crouch R, Woodfied H, Dae J, Pate A. Teephone assessment and advice: a training programme. Nurs Stand 1997;11: Derkx HP, Rethans JJ, Muijtjens AM, Maiburg BH, Winkens R, van Rooij HG, et a. Quaity of cinica aspects of ca handing at Dutch out of hours centres: cross-sectiona nationa study. BMJ 2008;337:a Car J, Freeman GK, Partridge MR, Sheikh A. Improving quaity and safety of teephone based deivery of care: teaching teephone consutation skis. Qua Saf Heath Care 2004;13: Curre R, Urquhart C, Wainwright P, Lewis R. Teemedicine versus face to face patient care: effects on professiona practice and heath care outcomes. Nurs Times 2001;97: Giesen P, Ferwerda R, Tijssen R, Mokkink H, Drijver R, van den Bosch W, et a. Safety of teephone triage in genera practitioner cooperatives: do triage nurses correcty estimate urgency? Qua Saf Heath Care 2007;16: Pain Heathcare. Odyssey Cinica Decision Support Software URL: software.com/ahc/ (accessed 12 November 2014). 27. Bunn F, Byrne G, Kenda S. The effects of teephone consutation and triage on heathcare use and patient satisfaction: a systematic review. Br J Gen Pract 2005;55: Venning P, Durie A, Roand M, Roberts C, Leese B. Randomised controed tria comparing cost effectiveness of genera practitioners and nurse practitioners in primary care. BMJ 2000;320: Kinnersey P, Anderson E, Parry K, Cement J, Archard L, Turton P, et a. Randomised controed tria of nurse practitioner versus genera practitioner care for patients requesting same day consutations in primary care. BMJ 2000;320: Shum C, Humphreys A, Wheeer D, Cochrane MA, Skoda S, Cement S. Nurse management of patients with minor inesses in genera practice: muticentre, randomised controed tria. BMJ 2000;320: NHS Institute for Innovation and Improvement. Stour Access System: A New Way to Manage GP Appointments. Better for GPs, Better for Patients, Better A-round URL: co.uk/website/j00700/fies/stouraccesssystembrochure.pdf (accessed 12 November 2014). 32. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Deveoping and evauating compex interventions: the new Medica Research Counci guidance. BMJ 2008;337:a Campbe JL, Britten N, Green C, Hot TA, Lattimer V, Richards SH, et a. The effectiveness and cost-effectiveness of teephone triage of patients requesting same day consutations in genera practice: study protoco for a custer randomised controed tria comparing nurse-ed and GP-ed management systems (ESTEEM). Trias 2013;14: NIHR Journas Library

147 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO Snooks HA, Carter B, Dae J, Foster T, Humphreys I, Logan PA, et a. Support and Assessment for Fa Emergency Referras (SAFER 1): Custer Randomised Tria of Computerised Cinica Decision Support for Paramedics. PLOS ONE 2014;9:e Campbe MK, Snowdon C, Francis D, Ebourne D, McDonad AM, Knight R, et a. Recruitment to randomised trias: strategies for tria enroment and participation study. The STEPS study. Heath Techno Assess 2007;11(48) Ukoumunne OC, Guiford MC, Chinn S, Sterne JA, Burney PG. Methods for evauating area-wide and organisation-based interventions in heath and heath care: a systematic review. Heath Techno Assess 1999;3(5). 37. Association of Pubic Heath Observatories (APHO) URL: genera-practice (accessed 12 November 2014). 38. Curtis L. Unit Costs of Heath and Socia Care Canterbury: PSSRU; EuroQo group. EuroQo: a new faciity for the measurement of heath-reated quaity of ife. Heath Poicy 1990;16: McHorney CAJ, Ware JE Jr, Lu JFR, Sherbourne CD. The MOS 36 item Short Form Heath Survey (SF-36):III. Tests of data quaity, scaing assumptions, and reiabiity across diverse patient groups. Med Care 1994;32: Baker R. Deveopment of a questionnaire to assess patients satisfaction with consutations in genera practice. Br J Gen Pract 1990;40: Mercer SW, Howie JGR. CQI-2: a new measure of hoistic interpersona care in primary care consutations. Br J Gen Pract 2006;56: NHS Engand/Ipsos MORI. GP Patient Survey URL: (accessed 12 November 2014). 44. Bower P, Roand M, Campbe JL, Mead N. Setting standards based on patients views on access and continuity: secondary anaysis of data from the genera practice assessment survey. BMJ 2003;326: Campbe JL, Roand MO, Richards S, Dickens A, Greco M, Bower P. Users reports and evauations of out-of-hours heathcare and the UK nationa quaity requirements. A cross sectiona study. Br J Gen Pract 2009;59:e Campbe JL. Patients perceptions of medica urgency: does deprivation matter? Fam Pract 1999;16: Howie JGR, Heaney DJ, Maxwe M. Measuring Quaity in Genera Practice. Occasiona Paper No. 75. London: Roya Coege of Genera Practitioners; Adams G, Guiford MC, Ukoumunne OC, Edridge S, Chinn S, Campbe MJ. Patterns of intra-custer correation from primary care research to inform study design and anaysis. J Cin Epidemio 2004;57: Hippisey-Cox J, Fenty J, Heaps M. Trends in Consutation Rates in Genera Practice 1995 to 2007: Anaysis of the QRESEARCH database. Nottingham: QRESEARCH; Jacoby A. Possibe factors affecting response to posta questionnaires: findings from a study of genera practitioner services. J Pubic Heath Med 1990;12: Smith WC, Crombie IK, Campion PD, Knox JD. Comparison of response rates to a posta questionnaire from a genera practice and a research unit. Br Med J (Cin Res Ed) 1985;291: Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 117

148 REFERENCES 52. Saisbury C. Posta survey of patients satisfaction with a genera practice out of hours cooperative. BMJ 1997;314: McCo E, Jacoby A, Thomas L, Soutter J, Bamford C, Steen N, et a. Design and use of questionnaires: a review of best practice appicabe to surveys of heath service staff and patients. Heath Techno Assess 2001;5(31) UK Government. Indices of Mutipe Deprivation URL: engish-indices-of-deprivation-2010 (accessed 12 November 2014). 55. Campbe MK, Piaggio G, Ebourne DR, Atman DG. Consort 2010 statement: extension to custer randomised trias. BMJ 2012;345:e Zwarenstein M, Treweek S, Gagnier JJ, Atman DG, Tunis S, Haynes B, et a. Improving the reporting of pragmatic trias: an extension of the CONSORT statement. BMJ 2008;337:a Bennett S, Parpia T, Hayes R, Cousens S. Methods for the anaysis of incidence rates in custer randomized trias. Int J Epidemio 2002;31: Pacheo GD, Hattendorf J, Coford JM, Mäusezah D, Smith T. Performance of anaytica methods for overdispersed counts in custer randomized trias: Sampe size, degree of custering and imbaance. Stat Med 2009;28: White IM, Royston P, Wood AM. Mutipe imputations using chained equations: issues and guidance for practice. Stat Med 2011;30: Lyratzopouos G, Eiott M, Barbiere JM, Henderson A, Staetsky L, Paddison C, et a. Understanding ethnic and other socio-demographic differences in patient experience of primary care: evidence from the Engish Genera Practice Patient Survey. BMJ Qua Saf 2012;21: /bmjqs Craig D, Rice S. CRD Report 6: NHS Economic Evauation Database Handbook. York: Centre for Reviews and Dissemination, University of York; Green J, Thorogood N. Quaitative Methods for Heath Research. London: Sage; Pope C, Zieband S, Mays N. Quaitative research in heath care. Anaysing quaitative data. BMJ 2000;320: Campbe JL, Fetcher E, Britten N, Green C, Hot TA, Lattimer V, et a. Teephone triage for management of same-day consutation requests in genera practice (the ESTEEM tria): a custer-randomised controed tria and cost-consequence anaysis. Lancet 2014;384: NHS Empoyers. 2014/15 Genera medica Services (GMS) Contract Quaity and Outcomes Framework (QOF). Guidance from GMS contract 2014/ PAYANDCONTRACTS/GENERALMEDICALSERVICESCONTRACT/QOF/Pages/ChangestoQOF aspx (accessed 1 November 2014). 66. Department of Heath. The NHS Pan. London: HMSO; Pope C, Banks J, Saisbury C, Lattimer V. Improving access to primary care: eight case studies of introducing Advanced Access in Engand. J Heath Serv Res Poicy 2008;13: /jhsrp Huibers L, Smits M, Renaud V, Giesen P, Wensing M. Safety of teephone triage in out-of-hours care: a systematic review. Scand J Prim Heath Care 2011;29: NIHR Journas Library

149 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO Gaagher M, Huddart T, Henderson B. Teephone triage of acute iness by a practice nurse in genera practice: outcomes of care. Br J Gen Pract 1998;48: Darne JC, Hiner SL, Nei PJ, Mamin JJ, McDonad CJ, Hui SL, et a. After-hours teephone access to physicians with access to computerized medica records. Experience in an inner-city genera medicine cinic. Med Care 1985;23: Vedsted P, Christensen MB. The effect of an out-of-hours reform on attendance at casuaty wards. The Danish exampe. Scand J Prim Heath Care 2001;19: McKinstry B, Hammersey V, Burton C, Pinnock H, Eton R, Dowe J, et a. The quaity, safety and content of teephone and face-to-face consutations: a comparative study. Qua Saf Heath Care 2010;19: Stirewat CF, Linn MW, Godoy G, Knopka F, Linn BS. Effectiveness of an ambuatory care teephone service in reducing drop-in visits and improving satisfaction with care. Med Care 1982;20: Key M, Egbunike JN, Kinnersey P, Hood K, Owen-Jones E, Button LA, et a. Deays in response and triage times reduce patient satisfaction and enabement after using out-of-hours services. Fam Pract 2010;27: Smits M, Huibers L, Oude BA, Giesen P. Patient satisfaction with out-of-hours GP cooperatives: a ongitudina study. Scand J Prim Heath Care 2012;30: Leibowitz R, Day S, Dunt D. A systematic review of the effect of different modes of after-hours primary medica care services on cinica outcome, medica workoad, and patient and GP satisfaction. Fam Pract 2003;20: Greenhagh T, Robert G, Macfarane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Mibank Q 2004;82: Kernick D. Wanted: new methodoogies for heath service research. Is compexity theory the answer? Fam Pract 2006;23: GP Access. PatientAccess URL: (accessed 12 November 2014). 80. Productive Primary Care. Doctor First URL: (accessed 12 November 2014). 81. Nationa Institute for Heath Research (NIHR). Heath Services and Deivery Research Programme URL: (accessed 12 November 2014). 82. Chader M, Wies NJ, Campbe J, et a. A pragmatic randomised controed tria to evauate the cost-effectiveness of a physica activity intervention as a treatment for depression: the treating depression with physica activity (TREAD) tria. Heath Techno Assess 2012;16(10). Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 119

150

151 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Appendix 1 ESTEEM tria: piot study report A 12-month, preiminary piot custer RCT and parae process evauation was undertaken in six practices, and aspects of this work were informative to the main tria protoco. The key earning outcomes incuded confirmation of the feasibiity of recruiting practices, the feasibiity and acceptabiity of the triage interventions, aterations required for the proposed data coection methods and the eve of custering of outcomes. Methods Six piot practices were recruited across Devon and Bristo, and were randomy aocated to each of GP-ed triage (GPT), nurse-ed triage (NT) and usua care (UC). Practices received training on the triage interventions, as described for the main tria. Practices coected outcome data for a 2-week period of patient recruitment. During this period, practices were each asked to recruit 200 patients. Data coection Capturing the patient sampe Receptionists competed a handwritten og sheet, recording a teephone cas received on each day of the data coection period. In order to capture patient information, the receptionist competed a row of data for each patient who caed to request a same-day appointment. For each ca, the receptionist recorded their own initias, and the date and time of the ca, and then competed four response options to determine eigibiity for ESTEEM ( appointment request? yes/no; same-day request? yes/no; patient we enough for tria? yes/no; patient communicates without difficuty? yes/no). If a four response options were yes, the receptionist went on to compete the patient s unique practice identification number, date of birth and gender. A further coumn was competed to indicate if the eigibe patient was added to the triage ist (yes/no) and, if not, a reason was recorded for no triage. The function of this daiy og sheet was twofod: 1. To provide a sampe frame an indication of the proportion of patients who were eigibe for triage as we as those who were eigibe and actuay entered into the tria Key demographic variabes, incuding date of birth and gender, were captured so that we coud assess quaitative differences between patients eigibe and entered, and those eigibe and not entered (as we as comparing against ineigibe patients). 2. To serve as a database of patients from which to manage research activities (e.g. questionnaire mai-outs) A member of practice staff coected a og sheets together at the end of each day and assigned a study-specific Read Code to the eectronic record of a patients who were marked on the og sheets as eigibe for ESTEEM. This faciitated eectronic searches of the practice systems to generate patient ists for the purpose of maiing foow-up study questionnaires and conducting case note reviews. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 121

152 APPENDIX 1 Cinician data coection forms During both the piot study and main tria, cinicians competed a short data coection form ( Cinician Form ) at the time of the initia consutation foowing a patient s same-day request (triage or usua care contact). This form captured detais of the consutation, incuding which heath professiona undertook the consutation, whether the patient did not attend, a measure of case mix, treatment and management options chosen (e.g. ordering of tests, recommending subsequent appointment or referra), and the start and end time of the consutation. Questionnaires Patients were sent a questionnaire 4 weeks after their index consutation. The questionnaire incuded questions on satisfaction and heath status (described in Secondary outcomes). The covering etter contained a section for patients who did not wish to participate or receive any further contact in reation to the study. These patients were asked to send back a bank questionnaire in a prepaid enveope as an indication that they wished to have no further contact. These patients did not receive any further communication in reation to the study. The questionnaire contained a section at the end, which offered the opportunity for patients who were wiing to compete a questionnaire to opt out of having their case note reviewed. Those who did not respond were sent a reminder etter with another copy of the questionnaire 2 weeks after the previous mai-out. Transition from piot to main tria Certain important aspects of the methodoogy were atered as a resut of observations during the piot study. The key earning outcomes reated to stopping rues, which incuded confirmation ahead of the main tria of the feasibiity of recruiting practices, the feasibiity and acceptabiity of the triage interventions, aterations required for the proposed data coection methods and the eve of custering of outcomes. Stopping rues Progression from the piot study to the main tria was dependent on the achievement of predefined stopping rues. These were as foows: 1. Recruitment of genera practices 70% of target number (i.e. 29/42) of main tria practices signed up to participate in the main tria by the end of the piot. 2. Coection of outcomes 55% of patient-reported outcomes (questionnaire response rate) and > 70% of patient questionnaire respondents providing written consent to a case note review. 3. Custering Design effect attributabe to custering shoud not increase the proposed sampe size by > 20%. The atter may have been satisfied through modification of the estimated ICC based on piot data, atering the number of custers (practices) recruited into the study, or a combination of both measures. Tabe 37 iustrates the performance of the piot study against these stopping rues and action taken to ensure progression to the main tria phase when any stopping rue was not met. Main tria practice recruitment A tota of 29/42 (70%) practices were to be recruited by the end of the feasibiity phase. Figure 10 outines progress with main tria practice recruitment at the time of competing the piot study. We had recruited 38 participant practices, most of which are prepared to begin the tria in either spring or autumn of Recruitment to this compex tria was chaenging, as participant practices were required to reorganise deivery of services at the reception desk, which was reiant on the commitment and understanding of a members of practice staff. 122 NIHR Journas Library

153 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 TABLE 37 Summary tabe of piot study outcomes against stopping rues Piot study outcome Stopping rue Piot study performance Action taken Main tria practice recruitment 70% (29/42 practices) 90% (38/42 practices) None required Questionnaire response 55% 38% Shortened questionnaire (use of EQ-5D rather than SF-36 measure of heath status) Incuded an incentive for respondents Incuded a second reminder maiing Consent to case note review 70% 86% Approva obtained from REC for cinicians to record patients initia verba consent to notes review at the end of the same-day contact, to be foowed up by posta questionnaire at 4 weeks aowing opt-out ICC Piot ICC requiring an increase in sampe size for main tria by < 20% Piot ICC = 0.03 (vs. origina estimate of 0.05) No increase in sampe size required Common reasons for practices decining participation incuded: not being interested in the research question or running a triage system (n = 13) imited staffing capacity to take on tria or to operate the tria if randomised to a particuar arm (n = 11) recent or imminent change to IT system (n = 1) partner surgeries getting invoved in the tria (n = 2) decining without specifying a reason (n = 16). Questionnaire response The ower-than-anticipated questionnaire response rate observed during the piot study (38% against the stopping rue of 55%) prompted a revision of the patient questionnaire, incuding incorporation of the EQ-5D measure of heath status rather than the SF-36, to shorten the overa ength and to incorporate a second reminder maiing. After the initia maiing of the questionnaire 4 weeks after the same-day consutation request, two reminders were sent to non-respondents at 6 and 8 weeks. In addition, the ethica approva granted at the beginning of the piot incuded an option to provide an incentive for questionnaire respondents. As such, main tria patients received an additiona eafet within the questionnaire pack offering the chance of being entered into a prize draw of 20 prizes of 25 worth of shopping vouchers. During the piot study, obtaining patient consent to the case note review was dependent on the return of the questionnaire, as we had not yet incorporated a stage for cinicians to request initia verba consent at the end of the index consutation. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 123

154 APPENDIX 1 Devon Bristo Warwick Norwich Approached Positive response Meetings hed no response 36 decined 3 ineigibe 2 may consider ater 10 decined 4 in correspondence no response 29 decined 8 ineigibe 6 may consider ater 5 ineigibe 1 may consider ater 2 decined 1 meeting schedued no response 7 in correspondence 43 decined 2 ineigibe 1 in correspondence 4 meeting schedued 4 decined ineigibe 1 considering 1 decined 2 considering 5 decined Agreed awaiting randomisation 3 awaiting randomisation 5 awaiting randomisation Randomised GPT NT UC GPT NT UC GPT NT UC GPT NT UC FIGURE 10 Main tria practice recruitment at end of piot (Juy 2010 to February 2011). 2 decined 2 ineigibe 2 considering 2 in correspondence 1 decined 5 awaiting randomisation 124 NIHR Journas Library

155 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Consent to case note review The acquisition of initia verba consent to the case note review by the cinician (with subsequent opt-out option within the posta questionnaire) is a departure from the origina piot methodoogy. The ower-than-anticipated piot questionnaire response rate indicated that we may not be abe to achieve sufficient numbers of main tria patients providing consent to the case note review if we were to be dependent on patients returning a questionnaire and providing written consent. The introduction of the verba consent stage at the end of the index consutation ensured that we woud secure sufficient numbers of patients for whom we coud coect our primary outcome data. Intracuster correation coefficient Against our origina estimate of ICC = 0.05, proposed in our piot study outine, the ICC based on the primary outcome is cacuated as 0.03 (95% CI 0.00 to 0.08, see Stata output at end of this short report); our statisticians (Tayor, Warren) note: The origina stopping rues were based on 55% questionnaire response, foowed by 70% consent to notes review, i.e. approx. 38.5% of the origina sampe consenting to notes review. Even if we required a sampe size of 5819 per arm (according to the [worst-case scenario] ICC of 0.08), this woud have produced 2240 respondents consenting to notes review. With the current arm size of 3751, if we assume 85% consent to notes review based on verba consent and questionnaire consent, this gives a tota of 3188 consenting to notes review per arm, which is we above the worst-case ICC sampe size. We were therefore reassured by these cacuations and saw this as attaining the fourth of the origina stopping rues. Method of capturing the patient sampe The piot study aowed the deveopment of a method to ensure correct patient identification. The piot study initiay invoved receptionists competing a handwritten og sheet, recording a teephone cas received each day of the data coection period. It became evident during the piot study that despite standardised training in competing the og sheets, they were not competed consistenty by receptionists within each practice and aso across practices. Constraints on receptionist time, especiay during periods of heavy demand, made competing the og sheet a rea chaenge. Accordingy, in the main tria the study database itsef was subsequenty popuated directy from practice appointment records, which directy captured the patients being identified for the study, rather than being reiant on the daiy handwritten receptionists og sheets. We aso introduced monitoring procedures to ensure high eves of patient identification (see Chapter 2, Integrity checks). Issues reported to Heath Technoogy Assessment at end of piot A number of factors experienced during the feasibiity phase and some subsequent proposas are important to mention here, as foows: Deay to start in Warwick and Norwich The estabishment of research staff in the recruitment sites of Warwick and Norwich has been affected by the extension to the feasibiity phase, to aow for the deveopment work for the piot study in Exeter and Bristo to proceed. A appointments were deayed by 1 month, such that a researcher and administrator at each site woud have been in post during June and December 2010, respectivey. Difficuty in appointing to these posts has meant that the Warwick researcher (Vicky Bowyer) and the Norwich administrator (Anna Varey) started in October The post of administrator in Warwick is yet to be fied, as a resut of organisationa changes within the University of Warwick, and the post of researcher in Norwich is being covered in the interim by Jamie Murdoch, a Research Feow in Urgent Care at University of East Angia (UEA). In addition to this, Professor Lattimer s move to UEA from the University of Southampton resuted in the transferra of a tria arrangements from Southampton to UEA and the reappication for research and deveopment (R&D) and funding approvas in PCTs around Norwich. Recruitment of genera practices in Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 125

156 APPENDIX 1 both Warwick and Norwich has presented a chaenge as a resut of these factors. However, staff at these sites have worked very hard and with enthusiasm to produce encouraging progress. Excess treatment costs from Primary Care Trusts Significant difficuty has been experienced in agreeing Excess Treatment Costs from a number of PCTs from whom R&D approva has been obtained. Without secure agreement of this funding, recruitment of practices and operation of the tria wi be at risk. Where funding has been agreed, an approach has been made to each PCT for reassurance that the agreed funds wi be made avaiabe to the research team, and responses are awaited. A significant shortfa remains ( 88,000) and a subvention appication is being prepared for submission to DH, foowing advice from Trudi Simmons (DH R&D). Eection and White Paper During the feasibiity phase, the eection of the coaition government, particuary with the reease of the White Paper in Juy 2010, is ikey to affect ongoing recruitment and may infuence operation of the tria during 2011 and Ethics We undertook substantia pretria work with respect to ethics (which was funded internay at cost to the Department but at no cost to the HTA). This meant that we secured ethica agreement for the tria at a very eary stage in the process. Foowing fu ethica approva, we have submitted and had approved 4 amendments to our protoco we have a minor amendment currenty under consideration. Each of these submissions does of course require significant attention to detai and reevant documentation using standard Integrated Research Appication System (IRAS) procedures. Capacity: possibe need for extension in time and resource This study started on 1 November 2009, and was funded for a period of 45 months. The origina design invoved a 12-month feasibiity phase, foowed by a 33-month main tria phase. Cear stopping rues were set to demarcate progress to the main tria phase. It was perhaps a mistake that there was no fexibe time to aow for carefu review of data accumuated during the feasibiity phase prior to commencement of the main tria phase. Foowing personne probems encountered within the first few weeks of the tria, and the subsequent departure of the tria manager, we were granted a 3-month extension to the feasibiity phase. At the time this was discussed, we did not request an extension to the overa ength of the tria itsef, athough we did note that the extension in feasibiity may have to be matched by an extension to the overa tria. Our experience of conducting the feasibiity work has identified the very substantia compexity of this tria. Athough we have competed a of the recruitment we set out to achieve, we have in addition coected, coated, and are in the process of anaysing the feasibiity data. This report represents an important communication with HTA, but ideay, we woud have buit in some time at this stage to ensure that a the essons from the feasibiity phase have been carefuy earnt, incorporated within the main tria documentation, and disseminated out to the researchers and principa investigator across the four sites. Recognising this, we have buit in an additiona 1-month period to aow the essons from the feasibiity to be earnt and disseminated. However, some additiona time woud have been extremey hepfu, aowing us to further refine our intervention manuas, standard operating procedures, and supporting data coation and anaytica materias. In summary therefore, athough we may be abe to achieve the main tria within the origina resource, there is aso the serious possibiity that, simpy on account of the compexity of the work being undertaken, an extension of 4 6 months may be necessary. This is an area I woud wish to discuss with HTA. 126 NIHR Journas Library

157 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Personne In genera, the tria is reasonaby we resourced. Processes around supporting practices, undertaking the (highy compex) research training procedures, for exampe targeting reception and cinica staff, is, in its own right, an important and compex process requiring carefu support from research staff. At the same time, those research staff are undertaking maiing of questionnaires, and the review of cinica notes. These processes are proving chaenging to manage within the resource avaiabe. Given the difficuties encountered, we woud ike to propose a soution (which requires an additiona staff member to be costed into our work). We beieve that, with the aocation of this modest additiona resource, we may be abe to compete the tria within the origina time frame. The data coection is schedued to take pace in three separate waves within each of the four recruitment centres. Each wave of the intervention is associated with a period in which a review of cinica notes takes pace. It woud be of great hep to aow us to empoy somebody a suppementary research administrator post on an intermittent basis. That individua woud be specificay trained in cinica note review and undertake the buk of that work, but supported by oca research staff. Essentiay they woud be empoyed for two months, haf time, for each of the three waves in each of the four centres. This amounts to the equivaent of a 1.0 fu-time research administrator over 24 months (athough this resource wi be distributed between the four centres on an intermittent basis). Trave We have been outstandingy successfu in recruiting practices associated with each of the four research centres. Recruitment, however, has been somewhat suggish in some centres and this has necessitated casting the net wider in terms of the geography. We do therefore anticipate the requirement for an increase in the trave budget to aow research staff to visit the practices on a reguar basis, supporting practice staff personay and face to face, ensuring continuing compiance with the intervention itsef, undertaking data coection, and generay maintaining the profie of the research and research processes within the practice. Across the four centres we beieve that an additiona 15,000 woud aow for that resource to be met. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 127

158 APPENDIX 1 ESTEEM piot study ICC anaysis (17 February 2011) A NHS primary heath care contacts (incuding did not attends ). oneway sum4weekcontactsincudingdnas practiceid. One-way Anaysis of Variance for s incudin s: Sum 4-Week contacts (INCLUDING D Number of obs = 346 R-squared = Source SS df MS F Prob > F Between practiceid Within practiceid Tota Intracass Asy. correation S.E. [95% Conf. Interva] Estimated SD of practiceid effect Estimated SD within practiceid Est. reiabiity of a practiceid mean (evauated at n = 55.48) A NHS primary heath care contacts (excuding did not attends ). oneway sum4weekcontactsexcudingdnas practiceid One-way Anaysis of Variance for s excudin s: Sum 4-week Contacts (Excuding D Number of obs = 346 R-squared = Source SS df MS F Prob > F Between practiceid Within practiceid Tota Intracass Asy. correation S.E. [95% Conf. Interva] Estimated SD of practiceid effect Estimated SD within practiceid Est. reiabiity of a practiceid mean (evauated at n = 55.48) 128 NIHR Journas Library

159 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 EQ-5D index score. oneway eq51_index practiceid One-way Anaysis of Variance for eq51_index: Number of obs = 266 R-squared = Source SS df MS F Prob > F Between practiceid Within practiceid Tota Intracass Asy. correation S.E. [95% Conf. Interva] Estimated SD of practiceid effect Estimated SD within practiceid Est. reiabiity of a practiceid mean (evauated at n = 42.01) Sensitivity anayses assuming data is dichotomous. oneway sum4wk_dna_cat practiceid One-way Anaysis of Variance for sum4wk_dna t: Number of obs = 346 R-squared = Source SS df MS F Prob > F Between practiceid Within practiceid Tota Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 129

160 APPENDIX 1 Intracass Asy. correation S.E. [95% Conf. Interva] Estimated SD of practiceid effect Estimated SD within practiceid Est. reiabiity of a practiceid mean (evauated at n=55.48). oneway eq51_index_cat practiceid One-way Anaysis of Variance for eq51_index t: Number of obs = 266 R-squared = Source SS df MS F Prob > F Between practiceid Within practiceid Tota Intracass Asy. correation S.E. [95% Conf. Interva] Estimated SD of practiceid effect Estimated SD within practiceid Est. reiabiity of a practiceid mean (evauated at n = 42.01) 130 NIHR Journas Library

161 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 The process evauation piot study Aims and objectives The process evauation consisted of two discrete phases; an initia 1-year piot study conducted aongside, and feeding into the piot of the main tria, foowed by the main process evauation study. This is the report from the process evauation component of the piot. The specific aims of the process evauation piot were to (1) ensure that the triage training and interventions coud be feasiby deivered by practices and (2) test and fine-tune data coection methods. Methods The piot study was conducted in six genera practices in Bristo and Devon between 2010 and It was designed to inform both the subsequent process evauation and the main ESTEEM tria. The practices had been randomy aocated to two each of GPT, NT and UC. They received training on the triage interventions and coected outcome data as they woud in the main tria, but using ony a 2-week period of patient recruitment. Data coection for the piot consisted of systematic observation and in-depth interviews. A tota of 49 hours was spent observing reception and triage staff activities in the six practices. In addition, 47 semistructured interviews were hed with a purposivey seected sampe of patients and staff. The week-ong observations of practice staff, patient actions and interactions, and other naturay occurring events were undertaken during the impementation period of the study interventions. Data were recorded in the form of brief paper-based fied notes. Observation took pace mainy in communa areas of the practices, such as patient waiting rooms, but aso in more private spaces, such as receptionists areas. Locations were chosen in order to gain a good appreciation of the variety of actions, interactions and events occurring in each practice, whie remaining sensitive to issues of participant trust. Observation focused on factors known to infuence the impementation and incorporation of compex heath interventions, aiming to iustrate the context and mechanisms operating and interacting in the management of patients same-day consutation requests. Written or audio-taped fied notes recorded activities reating to the running of the study and the deivery of the interventions rather than the usua business of the practices. Interview data were transcribed verbatim and anaysed thematicay across tria arms. Observations and findings were summarised systematicay and reported at reguar intervas to the ESTEEM Tria Management Group. Successfu aspects of tria protoco and procedure were noted and retained and the team deveoped soutions to any identified probems, which were triaed and re-reported as an iterative process of refining procedures. Semistructured interviews were conducted with six patients and 37 practice staff members, using an interview schedue deveoped by the Research Team and the Patient Group (Tabe 38). The patients were from three practices, two providing NT and one providing GPT. Participant patients were of both sexes across a range of ages. Patients were asked to refect on their experiences of accessing and receiving care. Staff interview participants comprised practice managers, receptionists, and nurses or doctors deivering the interventions in six practices: two NT practices, two GPT practices and two UC practices. TABLE 38 Staff interview participants in the piot study Practices GPs Practice nurses/nurse practitioners Receptionists Practice/reception managers Totas GPT NT UC Totas Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 131

162 APPENDIX 1 Fifteen cinicians and 22 administrative staff from across six tria practices participated. Staff were asked about the feasibiity and practicaity of the data coection methods and the impementation of the intervention. Fied notes and interview notes were anaysed systematicay to identify successfu and ess successfu aspects of the tria protoco. Notes were made of what did and did not work we: probems noted by each group of practice staff were recorded, with reasons for any probems, if discernibe, and things that went we were aso noted. These notes and exampes were discussed by the research team, and potentia probems, and soutions to probems, were identified. Where appropriate, such soutions were fed back to participating practice teams for consideration, comment and ratification before proceeding to the fu tria. Resuts The report on the substantive findings from the piot research interview data on how triage and the tria were experienced can be seen at the beginning of this report. This section reports on how findings from the piot informed the deveopment of the tria protoco, procedures and paperwork in preparation for the main tria. Engagement in the tria was a probem for some practice staff. Some receptionists in particuar had anxieties about their roe in teephone triage, perceiving it in as forcing them into the roe of gate-keepers, representing erecting a barrier to access to doctors appointments. This coud ead to them inadvertenty subverting the protoco, for instance by putting patients down for triage as we as booking them an appointment; reverting to usua care; omitting to inform patients they were part of a tria and that they woud receive questionnaires; and not expaining triage in a cear and effective way. It was aso observed that og sheets were not being competed with accuracy. These issues were addressed by encouraging managers to invove and communicate effectivey with receptionists, and in particuar to stress the positive aspects of the tria, and by deveoping, with the receptionists coaboration, a script for expaining the tria to patients. The tria fow chart was improved to make it easier to understand, and these measures were integrated into the training strategy. Training was aso amended to address motivationa probems, particuary for receptionists, whose roe is crucia because they faciitate the point of patient entry into the tria. Emphasis was paced on such positive benefits of triage as patient equity and the reduction of pressure on receptionists themseves in not having to find appointments. As competion of the ogs was reviewed, the need was identified for them to be amended to differentiate between patients who are given same-day appointments, patients who request same-day appointments but do not get one, and patients requesting a teephone consutation but not a face-to-face appointment. The reevant changes were made. It became cear that not a practice staff members had received adequate information about the tria, and consequenty fet aienated or uninvoved. Procedures were revised to ensure that staff members were each sent a personay addressed, roe-specific information pack. It was observed that to increase the fuency and ownership of the tria with receptionists and other staff then practices need to have a key senior person, a champion, to direct and drive it and address staff queries and concerns, and it was agreed that this woud be encouraged by the researchers when estabishing the tria in the practices. A number of other training issues were identified. GPs needed to be confident with the system for booking appointments. Triaging nurses shoud be computer iterate and confident in using the decision support software, which itsef shoud be ensured as working we before commencement of triaging. The piot reveaed ambiguities for which guideines needed to be deveoped. One such was the reuctance of some practices to incude infants and young chidren in the triage protoco, and it was decided that practices coud excude them they wished. Poicies were aso deveoped for the procedure if patients refused to be triaged. Another ambiguity was defining what exacty constitutes a same-day appointment request, for exampe when the patient does not specify. Poicy on whether use of the decision support software was mandatory for triaging nurses aso had to be defined, and it was decided that nurses woud be encouraged, but not compeed to use the software. 132 NIHR Journas Library

163 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Processes for checking compiance with tria procedures were aso deveoped. Effective communication channes between the practices and tria researchers were put in pace, in particuar a named key person to contact in each organisation, foowing suggestions from practice managers. Guidance and cear definitions, such as on what is the end time were put in pace to ensured harmonisation of the competion of Cinician Forms, which had been shown to be done somewhat haphazardy. Cinicians concerns about the eiciting of verba but not written consent for the team to access patient notes were addressed by reassurance and evidence of ethica approva for this, and, in one case, a persona visit from a senior member of the study team. Combining triage with the duty doctor roe was observed to potentiay compromise safety, and practices were advised in training not to do this, athough some continued to do so. Practices were advised to triage consistenty during set periods, which had been observed not to be happening. Concusion The quaitative piot process evauation provided insights into the processes and experiences of estabishing a tria in practice microcontexts, which enabed amendments and adjustments to be made to the tria protoco, procedures and materias to faciitate the smooth introduction and conduct of the main RCT. It enabed protoco amendments to be submitted for ethica approva in a timey manner, which did not deay or compromise the smooth running of the tria once it was under way. Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 133

164

165 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Appendix 2 Summary of changes to origina ESTEEM protoco Change to protoco Date Before piot study Amendments to the origina patient information eafet [Tria], patient invitation etter [process evauation] and the thanks but not needed etter to patients [process evauation] and incusion of an incentive for questionnaire respondents 16 October 2009 Between piot study and main tria Amendments to process evauation recruitment materias advised by PPI group 2 August 2010 Amendments to Main Tria recruitment materias advised by PPI group 26 August 2010 Use of verba consent to case note review 29 December 2010 Transfer of patient names and addresses 18 March 2011 Amendments to patient documentation foowing end of piot and use of Waiting Room Posters 1 June 2011 During main tria Use of amended mai-out materias for missed patients at one practice 30 Juy 2012 Use of postcode data to cacuate a deprivation score 3 September 2012 Change of study Sponsor from Devon PCT to Roya Devon & Exeter NHS Foundation Trust 30 October 2013 Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 135

166

167 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Appendix 3 Audit Log Sheet JS 9:06 Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 137

168

169 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Appendix 4 Receptionist fow charts Receptionist fow chart (genera practitioner triage) Phone ca received Is the patient phoning for a face-to-face consutation with a GP or nurse practitioner at the surgery? Yes Does the patient request/prefer to be seen today? Is the patient we enough for the tria? Too i incudes severe chest/abdomina pain, severe difficuty breathing, vomiting bood, severe psychiatric probems, atered consciousness, seizures, pregnancy probems Yes Yes Can the patient communicate without difficuty? Yes Patient IS ELIGIBLE for the tria No No No No Patient is NOT ELIGIBLE for the tria Dea with in usua way Book a triage appointment Triage appointment not booked Te the patient about the triage We re trying out a new system where patients asking to be seen today wi be caed back by a doctor, who may be abe to hep over the phone. If you and the doctor decide that you do need to be seen today, the doctor wi make an appointment. We may send you a questionnaire in 4 weeks time. Add patient to the triage ist Note the patient s ca back teephone number Ask if s/he woud mind sharing nature of probem Add patient to triage ist and make notes on screen Mark patient as ESTEEM DD/MM Te patient roughy when doctor wi ca back (if known) Book face-to-face or teephone appointment for TODAY? Book appointment for today Mark patient as ESTEEM DD/MM Book face-to-face or teephone appointment for ANOTHER DAY? Book appointment on another day Mark patient as ESTEEM DD/MM No appointment booked/other outcome? Add patient to the unbooked session Mark patient as ESTEEM DD/MM Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 139

170 APPENDIX 4 Receptionist fow chart (nurse triage) Phone ca received Is the patient phoning for a face-to-face consutation with a GP or nurse practitioner at the surgery? Yes Does the patient request/prefer to be seen today? Is the patient we enough for the tria? Too i incudes severe chest/abdomina pain, severe difficuty breathing, vomiting bood, severe psychiatric probems, atered consciousness, seizures, pregnancy probems Yes Yes Can the patient communicate without difficuty? Yes Patient IS ELIGIBLE for the tria No No No No Patient is NOT ELIGIBLE for the tria Dea with in usua way Book a triage appointment Triage appointment not booked Te the patient about the triage We re trying out a new system where patients asking to be seen today wi be caed back by a nurse, who may be abe to hep over the phone. If you and the nurse decide that you do need to be seen today, the nurse wi make an appointment. We may send you a questionnaire in 4 weeks time to see how you found the new system. Add patient to the triage ist Note the patient s ca back teephone number Ask if s/he woud mind sharing nature of probem Add patient to triage ist and make notes on screen Mark patient as ESTEEM DD/MM Te patient roughy when nurse wi ca back (if known) Book face-to-face or teephone appointment for TODAY? Book appointment for today Mark patient as ESTEEM DD/MM Book face-to-face or teephone appointment for ANOTHER DAY? Book appointment on another day Mark patient as ESTEEM DD/MM No appointment booked/other outcome? Add patient to the unbooked session Mark patient as ESTEEM DD/MM 140 NIHR Journas Library

171 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Receptionist fow chart (usua care) Phone ca received Yes Is the patient phoning for a face-to-face consutation with a GP or nurse practitioner at the surgery? No Does the patient request/prefer to be seen today? Is the patient we enough for the tria? Too i incudes: severe chest/abdomina pain, severe difficuty breathing, vomiting bood, severe psychiatric probems, atered consciousness, seizures, pregnancy probems Yes Yes Yes No No Patient is NOT ELIGIBLE for the tria Dea with in usua way Can the patient communicate without difficuty? No Yes Patient IS ELIGIBLE for the tria Te the patient We re currenty evauating our same-day appointment system and you may be sent a questionnaire in 4 weeks time to ask for your views. Book face-to-face or teephone appointment for TODAY? Book face-to-face or teephone appointment for ANOTHER DAY? No appointment booked/other outcome? Book appointment for today Mark patient as ESTEEM DD/MM Book appointment on another day Mark patient as ESTEEM DD/MM Add patient to the unbooked session Mark patient as ESTEEM DD/MM Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 141

172

173 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Appendix 5 Questionnaire for withdrawn practices QUESTIONNAIRE FOR WITHDRAWN PRACTICES To hep us understand the reasons why practices were unabe to continue participation in ESTEEM pease compete the form beow: Practice Name: Randomised to: Competed by: Date: What were the main reasons for withdrawing from ESTEEM? (tick a that appy): 1. Dissatisfaction with the Tria arm randomised to. 2. The practice was under-resourced to cope with the demands of ESTEEM (tick a that appy): Difficuty recruiting additiona staff Difficuty extending staff hours as needed Other, pease provide detais 3. Your practice experienced unexpected extenuating circumstances (tick a that appy): Long-term sickness of key staff member(s) Retirement/resignation of key staff member(s) Other, pease provide detais 4. Insufficient buy-in from a members of practice staff 5. Lack of support from the research team 6. Change of practice IT system 7. Change of practice premises 8. Triage system impemented between being recruited and starting the tria 9. Other, pease state: Pease add any further comments overeaf Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 143

174

175 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Appendix 6 Covering invitation etters Covering etter to patients, on practice-headed note paper <Surgery address> <Patient address> <Date> <patient computer ID> Dear <Name of patient>, About four weeks ago you teephoned our surgery requesting a same-day consutation. The doctor or nurse you saw or spoke to may have briefy mentioned that our surgery is invoved in a research study. The study, ed by the Peninsua Medica Schoo in Exeter, ooks at how we provide care for patients wanting a same-day consutation. In particuar it ooks at how we might continue to provide high-quaity care when there are not enough same-day consutations to meet demand. Our surgery is fuy in support of this important piece of research. The resuts coud hep to improve the way that heath services are deivered by the NHS, ocay and nationay. I have encosed a brief questionnaire about the above arrangements. Pease hep us by competing it and returning it in the pre-paid enveope suppied. If you saw or spoke to one of our doctors or nurses, at the end of your consutation they may have asked for your permission to aow one of the researchers to ook at your medica records. At the end of the questionnaire you wi find a consent form. If you have now changed your mind, and woud or woud not ike a researcher to ook at your medica records, you can et us know by competing this form. If you did not have the opportunity to agree or disagree to your records being reviewed, pease compete the consent form now. The encosed information eafet contains further information about the study. If you have any queries, pease do not hesitate to contact the researchers directy, using the detais given at the end. If you do not wish to take part in the study, I woud be gratefu if you woud sti return the bank questionnaire in the enveope provided. We wi then know that you do not wish to receive any further contact from us about this study. Your assistance woud be of great vaue and very much appreciated. Yours sincerey, <Name of GP or Practice Manager> Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 145

176 APPENDIX 6 Covering etter to parents, on practice headed note paper <Surgery address> <Patient address> <Date> <patient computer ID> Dear parent/guardian of <Name of patient>, About four weeks ago you teephoned our surgery requesting a same-day consutation for your chid. The doctor or nurse you saw or spoke to may have briefy mentioned that our surgery is invoved in a research study. The study, ed by the Peninsua Medica Schoo in Exeter, ooks at how we provide care for patients wanting a same-day consutation. In particuar it ooks at how we might continue to provide high quaity care when there are not enough same-day consutations to meet demand. Our surgery is fuy in support of this important piece of research. The resuts coud hep to improve the way that heath services are deivered by the NHS, ocay and nationay. I have encosed a brief questionnaire about the above arrangements. Pease hep us by competing it on your chid s behaf and returning it in the pre-paid enveope suppied. If you saw or spoke to one of our doctors or nurses, at the end of your consutation they may have asked for your permission to aow one of the researchers to ook at your chid s medica records. At the end of the questionnaire you wi find a consent form. If you have now changed your mind, and woud or woud not ike a researcher to ook at your chid s medica records, you can et us know by competing this form. If you did not have the opportunity to agree or disagree to your chid s records being reviewed, pease compete the consent form now. The encosed information eafet contains further information about the study. If you have any queries, pease do not hesitate to contact the researchers directy, using the detais given at the end. If you do not wish to take part in the study on your chid s behaf, I woud be gratefu if you woud sti return the bank questionnaire in the enveope provided. We wi then know that you do not wish to receive any further contact from us about this study. Your assistance woud be of great vaue and very much appreciated. Yours sincerey, <Name of GP or Practice Manager> 146 NIHR Journas Library

177 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Appendix 7 Patient information eafet HTA Tria No. 08/53/15 A study about your experience of getting care from a GP or Nurse PATIENT INFORMATION LEAFLET Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 147

178 APPENDIX 7 This eafet contains an invitation to take part in the above study. It tes you why the study is being done and what it woud invove for you. Pease take time to read it through carefuy, so that you can decide whether or not you woud ike to take part. THE RESEARCH STUDY Patients often ring their surgeries asking for same-day consutations. Not a such patients want or need to see a doctor or nurse face-to- face on each occasion. Some peope, such as those who work during the day or have caring responsibiities, may find it easier to get prompt advice over the phone, rather than having to visit the practice in person. This system of assessing how best to meet patients needs is caed teephone triage. Our study ooks at how teephone triage systems ed by doctors and nurses work. We are particuary interested in how these systems affect patients experiences, safety and heath, as we as practices workoads and costs. Your surgery is one of 42 across the country that has kindy offered to hep us with this study. Each of the 21,000 or so patients from these surgeries who are invited to participate wi have asked for a same-day consutation. This eafet invites you to join the study. If you decide to take part, we woud ike you to fi in the short questionnaire encosed with this eafet. This wi te us how you got on with the system used by your surgery to manage same-day consutations. This study is being carried out by a team of experienced researchers based in universities in Bristo, Exeter, Norwich and Warwick. We are avaiabe to answer your questions. You wi find your oca research teams contact detais at the end of this eafet. Thank you for taking the time to read this eafet 148 NIHR Journas Library

179 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 What is the purpose of this study? This study ooks at new ways to improve patients access to services when they ask for same-day consutations. In particuar, it aims to find out how we teephone triage systems ed by doctors and nurses work. We are especiay interested in any differences that these systems make to patients experiences of care, safety and heath, as we as to surgeries workoads and costs. Your surgery is taking part in this study. Whie the study is running, when you phone to ask for a same-day consutation, a doctor wi phone you back. S/he wi tak to you about your needs, to see if s/he can hep you over the phone. If you and the doctor decide that you sti need to come into the surgery, then s/he wi make an appointment for you. Who is taking part in this study? A adut patients or parents of chidren who have contacted their doctors surgeries asking for same-day consutations with a doctor or nurse during the period of the study may be asked to take part. What wi happen to me if I decide to take part in this study? If you decide to take part in this study, we woud ike you to compete and return the brief questionnaire encosed. This shoudn t take more than about 15 minutes of your time. The questionnaire asks for your views on your surgery s same-day consutation service. On the day of your consutation, a heathcare worker may have asked if you woud give permission for one of our researchers to ook at your medica records. The encosed questionnaire asks the same question. If you originay said that that it woud be OK for us to ook at your records, but have now changed your mind, you can opt out. Simiary, if you originay said no to us seeing your records, and have now changed your mind, you can opt in. If you et us ook at your records, we wi ony coect information on the number and ength of any visits that you made to primary care services (e.g. your surgery, A&E, a wak-in centre) within a month of making your same-day consutation request. Doing this wi et us compare the effects of the different systems that we are studying. The questionnaire aows you to say yes or no to having your records reviewed. No sensitive information about your detaied medica history wi be recorded, either way. Do I have to take part? Wi my decision affect the services that I receive? We do hope that you wi be wiing to compete the encosed questionnaire, but your decision about whether to take part in the study wi in no way affect the services that you receive. It is entirey up to you whether or not you decide to take part in the study. You may aso withdraw from the study at any time, without giving any reason. What are the possibe risks or disadvantages in taking part? As far as we know, there are no particuar risks or disadvantages invoved in taking part in this study. The surgeries taking part in the study have processes in pace to ensure that Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 149

180 APPENDIX 7 any probems that might arise from the triage systems being used are spotted eary on and resoved. What are the possibe benefits of taking part? Taking part in this study woud aow you to have your say about your heathcare services, heping us to ensure that such services are provided in the best possibe way. Your contribution coud therefore hep to improve things for other peope at your surgery and/or using genera practice services nationay. Wi the information I provide be kept confidentia? Yes. A information about you wi be handed in strict confidence in accordance with ethica and ega practice. Your decision to take part in the study wi aso be kept confidentia. The ony time that we woud pass on information, for exampe to your doctor, woud be if something in your questionnaire made us think that you, or someone ese, was at risk of serious harm. Normay, we woud ony pass on information in this way after discussing the situation with you. Your medica records wi ony be reviewed by a study researcher if you give us permission to do so. A persona information wi be stored securey in ine with the Data Protection Act (1998) for up to a year and then destroyed. Competed questionnaires wi aso be destroyed after 5 years. What wi happen if I don t want to carry on with the study? If you decide to take part in the study and then change your mind, you may stop taking part at any time without having to give a reason. If this shoud happen, genera information coected up to that time, which does not identify you (such as your gender and age) might sti be kept. This woud et us see if patients who eft the study differed in some way from those who stayed. It woud be important to know this because, if the peope who eft and stayed were different, it might bias the study resuts. If you do decide to withdraw from the study then pease ca a member of the research team our contact detais are at toward the end of this booket. What if there s a probem? Any compaint about the way that you have been deat with during this study wi be prompty and thoroughy addressed. If you have any compaints or concerns about any aspect of the study itsef or the research team s work, pease fee free to discuss them with us. You wi find our contact detais at the end of this eafet. If you have more genera concerns about the way that you have been approached or treated during the study, you can access the NHS compaints mechanism via the Patient Advice and Liaison Service (PALS) on Cas made to this number are free. Aternativey, you can visit the PALS website at: How wi the resuts of the research study be used? The resuts of this study wi interest many different peope and organisations. We wi share them with the surgeries invoved and with other NHS bodies, to hep them to 150 NIHR Journas Library

181 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 improve their services. We wi aso write them up for the study s funding body, for academic pubications and conferences as we as for the genera pubic. The resuts wi aso be avaiabe on the foowing website: No individua patient wi be identifiabe in anything we write. Who is organising/funding the study? Who has reviewed it? Our study is funded by the Nationa Institute for Heath Research Heath Technoogy Assessment Programme The study has been reviewed, and its quaity assured, by the Institute s team of scientific experts and the South West 2 NHS Research Ethics Committee (Ref. 09/H0202/53). The research is sponsored NHS Devon Primary Care Trust. Which organisations are carrying out the study? The study is being ed by the Peninsua Medica Schoo, based in Exeter, and carried out by research teams based in universities in Bristo, Exeter, Norwich and Warwick. Want to know more about the study or questionnaire? If you have any questions about this study, pease do not hesitate to contact us. Dr. Raff Caitri Researcher (ESTEEM) Peninsua Medica Schoo Smea Buiding St. Luke s Campus University of Exeter EX1 2LU Mrs Emiy Fetcher Tria Manager (ESTEEM) Peninsua Medica Schoo Smea Buiding St. Luke s Campus University of Exeter EX1 2LU Emai: raff.caitri@pcmd.ac.uk Emai: emiy.fetcher@pcmd.ac.uk Te: Te: Thank you for taking the time to read this information eafet Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 151

182 APPENDIX 7 Sti not sure whether to take part? For independent advice about taking part in research studies in genera you can contact 'INVOLVE', an organisation that offers advice and information on patient/pubic research in the NHS, at: Wessex House, Upper Market Street, Easteigh, Hampshire, SO50 9FD Teephone: Textphone: Fax: Emai: admin@invo.org.uk Website: If you ive in Devon, 'Fok-Us' offers simiar advice & information at: Room 407, Noy Scott House, Roya Devon & Exeter Hospita, Barrack Road, Exeter, Devon, EX2 5DW. Teephone: Fax: Emai: fok.us@exeter.ac.uk Website: NIHR Journas Library

183 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Appendix 8 Patient questionnaire Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 153

184 APPENDIX NIHR Journas Library

185 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 155

186 APPENDIX NIHR Journas Library

187 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Appendix 9 Receptionist Tria Log Sheet JS 9:06 Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 157

188

189 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Appendix 10 Adverse events reporting procedure Report of a probem/compaint received by practice about a patient from any source during tria period Cinica ead prompted via emai each week to compete and return the Adverse Events og (even if zero) to the research team Principa investigator determines whether compaint/report is Serious adverse Adverse Attributabe to ESTEEM Not attributabe to ESTEEM Attributabe to ESTEEM Not attributabe to ESTEEM Agreed with the cinica ead in the practice Serious adverse ESTEEM Serious adverse not ESTEEM Adverse ESTEEM Adverse not ESTEEM Compete serious adverse event form and send to DMC, REC, RM&G and sponsor Routine adverse event records form report to DMC at the end of each wave Case note review at 12 weeks deaths and emergency admissions within 7 days of initia consutation FIGURE 11 Fow diagram representing the adverse events reporting procedure for the ESTEEM tria. DMC, Data Monitoring Committee; REC, Research Ethics Committee; RM&G, Research Management & Governance. The purpose of this process is to detect any probems during the period that the intervention is running, through reports of probems/compaints received by the practice. A reports of events wi be coected by the research team, prompting the Cinica Lead to review probems/compaints recorded by the practice and to compete (even if to state no events ) and return the Adverse Events og. Practices wi be asked to report anything that is of major concern to the research team in rea time. The assessment of seriousness and attribution of probems wi be undertaken by the principa investigator using the foowing rues: Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 159

190 APPENDIX 10 Seriousness A serious adverse event incudes: i. death ii. serious iness or substantia deterioration in heath eading to A&E attendance and/or emergency hospita admission. An adverse event incudes: i. deay in treatment with no direct resutant harm ii. a patient or heath professiona making a compaint to the practice regarding process of care. Attribution Events attributabe to ESTEEM Events that occur as a suspected resut of the mode or timing of the initia consutation coud incude: misdiagnosis/assessment of the severity of a condition or deay in treatment due to not seeing a patient face to face (e.g. skin maignancy, giving advice/prescribing antibiotics over the teephone for a chest infection which is actuay more severe, meningitis abeed as vira iness, dyspepsia that turns out to be cardiac ischaemia, abdomina pain that becomes a ruptured appendix, deay in treatment for a stroke/meningitis/myocardia infarction) faiure to manage infections adequatey over the teephone, for exampe not recognising the need for antibiotics or prescribing antibiotics unnecessariy misunderstanding of medication instructions over the teephone, eading to side effects patient confusion over changes to the appointment system, for exampe patients not turning up for appointments deay in treatment of patients at the practice caused by safeguarding the time of a cinician, for exampe a nurse who is responsibe for cervica smears may not have as much time to do them if time is taken up triaging for ESTEEM increase in cinician stress, deterioration in heath and/or absence from work owing to initia experience with an intense workoad of teephone cas compaint reported by a patient or heath professiona regarding the initia consutation, for exampe timing or content. 160 NIHR Journas Library

191 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Events not attributabe to ESTEEM Events that are unreated coud incude: death, A&E attendance or hospita admission due to a cause unreated to that discussed during teephone triage an event that woud have happened regardess of the mode of consutation, for exampe drug-reated reaction. If a serious adverse event is deemed to be attributabe to ESTEEM, the Tria Manager wi report this within 24 hours, via fax, using the serious adverse event form, to the Research Ethics Committee (REC), Research Management & Governance (RM&G) units, the Data Monitoring Committee (DMC), and sponsor. A retrospective search of practice records wi coect data on patient deaths during the tria, to undergo the same process of determining attribution. A separate process of case note review wi be undertaken by the research team 12 weeks after patients are entered into the tria, which wi assess A&E attendances and emergency hospita admissions. Case note review at 12 weeks The study protoco states deaths within 7 days of same day consutation request (from practice records), and attendance at A&E within 4 weeks and number and ength of stay of emergency hospita admissions within 7 days of index consutation (from primary care records examined 12 weeks after same day consutation request). The events isted above that are found at case note review wi not be subject to the same assessment of seriousness and attribution as detaied for the events reported weeky by practices to the research team. Count data of events recorded at case note review wi be presented to the DMC as the tria progresses and reported in fu at the end of the tria. Recording emergency hospita admissions The researcher conducting the case note review wi record a hospita admissions within 7 days of the index consutation. An admission wi be detected by the presence of a discharge summary in the patient s record. Notes: Patient admitted to a hospita ward via A&E If there is ony one discharge summary = one contact (admission) If a separate discharge summary reating to the A&E contact has not been generated in addition to the ward admission, this may indicate that the admission via A&E was arranged by a GP or admitting team. If there are two discharge summaries = two contacts (one A&E pus one admission) A separate discharge summary generated for the A&E contact and for the ward admission may indicate that the patient presented themseves to A&E and was subsequenty admitted to a ward. Patient admitted directy to a hospita ward, Medica Assessment Unit or equivaent (not via A&E) Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 161

192 APPENDIX 10 This counts as one contact (admission), even if the patient was admitted and discharged on the same day. Patient admitted via A&E, but did not progress to a hospita ward before discharge Even if a discharge summary has been generated to suggest that a hospita admission occurred, this counts as one contact (A&E), even if the patient was admitted and discharged on the same day or on different dates. Defining emergency hospita admissions The researcher wi present a admissions found in the notes to the ead GP in the practice, for advice on whether an admission was an emergency, using the foowing definition: An emergency hospita admission for the tria is an UNPLANNED admission, irrespective of cinica need. An unpanned admission is any admission (through any route, e.g. A&E, Bed Manager, GP contacting the hospita team), irrespective of cinica need, which was arranged on the same day as the admission occurred and not before that date. An admission on the same date foowing an outpatient s appointment woud count as unpanned. In cases in which a patient is seen (e.g. by out-of-hours service) and admitted the next day, but within 24 hours, these scenarios woud count as unpanned. A panned admission is any admission arranged for further into the future, even if the person is acutey unwe. 162 NIHR Journas Library

193 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Appendix 11 Cinician Form Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 163

194

195 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Appendix 12 Case note review Case Notes Review Form Practice ID Patient ID Number DOB Gender Review Date / / Is the patient sti registered with the surgery? Yes No If no, has the patient: Moved away? (date) / / Died? (date) / / Date of same day appointment request / / Contacts within 4 weeks up to and incuding: / / Emergency Hospita Admissions within 7 days up to and incuding: / / No contacts after index consutation GP Surgery Contacts GP: In Surgery GP: Teephone GP: Home Visit GP: Unspecified Nurse: In Surgery Nurse: Teephone Nurse: Home Visit Nurse: Unspecified Genera Unspecified DNA WIC Contacts Doctor Nurse Unspecified Date Date Date Date Date Date Date Date Date Date Date / / / / / / / / / / / / / / / / / / / / / / Contact 1- Index Contact 2 Contact 3 Contact 4 Contact 5 Contact 6 Contact 7 Contact 8 Contact 9 Contact 10 Contact 11 Not Found Index contact -Shoud have Cinician Form- OOH Contacts Doctor: In Surgery Doctor: Teephone Doctor: Home Visits Nurse: In Surgery Nurse: Teephone Nurse: Home Visit Unspecified A&E Contacts A&E contact Hospita Admission** Emergency Admission Bed days Researcher Comments ** Hospita Admission Codes: Unre = Unreated admission; R-P = Reated, panned admission; R-U = Reated, unpanned admission Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 165

196

197 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Appendix 13 Overview of practice integrity checks Run-in period Data coection period Practice Check 1 Check 2 Check 3 Check 4 a Check 5 Check b b b b Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 167

198 APPENDIX 13 Run-in period Data coection period Practice Check 1 Check 2 Check 3 Check 4 a Check 5 Check b , the integrity check was passed;, the integrity check was faied on one or more aspects. a When Check 2 was faied, practices remained in run-in and further checks were arranged. b When a practice proceeded to data coection and faied its fina integrity check during the first week of data coection, it was permitted to continue and the tria researcher had further communication with the practice regarding research procedures. 168 NIHR Journas Library

199 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Appendix 14 Assessment of nurses use of computerised decision support software during ESTEEM The aim of ESTEEM was to compare GPT with NT and both of these with UC. As part of the NT arm, a CDSS package was instaed in practices. A nurses invoved in the triage were provided with fu training on the use of the software and teephone triage consutation skis. They were tod that the software was there as a support too for them to use and, athough the aim of the tria was not to evauate the software, they shoud at east open the software for a ESTEEM patients. The aim of this paper is to review the use of this software by the nurses invoved in the tria. We have CDSS data for 14 of the 15 NT practices; at one practice the software was removed before it was possibe to run the reports needed (Tabe 39). The software was instaed in a of the NT practices. Where possibe, the CDSS software was embedded into the practices current system. If this was possibe the nurse was abe to open the patient s record and then open the CDSS, meaning that the patient s demographic information was automaticay entered into CDSS from his/her medica records. For these patients we are abe to match them to the ESTEEM database using their unique computer ID number. Owing to the different systems used by practices it was not aways possibe to embed the software for a of them. For these patients we therefore do not have their computer ID number. When opening the software the nurse was required to manuay enter the patient s demographic information. This incuded either his/her age or date of birth, aong with his/her gender. These patients have been matched to the ESTEEM database using date of birth where possibe. In addition, the date of ca was compared to ensure that the correct patient was identified. When ony age was entered this was used aongside patient s TABLE 39 Summary of practices and patients for whom CDSS data were avaiabe ID Location List size ESTEEM size group Patients recruited Compete data Incompete data Incorrect date No data 103 Urban 7981 Large Urban 5974 Medium Rura 6037 Medium Urban 10,500 Large Rura 11,267 Large Urban 15,800 Sma Urban 9233 Large Rura 5949 Medium Rura 3227 Sma Inner city 5300 Medium Urban 12,000 Medium ,500 Large Urban 9400 Large Rura 3100 Sma Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 169

200 APPENDIX 14 gender and date of ca. Occasionay, it was the case that, for exampe, two 50-year-od femaes woud ring on the same day; in this case their Cinician Forms were used to identify the start time of the consutation to differentiate between them. In a sma number of cases the nurses initias were aso used if more than one nurse was triaging on any particuar day. Any dupicate contacts (cas after date of the index consutation) were removed from the database aong with any non-esteem patients. Therefore, each patient had a singe row of data in the spreadsheet. The reports pued from the software incuded a wide range of variabes, which were used to cacuate the eve of urgency of the ca and therefore indicate to the nurses how quicky the patient needed to be seen. The majority of these variabes were purey computationa and have itte or no meaning by themseves. In addition, the outcome of the ca was based on whether the nurses used the software and, if so, to what extent. However, these reports did incude the ength of the consutation and the number of questions asked by the nurse. The aim is to use these variabes to devise a measure of intensity and a proxy variabe for use of the software. The ength of assessment/number of questions asked is used as a measure of intensity. In addition, we coud group consutations into one of three groups: software opened and used, software opened but not used, and software not opened or used. Compete data means that the reports contained both ength of assessment (in seconds) and number of questions asked (this impies that the CDSS was opened and used). Incompete data means the report contained the ength of the assessment but as no presenting compaint was entered no information is avaiabe regarding the number of questions asked (this impies the software was opened but not used). Incorrect date means that athough there are compete data avaiabe for this patient the data do not reate to the index consutation. Finay, No data reates to patients for whom the CDSS was not opened or used. Tabe 40 shows the average ength of ca, number of questions asked and intensity score for each practice, aong with the percentage of patients within each practice that the software was used for. The average ength of cas was seconds or just over 4 minutes. During this time an average of 14 questions was asked, meaning that the nurses spent approximatey 23 seconds on each question. However, there does not appear to be any correation between usage of the software and the ength of ca or number of questions asked. This shows that overa the CDSS was at east opened for 71% of patients, athough these vaues varied from 95% down to just 16%. Athough the nurses were asked to open the software for a patients, the software was provided as a decision support too and therefore it was at each nurse s discretion how much they used the software. Feedback from the nurses who participated in the tria indicated that sometimes the software was not opened as it coud not be used for a particuar patient. The software was initiay deveoped for out-of-hours primary care services as opposed to a primary care setting, and, therefore, there was no option for medication queries and potentia interactions. One exampe of this was contraception, the ony question set within the software reated to emergency contraception as opposed to probems with current medication, i.e. forgetting to take a tabet. The CDSS provided considerabe support to the nurses, especiay at the beginning of the tria; however, if it was to be used as part of a wider impementation of teephone triage it woud be worth investing in software that is specificay designed for a primary care setting. 170 NIHR Journas Library

201 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 TABLE 40 Average ength of ca, number of questions, intensity score and percentage of patients for whom CDSS was used by practice Practice Average ength of ca Average number of questions asked Intensity % used % opened, not used % not used (n = 556) (n = 539) (n = 536) (n = 140) (n = 135) (n = 134) (n = 538) (n = 531) 27.2 (n = 516) (n = 494) (n = 473) (n = 467) (n = 514) (n = 480) (n = 468) (n = 435) 7.63 (n = 406) (n = 398) (n = 421) (n = 418) (n = 400) (n = 373) (n = 367) (n = 365) (n = 441) (n = 428) (n = 421) (n = 195) 9.96 (n = 195) (n = 190) (n = 131) (n = 121) (n = 121) (n = 146) (n = 140) 9.66 (n = 140) (n = 91) (n = 81) (n = 77) (n = 353) 9.5 (n = 2) (n = 1) Mean (n = 4828) (n = 4316) (n = 4234) Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 171

202

203 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO. 13 Appendix 15 Practice Profie Questionnaire The foowing information wi be used to hep the research team check whether you are eigibe to participate in the ESTEEM tria. The information wi ony be used by the research team for the purpose of the tria. Pease compete a the questions as accuratey as possibe 1. PRACTICE CONTACT DETAILS ESTEEM TRIAL PRACTICE PROFILE QUESTIONNAIRE Practice name: Practice Manager s name: Practice ID: Address: Teephone: Post code: PCT : Fax: Emai: Aternative contact name: Teephone: Emai: 2. LIST SIZE AND STAFFING Q1. What is your current practice ist size? Q2. Pease fi in the numbers of staff currenty empoyed within the practice: (incude a heath professionas working mainy in the practice, whether empoyed by the practice or PCT) Fu time Part time Fu time Part time GPs (principas or saaried) District nurses Practice nurses Midwives Nurse practitioners Heathcare assistants Other Other.. Physiotherapists Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 173

204 APPENDIX PRACTICE CHARACTERISTICS Pease put a cross to indicate your answers Q3. Woud your cassify your practice as: Rura Urban (Inner) city Q4. What computer system is used for patient records? EMIS LV EMIS PCS Microtest Vision Synergy SystmOne Other.. Q5. Pease indicate whether your practice offers any of the foowing services aimed at improving access to primary care: Nurse-ed wak-in cinic Minor injuries cinics Other service (pease specify) Other service (pease specify Q6. Pease describe the number of sessions that the practice regards as a Whoe Time Equivaent (WTE) doctor: 1 session = 4 hours of cinica activity Less than 7 Seven Eight Nine Ten More than ten 174 NIHR Journas Library

205 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO TRIAGE For the purpose of this study, we are defining teephone triage as occurring when doctors or nurses make a decision about the heathcare needs of a patient after/whie taking to them on the teephone. Q7. Pease coud you answer the foowing question by circing yes or no AND describing the typica management of same-day consutation requests at your practice for A) and B): Does the practice routiney triage on the teephone ALL patients making a same day consutation request even when you have un-booked consutation sots avaiabe? YES NOT eigibe NO Probaby eigibe Pease describe how you woud manage a same day consutation request: A) When you have avaiabe consutation sots B) When you do not have avaiabe consutation sots Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 175

206 APPENDIX 15 Pease put a cross to indicate your answers Q8. Does the practice aocate time in the diary for doctors or nurses to carry out teephone triage? Yes No (if no, pease go to Q12) Q9. How many doctors or nurses currenty carry out teephone triage at the practice? Doctors Nurses Q10. Which one of the foowing best describes how teephone triage is used by doctors/nurses at your practice: A patients seeking a consutation are triaged on the teephone A patients seeking a same-day consutation are triaged on the teephone Patients seeking a same-day consutation are triaged on the teephone ony if a the appointment sots are taken Teephone triage is not used across the whoe practice but at east one doctor triages his/her own patients Q11. What proportion of patients seeking a same-day consutation are triaged on the teephone (pease estimate)? < 25% 26-50% 51-75% % Cannot estimate Q12. Pease describe beow the prescribing responsibiities and other activities that nurses at the practice are currenty invoved in: NIHR Journas Library

207 DOI: /hta19130 HEALTH TECHNOLOGY ASSESSMENT 2015 VOL. 19 NO CONSULTATIONS/ APPOINTMENTS Pease put a cross to indicate your answers Q13. How woud you describe your consutation management / appointment system? Stricty persona ist (patients may ony see their named doctor) Patients are encouraged to see their named doctor but may see other doctors if they wish Patients may see any avaiabe doctor Other arrangement pease describe beow 6. TYPE OF APPOINTMENT SYSTEM Q14. Pease estimate what proportion of patients is seen with the foowing arrangements: Turn up and wait to be seen % Pre-booked appointment (booked at east one day in advance) Appointments booked on the day Other (specify) Other (specify) TOTAL 100 % Queen s Printer and Controer of HMSO This work was produced by Campbe et a. under the terms of a commissioning contract issued by the Secretary of State for Heath. This issue may be freey reproduced for the purposes of private research and study and extracts (or indeed, the fu report) may be incuded in professiona journas provided that suitabe acknowedgement is made and the reproduction is not associated with any form of advertising. Appications for commercia reproduction shoud be addressed to: NIHR Journas Library, Nationa Institute for Heath Research, Evauation, Trias and Studies Coordinating Centre, Apha House, University of Southampton Science Park, Southampton SO16 7NS, UK. 177

208 APPENDIX 15 Thank you for your time Pease return this questionnaire to the Peninsua Medica Schoo in the repy paid enveope provided. OR Fax to: NIHR Journas Library