A systematic review for palliative care clinical indicators for pain. Final report

Transcription

1 A systematic review for palliative care clinical indicators for pain Final report

2 This report was prepared for the Department of Health by: Pain & Palliative Care Department Peter MacCallum Cancer Centre St Andrews Place East Melbourne Victoria Locked Bag 1 A Beckett Street Victoria 8006 Australia Phone Authors: Alex Brando and Odette Spruyt If you would like to receive this publication in an accessible format please phone using the National Relay Service if required. This document is available as a PDF on the internet at < Copyright, State of Victoria, Department of Health 2013 This publication is copyright, no part may be reproduced by any process except in accordance with the provisions of the Copyright Act Authorised and published by the Victorian Government, 50 Lonsdale St, Melbourne. Except where otherwise indicated, the images in this publication show models and illustrative settings only, and do not necessarily depict actual services, facilities or recipients of services. May 2013 ( )

3 A systematic review for palliative care clinical indicators for pain Final report

4 Acknowledgements The Systematic Review of Palliative Care Clinical Indicators for Pain project was funded by the Victorian Department of Health to support the work of the Palliative Care Clinical Network. The project team acknowledges the assistance of CareSearch in conducting the literature review. We also thank UltraFeedback for their pro bono support in developing and hosting the web survey. The project team would like to express its appreciation to the project advisory committee for their guidance and support. The project team is also grateful to those who contributed greatly to this project by participating in interviews and the online survey. Successful completion of this project would not be possible without their kind support.

5 Contents Executive summary 1 Introduction 3 Background 3 Project objectives 3 Project scope 3 1. Methodology Literature review Expert panel Round 1 appropriateness voting Round 2 necessity voting Result validating In-depth interview and survey Survey design Data analysis 8 2. Results and discussion Literature review result Expert panel result Outcomes from round 1 appropriateness 12 Outcomes from round 2 necessity Validating voting results Result and discussion Survey result Respondent profi le Tools and quality improvement initiatives in use Data entry Top benefi ts and barriers Cluster analysis result Recommendations General recommendations Minimum requirement Individual organisation Statewide implementation 35

6 3.2 Recommended solutions to barriers General resource issues Lack of IT support Low awareness within an organisation Leads to extra work for staff Lack of ongoing funding to support data entry Lack of staff with the appropriate skills Lack of support from staff Lack of support from management Low priority within our organisation Not related to patient care Suggestions for services within each group characterised in the survey General suggestions The enthusiast group The conservative group The cautious group 42 Appendix 1: Databases searched 44 Appendix 2: Terms used for literature search and results 45 Table A1: Search strategy applied to all databases 45 Table A2: The PubMed search strategy incorporating the CareSearch palliative care fi lter 45 Appendix 3: Candidate indicators for expert panel 46 Appendix 4: Survey content 68 Appendix 5: Data specifi cation for clinical indicators for pain 70 Defi nitions and benchmarks of recommended indicators 70 Data elements required for each indicator 73 Data elements defi nition 74 References 75

7 List of tables Table 1: List of agreed indicators from round 1 12 Table 2: List of changes to indicators from round 1 13 Table 3: List of indicators complied with the relaxed rule from round 1 13 Table 4: List of highly discordant indicators from round 1 14 Table 5: List of indeterminate indicators from round 1 14 Table 6: List of agreed indicators from round 2 15 Table 7: List of changes to indicators from round 2 15 Table 8: List of indicators complied with the relaxed rule from round 2 16 Table 9: Discipline 20 Table 10: Top three benefi ts and barriers 21 Table 11: Group vs service type 24 Table 12: Group vs region 25 Table 13: Group vs service delivery areas 26 Table 14: Group vs staff number 27 Table 15: Group vs usage time of quality improvement projects 28 Table 16: Group vs usage time of LCP 29 Table 17: Group vs usage time of assessment tools 30 Table 18: Indicative coordination time for the three phases 34 Table 19: Breakdown of indicative time requirement 35 Table 20: Breakdown of indicative time requirement of network coordination 37 v

8 List of figures Figure 1: Indicators by type 10 Figure 2: Indicators by setting 11 Figure 3: Indicators by detail level 11 Figure 4: Service types of respondents 18 Figure 5: Service delivery areas of respondents 18 Figure 6: Health regions of respondents 19 Figure 7: Staff numbers of respondents 19 Figure 8: Tools and quality programs in use by respondents 20 Figure 9: Perception profi les of respondent groups 23 Figure 10: Perception profi les of respondent groups radar 23 Figure 11: Distribution of groups by service type 24 Figure 12: Distribution of groups by departmental region 25 Figure 13: Service delivery areas of groups 26 Figure 14: Distribution of Groups by staff numbers 27 Figure 15: PCOC participation time in groups 28 Figure 16: LCP participation time of groups 29 Figure 17: Assessment tools using time of groups 30 Figure 18: Factors and stages of implementation for individual organisations 32 Figure 19: Stages of implementation for organisational level and network level 35 vi

9 Executive summary This project is one of four initial projects undertaken on behalf of the Palliative Care Clinical Network (PCCN) to address a key priority area of Victorian palliative care strategic directions (Department of Health 2011), namely Providing quality care supported by evidence. This project has three main objectives. The fi rst is to undertake a review of the evidence base for clinical indicators for pain in palliative care practice. The second is to recommend a group of clinical indicators for pain to the PCCN, for statewide implementation. The third and major part of the project is to develop a business case for organisations to use in adopting these indicators into their organisational operational plans. A comprehensive literature review was conducted and an expert panel established to determine candidate clinical indicators to recommend. An important factor to note is that a Cochrane-style systematic review was not within the remit of this project. A survey was conducted to determine major perceived barriers and benefi ts to implementing these indicators. Service providers attitudes towards quality improvement programs were also assessed via the online survey. The literature review used a broad search strategy provided by CareSearch. Seventy-four references were identifi ed from several major health databases, including Medline ( ) and Embase ( ). An additional 22 references were identifi ed through reference tracking. From these references, together with guidelines and policies published by government departments and Australian and international organisations, the researchers identifi ed 113 clinical indicators for pain, which fall into four major categories: outcome, structure, process and symptom. This project focused on process indicators for clinical-level quality improvement to complement and avoid duplication with national projects such as the Palliative Care Outcomes Collaboration. From these, 29 indicators were chosen by the project team for consideration by a seven-member expert panel with backgrounds in medical, nursing and management, and derived mostly from the Victorian palliative care sector. The expert panel followed a modifi ed RAND/UCLA Appropriateness Method approach to achieve a fi nal set of indicators, rating the 29 indicators on the following criteria: appropriateness (assessing validity, feasibility and generalisability) and necessity (in terms of being fundamental to care and consistent with quality of care). An independent shadow rater with both a management and nursing background validated the expert panel s results. The resultant six indicators recommended for implementation by the Victorian Department of Health through the PCCN were pertinent to two areas of pain management, namely pain assessment and analgesic prescribing. The recommended indicators are as follows: pain assessment indicators use of a validated pain scale assessing pain for a new patient regular pain assessment analgesic prescribing indicators prescribing for breakthrough pain scheduled pain medication for severe pain providing a bowel regimen with an opioid. The online survey items were developed in conjunction with the concurrent PCCN assessment tools project and were based on interviews with fi ve palliative care managers and the research team. The survey included items about current use of tools and participation in quality improvement projects, and asked about barriers, benefi ts and attitudes towards clinical tools and quality improvement 1

10 projects in use. Respondents were asked to rate their level of agreement with the questionnaire items about specifi c barriers and benefi ts using a fi ve-point Likert scale. The survey was sent to 60 service providers, 38 of whom (63 per cent) responded. Responding services represented the range of Victorian palliative care service providers when compared with the Palliative care service delivery framework (SDF) (Aspex Consulting 2010). Compared with SDF, community services were more highly represented in survey respondents while consultancy and inpatient services were less represented. No publicly funded day hospice responded to the survey. Survey respondents service delivery areas and regions are distributed similarly to the SDF mapping. The analysis of the survey data revealed that the main benefi ts of quality improvement projects perceived by service providers were that they enable benchmarking, demonstrate care practices and upskill staff. The main perceived barriers were that they lead to extra work for staff, and that there is a lack of IT support and a lack of ongoing funds to support data entry. Three different types of services were identifi ed through cluster analysis of all perceptions and attitudes. They were named as enthusiast, conservative and cautious according to their characteristics. Locality, service type, service area and organisation size did not seem to have a very strong infl uence or correlation on perceptions. There was a pattern that those groups participating in most quality improvement projects had been using quality projects or assessment tools for a longer time than those less actively participating. To help Victorian palliative care service providers to implement the recommended clinical indicators, stages are identifi ed for both individual organisations and the statewide network. For an individual organisation, the recommended stages are: the pilot stage, the integration stage and the continuous improvement stage. Organisational change, culture building for quality improvement, education and communication are considered key factors throughout all stages. For the statewide network, the stages are referred to as: the modelling stage, the expanding stage and the collective improvement stage. Phased introduction of indicators according to site-specific capacity Members of the so-called enthusiast group would be well placed to act as models for other services in the modelling stage of statewide implementation. Organisations of the so-called conservative group will likely be ready to participate in quality improvement implementation in the expanding stage of implementation. Introducing validated pain tools into routine clinical care rather than undertaking to implement the recommended indicators is considered more suitable initially for the cautious group, with the purpose of increasing awareness and willingness to participate and graduated introduction. A detailed process and methodology are suggested for implementation in all settings, along with brief solutions against each barrier. The project team recommended the following as minimum requirements to introduce the recommended indicators: a validated tool for pain assessment to be adopted by each palliative care provider documentation of the required elements for the recommended indicators in a precise and timely way that facilitates quality improvement demonstration project(s) in typical palliative care settings to establish initial benchmarks and to showcase improvements in the quality of care for patients by implementing the recommended indicators. It is recommended that the PCCN facilitate and support the coordination of the demonstration project(s) for the modelling stage of statewide implementation. 2

11 Introduction Background As part of its strategic approach to addressing the goals of the national strategy, the Victorian Department of Health allocated funding to uncover areas of defi cit of care in the Victorian palliative care sector, and to develop strategies and identify aids that may contribute to realising the national goals. The Palliative Care Clinical Network (PCCN) was established by the Department of Health s Palliative Care team to oversee the clinical elements and implementation of: Strengthening palliative care: Policy and strategic directions (Department of Health 2011) a service delivery framework and service capability framework the Clinical Service Improvement program. In an evaluation of the palliative care policy s implementation, there were several recommendations about the key strategic priorities to be addressed for This project arises from the recommendation Establish a PCCN and a statewide program for the uptake of evidence into clinical practice under strategic direction 6 of the policy Quality care at all times. Project objectives This project has three components. The fi rst is to undertake a review of the evidence base for clinical indicators for pain in palliative care practice. The background review includes published guidelines and other key documents summarising expert opinion and recommended best practice, recognising that the evidence base for such indicators remains weak. The second component of the project is to recommend a group of clinical indicators for pain to the PCCN, for statewide implementation. The third and major part of the project is to develop a business case for organisations to use in adopting these indicators into their organisational operational plans. This component of the project focused on how the recommended indicators will become part of the quality framework for palliative care organisations across Victoria. The implementation of these indicators was tailored to different levels of palliative care service delivery, in keeping with the service delivery framework and service capability framework. Through establishing statewide clinical indicators for pain in palliative care practice, services will be supported in delivering the highest quality care for their patients. Project scope The project included a review of existing literature and quality improvement projects in Australia and internationally. The project did not include a Cochrane-style systematic review. The project recognised the quality improvement projects currently underway in Australia, including The National Standards Assessment Program (NSAP), Palliative Care Outcomes Collaboration (PCOC), Australian and New Zealand Society of Palliative Medicine (ANZSPM), CareSearch and the National Pain Summit s Pain and Palliative Care Working Group, and will draw from and align with the outcomes and recommendations of these projects wherever possible. The project aimed to provide Victorian palliative care organisations with a methodology by which the recommended indicators might be integrated into their existing quality improvement programs. 3

12 It recognised that many organisations in Victoria currently participate in PCOC, NSAP and other relevant quality improvement programs and that the recommendations of this project need to be sensitive to and support previous commitment to quality improvement. A business case was developed that will assist organisations to implement and evaluate the impact of introduction of the recommended indicators into their quality improvement activities. The project recommended to the department a frequency of reporting of data and methodology for benchmarking pain clinical indicators, working closely with departmental data analyst for palliative care. It also developed guideline(s) and requirement(s) to help organisations assess and collect clinical indicator data, as well as guideline(s) and requirement(s) for developing educational information about indicators and relative data collection tools. The project did not include implementing or piloting the tools in any palliative care setting, nor did it develop a database for organisations to collect the data. In recognition that the science of implementing medical knowledge and quality improvement is in its infancy, this project did not include research into the outcomes of implementing the recommended pain clinical indicators. 4

13 1. Methodology 1.1 Literature review A comprehensive literature review was conducted to identify current data collection and indicator use by Australian and international services. The literature review included a review of academic literature, including unpublished articles and conference abstracts, and grey literature, including reports and guidelines from service providers, governments and professional associations. CareSearch provided support for the academic literature search. A broad search strategy was applied to several major health databases (see Appendix 1) in order to identify the evidence base for clinical indicators of pain in palliative care. The references of identifi ed articles were also examined and articles included where they were unique. An internet search was also executed. Articles identifi ed were downloaded into an EndNote X3 Library for that database. After all searches were completed, individual EndNote Libraries were merged and duplicate records were deleted. Only the English language was applied. Only the Evidence-based Medicine (EBM) Reviews results list was screened for relevance. This was done because search terms had worked on the full text, rather than just title, abstract and subject fi elds, and the results were clearly not on topic. No consistent Medical Subject Heading term (MeSH term) could be identifi ed for the topic of clinical indicators, except within the Cumulative Index to Nursing and Allied Health Literature (CINAHL) database. A broad and sensitive search strategy was therefore constructed using text words, truncated where appropriate. This was applied consistently across all databases except PubMed. The PubMed search used a variation on this search. This was required for two reasons: PubMed does not allow proximity limiters and a more sophisticated palliative care fi lter can be applied in this database for the palliative care component. Please refer to Appendix 2 for detailed terms used. Identifi ed literatures were then evaluated by the project offi cer according to predefi ned procedure and research questions. A detailed literature review protocol was developed to ensure consistent quality of the review process. The review focused on the following questions: What What are the data being collected about pain to improve quality of care? What methods are people using to collect data on pain and pain management? What are they doing with those data? What settings are suggested for each data element (indicator)? Why Why are those elements chosen? What is the evidence? (literature review, expert panel decision, pilot run/trial) How strong is the evidence? (steps: classify, evaluate, summarise) How How are the data being collected? How are the data reported? How are the data analysed? How are the data used to assess and improve quality of care? Abstracts were evaluated by the project offi cer. A review of the full text was conducted where a literature was believed to be a close fi t for the purpose of answering the above research questions. 5

14 Some of literature was also reviewed by an independent reviewer according to a predefi ned literature review protocol to ensure consistency. 1.2 Expert panel A modifi ed version of RAND/UCLA Appropriateness Method (RAM) was used based on instructions from The RAND/UCLA appropriateness method user s manual (Fitch et al. 2001). Seven experts, the lower limit as recommended in the manual, were engaged in the process instead of the traditional nine, although there is no evidence showing a relationship between validity of results and the number of experts. One cycle of appropriateness voting and one cycle of necessity voting were conducted. Modifi cations of indicator defi nition, including numerator and denominator, were fi nalised between two rounds of voting. An initial list of indicators with detailed defi nition was provided before voting. The indicators and defi nitions were determined by the project team based on a previous literature review and internal discussion. Panel members were not provided with the literature reviewed by the project team. Rating decisions were expected to be made based upon the panel member s expertise and existing knowledge of clinical quality improvement, especially of current status in Victoria. Supporting literatures were only provided upon request Round 1 appropriateness voting The list of indicators was distributed together with a rating form. The fi rst round consisted of voting on the appropriateness of each indicator with consideration of the following factors: Validity: Is this indicator supported by the voter s knowledge? Feasibility: Is it feasible to use this indicator in practice? Generalisability: Is this indicator applicable for all settings or just a few of them? Each indicator was rated on a 1 9 scale, with 1 representing highly inappropriate and 9 highly appropriate. The criteria of agreement and disagreement between experts were defi ned according to The RAND/ UCLA appropriateness method user s manual. Defi nition of classifi cation rules: Strict agreement for appropriate: All seven ratings fall in the 7 9 region. Relaxed agreement for appropriate: Six ratings fall in the 7 9 region, one rating falls in the 4 6 region. Disagreement: At least one rating falls in each of two extreme regions (1 3 and 7 9). Indeterminate: All other scenarios not included above Round 2 necessity voting The result from round 1, with modifi cations based on recommendations, formed the basis for the second round of voting. This round consisted of voting for necessity, considering the following factors: Fundamental: This indicator is essential to establish quality improvement. Consistent quality of care: It will be hard to maintain quality of care to a good level without implementing this indicator. 6

15 Each indicator was rated on a 1 9 scale, with 1 representing highly inappropriate and 9 highly appropriate. The criteria of agreement and disagreement between experts were the same as round Result validating Since the project used a simplifi ed version of RAM, it is important to introduce a protection mechanism to minimise possible bias. To validate the result, a shadow rater was introduced later to go through the whole process again. Scoring results from all eight raters were examined against the same rules. 1.3 In-depth interview and survey A structured questionnaire was developed from in-depth interviews with fi ve targeted palliative care providers from diverse clinical and administrative backgrounds. The questionnaire explored the relevance of nine commonly perceived barriers and eight benefi ts. Three questions were added to explore the respondent s impression of the organisation s interest and the value given to the service provider s efforts to implement these tools and projects. The questionnaire was set up as a web survey and distributed online to 60 service providers across the state. Appendix 4 is a print version of the fi nal web survey Survey design The survey was jointly administrated by project offi cers of both projects and the survey service provider UltraFeedback. Survey purpose 1. To understand the current situation within Victorian palliative care service providers (including tools in use) and participation in quality improvement projects or initiatives. 2. To identify barriers, benefi ts and attitudes towards clinical tools and quality improvement projects in use. 3. The data will be used to form recommendations to the PCCN for implementing new tools and clinical indicators for pain. Target audience The survey was designed to be responded to by one representative from each palliative care service provider. Palliative care service provider refers to the following: if the organisation is for palliative care purpose only ( hospice or service provider means the whole organisation) if the organisation is multidisciplined ( a hospital with palliative care service or service provider means the palliative care department of the organisation). Selection criteria Participants were representatives from Victorian palliative care service provider who could best respond to the questions in the survey, ideally the person who was in charge of quality issues or head of the department or organisation. 7

16 Participant identification procedures The Department of Health provided a list with the contact details of all palliative care agencies. This list is available publicly. Survey delivery method The survey was delivered in the form of online web survey. A cover letter on behalf of the Department of Health and PCCN, plus a link to the online survey, were distributed by UltraFeedback. The survey information was included in the weekly newsfl ash provided by Palliative Care Victoria. Two reminder s were sent during survey collection. Consent process A cover letter sent with the survey link included the following statement, By fi lling out the survey you are consenting to participate. Privacy issues No individual participants were identifi ed in any publication or report arising from the survey. Ethical considerations This survey was targeted at palliative care service providers to collect information on tools and quality improvement initiatives they were using at the time. It did not involve any patient contact or intervention. Data collection was anonymous. Data will be stored for seven years and confi dentiality maintained throughout Data analysis Methods to identify the top three benefits and barriers Different methods were adopted to fi nd out the overall top three barriers and benefi ts perceived by respondents. Method 1 Original scores The average of a respondent s scores for the same question asked about each tool/initiative was used as this respondent s overall score for this question against all tools and initiatives. The average score of all respondents responses was used as the fi nal score for the overall answer to each question. The top three barriers and benefi ts were selected based on these average scores. Method 1.1 Original scores without RUG-ADL, Karnofsky Since Resource Utilisation Groups Activities of Daily Living (RUG-ADL) and Karnofsky scores are part of PCOC data collection, this method removed ratings for both and used the same calculation as in method 1 to choose the top three options. Method 1.2 Original scores for PCOC only Because of the high similarity between PCOC and this project, especially when compared with the rest of the available options, it is reasonable to use PCOC to identify potential barriers and benefi ts for implementing clinical indicators for pain. Method 1.3 Original scores for PCOC and LCP Liverpool Care Pathway (LCP) ranking is not analysable due to limited responses. Combined results of PCOC and LCP will give an overall ranking for quality improvement projects only. 8

17 Method 2 Weighted ranking It is possible that using methods will generate results with the same scores. In this case, it will not be possible to distinguish between options. Another problem is one particular question may only have one response with a very high score, which will reduce the credibility of the fi nal results. To minimise the infl uence of these problems, the top three benefi ts and barriers were identifi ed for each clinical tool or quality project, and they were given rank scores of 3, 2 and 1 according to their priorities. The fi nal score for a particular benefi t or barrier is calculated by summing up the products of rank scores and response numbers for all tools and projects. Method 2.1 Weighted ranking without RUG-ADL and Karnofsky Based on the same reason as method 1.1, RUG-ADL and Karnofsky are excluded from this analysis. The rest is the same as method 2. Method 2.2 Capped weighted ranking In the analysis of methods 2 and 2.1, it was found that it is sometimes diffi cult to distinguish between options, especially when the response rate for the particular tool or project is very low. Multiple options may take the same rank. For example, Enabling consistent care and Improves quality of care were ranked equally as second best benefi ts for Edmonton Symptom Assessment System (ESAS). Bias will be created if both were given rank scores of 2 and the third best option will be excluded. To solve these problems, a possible solution is to split the rank score evenly to those options gaining the same rank and keep the total scores for one tool as 6 ( ). In this example, both benefi ts were given a score of 1, keeping the total score of the second best benefi ts as 2 and the total score for all top benefi ts as 6. The rest of the analysis is the same as method 2. Method 2.3 Capped weighted ranking without RUG-ADL and Karnofsky Based on the same reason as method 2.1, RUG-ADL and Karnofsky are excluded from this analysis. The rest is the same as method 2.2. Method to classify palliative care service providers Though each palliative care service provider has its own characteristics, it will be worthwhile to classify them into a limited number of groups and to identify common themes within each group. All perceived benefi ts, barriers and attitudes are considered as factors to group service providers. With this many factors, it is not possible to classify service providers manually. Cluster analysis was used to group service providers. Due to the similarity of PCOC and LCP to the Pain Indicator Project, it is worthwhile to group service providers according to their responses to PCOC and LCP, and their overall responses to all tools and the overall project. The grouping results can be verifi ed against each other to ensure the groups are classifi ed reasonably. 9

18 2. Results and discussion 2.1 Literature review result The literature suggested different defi nitions and types of clinical indicators (Travaglia & Debono 2009). Four types of indicators were identifi ed as suitable for consideration for this project: Process Measures of the quality of the care provided, including any element in the interaction with patients, such as diagnosis and treatment. The aim is to measure whether clinicians are adhering to (evidence-based) practices that achieve the best outcome for patients. Outcome Measures or approximations of the effects of care on the health status of patients and populations. As multiple factors contribute to health care outcomes, evaluations of outcome indicators take into account differences in case mix and controls over other covariates. Symptom Measures of specifi c aspects of care related to predetermined diseases. Structure Measures of the attributes of settings within which healthcare occurs, including material and human resources and organisational structure. Seventy-four literatures were identifi ed through searching the data bases. By screening abstracts, 35 were classifi ed as irrelevant to the project. The full text of 39 articles classifi ed as relevant were reviewed according to protocol requirements and 22 more were identifi ed through reference-tracking and reviewed as well. Besides these academic literatures, guidelines and policies from government and non-government agencies, including Australian Institute of Health and Welfare, the Australian Council on Healthcare Standards, Australian and New Zealand Society of Palliative Medicine, Victorian Government, Palliative Care Victoria, Palliative Care Outcomes Collaboration, Palliative Care Australia, were also reviewed. A total of 113 indicators were identifi ed through the full text review process. After removal of duplicated indicators by reviewing description and context of every indicator, 55 distinct indicators were included in fi nal indicator set. These indicators were grouped by types, settings and levels of detail. Types are as defi ned above, namely process, outcome, symptom and structure. Figure 1: Indicators by type Symptom 16 Outcome 7 Structure 1 Process 31 10

19 Among these four types, symptom and structure indicators were considered irrelevant. Settings include hospital, hospice, LTCF (long term caring facility) and all. Indicators designed for ICU use are included into hospital setting while nursing home is classifi ed into LTCF. An indicator was classifi ed into one setting when it was either only discussed within that setting or recommended by the author for the setting. The setting all means that the indicators were not discussed for particular settings in an article but should not be interpreted as the indicator being suitable to use in all possible settings. In the case of an indicator being discussed in multiple settings separately in different articles, it was classifi ed under all. Figure 2: Indicators by setting Hospital 7 LTCF 12 Hospice 4 All 32 Levels of detail include high and low based on the indicator s defi nition in the original article. An indicator was classifi ed as high when a detailed requirement was specifi ed for the way the tool could be used or the indicator calculation method given; it was classifi ed as low when details of its implementation or calculation were not available. Figure 3: Indicators by detail level Low 23 High 32 The project team selected a fi nal list of 29 indicators from which the expert panel could select the fi nal group. Appendix 3 lists these candidate indicators. 11

20 2.2 Expert panel result Outcomes from round 1 appropriateness Voting result The following nine indicators entered round 2 of voting with modifi cations of defi nition and/or benchmark according to feedback from panel members. Table 1: List of agreed indicators from round 1 ID Type Indicator name Mean Standard deviation O1 Outcome Pain relief in specifi ed time P1 Process Regular pain assessment P2 Process Assessing pain for a new patient P4 Process Providing a bowel regimen with an opioid P6 Process Use of a validated pain scale P8 Process Prescribing for breakthrough pain P9 Process Documenting pain in end of life P13 Process Scheduled pain medication for severe pain P19 Process Documenting medicine used Among all selected indicators, Assessing pain for a new patient scored 9 by all raters. All factors related to this indicator were marked as Yes by all raters too, showing complete agreement in the fi elds of validity, feasibility and generalisability for this indicator. Scheduled pain medication for severe pain and Documenting pain in end of life were the two weakest indicators among those chosen. They all have a mean rating of 7.9, and only six out of seven were marked Yes for validity. All other indicators were given seven out of seven and Yes for validity, in other words, 100 per cent agreement. The ratings for all indicators tend to fall in the higher score region, with a mean score of 7.4 and standard deviation of This is expected because the indicators are screened by the project team. There are nine indicators classifi ed as having agreement with the strict rule, and six more will fall into this category if the relaxed rule is applied. Since no more than 10 indicators were expected to pass round 1 as defi ned in the terms of reference, the strict rule of agreement was used. Modifications to definition and benchmarks The defi nitions of some of the nine selected indicators were modifi ed following comments from the expert panel in round 1. No changes in the benchmarks were recommended during round 1. 12

21 Table 2: List of changes to indicators from round 1 ID Indicator name Changes O1 Pain relief in specifi ed time Numerator: added clarifi cation on 3 Denominator: added and the pain is identifi ed as a symptom P9 P19 Documenting pain in end of life Documenting medicine used Numerator: added The documentation should include assessment using validated pain scales. And complementary treatments also. This should be practice for all medications and therapies in all settings. Indicators to be included with the relaxed rule Table 3: List of indicators complied with the relaxed rule from round 1 Count: 6 ID Type Mean Standard deviation P5 Process Change pain regimen with severe sustained/ worsening pain P10 Process Patient education P18 Process Identifying cause of pain P22 Process Assessing impact of pain P23 Process Assessing effi cacy and side effects for new opioid therapy P24 Process Documenting pain management Indicators P5 and P10 actually had higher average score than P9 and P13. They were excluded because, although most raters rated them high enough, one rater gave a score less than 7. Although one of our raters gave systematically low scores for all the indicators, it was not this rater s score that made both P5 and P10 fall out of the 7 9 range. Therefore, it is reasonable to conclude that these two indicators cannot achieve strict agreement of appropriateness. 13

22 Highly discordant indicators Table 4: List of highly discordant indicators from round 1 Count: 8 ID Type Indicator name Mean Standard deviation O4 Outcome Optimal pain management O5 Outcome Effectiveness of treatment P7 Process Re-assessment with change of pain regimen within 4 hrs (inpatient)/next visit (outpatient) P11 Process Carer satisfaction P15 Process Pain treatment rate P17 Process Pain documentation fl ow sheet P20 Process Consistent opioid dose across setting P21 Process Complete pain assessment rate Among these eight indicators, it is evident that O4, O5, P7, P11, P15 and P17 are highly discordant because of similar low average scores and high variance. On the other hand, P20 and P21 are suspicious because their average scores are too high. The reason for classifying them as highly discordant is because the rater who gave systematically low scores, scored them extremely low, 1 for P21 and 2 for P20, while all other raters give them high scores within the 7 9 range. Further clarifi cation from the rater took place without resulting in a change in overall scoring because systematic low scoring did not result in rescaling of the scores to a signifi cant extent and the rater answered No for all the factors of these two indicators. Indeterminate indicators Table 5: List of indeterminate indicators from round 1 Count: 6 ID Type Indicator name Mean Standard deviation O2 Outcome Patient satisfaction O3 Outcome Stable pain in end of life P3 Process Staff education P12 Process Patient satisfaction P14 Process Quality improvement policy P16 Process Pain management policy Since indeterminate covered all scenarios outside the other three defi ned classifi cations, there was no particular scoring pattern observed. 14

23 Outcomes from round 2 necessity Voting result The following six indicators were chosen from round 2 necessity voting. Table 6: List of agreed indicators from round 2 ID Type Indicator name Mean Standard deviation P1 Process Regular pain assessment P2 Process Assessing pain for a new patient P4 Process Providing a bowel regimen with an opioid P6 Process Use of a validated pain scale P8 Process Prescribing for breakthrough pain P13 Process Scheduled pain medication for severe pain Among all six chosen indicators, Assessing pain for a new patient achieved maximum consensus with all raters scoring 9. And all raters considered it as both fundamental and essential for consistent quality of care. This indicator was the most agreed indicator in round 1 also. The weakest supported indicator among the six selected was Use of a validated pain scale. It had the lowest score (8.0) although it was rated third highest in round 1. And among the factors considered, one rater didn t consider it as fundamental, one rater didn t consider it as essential for consistent quality of care and a third rater answered no for both of them. This implied hidden disagreement on its necessity, which was far from that expected. As in the fi rst round, the ratings for all indicators tended to fall into the higher score region, with a mean score of 8.2 and standard deviation of The six selected indicators are identifi ed through the strict agreement rule. All other three indicators will be considered as necessary if the relaxed agreement rule is applied. Modifications to definition and benchmarks More comments were provided by raters during this round. Table 7: List of changes to indicators from round 2 ID Indicator name Clarification O1 Pain relief in specifi ed time More detail in numerator: For outpatients, if the pain regimen is changed or advice is given, the measurement should be followed up by phone within 24 hours. P1 Regular pain assessment This indicator needs to be further redeveloped. P9 P13 Documenting pain in end of life Scheduled pain medication for severe pain Numerator defi nition changed to: Number of deceased patients with pain assessment documented in last seven days of life. The documentation should include assessment using validated pain scales. Denominator defi nition changed to: Number of patients with severe pain documented with a validated pain score. 17

24 Although indicator O1 was not selected, it is still worthwhile recording the clarifi cation for it. For O1 Pain-relief in specifi ed time, a more detailed clarifi cation was needed to describe scenarios to obtain a post-treatment pain score. 1. Inpatient without regular pain assessment: Use last measurement within 48 hours of admission. 2. Inpatient with regular pain assessment: Use average pain scores after last pain-related prescription or treatment. 3. Outpatient with pain measured at each visit: Use measurement of the second visit. If pain regimen is changed or advice is given, the measurement should be followed up by phone within 24 hours. 4. Outpatient without pain measured at each visit: This is defi nitely considered as a failure to meet the requirement. A home visit will be treated the same as an outpatient. P2 Assessing pain for a new patient One rater suggested changing the timeframe for inpatients to 24 hours. The benchmarking criteria could be modifi ed during implementation without signifi cant effort required. Indicators to be included with the relaxed rule All nine proposed indicators would have been chosen if the relaxed rule was applied. The three additional indicators were: Table 8: List of indicators complied with the relaxed rule from round 2 Count: 3 ID Type Indicator name Mean Standard deviation O1 Outcome Pain relief in specifi ed time P9 Process Documenting pain in end of life P19 Process Documenting medicine used Overall rating pattern Overall, there is no systematic difference between the scoring patterns of individual raters. However, a clear pattern associated with professional background emerged, which was not so obvious in round 1 voting. Raters with a nursing background tended to give higher scores than those with a medical background, no matter what their current appointments were. Total scores by clinicians were all below 75 while total scores by nurses were all above 75. Although the sample size was too small to prove this hypothesis, it is a topic worth further consideration Validating voting results With the introduction of the shadow rater, there was no change of results in both rounds compared with the original voting results. 16

25 2.2.3 Result and discussion The expert panel showed high-level consensus over the fi nal set of indicators; the validating mechanism of shadow voter did not introduce any variance to the fi nal result. Both the expert panel and shadow voter followed the same rules defi ned prior to the voting process. The fi nal indicators are: regular pain assessment assessing pain for a new patient providing a bowel regimen with an opioid use of a validated pain scale prescribing for breakthrough pain scheduled pain medication for severe pain. It is interesting that, within these six indicators, three of them were recommended by the ANZSPM Clinical Indicators Working Group, although naming and defi nitions are slightly different. The ANZSPM indicators are: pain intensity quantifi ed plan of care for pain aperients/laxatives initiated in patients on opioids. It is very hard to justify to what extent the communication of these three indicators affects the judgement or opinions of all involved experts. Despite this concern, the ANZSPM recommendation could serve as evidence that the expert panel outcome from this project is aligned with outcomes from this national organisation. The backup grouping mechanism wasn t used because the results were as expected. However, an interesting phenomenon was observed. Experts with a nursing background tended to score higher than those with a medical background. Further research will be required to explore this observation as it may be very helpful for when implementing the recommended indicators. 2.3 Survey result Respondent profile Response rate Respondents invited 60 Respondents with answers 38 (63%) In all, 38 out of 60 (63 per cent) invited services responded to the survey, which is an acceptable response rate according to Mangione s classifi cation of response rates to postal questionnaires (Mangione 1995) and an excellent response rate according to general online surveys, which range from 44 to 60 per cent (Cobanoglu et al. 2001; Cook et al. 2000). 17

26 Service type Figure 4: Service types of respondents Vic DH SDF* 42% 20% 33% 4% Survey 68% 13% 18% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Community service Inpatient service Consultancy service Day hospice *Vic DH SDF Palliative care service delivery framework by the Department of Health Compared with the Palliative care service delivery framework (SDF) (Aspex Consulting 2010), community services were more highly represented in survey respondents while consultancy and inpatient services were less represented. No publicly funded day hospice responded to the survey. Survey respondents service delivery areas and regions are distributed similarly to the SDF mapping. Ethics approval did not include direct contact with respondents to protect the confi dentiality of responses. Service delivery areas Vic DH SDF* 33% 20% 65% 33% 4% 2% Survey 27% 68% 68% 13% 18% 5% 0% 20% 40% 60% 80% 100% Metropolitan State-wide Regional and rural Figure 5: Service delivery areas of respondents *Vic DH SDF Palliative care service delivery framework by the Department of Health When compared with SDF data, the survey respondents distribution is similar to the whole population of providers in terms of service delivery area. It is worth noting here that the survey asked about regional service and rural service separately and some service providers deliver services in multiple types of areas; this data is consolidated to be comparable to SDF data. Since the survey is anonymous, it is possible that there might be errors when consolidating data. For agencies delivering services in multiple types of areas, one will be classifi ed as metropolitan when it claims delivering 18

27 services to both metropolitan and regional areas. The general principle of assumption is an agency will be based in the more populated area. Vic DH SDF* 14% 10% 16% 11% 13% 12% 12% 11% Survey 16% 5% 16% 11% 13% 13% 13% 13% 0% 20% 40% 60% 80% 100% Barwon-South Western Region Gippsland Region Hume Region North & West Metropolitan Eastern Metropolitan Region Grampians Region Loddon Mallee Region Southern Metropolitan Region Service region Figure 6: Health regions of respondents *Vic DH SDF Palliative care service delivery framework by the Department of Health Geographic distribution of survey respondents is similar to that from the SDF. The Royal District Nursing Service (RDNS) is not counted, although it provides services for multiple agencies in metropolitan areas % 3% % % % % % 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Staffing profile Figure 7: Staff numbers of respondents From examining raw survey data, it is very possible that this question was interpreted into two different meanings: number of dedicated palliative care staff and number of the hosting organisation s staff where the palliative care unit is part of a larger organisation. It is not possible to distinguish between these two interpretations. This information is hence used for reference only. 19