Healthcare Improvement and Rapid PDSA Cycles of Change: A Realist Synthesis of the Literature

Transcription

1 FINAL REPORT Healthcare Improvement and Rapid PDSA Cycles of Change: A Realist Synthesis of the Literature Esther Curnock, MPH MRCGP Julie Ferguson, PhD John McKay, MD FRCGP Paul Bowie, PhD FRCP (Edin) NHS Education for Scotland Department of Postgraduate General Practice Education 2 Central Quay GLASGOW G3 8BW Further information: paul.bowie@nes.scot.nhs.uk The study was commissioned and funded by the Patient Safety Multi- Professional Steering Group of NHS Education for Scotland.

2 Abstract Background Plan-Do-Study-Act (PDSA) is a quality improvement (QI) tool used in healthcare. It pursues effective changes in healthcare processes to enhance health outcomes, using rapid small-step change cycles. There is limited research into the effectiveness of QI programme components. This systematic realist synthesis aimed to identify and summarise the implementation and contextual factors that may influence the effectiveness of PDSA. Methods We established outline theories relating to whether, how and why PDSA works by reviewing recent papers. We systematically searched for articles in Medline, Embase, British Nursing Index, PsycINFO, HMIC, MIDIRS and EBM ( ). Articles referencing or assumed to use PDSA (e.g. having employed the Institute of Healthcare Improvement (IHI) collaborative methodology) were included. Any outcome measure reported as leading to QI was included, whether or not it was clinical. A data extraction form, informed by PRISMA guidelines was employed. Using agreed inclusion and exclusion criteria we selected papers which would add to, or refine, the theories. Articles were added through snowballing from citation lists. The data/findings were extracted and synthesised according to the contextual and implementation issues which facilitated or limited successful PDSA outcomes. Results 94 articles were reviewed as relevant to our research question with a total of 44 contributing to our evaluative framework and full theory development. PDSA has led

3 to changes in practice and health outcomes in a variety of settings, geographies and patient groups. Several factors were identified as influencing effectiveness. These included: organisational size, context and stability; setting clear and measurable improvement goals underpinned by evidence; staff engagement and ownership; speed and size of cycles; actions taken to spread and sustain changes; and resources. Conclusions PDSA may be more successful in smaller, defined settings with stable staff and patient groups. Culture seems more important than size. Success is more likely where: evidence linking process goals to intended outcomes exists; is understood and visible to staff; the workforce is actively engaged; and, staff have prior QI experience. Although difficult to achieve, quicker and shorter PDSA cycles may lead to faster/more sustained practice change. Improvements are spread and sustained more when progress and associated data are shared and visible. Existing support structures and additional resources lead to greater success. Registration: Not applicable Keywords: PDSA, Quality Improvement, Patient Safety, Realist Synthesis, Evidence Review, Healthcare

4 Background PDSA (Plan-Do-Study-Act) is a tool that is utilised in a diverse range of quality improvement (QI) programmes and projects in healthcare systems worldwide. The aim of PDSA is to pursue, sustain and rollout effective changes in care processes that favourably affect outcomes, using rapid small-step change cycles. The model involves four cyclical stages: hypothesis formation (Plan); implement the new process with data collection (Do); interpreting the results (Study); and a decision as to what to do next (Act) [1] PDSA (also known as Plan-Do-Check-Act, PDCA) was developed as a method for achieving efficiency in Japanese car manufacturing [2] and was influenced by earlier work on industrial statistical quality control [3]. It was one of the key tools developed as part of the Total Quality Management movement [4]. Langley et al [5] used PDSA as the central component of their Model for Improvement, and were arguably the first to propose the use of PDSA cycles in health care QI. The Model for Improvement is popularised by the Institute of Healthcare Improvement (IHI) through their Breakthrough Series collaborative approach [6]. IHI methods are strongly influenced by industrial continuous QI methods that originated in manufacturing industries and organisational change theories that are perceived to be complementary [7]. In the United Kingdom (UK) in recent years several large-scale QI programmes such as the Scottish Patient Safety Programme [8] and the Health Foundation s Pilot Safer Patients Initiative [9] have been implemented in partnership with the IHI. However, empirical evidence of the effectiveness of QI programmes, as

5 well as evaluation about their impact on healthcare quality is limited [10]. Few controlled studies of QI collaboratives have been undertaken, and the results are equivocal. For example, Landon et al [11] evaluated the effectiveness of a QI collaborative to improve HIV patient care using a prospective controlled study design, involving 9986 patients in 69 clinics. No significant differences in quality of care measures were found between control and intervention clinics). A systematic review of QI collaboratives from 1995 to 2006 had limited scope. In it a small number of controlled trials showed limited positive effect and two showed no significant benefit [12]. Their objective was specifically to assess for empirical evidence of effectiveness, but they were unable to draw any conclusions from the vast majority of reports due to uncontrolled study design. A systematic review of the effectiveness of teaching QI methods to clinicians also had mixed results [13]. A common criticism of QI initiatives has been their tendency to favour action over evidence, with the rapid dissemination of innovative but unproven initiatives, with potential unintended consequences of harm and opportunity cost [14-16]. Although evidence of the effectiveness of many QI approaches is lacking [17] several factors that appear to be critical to the success or failure of QI collaborative projects have been identified. A review [13] found that using incremental tests of change (i.e. the PDSA approach), and having access to performance data appeared to be important factors, but were themselves insufficient for ensuring beneficial outcomes. The review was unable to analyse the determining factors any further or make specific recommendations regarding curricula.

6 Using randomised control trial (RCT) designs to evaluate interventions where the context, content, and implementation have a high degree of heterogeneity can be problematic. Such methods when applied to less stable and complex interventions, and when unaccompanied by a detailed process evaluation, may fail to identify that interventions work in some instances rather than others, or to explain why they work (i.e. the potential casual mechanisms). This reduces the generalisability of both research and subsequent review conclusions. In addition, strict eligibility criteria mean that much useful contextual information has been lost in the existing traditional systematic review literature, including information required by healthcare managers and policy makers to help them decide whether to implement a specific QI approach in their local setting [18]. The need for an alternative approach to evaluation and the systematic review of QI programmes to elicit this contextual and generalisable learning is recognised by many authors [1, 16, 18-22]. It has been argued that because programmes are embedded in a range of attitudinal, individual, institutional, and societal processes they are not things that can be claimed to work or not work : instead they contain certain ideas that work for certain subjects in certain situations [23]. Theory driven approaches to both evaluation and evidence review that try to understand the links between context, content, implementation and outcomes are being developed and tested to see if they can reduce such problems [15, 18].

7 At the point of embarking on this review in 2010 there was to our knowledge no systematic theory informed review of the evidence available which identifies the key mechanisms within many QI programme initiatives - PDSA - that are likely to lead to practice and health outcome change and improvement. Review purpose Given the above limitations in current knowledge and by using elements of a realist synthesis approach, as espoused by Pawson et al [24], this review attempts to address the following question: What are the implementation and contextual factors that may enhance or reduce the effectiveness of PDSA as a QI tool in healthcare settings? The review was commissioned and funded by NHS Education for Scotland, a special health board with responsibility for the education, training and life-long learning of the healthcare workforce in Scotland. Training in QI concepts and methods - including PDSA is a learning need that can be identified across the health care workforce. If such interventions are to be rolled out it is vital to understand how best to implement them and in what circumstances and contexts they could work. We present the discussion and learning in relation to the key theories uncovered via the review processes. This learning then informs our recommendations regarding the training of health care professionals in PDSA methods, as well as our guidance on the effective implementation of the technique as part of ongoing improvement initiative.

8 Methods Key theories to be explored An initial background search was conducted examining 37 papers on PDSA published over the previous year (2009) from study commencement, and browsing references identified from these papers. This allowed mapping of the diverse perspectives and theories about PDSA. Inclusion criteria for the review were refined in light of emerging learning from this process. Informal networking with known experts in the field also informed this process. The initial resultant theories derived from this initial process are shown in Table 2 and more refined versions in the findings section. These informed our data search and extraction processes. Determining the search strategy We subsequently conducted a systematic search of papers using the terms in Figure 1 See Figure 1: Formal Search Criteria (additional file 1) Only English language papers were included from over a five year period from 2005 to the time of the search in September These time limits were selected due to resource limitations and pragmatism. Either the setting or the participants had to be directly involved in healthcare. Any outcome measure in the papers that led to QI was included, regardless of whether or not it was clinical. Study Quality Assessment and data extraction

9 An initial process was carried out to screen abstracts in order to exclude papers of low relevance or low worth. Two questions were applied: 1. Do the authors refer to either a PDSA or PDCA process, or IHI collaborative methodology for use in QI in healthcare? 2. Does the paper go beyond superficial description or commentary, i.e. is it a broadly competent attempt at research, enquiry, investigation or study? Papers were rejected at this stage if the answer to either of these questions was no and the rationale for this was documented. Papers for which the abstract did not offer enough information to determine eligibility were retrieved for full text review. The same initial questions were asked of all full texts obtained for review, and papers were subsequently rejected if either answer was no. Papers that met the criteria of relevance and worth underwent full data extraction. A data extraction form shown in Figure 2 was developed. See Figure 2: Data Extract Form (additional file 2) This data extraction form was informed by PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) guidelines [25], and the data extraction framework used in a systematic review with a similar approach [26]. The SQUIRE (Standards for QI Reporting Excellence) guidelines [27] were used to establish a guide to quality criteria however no formal quality scoring system was employed, in keeping with similar systematic realist reviews [28]. Excluding a large number of relevant studies on grounds of rigour would reduce the validity and generalisability of the

10 learning from realist synthesis review findings [29]. The form went through an iterative cycle of improvement in a piloting phase involving the three reviewers until a final format was agreed upon. The data extraction variables are shown in Table 1 Table 1: Data Extraction variables Data Extract Variables Year of Publication Paradigm (theoretical lens used) Type of Paper Perspective (unit of analysis) Aim & Objectives Design o Study design o QI methods used o Method of evaluating outcomes Funding Key actors o who supporting how, why o who delivering roles, training o who receiving o who comparing to Nature of the Improvement Context factors o Country / region o Urban / rural o Primary / secondary healthcare o Other local / structural factors of influence Implementation process o Costs of implementation o Plans for ongoing monitoring Data Collection o Timeline Data Analysis o Handling of disconfirming observations Results o Summary of main results o Facilitators / strengths / successes of implementation o Barriers / weaknesses / difficulties of implementation o Strength of relationship between intervention & changes Dissemination Process Conclusions Reflexivity (consideration of potential biases & conflicts of interest)

11 Ethical implications The principal summary measures were factors influencing and critical to the implementation of the PDSA process (see Table 2) and the associated theories in the findings section below. A final summary page asked the reviewer the following questions: 1. Does the paper have an important message for our review question? 2. Does the paper fulfil the established quality criteria in its domain? 3. What factors does the paper identify as being critical to the use of PDSA or to the delivery of education on PDSA? Papers were randomly allocated to three reviewers (EC, JM, and JF). Twenty percent of papers were extracted independently by two reviewers (pairing between reviewers varied to ensure consistency across the group). Reviewers met in pairs after each had completed extraction of a cycle of 10 papers. Any disagreements on the data in the papers extracted in duplicate were then resolved by consensus. The meeting also provided the opportunity for discussion of uncertainties arising with any of the other papers completed by each reviewer. The approach taken in a realist review makes such dialogue between researchers essential for the iterative process of classification. For each article, efforts were made to uncover any sources of bias from either the authors or the main support team of the initiative. Each article was examined for evidence of who or what had triggered the decision to initiate the QI project; why it had occurred at that time; funding sources; and whether the authors had commented on

12 potential conflicts of interest. A judgement was made on the transparency of data presentation and the degree to which the reader could analyse results objectively. Finally we used a snowballing technique to scan the references of the most relevant articles identified in the systematic search. This technique was used to identify key papers prior to 2005, and any further papers not identified by the formal search process that could contribute to populate the synthesis framework. Following this a final search for additional studies from the grey literature was conducted to strengthen specific aspects of the evaluative framework (Table 2) and theories. Additional studies considered included sources such as Government Reports and national and international PDSA/Collaborative group papers (e.g. National Primary Care Collaborative [English], Australian Primary Care Collaborative [part of Improvement Foundation], and Health Foundation Reports). Evidence/data synthesis To facilitate the synthesis process an evaluative framework that informed the development of our initial six theories was developed and populated using the evidence/learning. This framework identified crucial dimensions of the theories that related to mechanisms (core elements of the PDSA cycle, or how it was applied) and contexts (staff motivation, stability, etc). We were seeking data and learning on these areas to allow us to test and further refine our initially identified theories. The framework is illustrated in Table 2. Data and learning were therefore synthesised in order to determine what implementation and contextual factors (circumstances and constraints) that may enhance or reduce the

13 effectiveness of PDSA as a QI tool in healthcare settings. Contradictory evidence was used to generate insights about the influence of the PDSA mechanisms or contexts. To ensure transparency within the subsequent discussion and findings sections we have referenced the authors which support our assertions. In addition we have used codes which can be used to identify the actual key finding/learning from within each paper that has been used to refine our theories and inform our conclusions. The T codes identify which of our contexts/mechanisms the text relates to (e.g. T1 T9 shown in Table 2 and the C code relates to the precise contextual issues (or constraining factors) identified by the author that has influenced the success of the PDSA intervention studied/reviewed. The author, paper and constraint or contextual factor relating to each code can be seen by referring to Figure 5. Conclusions were drawn by synthesising the data in relation to the evaluative framework (Table 2) and are presented as a narrative account. Draft recommendations and conclusions were tested with key stakeholders, and the review team worked alongside practitioners and policy-makers to apply recommendations in particular contexts. Results Review statistics/ initial theories and synthesis framework. Recent relevant papers (n=37) from the year preceding the review start date were identified and reviewed and from this process several draft implementation and

14 contextual mechanisms were identified which might influence the effectiveness of PDSA. These are shown in Table 2: Table 2: Data synthesis framework based on initial draft identified mechanisms and contexts MECHANISM enhancing outcomes practice changed/sustained T1. T2. T3 T4 Measurable process goals that are linked to outcomes via evidence Changes are more likely to be sustained by ensuring the impact of change (data/progress) is visible Changes broken down into small-step cycles completed in rapid succession lead to faster sustained change CONTEXT (Constraining and facilitating Factors) Local spread to surrounding settings/context can be achieved via key actions T5 T6 T7 T8 T9 Frontline staff may have relevant knowledge of changes & implementation skills that can influence success Engagement of frontline staff can influence participation and impact Existing resources can sustain PDSA/ changes The tool is can achieve practice change/outcomes in multiple disciplines & organisational contexts The context/stability/ size of the organisation can impact on PDSA cycles/impact These mechanism and contexts informed the data extraction process. They were subsequently integrated and reordered into six key theories about what influences the effectiveness of PDSA. We have kept the initial codes above however to allow reference back to the data extraction process. The more collapsed /integrated theories are:

15 Theory One: PDSA is can achieve practice change/outcomes across different health care disciplines, organisational contexts and settings (see T8/9 in Table 2). Theory Two: PDSA interventions are more likely to lead to practice change when the intended changes are measurable in the short term and when practitioners can see the changes are evidence based (see T1 & T2 in Table 2) Theory Three: PDSA leads to more substantial and sustained practice change when frontline staff are fully engaged with the process (see T5 & T6 in Table 2) Theory Four: Rapid PDSA cycles lead to faster improvement, longer-term change and sustained momentum for change (see T3 above Table 2 in Table). Theory Five: Achieved changes can spread into other settings and contexts (see T4 above in Table 2). Theory Six: Improved practice in PDSA can be achieved and sustained through existing resources (see T7 in Table 2) Figure 3 shows the results from the data base search (from ) and the reasons for exclusion of papers. See Figure 3 Data base search results and exclusions (additional file 3) The search identified a total of 762 papers of which 94 were eligible after the application of our review criteria. These 94 papers were identified as relevant to our research question and one or more of our subsidiary theories and contained some worthwhile information. Of these, a significant proportion described before and after studies, using baseline data as the comparison group. The quality of these papers varied

16 significantly, but nevertheless many were able to contribute useful information to refute or verify our postulated theories and add new knowledge to existing evidence [7]. Of these 94 papers, 32 were identified as holding relevant information to populate the evaluation synthesis framework that informed our initial theories (Table 2). The remaining 62 papers provided information on the general effectiveness of PDSA (e.g. Theory one only) and detailed information on the settings it was applied within, but did not provide sufficient detail on any individual PDSA components (e.g. cycle speed) or contexts (e.g. previous QI culture) to add substantially to our synthesis framework. The snowballing technique used to identify key papers prior to 2005 involved scanning the references of the most relevant 32 articles identified in the systematic search that populated the synthesis framework. This technique identified a further 12 papers. Therefore, a total of 44 (32 plus 12) papers were drawn on to populate the evaluation synthesis framework and refine our theories. The discussion and findings in this paper are based mainly on these 44 papers. Review Limitations This review has focussed on how PDSA can be used to best effect in healthcare settings. However PDSA is often used in conjunction with other components of QI. In particular it has been used most widely within the collaborative approach, which can involve multiple sites sharing knowledge, learning days, and external support from experts. It is therefore difficult to disentangle the dimensions which influence the effectiveness of PDSA from the dimensions that effect the wider approach utilised.

17 We have been unable to produce definite statements about what works ; instead, we have added further detail to our initial theories and produced advice on the likely implications of different contexts on the success of PDSA. This outcome is a natural consequence of the type of question being asked, rather than a failing of the scope of the review or its methodology. The contextual details within individual studies were limited in many cases. It is unclear whether we have omitted other important contextual factors that either aren t identified by study authors, or are considered not influential enough to report. However by collecting the data available on a large number of papers we hope we have been able to identify most of the significant elements that influence the effectiveness of PDSA. Although we were systematic in our data extraction processes and the application of our criteria, it is feasible that we missed some relevant articles. This will be particularly true for articles published prior to 1995, as prior to this date we used a snowballing process for papers cited in our most relevant articles identified in the systematic search. Publication bias is a significant concern and one that is difficult to quantify. Successful organisations have a vested interest in promoting their successes [14]. Once a QI initiative has been undertaken there is little incentive to devote resources to investigating its efforts, particularly where private resources have been invested. Financial interests may be an influence in the popularity of specific QI methods [18]. Mittman (2004) suggests objective evaluations of QI methods are lacking due to demand-led bias and points out that much of the published information is located in management and practitioner-orientated journals such as the The Joint Commission Journal on Quality in Healthcare, which tend to emphasise only successful accounts

18 [16]. Our findings should be considered in light of the likelihood of significant publication bias. However due to the nature of our question, we would argue that it is less of a concern than if we had attempted to answer the question does PDSA work? Some common criticisms of realist synthesis are that: the iterative approach can introduce bias; there can be a tendency to treat all forms of evidence as equally authoritative; and this can lack transparency in the choice of evidence used. Several steps were taken to minimize these weaknesses. A significant period of time was spent refining the research question at the scoping stage, giving stability to the question being asked at the point data was extracted, and therefore reducing risk of bias introduced. SQUIRE guidelines were used to determine the authoritativeness of different sources of evidence. In the first phase of data extraction the reviewers examined all eligible studies within a specified time frame, rather than take a purposive sample, in order to ensure the transparency of study selection. In taking these steps the realist approach was modified to suit our specific needs and resources. This is in part a reflection of the complexity of synthesising the evidence on organisational interventions. If complex interventions are by nature leaky and susceptible to local adaptation, it is perhaps inevitable that the review methodology applied to them will also need local adaptation. Discussion Theory One: PDSA can achieve practice change/outcomes across different disciplines and organisational contexts/settings. Figure 4; Illustrates the numbers and proportion of studies used for data extraction that detailed specific factors of interest to the review question, theories and our discussion.

19 Figure 4 Number and proportions of the 94 studies used for data extraction studies detailing specific factors of interest to the review question (additional file 4) Application of PDSA across settings and organisations and patient groups PDSA is a mechanism for QI that has been used to change practice in a wide variety of disciplines and organisational contexts. The examination of 94 papers included found that in most cases the context was sufficiently well described that the findings could be related to other settings (Figure 4 Table 26). 70% of papers were based in either the USA or Canada, and a further 12% were UK based (see Figure 4 Table 27). Both urban and rural contexts were described (see Figure 4 Table 28). The projects were based in secondary care in 63% of cases, and primary care in 18% of cases (see Figure 4 Table 29). Of the 29 hospitals that gave additional descriptive information 12 were described as a tertiary centre, and a further 12 were described as academic or teaching hospitals. The patient group receiving the improvement were sometimes defined by clinical diagnosis (e.g. diabetes, asthma, depression, stroke), some by a procedure or risk factor (e.g. requiring ventilation, at risk of falling), and some by location of setting (e.g. ITU, radiology, A&E, haemodialysis, oncology ward). Overall there was a very wide spectrum of clinical problems and scenarios in which PDSA was used as a tool for healthcare improvement in some capacity (see notes at the end of Figure 4). Figure 5 details the references and the associated codes for the information contained in the subsequent discussion. These codes can be used to identify the key findings from the review used in our evaluation synthesis (Table 2) to support our discussion and findings. All of the codes in the subsequent discussion therefore relate back to Figure 5

20 See Figure 5: Codes for evidence used in the evaluation synthesis (additional file 5) Organisational culture, characteristics and stability The evaluation of the UK patient safety project found no specific organisational characteristics associated with greater perceived value of PDSA [30, T8/C17], and no association was found between site characteristics and the achievement of sustaining and spreading change in a US mental health project [31]. Whilst there is no clear characterisation of the organisation that predicts effective use of PDSA there are some organisational contexts and cultures in which PDSA appears to be more or less effective. A small community hospital A&E department found using PDSA cycles to be a vey useful tool for achieving improvements [32, T8/C5]. Projects that involve changes in a well defined geographical area such as an ICU or a single clinic [33, T2/C3] appear ideal. This may relate at least in part to the visibility of practice change as staff feedback can be centralised and focused. Although larger settings may be more likely to contain internal QI expertise, it can be harder to sustain and spread changes in these contexts possibly due to difficulty obtaining a critical mass that makes changes routine. The impact of any small changes in larger settings can become diluted, and the changes less visible [31, T8/C3].

21 A lack of on-site QI expertise may lead to teams struggling [34, T8/C4]. Contexts which contain QI expertise, most likely larger settings, can be supportive of change cycles, especially if PDSA is integrated into existing QI programmes. Leadership and management may also be more supportive where QI is routine. However as identified above larger settings can have drawbacks. Organisations with a developmental rather than hierarchical culture were associated with greater staff motivation to conduct PDSA cycles [35, T8/C6]. In contexts where staff are less familiar with discussing care processes the tool may be less applicable. A strong culture of clinician autonomy can also be a barrier [30, T8/C13]. A Dutch mental health project using PDSA found the model did not fit in to the existing culture of primary care professionals, who found it difficult to apply [36, T8/C7]. A lack of staff with a prior interest in QI was found to be an important factor in an initiative with clinical trainees where the PDSA model did not translate into rapid cycles of change [37, T8/C8]. Organisations able to combine the use of PDSA with other features of QI projects where information and reporting is shared between other organisations may find it more effective [38, T8/C16]. A primary care diabetes QI project also found that some changes were implemented simply as a result of discussion and agreement. There was no requirement in this small setting to test the changes and roll them out gradually [39, T3/C18]. In these situations PDSA as a QI tool is made redundant.

22 Staff, disciplines and management support When several disciplines were involved poor communication between groups was identified as a barrier [40, T8/C14]. Offering a virtual environment to facilitate communication does not appear to have been helpful if the staff were used to such technology [36, T8/C15]. Leadership distraction with other priorities was a significant barrier [41, T8/C2]. Poor leadership support had a significant impact on the effectiveness of a US mental health QI project [31, T8/C11]. Senior support is often assessed as crucial [42, T8/C1], with the possible exception of units which are self-contained or operate fairly independently such as a neonatal ICU [43, T8/C12]. Organisational and staff stability As PDSA cycles start small and gradually expand they may be more resilient to instability and more suitable for changing situations than other QI tools. However large-scale organisational changes will adversely affect staff motivation to implement and sustain cycles [34, T9/C1; 42, T9/C2]. Staff turnover was a key barrier to implementation of cycles [44, T9/C3; 45, T9/C4], particularly if there was reliance on a small number of champions [34, T9/C5; 46, T9/C6]. Staff turnover and retention were often a barrier in contexts such as teaching hospital [47, T9/C7] and a US mental health collaborative [31, T9/C9] and settings with a high proportion of part-time staff [34, T9/C8].

23 Goal definition PDSA is more effective if goals have clear operational definitions [48, T1/C13]. This distinction can be described as the difference between control and learning QI projects [4, T1/C11]. Goals that are applicable in all situations (rather than discretionary) appeared to be more successful in terms of achieving and sustaining changes when using PDSA [4, T1/C11]. Such goals are more common and appropriate in settings with high control, for instance ICU [49, T1/C12]. Stability of patient population The impact of changes were harder to measure (and so visibility of success reduced) when the patient population was very transient [44, T2/C9], or when an acute clinical problem left little time for staff to document an implemented change [46, T2/C10; 40, T2/C11]. Measuring and documenting the impact of changes may be easier in contexts where patient populations are more stable, less transient and where patient turnover is generally lower. Staff skills, capacity and training Implementation of the PDSA cycle is difficult without some degree of process-thinking skill [34, T5/C2]. Some form of training was given to staff in 68% of papers used for data extraction (see Figure 4, Table 21), and in one third the training covered both the intervention itself and some type of input on the use of PDSA. In a further third the training focussed only on the intervention (see Figure 4, Table 22). The concept of identifying appropriate measures for PDSA can be difficult to grasp [50, T5/C13; 41, T5/C14]. Skills were also needed to collect and present performance data and these

24 could also be difficult to learn [50, T5C15]. Training on basic concepts of the process may need to be consolidated once put into practice [51, T5/C16; 41, T5/C17]. A UK Accident & Emergency-based project found the medical consultants grasped the concept more easily than senior managers [52, T6/C13]. Ideas regarding implementation strategies and appropriate measures did not always reside within staff knowledge bases [53, T5/C5; 50, [T5/C6], although frontline staff took ownership of sourcing implementation ideas within an ICU based project [49, T5/C4]. External input regarding knowledge of scientific literature and clinical organisational features from a support team were sometimes required [54, T5/C7]. External support may also be required for goal setting, for making recommendations about which tools might be used, and for supplying template tools to implement [31, T5/C10; 43, T5/C11; 50, T5/C9; 55, T5/C12]. Top-down instruction was favoured in a primary care mental health collaborative [48, T5/C1]), but reduced frontline ownership. Use of external expertise was cited as key in helping motivate staff in a US secondary care project over 58 sites [42, T6/C24], and staff learning sessions were rated as essential to staff motivation in a UK patient safety collaborative [30, T6/C25]. Financial incentives for staff could help overcome initial resistance, and were critical to frontline staff engagement in a Canadian primary care collaborative project [56, T6/C26]. A lack of dedicated time to conduct a QI project using PDSA can hinder progress [36, T7/C2]. If staff time is already saturated building in additional processes to implement

25 change can lead to staff having to work in their own time [34, T7/C3]. A project based on an in-patient cardiology ward found that the equivalent of a 1.5 full-time nurse s post was required for the time spent on daily monitoring, reviewing results, and giving staff feedback [47, T7/C4]. The additional staff time required was found to be a limiting factor in several projects [51, T7/C7; 57, T7/C5; 58T7/C6;]. A project based in nursing homes found that other demands on staff time led to difficulty in persuading staff of the importance of the QI project [59, T7/C8]. However as the extra work associated with a change became part of normal routine one project observed that workload was reduced as staff worked more efficiently as a consequence of the improvements [51, T7/C7]. Staff turnover is a key barrier, particularly if there has been an over reliance on local champions. As such, the engagement of a range of committed staff may help sustain and spread change to other suitable settings. Theory Two: PDSA interventions are more likely to lead to practice change when the changes are measurable, visible and evidence based/plausible. Theory two postulates that PDSA interventions are more likely to lead to practice change amongst staff when the changes anticipated (practice changes or changes in patient outcomes) are measurable in the short term (e.g. within each/early PDSA cycles) and when practitioners can visibly see that the anticipated changes are taking place, are based on evidence and plausibly linked to longer term changes in patient/health care outcomes. Measuring short term practice outcomes or longer-term health care outcomes

26 QI projects using PDSA cycles often have an overall aim with a patient or clinical care focus. However initiatives are often analysed in terms of processes-of-care measures rather than clinical outcomes. One advantage of process-based measures is that it is much more likely that an improvement can be demonstrated within a short-time frame: goals can be defined tightly and specific targets set. A major disadvantage of focussing on processes-of-care is that the link with clinical benefit can often be unclear, which has a significant impact on staff motivation. Additionally, although the evidence base for the overall clinical aim may be robust, the evidence relating the achievement of the practice change measure to this can be lacking. The review learning suggests that the most successful projects are those in which the relationship between process goals and clinical benefit was evident, and in which clinical benefit for patients could be observed by staff. Translation of process improvements into clinical improvements may be difficult in the short timeframe of a specific project however [36, T1/C15; 50, T1/C14], and a cause and effect relationship between the change made and clinical benefit can be difficult to establish [30, T1/C17; 48, T1/C17]. Both a paediatric asthma QI programme and a project to reduce hospital drug errors found that outcomes relating to processes of care were more successful than patient level outcomes [50, T1/C4; 60, T1/C3). Identifying and implementing appropriate measures could be a difficult task however [30, T1/C5]. It can also be unclear as to what constitutes enough evidence when clinical aims are being set. In a critical care QI project evidence emerged suggesting they might actually

27 be causing harm instead of benefit [46, T1/C7]. In reality clinical goals were chosen on the basis of both evidence base and stakeholder priorities [50, T2/C8]. When the risk of harm was low, for instance improving patient knowledge or reducing patient waiting times, an evidence base was less important as these non-clinical aims were deemed beneficial in themselves [38, T1/C9; 61, T1/C10. Providing feedback on goals and keeping progress visible Ensuring visibility of data relating to the goals set was also important [48, T2/C4]. A smaller setting/ sub unit also enables the display of data in a central location that is visible to all relevant staff, for example a data wall displaying progress towards the goals set [49, T4/C5]. Visually displaying baseline measures can demonstrate the need for change to staff and build motivation [48, T6/C1], as can clear presentation of the evidence-base for the improvement goals [40, T6/C2]. Initial staff resistance can be overcome by ensuring positive changes are highly visible [62, T6/C30]. Celebrating successes even when small and creating a sense of competition between departments were found to be effective [46, T6/C34]. However high visibility acted as a discouragement to staff where little progress was made towards the goals despite staff effort [32, T6/C32]. A neonatal pain management project highlighted the need for perseverance even when early results may have been perceived as a failure, as follow-up data demonstrated success [40, T6/C33]. The difficultly with interpreting early results has been viewed as demoralising and confusing in other projects [53, T6/C31].

28 One project using PDSA found the focus on individual patients was very beneficial for gaining staff enthusiasm as the benefits were directly seen, however without processthinking skills the motivation gained did not translate into action with a wider group of patients [34, T6/C27]. Focussing on small scale changes can lower expectation of the eventual clinical benefit, and staff may be less motivated to be involved [61]. Other projects found small changes fuelled enthusiasm that led on to motivation to see larger changes [50, T6/C28]. Sustaining the changes Regular feedback is associated with greater ability to sustain change [31, T2/C7], and in some studies there were periods where the frequency needed to be particularly high [57, T2/C8]. Even when changes had become part of accepted routine care, ongoing monitoring and display of measures were necessary to sustain the change [57, T2/C12; 48, T2/C13]. Direct comparison between patients that were included in a PDSA cycle and those that had not was helpful for sustaining momentum, especially when the link with clinical benefit was clear, such as a care bundle for ventilated patients [48, T2/C1]). Improvements are of most value when they are sustainable, and when secondary spread to other relevant areas is achieved. Sustainability appears to be aided by ensuring feedback to staff is given regularly, and progress towards goals is kept visible. This is much more likely to be achieved in situations when the improvements are being carried out within a defined team in a defined setting.

29 Theory Three: PDSA leads to greater and more sustained practice change when frontline staff take ownership of and are engaged with the process Frontline clinical staff were responsible for delivering the improvement process in 78% of papers used for data extraction (see Figure 4, Table 23). Active engagement of frontline clinical staff seemed to be important for the success of projects [63, T6/C6], and participation itself may have been perceived to create a sense of ownership amongst staff [50, T6/C9]. Achieving engagement and buy in Although staff buy-in to participating in QI in their area of work is crucial to its success it is not always easy to achieve. It is unlikely that this difficulty is particular to PDSA cycles. The small-step cycles concept suggests that it achieves a greater level of staff empowerment than more radical top-down changes. However the reality appears to be that this is often not the case. Staff engagement and motivation can be essential in championing the spread of changes to other sites. Capturing the interests of different staff groups may require a variety of techniques. For example, busy leadership personnel will need to be updated using a time-effective method, such that their attention is not lost. However, clinicians may be more engaged if outcomes of clinical interest and relevance are emphasised. Front line staff have been found to be more engaged when they have a sense of ownership over the change outcome decisions. Factors associated with staff resistance

30 Staff can also demonstrate resistance due to concerns regarding loss of clinical judgement and control, particularly when changes were protocol and target driven [64, T6/C4]. Resistance from clinical staff in particular was an issue [50, T6/C13; 31, T6/C14]. An evaluation of a UK based orthopaedic collaborative using PDSA cycles found clinicians did not appear to wish to lead [53, T6/C10]. The evaluation of the orthopaedic project found that of 17 components of the collaborative initiative, PDSA was rated the lowest, with many unconvinced of its value for driving change [53]. However the PDSA process was rated as highly important by both clinical staff and board-level staff in a UK based patient safety project [30]. A different picture emerged from a UK Accident & Emergency project in which staff motivation and empowerment amongst frontline staff was high [52, T6/C5]. Motivation can be difficult to achieve, particularly if more than one hospital department or team is involved [61, T6/C17; 37, [T6/C18]. Process measures were often judged on documentation and if there was a perception these tasks were already being done (but not documented) motivation was poor as there were low expectations of clinical improvement [57, T6/C4]. Allowing the staff team to decide on change ideas was effective in building ownership, but was also inefficient as lessons learned elsewhere are re-learned [41, T6/C8]. A further consequence of a strong sense of ownership was slowing down of the improvement process as everyone has their say [58, T6/C7]. It seems that there needs to be a balance between frontline ownership and stalling progress as everyone has their say [58, T3/C22]. Although staff motivation or buy-in was critical to achieving

31 improvement goals, it varies widely between contexts and was not easily predicted [54, T6/C15]. Participation on a voluntary basis was found to increase motivation [65, T6/C16]. Ensuring a broad range of staff were on the improvement team was important if the initiative involved multiple sites and disciplines, for example using PDSA cycles to improve patient transportation [66, T6/C19]. Focussing on early adopters or those staff most favourable disposed to PDSA at the start promoted the involvement of other staff [48, T6/C20]. Supervisory and management support Where supervisor support is perceived by staff it facilitates empowerment [35, T6/C21], but as empowerment is associated with increased scrutiny and responsibility if things go wrong, staff can be ambivalent about taking greater ownership, and managers may be reluctant to release staff if they feel it could threaten the security of their role [4, T6/C22]. The UK A&E project met resistance from the senior management who appeared threatened by the empowerment of frontline staff. Theory Four: Rapid PDSA cycles lead to faster improvements and sustained momentum for change There are two opposing philosophies in QI, that of gradualism or incrementalism (to which PDSA belongs), and that of radical transformational change. Bate et al (2002) suggest much hangs on which is correct [53]. Plsek (1999) suggests that pace is

32 crucial in the utilisation of PDSA cycles [67], however the reality captured in our review seems to be that momentum can be difficult to both achieve and sustain. The likely speed of PDSA cycles and of change There was reasonable evidence of the relationship between changes observed and use of PDSA cycles in 46% of papers used for data extraction (see Figure 4, Table 19). There were examples of projects having been able to conduct cycles rapidly, for instance a US drug error QI project conducted 739 cycles over 15 months, of which 63% were described as real tests of change, the remainder being educative [50, T3/C2]. There were, however, numerous examples in the literature of single PDSA cycles taking place over a number of weeks or months [37, T3/C6; 53, T3/C7; 68, T3/C8; 69, T3/C8; 45, T3/C9; 57, T3/C10; 70, T3/C11; 71, T3/C13]. Along with other enthusiasts of the PDSA method Kilo (1998) states dramatic, rapid changes in outcomes can (and do) occur within months [72]. Our review suggests that this expectation is not often met, and it is often more realistic to prepare for slower and less radical change. This somewhat contradicts the rapid-cycle terminology used to describe PDSA. Reasons for the relatively slow change There seems to be a variety of reasons for slower cycle speeds such as difficulties translating the goals of a project into small steps that can be tested within cycles conducted in rapid succession. Concerns exist that as cycles involve increasing numbers of patients, inadequate time for learning may be given if cycles are conducted too rapidly, especially when new skills amongst staff need to be developed [33, T3/C14]. Process changes that involve a small number of patients (for instance a

33 clinical problem which presents less than once a week) means that changes are harder to break down into small steps and cycles are slower [61, T3/C19]. Spending too long collecting baseline data was thought to be a common cause of loss of momentum in a US drug error initiative [50, T3/C20]. Projects that were required to generate ideas internally were found to conduct cycles more slowly [43, T3/C1]. In addition inclusivity and engagement of a wide range of staff all slow cycles down. Some changes necessitated large-scale disruption to supporting services, even when applied to small number of patients. In these cases the disruption to the system was greater if tested in small steps than if changes were made in one large cycle 61, T3/C15]. A UK orthopaedic QI initiative found that specifying changes must be achieved in small steps limited the changes that were achievable [53, T5/C16]. There were also some changes which were implemented immediately on a wide scale with little difficulty, for instance making a decision to remove bags of potassium chloride from ward supplies [50, T3/C17]. A primary care diabetes QI project also found that for some changes it was sufficient to discuss and agree on the changes that were needed in order to see them implemented. There seems also to be a relationship between this and Theory Two concerning the importance of the visibility of changes and cycle speed. Visibility of the changes was co-dependent on the speed & size of the cycles a large change would go through the cycle at a lower speed, reducing the visibility of the change due to too little timely feedback [61, T2/C6]. Given this slower cycle speeds may impact on both effectiveness and sustainability of resultant changes.