Optimizing Audit and Feedback Interventions to Improve Quality in Primary Care

Transcription

1 Optimizing Audit and Feedback Interventions to Improve Quality in Primary Care by Noah Michael Ivers A thesis submitted in conformity with the requirements for the degree of doctorate in clinical epidemiology Institute of Health Policy Management and Evaluation University of Toronto Copyright by Noah Ivers 2014

2 Optimizing Audit and Feedback to Improve Quality in Primary Care Abstract Noah Michael Ivers Doctorate in Clinical Epidemiology Institute of Health Policy Management and Evaluation University of Toronto 2014 One of the most consistent findings in health services research relates to gaps between ideal and actual care. Audit and feedback is frequently implemented to help providers identify these gaps and to subsequently improve quality of care, with widely varying results. The overall aim of this thesis was to examine how to optimize audit and feedback interventions provided to family physicians to more reliably result in professional behaviour change that will benefit patients. A systematic review, meta-analysis, and meta-regression was conducted to determine the effect of audit and feedback interventions on quality of care and to identify effect modifiers. The review included 140 trials and found that audit and feedback works best when the source is a supervisor or colleague, it is provided more than once, it is delivered in both verbal and written formats, and when it includes both explicit targets and an action plan. Based on the latter finding and applying principles from relevant behaviour change theories, a worksheet was developed to guide feedback recipients in setting appropriate goals and action plans in response to identified gaps in care. This worksheet was tested in a pragmatic, cluster-randomized trial including 53 family physicians from 14 practices across Ontario. All participating family physicians received feedback every six months regarding the proportion of patients with diabetes and/or heart disease receiving guideline-recommended care. Family physicians in the intervention group also received the worksheet appended to the feedback reports. After two years, no significant effects differences were found between the groups, in part because of poor uptake of the worksheet. An embedded qualitative evaluation examined the barriers and facilitators to improving quality of care for chronic disease management perceived by family physicians who received the feedback reports. Findings highlighted the importance of matching the targeted behaviour change(s) with the priorities and capabilities of recipients and their organizations. ii

3 Acknowledgments The systematic review was supported by the Cochrane Effective Practice and Organization of Care Group and by Andy Oxman s team at the Norwegian Knowledge Institute. The conduct and analysis of the trial was supported by a grant from the Canadian Institutes of Health Research (CIHR) funding reference number, The development of the intervention and the embedded qualitative study was supported by a team grant from CIHR, Knowledge Translation Improved Clinical Effectiveness Behavioural Research Group (KT- ICEBeRG). I would like to thank the physicians participating in EMRALD. EMRALD infrastructure is supported by research grants, but all work is conducted at ICES, which is funded by an annual grant from the Ontario Ministry of Health and Long Term Care (MOHLTC). The opinions, results, and conclusions reported in this article are those of the authors and are independent from the funding sources. No endorsement by ICES or the Ontario MOHLTC is intended or should be inferred. I m very lucky that my research interests seem to dovetail with some popular trends and that this has opened up a number of funding opportunities that may not have been present before. I am grateful for those who have come before me, and the efforts they have put forth to develop research funding opportunities for new family physician clinician scientists. During the course of this dissertation, I was fortunate to be awarded a training award from the C-CCORT CIHR team grant led by Jack Tu, then the TUTOR-PHC program led by Moira Stewart, and then a fellowship award from CIHR. I was also well supported by the practice plan of the Department of Family Medicine at Women s College Hospital and provided a stipend from the Department of Family and Community Medicine, University of Toronto. In all, a great deal of funding from taxpayer dollars has been committed to allow me to pursue this road. I intend to generate a good return on this investment, by every metric. I would like to thank Dr. Jim Ruderman, Dr. Sheila Dunn, Dr. Nick Pimlott, and Dr. Carol Kitai for their clinical mentorship and guidance as I meandered down this path toward becoming a clinician scientist. I tell anybody who will listen that I ve got a pretty great job it s mostly thanks to the support I ve received at Women s College Hospital that this has been possible. iii

4 I am grateful to each of the co-authours of the publications that form the meat of this dissertation. I would also like to thank my entire thesis committee, Dr. Merrick Zwarenstein, Dr. Jeremy Grimshaw, Dr. Jan Barnsley, Dr. Ross Upshur, Dr. Baiju Shah, and Dr. Karen Tu each member played a vital role and each brought a unique and helpful perspective. I m grateful that both Jan and Ross were able to (gently) open my eyes to issues of epistemology and ontology understanding that multiple worldviews can be simultaneously valid has made me a better physician and more inquisitive researcher. Baiju s smiling face, supportive approach, and willingness to offer practical advice has come in handy throughout. Last, but not least, it has been (and continues to be) a great learning experience to see Karen turn her idea into a sustainable team, through sheer will and determination. EMRALD has the potential to be a game-changer in primary care research and I m grateful to have played a role in its development. Clearly, without Merrick putting his faith in me when I cold-called him, none of this could have been possible. That cold-call set me on a path that has been exciting, challenging, and at times frustrating, but always intellectually fulfilling. I look forward to many more collaborative efforts with him in future. Merrick put me in touch with Jeremy, who has been a willing and able mentor in every sense of the word and who continues to open doors for me in the field of implementation science. I m not sure how I ll ever return the favour. I feel lucky to have mentors who have become friends and role models who have become colleagues. I d like to thank my parents who told me I could do anything I wanted and who worked hard to ensure I believed it. They taught me to love learning what a wonderful gift. Finally, I d like to thank my wife, who reads nearly everything I write and offers critical insights in a supportive, constructive manner. She understands when I m distracted by my work and allows me the space to be ambitious. She loves me in spite of my faults. iv

5 Table&of&Contents& Acknowledgments... iii! List of Tables... xi! List of Figures... xiii! List of Plates... xiv! List of Appendices... xv! 1! Introduction... 1! 1.1! Using audit and feedback to improve quality of care... 1! 1.1.1! Knowledge translation and changing healthcare-related behaviours... 1! 1.1.2! Development of audit and feedback for health provider behaviour change... 4! 1.1.3! Systematic reviews of audit and feedback in health care... 5! 1.1.4! Best practices for audit and feedback: theory and empirical evidence... 6! 1.1.5! Effect modifiers of audit and feedback: targeted behaviour and context... 9! 1.2! Audit and feedback in primary care... 11! 1.2.1! Complex patient populations with competing priorities... 11! 1.2.2! Shifting sands: primary care in Ontario... 12! 1.2.3! Existing audit and feedback programs in Ontario primary care... 14! 2! Objectives and approach... 16! 2.1! Overall aim and specific study questions... 16! 2.2! Approach... 18! 2.2.1! Multiple methods... 18! 2.2.2! A pragmatic paradigm... 19! 3! Papers... 21! 4! Paper 1: Audit and feedback: effects on professional practice and healthcare outcomes... 22! 4.1! Background... 22! v

6 4.1.1! How the intervention might work... 22! 4.1.2! Why it is important to do this review... 24! 4.2! Objectives... 25! 4.3! Methods... 28! 4.3.1! Search methods... 28! 4.3.2! Selection criteria... 28! 4.3.3! Data collection and analysis... 28! 4.3.4! Types of studies... 29! 4.3.5! Types of participants... 29! 4.3.6! Types of interventions... 29! 4.3.7! Types of outcome measures... 30! 4.3.8! Search methods for identification of studies... 31! 4.3.9! Electronic searches... 31! ! Searching other resources... 32! ! Data collection and analysis... 32! ! Selection of studies... 32! ! Data extraction and management... 33! ! Assessment of risk of bias in included studies... 34! ! Measures of treatment effect... 34! ! Unit of analysis issues... 35! ! Cluster randomised trials... 35! ! Studies with more than two arms... 36! ! Dealing with missing data... 36! ! Assessment of heterogeneity... 36! ! Data synthesis... 36! ! Subgroup analysis and investigation of heterogeneity... 37! vi

7 4.3.17! Sensitivity analysis... 40! 4.4! Results... 41! 4.4.1! Description of studies... 41! 4.4.2! Characteristics of setting and professionals... 45! 4.4.3! Targeted behaviours... 45! 4.4.4! Characteristics of interventions... 45! 4.4.5! Outcome measures... 46! 4.4.6! Risk of bias in included studies... 46! 4.4.7! Effects of interventions... 47! ! Dichotomous measures of compliance with desired practice... 49! ! Continuous measures of compliance with desired practice... 50! ! Patient outcomes... 51! ! Investigation of heterogeneity... 51! ! Exploratory analyses... 61! 4.5! Discussion... 79! 4.5.1! Summary of main results... 79! 4.5.2! Overall completeness and applicability of evidence... 82! 4.5.3! Quality of the evidence... 82! 4.5.4! Potential biases in the review process... 83! 4.5.5! Agreements and disagreements with other studies or reviews... 84! 4.5.6! Implications for practice... 87! 4.5.7! Implications for research... 88! 4.5.8! Conclusions... 89! 5! Paper 2: Feedback GAP: pragmatic, cluster-randomized trial of Goal setting and Action Plans to increase the effectiveness of audit and feedback interventions in primary care ! 5.1! Background... 90! vii

8 5.2! Methods... 92! 5.2.1! Study design... 92! 5.2.2! Setting... 92! 5.2.3! Participants and data collection... 93! 5.2.4! Allocation... 93! 5.2.5! Intervention... 94! 5.2.6! Outcomes... 95! 5.2.7! Analysis... 96! 5.3! Results... 98! 5.4! Discussion ! 6! Paper 3: My job is one patient at a time : perceived discordance between populationlevel quality targets and patient-centered care inhibits quality improvement ! 6.1! Background ! 6.2! Methods ! 6.2.1! Setting and Context: ! 6.2.2! Participants: ! 6.2.3! Intervention: ! 6.2.4! Data Collection ! 6.2.5! Analysis ! 6.3! Results ! 6.3.1! Participant characteristics ! 6.3.2! Usefulness of the feedback for systematic chronic disease management ! 6.3.3! Barriers to QI efforts in response to feedback ! 6.3.4! Preferences for intervention design to support QI ! 6.4! Discussion ! 7! Overall thesis discussion ! viii

9 7.1! Summary of findings ! 7.1.1! Effectiveness of audit and feedback is partially explained by how it is designed and delivered ! 7.1.2! Intervention components must be implemented in a manner that fits the context ! 7.1.3! Capacity for accomplishing the targeted behaviour change must be clearly understood ! 7.2! Strengths and limitations ! 7.2.1! Systematic review, meta-analysis and meta-regression ! 7.2.2! Development and evaluation of the goal setting and action planning intervention ! 7.3! Implications ! 7.3.1! Lack of Progress in the field ! 7.3.2! Implications for further A&F research ! 7.3.3! Implications for conducting audits ! 7.4! Conclusions ! 8! References ! 9! Appendices ! 9.1! Appendix ! 9.1.1! Electronic Search Strategies ! 9.1.2! List of Included studies ! 9.1.3! List of excluded studies ! 9.2! Appendix ! 9.2.1! Goal-setting and Action-plan Worksheet for Intervention Arm ! 9.2.2! Prototype of feedback report that all participants received ! 9.3! Appendix ! 9.3.1! Original CME surveys ! ix

10 9.3.2! Revised CME surveys ! 9.3.3! Information provided to explain feedback reports ! 9.4! Appendix ! 9.4.1! Feedback preferences ! 9.4.2! Initial response and reaction ! 9.4.3! Personal barriers ! 9.4.4! QI infrastructure ! x

11 List of Tables Table 1: Selected variables considered for inclusion in meta-regression analysis... 39! Table 2: Description of included trials (N=140)... 44! Table 3: Summary of findings: Audit and feedback for health professionals... 48! Table 4: Assessment of Heterogeneity: results from meta-regression... 53! Table 5: Exploratory analysis, meta regression with targeted behaviour... 62! Table 6: Intervention components and description... 94! Table 7: Composite process score calculated for each patient as primary process outcome ! Table 8: Baseline characteristics of clinics, and family physicians ! Table 9: Baseline characteristics of patients ! Table 10: Outcomes for patients receiving feedback plus the goal-setting and action-planning worksheet versus feedback alone ! Table 11: Change in outcomes over time by intervention arm, adjusted for effects of clustering ! Table 12: Outcomes for patients with diabetes but not IHD, adjusted for effects of clustering. 107! Table 13: Outcomes for patients with IHD but not diabetes, adjusted for effects of clustering. 108! Table 14: Outcomes for patients with both DM and IHD, adjusted for effects of clustering ! Table 15: Characteristics of twelve participants selected for interviews and of fifty-four potential participants ! Table 16: Selected barriers and suggested areas for future research when conducting audit and feedback for quality improvement in primary care ! xi

12 Table 17: Consistent effect sizes observed multiple versions of Cochrane review of A&F ! xii

13 List of Figures Figure 1: Study flow diagram... 42! Figure 2: Risk of bias graph: review authors judgements about each risk of bias item presented as percentages across all included studies ! Figure 3: Bubble plot, adjusted risk difference by baseline performance... 54! Figure 4: Box plot, comparing adjusted risk difference by format of feedback... 55! Figure 5: Box plot, comparing adjusted risk difference by source of feedback... 56! Figure 6: Box plot, comparing adjusted risk difference by frequency of feedback... 57! Figure 7: Box plot, comparing adjusted risk difference by presence/extent of instructions for improvement... 58! Figure 8: Box plot, comparing adjusted risk difference by direction of change required by the feedback... 59! Figure 9: Box plot, comparing adjusted risk difference for Comparison B (audit and feedback alone versus usual care) and Comparison C (multifaceted intervention featuring audit and feedback versus usual care)... 60! Figure 10: Cluster-Patient flow diagram ! Figure 11: Study feedback report provided to participating family physicians with aggregated proportion of patients meeting nine quality indicators ! Figure 12: Ontario Ministry of Health feedback report with patient-specific data provided to all family physicians across the province for three process measures ! xiii

14 List of Plates Box 1: Definitions of intensity of interventions as operationalized in the 2006 Cochrane review of audit and feedback... 6! Box 2: Usefulness of the feedback reports for QI ! Box 3: Barrier: Participants challenges leveraging the EMR for action ! Box 4: Barrier: Tension between population-level targets and individualized clinical decisions ! Box 5: Barrier: Challenges with priority setting in primary care ! Box 6: Desire to focus feedback on higher-risk patients ! Box 7: Desire for additional resources to manage chronic disease initiatives ! xiv

15 List of Appendices 9.1! Appendix ! 9.2! Appendix ! 9.3! Appendix ! 9.4! Appendix ! xv

16 1 Introduction 1.1 Using audit and feedback to improve quality of care Knowledge translation and changing healthcare-related behaviours Health, economic, and moral arguments make the case for spending less on technological advances and more on improving systems for delivering care. 1 Evidence of best practices in health care can only benefit patients if the knowledge is converted into practice. Yet one of the most consistent findings in health services research is a large gap between ideal care and actual care provided to patients. Suboptimal practices have been observed for every type of patient problem, from primary prevention to trauma care, and for every type of professional practice, from investigations to prescribing. In fact, the gap between ideal and actual care was described in a 2001 Institute of Medicine report as a Quality Chasm. 2 Consider patients with diabetes and/or heart disease who have elevated risk of cardiovascular events. 3 Guidelines provide strong recommendations based on high-quality evidence regarding pharmacological and non-pharmacological strategies to reduce this risk. Unfortunately, despite widespread awareness of these guidelines, many of these patients receive inadequate management of cardiovascular risk factors 4-8. The scientific field that aims to address the problem of sub-optimal implementation of evidence-based practices in health care is known in Canada as knowledge translation (KT). CIHR defines KT as a dynamic and iterative process that includes the synthesis, dissemination, exchange, and ethically-sound application of knowledge to improve the health of Canadians, provide more effective health services and products, and strengthen the health care system. 9 Various KT-related terms may be used to describe the concept of moving from knowledge to action. 10 Although variably defined, the knowledge to action cycle generally involves the following (non-linear) phases: problem identification, assessment of barriers to knowledge use, tailoring interventions, monitoring knowledge use, and evaluating outcomes to inform sustained use and/or revision of the cycle. 11 1

17 Ultimately, accomplishing KT requires a change in behaviour and therefore implementation of best practices (ensuring the right care for the right person at the right time) depends on understanding the determinants of desirable behaviours by health professionals (and their patients). Many studies have examined why gaps in care exist. A 1999 systematic review of 76 studies formulated findings regarding more than 250 types of barriers into the categories of knowledge (i.e., lack of familiarity with specific recommendations), attitudes (i.e., lack of agreement that application of the guideline will lead to desired outcomes in a particular patient), and behaviour (i.e., inability to accomplish the recommendations despite intentions to do so). 12 A Cochrane review of 26 trials indicates that interventions tailored to specific barriers can lead to improvements in professional behaviour. 13 This underscores the importance of understanding why gaps in care exist in order to develop effective interventions that will improve adherence to guideline recommendations. To develop evidence for strategies to translate guideline recommendations in practice that are as robust as the evidence guiding the recommendations themselves, it is crucial to understand the active ingredients of interventions aiming to change practice. 14 However, it remains unclear how to best identify modifiable determinants of health professional behaviour and use this information to develop successful interventions. Numerous theories from a variety of fields such as psychology, sociology, economics, marketing, education, and organizational behaviour have been used to understand professional behaviour and to develop improvement strategies. 15 Some theories focus on professional behaviours for specific tasks (i.e., micro-level theory), others on change within the organizational setting (i.e., meso-level theory), and still others consider broader social and economic contexts (i.e., macro-level theory). Many of these theories offer useful insights indicating potential effect modifiers and active components of KT interventions. A potential advantage of using such theories of human behaviour to understand the determinants of health professional behaviour is that it offers a basis for generalization across various contexts and targeted behaviours. 16 With this in mind, the UK Medical Research Council guidance on developing complex interventions (e.g., KT initiatives) recommends drawing on existing evidence and theory to develop a clear rationale for how an intervention will act on determinants of health professional and/or patient behaviour to result in desirable behaviour changes (i.e., developing a hypothesis regarding mechanism of action). 17 2

18 As of yet, there is no widely accepted taxonomy to describe improvement strategies or their components (though efforts are in process to address this issue 18 ). Commonly, KT interventions are categorized into those directed at the level of the patient (e.g., education, reminders, development of self-management skills, etc.), the health professional (e.g., education, decision support, audit and feedback, etc.), the organization of care (e.g., revision of professional roles, case management, shared care, etc.), and/or relevant regulatory or policy factors (e.g., accreditation, reimbursement, coverage inclusions, etc.). 19, 20 Although intuitively appealing, this type of high-level categorization has important limitations in that it tends to conflate features of intervention design with features of delivery and does not tend to emphasize the identification of components necessary to enact the desired behaviour change (i.e., the active ingredients). Thus, active components are variably described in the KT literature and single labels are given to interventions with different active components. 21 The lack of consensus regarding KT intervention taxonomy, 22 combined with uncertainty in best methods to identify determinants of behaviour, and challenges in linking these determinants to intervention components, 23 likely contributes to uneven results in the literature and reflect a field that is still in its infancy. Nevertheless, a large (and growing) evidence-base exists for stakeholders considering the development of interventions to close the gap between ideal and actual care. 24 In fact, a variety of KT interventions have been used to accomplish similar end-goals with respect to patient outcomes. Therefore, the Cochrane Effective Practice and Organization of Care group develops two types of systematic reviews of KT interventions: reviews of different improvement strategies aiming to improve a particular outcome and reviews summarizing the effect of specific strategies regardless of the type of practice or behaviour involved. 25 This thesis focuses on a particular type of KT intervention: audit and feedback (A&F). As described in Section 2 (Objectives), I set out to examine the effectiveness of A&F as a means to improve quality of care and to examine potential effect modifiers, using constructs from relevant behavioural theories to inform the questions and analyses. Below, I outline my justification for examining A&F both in general and specifically in primary care, and I describe the state of knowledge prior to this thesis regarding the effectiveness of A&F in healthcare. To provide context, I also briefly describe the state of primary care in Ontario and existing A&F initiatives in the province. This background information sets the stage for the detailed project objectives and the subsequent discussion of my approach to this research. 3

19 1.1.2 Development of audit and feedback for health provider behaviour change To assess whether patients are receiving the best quality of care it is essential that we measure practice to know when we need to change it 26 Historically, continuing professional development programs for health professionals have emphasized the tenets of adult learning theory. 27 In this approach, it is felt that health professionals are self-directed learners, who actively evaluate their progress and their abilities in order to plan future learning projects. Unfortunately, measures of adherence to guideline recommendations based on physician self-report are so inaccurate that they are felt to be invalid as a measure of performance. 28 Furthermore, systematic reviews indicate that traditional, passive approaches to continuing medical education rarely lead to behaviour change In fact, rather than constantly trying to address gaps in learning, physicians often resist learning in areas of weakness. 31 Therefore, audit and feedback is felt to be effective as a tool to improve professional practice because it may overcome physicians limited ability to accurately self-assess. 32 A clinical audit measures a set of clinical practices against standards. 26 Aspects of professional processes and patient care which may include measures of structure, processes, and/or outcomes 33 - are selected and evaluated against explicit criteria. The strategy is based on the belief that health professionals are prompted to modify their practice when shown that their clinical practice is inconsistent with recommendations or not meeting relevant benchmarks. Future audits should then be conducted to see if the initial audit led to improvements in healthcare delivery. 34 Audit and feedback can be organized internally, (i.e. by local groups of practitioners), externally, (i.e. run by centralized professional bodies, research groups or governmental structures), or a combination of both. How the audit and feedback is organized may be related to the underlying objectives of the program. For example, internal audits may be conducted in pursuit of continuous quality improvement or continuing professional development, while external audits may be initiated to encourage accountability, linked to accreditation and certification, public reporting, or to economic incentives and reimbursement schemes. Regardless of the program objectives, their achievement is dependent upon the availability of accurate, relevant data. Performance data may be analyzed using information from electronic patient records, medical registries, administrative databases, standardized observations, and/or patient questionnaires. If data are available, the feedback may include a comparison of 4

20 individuals performance patterns with those of immediate peers, or with aggregate data from large groups of providers, units or hospitals. Once data are collected, decisions must be made about who should receive the feedback, how the data should be presented, how it should be delivered, and how often. Considerations for these decisions may include the expertise of those conducting the program, the feasibility and/or cost of different options, recipient interest and preferences, as well as policy issues such as privacy legislation. In other words, A&F is not a simple, standardized intervention its form (and function) varies widely Systematic reviews of audit and feedback in health care Audit and feedback can be a useful intervention, but the effects of audit and feedback vary widely 35 In 1996, Balas published a systematic review of physician profiling, 36 finding a small improvement across twelve trials for test utilization and prescribing (odds ratio 1.091, 95% confidence interval to 1.136), but they did not examine effect modifiers. In 2004, Grimshaw reported a systematic review of 235 studies of 309 interventions to improve adherence to guidelines. 37 This included twelve studies that tested A&F alone against usual care and an additional 73 that tested A&F as part of a multifaceted intervention. For the six A&F trials with dichotomous outcomes, the median effect was a 7% increase in guideline-intended processes of care (range 1.3% to 16%). Sources of heterogeneity were not explored, other than the presence of co-interventions. In 2006, the Cochrane review of A&F concluded that A&F can effectively improve health professional processes of care. 38 The meta-analysis included 85 trials, finding a median adjusted increase in compliance with recommended practice of 5% for dichotomous outcomes (inter-quartile range (IQR) 3%-11%) and a 16% median adjusted percentage change related to control for continuous outcomes (IQR 5%-37%). The authors found that the effectiveness of feedback was greater when baseline performance was lower and also when the A&F intervention had greater intensity. The latter finding was noted in only one of their analyses, and the team defined intensity in a manner that made it difficult to identify individual active components (see Box 1), meaning that the results could not easily be used to design new interventions. 39 5

21 Box 1: Definitions of intensity of interventions as operationalized in the 2006 Cochrane review of audit and feedback Intensive: (individual recipients) AND [(verbal format) OR (a supervisor or senior colleague as the source)] AND (moderate or prolonged feedback) Non-intensive: [(group feedback) NOT (from a supervisor or senior colleague)] OR [(individual feedback) AND (written format) AND (containing information about costs or numbers of tests without personal incentives)] Moderately intensive: any other combination of characteristics Two other systematic reviews of A&F have been conducted that considered effect modifiers for the intervention. In 1991, Mugford conducted a narrative review of 36 studies. This review emphasized the importance of focusing on indicators of interest to the recipients and stressed that feedback would be more effective when relevant information is delivered to those with the power to act, as close as possible to the time of decision making. 40 In 2010, Van der Veer published a systematic review of the impact on quality of care of using medical registries to produce feedback reports to health professionals. 41 This review included 53 studies of widely varying quality and considered both quantitative and qualitative data. They noted that important effect modifiers seemed to be the quality of the data provided to recipients, the motivation and interest of recipients, and the organizational support for quality improvement Best practices for audit and feedback: theory and empirical evidence Audit and feedback will continue to be an unreliable approach to quality improvement until we learn how and when it works best. 39 In general, for audit and feedback (or any other quality improvement intervention) to be successful, those expected to change their behaviour must have motivation, capability, and 6

22 opportunity to improve. 42 However, even when motivation, capability, and opportunity are seemingly present, feedback does not consistently lead to improvement, and this may be attributable, at least in part, to sub-optimal design and/or implementation of the intervention. 43 Unfortunately, few studies have directly tested different strategies for the design and/or delivery of feedback, making it difficult to determine best practices for conducting A&F in health care. 39 One aspect of feedback that has been examined in trials is the type of benchmark or comparison data provided. Two trials have shown that feedback with peer comparisons were slightly more effective than feedback without normative benchmarks. 44, 45 One head-to-head study found that when aggregate data is used in feedback reports to physicians, reports comparing performance to a relevant, achievable benchmark (i.e., the scores achieved by the top decile of peers) resulted in greater improvements than reports comparing to the mean performance of peers. 45 A plausible explanation for this finding is that recipients with performance near the mean may be reassured that their score is not substandard when the mean is used as a benchmark. Many other feedback components, such as whether feedback should include specific action plans, have less direct evidence to inform their design. In 1996, Kluger and DeNisi conducted a systematic review and meta-analysis of feedback interventions, combining over 600 standardized effect sizes and found a moderate improvement in performance (Cohen s d=0.41), but noted that 1/3 of effects were negative. They developed the Feedback Intervention Theory (FIT) to explain their findings. 46 The FIT posited that the effectiveness of feedback interventions, when salient, depends on the locus of attention of the recipient. For instance, feedback will be less effective if the feedback shifts the recipient s attention away from the task and especially if the feedback generates self-doubt or self-concern. In other words, FIT posits that activation of emotional responses diverts attention from the task. Kluger and DeNisi proposed that in addition to the nature of the feedback design and delivery, characteristics of the targeted behaviour and of the recipient (and their context) each play a role in determining how the recipient will respond to feedback. A re-analysis of the 2006 Cochrane review testing prepositions from FIT found that feedback was more effective when it included instructions about the how to carry out the desired behaviour change (e.g., action plans). 47 The findings were in keeping with FIT, since correct solution information can help focus attention on the targeted behaviour change while minimizing cognitive overload, and verbal delivery risks an affective response, potentially distracting from the intended task. 7

23 Since feedback can lead to decreased performance (especially if the recipient feels threatened), feedback when delivered verbally should be offered in a formative, supportive manner 48 and should feature learning tools or suggest training options for areas that need improvement, especially when the goal is difficult to attain. 49 Facilitating recipients reflection upon feedback can help with reconciliation of any discrepancy between ideal and actual performance and enable a productive reaction. 50 Nevertheless, when facilitated reflection is not feasible (e.g., when feedback is provided at broad scale), empirical evidence from the management literature suggests that goal setting can increase the effectiveness of feedback, 51 especially if specific and measurable goals are used, by increasing goal-commitment. Likewise, action plans can facilitate success in reaching a goal both by increasing goal-commitment and by overcoming barriers such as distraction or fatigue. 52 Control Theory 53 suggests that discrepancy between desired and actual performance leads to a response by recipients to reduce the cognitive dissonance, 54 either by discounting the feedback (e.g., by questioning its validity), or by taking action to resolve the discrepancy (e.g., improve performance). Those who are dissatisfied with their performance will be more likely to develop a change in behaviour if they are committed to the goal and if they meet a threshold level of self-efficacy for that task. 55 Involvement in the audit process may decrease the likelihood that recipients will discount the veracity of the data when discrepancies are highlighted, as noted in one previous trial, where participating the audit process was more effective than simply receiving data. 56 However, significant improvement was not seen across all targeted behaviours in that trial, and one other trial found that involvement in setting targets actually was associated with decreased performance. 57 Another reanalysis of the 2006 Cochrane review that applied Control Theory to test target-setting and action plans as effect-modifiers of feedback was inconclusive because very few studies explicitly described their use of targets or action plans. In the management literature, successful feedback (leading to behaviour change) is thought to require high perception of accuracy and a positive overall appraisal reaction. 58 The appraisal reaction is essentially a function of the learner s satisfaction with and acceptance of the feedback. To some extent, this may reflect the recipient s receptivity to feedback, which itself is thought to depend on individual differences in sensitivity to others views of oneself, perceived utility of feedback to help accomplish goals, sense of accountability to act on the feedback, and self-efficacy to accomplish the desired task. 59 The legitimacy of the source of feedback also 8

24 plays an important role; there is some evidence that feedback interventions may be more effective when delivered by peers 60 and/or opinion leaders. 61 In addition, individual recipients likelihood of seeking feedback is correlated with the extent to which constructive feedback is readily available from supervisors and delivered in a supportive manner. 62 These findings may be interpreted as further support for the importance of providing feedback in a manner that focuses attention on the task rather than the self, as suggested by FIT. A qualitative study collecting data from interviews of over 100 health care providers in the American Veterans Administration health system examined how higher and lower performing organizations use clinical audit data. 63 This study found that higher performing facilities were more likely to have feedback that featured a non-punitive tone, again in keeping with FIT. Other key features included timeliness (so that data are relevant), individualization (as opposed to aggregated or group-level data), and customizability (to alter the level of detail). Feedback with all these features was more likely to be perceived as actionable. The management literature echoes this emphasis on action-ability, and therefore urges a focus on process metrics in feedback rather than outcomes. 64 Recognizing that feedback alone is sometimes not enough to change provider behaviours, further improvements have been sought by pairing feedback with more intensive (and generally more expensive) co-interventions, such as educational outreach visits. 65 Although these intensive interventions are rarely informed by explicit theory 66 it is reasonable to believe the hypothesized mechanism of action is to help participants to focus attention on tasks required to address gaps in care identified by the feedback reports. This type of co-intervention may be based on beliefs that health care providers are adequately motivated to provide consistently high quality of care, and that feedback may help them identify opportunities for improvement, but that co-interventions are required to enable specific actions Effect modifiers of audit and feedback: targeted behaviour and context There are countless attractive options people do not pursue because they judge they lack the capabilities for them 55 9

25 The 2006 Cochrane review of A&F found an inverse relationship between baseline performance and effect size, which could reflect room for improvement and ceiling effects. 35 However, strength of the relationship between baseline performance and effectiveness of feedback is likely moderated by other factors related to the targeted behaviour, including the complexity 67 and trialability 68 of the change(s) required and the fit within normal workflows. 69 In addition, some portion of feedback response is dependent on the characteristics of the recipient and their disposition toward the targeted behaviour. Recipients are likely to increase effort toward a goal only to the extent that they believe that the increased effort will lead to increased performance (or improved outcomes for patients). 70 Therefore, feedback may be most effective when targeted changes in behaviour(s) that recipients perceive to be both achievable and connected to desirable results. Certain recipient characteristics that may moderate the impact of feedback interventions on motivation to perform the targeted behaviour warrant consideration. 70 For example, those with external (versus internal) locus of control may be more responsive to feedback with normative comparisons and information about extrinsic rewards. In the educational literature, it is suggested that low achieving learners struggling with new tasks should receive more directive, immediate feedback, while for higher achieving learners, delayed feedback to provide hints or verify progress is preferable. 43 Thus, if A&F interventions have the capacity to consider recipient characteristics and to tailor the intervention accordingly, greater success may be achievable. It is also known that people are more likely to try to accomplish a feedback-related goal if they have self-efficacy for the targeted behaviour. 55 This may indicate that less complex tasks could be more amenable to improvement through feedback, especially when correct solution information (i.e., action plans) may be easier to communicate. Even if the recipient s intention to change behaviour is strong, the desired action may depend on external constraints related to the recipient s opportunity for improvement and capability to enact changes. Organizational theories focused on quality improvement and/or innovation offer clues regarding potential important contextual effect modifiers for A&F. 71 Numerous conceptualizations of organizational readiness for change have been proposed, 72 and key concepts include managerial commitment and availability of resources. A cross-sectional study in 37 primary care practices in Quebec found an association between the extent of shared administrative resources and quality of care. 73 The extent of quality improvement infrastructure 10

26 and its role in quality for primary care illustrates the complex relationship between the recipient s self-efficacy for improvement and their potential reaction to A&F. 1.2 Audit and feedback in primary care Complex patient populations with competing priorities If it is patient health (rather than disease processes or outcomes) that is of interest as the proper focus of health services, a broader view of quality of care provided to patients is in order. 74 As indicated above, measurement of quality of health services is often operationalized as adherence by health professionals to guideline-recommended processes of care. A number of studies have indicated that patients followed by specialists are more likely to receive guidelinerecommended prescriptions for specific chronic disease conditions. 75, 76 However, adherence to single disease-oriented guideline recommendations may be inadvisable in certain circumstances. In primary care, patients are increasingly complex, presenting with multiple interacting biopsycho-social issues and the evidence used to develop disease-specific guideline recommendations may not apply. 77 A meta-synthesis of qualitative studies examining primary care physicians attitudes toward implementing guidelines highlighted concerns about applicability, especially in cases of multi-morbidity. 78 Providing high quality care defined according to guideline adherence in this setting is also challenging because recommendations from different guidelines may contradict each other. 79 Furthermore, even if the intention to adhere to guideline recommendations is present, there simply may not be enough time to implement all guideline recommended actions for prevention and management of common conditions in primary care. One study suggested that in a standard sized practice, with average disease prevalence, it would take up to 10.5 hours per day to manage the top 10 chronic conditions. 80 This does not account for any time taken to address the actual reasons for the patient s visit. This situation makes it inevitable that some recommendations will not be implemented; prioritization of patient needs is an essential role of the primary care provider. 11

27 Furthermore, adherence to clinical recommendations in guidelines generally represents only one aspect of quality in health care. In the Quality Chasm report referenced earlier, the Institute of Medicine proposed six domains 2 : patient-centeredness, equity, timeliness, safety, effectiveness and efficiency. International comparisons suggest that additional key attributes for successful primary care systems include longitudinal relationships, comprehensiveness, and coordination. 81 Ready access to patient-centered primary care meeting these attributes provides the necessary foundation for health systems to efficiently improve population health. Therefore, it is important to consider the risk that overemphasis of one domain of quality (e.g. clinical effectiveness) could drown out others (e.g. patient centeredness). 82 For instance, in one study where primary care physicians were mailed a clinical vignette of a patient with diabetes whose main issue was worsening chronic low back pain, only 44% of American primary care physicians prioritized management of pain over management of diabetes. 83 Despite this finding, the values and norms of family physicians, who provide the vast majority of primary care in Ontario, prioritize patient-centeredness. 84 Of the four principles of family medicine, only one ( the family physician is a skilled clinician ) may be taken to reflect the clinical effectiveness domain, and it does so while emphasizing the importance of reaching common ground with respect to goals of care and treatment options. 85 Thus, tensions arise when patient preferences do not align with guidelines, 86 recognizing that in multi-morbidity what s good for a specific disease is not always best for the whole patient Shifting sands: primary care in Ontario Family Health Teams should become the norm for primary care. They need to be big enough to support a wide range of care providers and the number of other staff needed to track people through the system. They should offer better after-hours care and add more specialists to their teams 87 Many authors, both in the scientific literature 88 and popular media 89 have identified a mismatch between changing demographic trends and health system resources. In particular, Ontario has an aging population, and an increasing proportion of patients with multiple chronic conditions. 90, 91 As the complexity of patients in the community has increased, new models of 12

28 primary care have been developed and promoted in Ontario aiming to facilitate improved management of patients with multiple chronic conditions. 92 Fortunately, there is increasing evidence that ambulatory care can be redesigned to achieve better outcomes for such patients. In particular, the Chronic Care Model, which incorporates self-management support, decision support, delivery system re-design, clinical information systems, healthcare organization, and community resources, has proven an extremely popular (and potentially effective) approach. 93 Results of health system re-design based on the Chronic Care Model vary depending on the specific intervention components included, characteristics of the patients targeted, and nature of the setting. 94 Nevertheless, the tenets of the Chronic Care Model are reflected in the movement toward implementation of Patient Centered Medical Homes, which tend to require enhanced staffing, meaningful use of electronic health records, and attention to change management for organizational transformation to a new practice model. 95 Early evaluations of Medical Homes indicate the potential for improved patient experience and cost 96, 97 savings, as well as reduction in emergency department visits and hospitalizations. In Ontario, the primary care model most in keeping with the patient centered medical home (known as Community Health Centres (CHCs)) has salaried physicians, multiple allied care providers, and governance models that attempt to align services with community needs. Compared to other models of primary care in Ontario, CHCs achieve lower than expected emergency department utilization amongst their population. 98, 99 While CHCs were developed specifically to service vulnerable populations, another model known as Family Health Teams (FHTs) was developed to incorporate allied health professionals into the broader primary care context. FHTs represent the most recent stage of a primary care reform process in Ontario from exclusively fee-for-service models toward patient enrolment models with multidisciplinary teams, and support for both IT infrastructure and quality improvement. 100 However, this is a process that is far from complete. Even FHTs with EMRs do not necessarily have the capacity to evaluate quality of care and even those who can acquire usable data may not have the capacity to act upon it. Anecdotally, many FHTs are teams only in name; a group of family physicians paid mostly by capitation sharing administrative resources and access to allied health professionals, but with mostly unchanged patterns for provision of care. 13

29 Furthermore, despite massive investments, relatively few patients are seen in team-based models. 101 Approximately 25% of Ontario family physicians continue to work as solo providers practicing under the traditional fee-for-service model. 92 An additional 25% of family physicians practice in putative patient enrolment and team-based models (known as Family Health Groups (FHG)). These tend to take the form of large groups of physicians that are not co-located, with minimal evening walk-in hours staffed by physicians not known to the patients in locations that may not be convenient. Despite receiving fewer resources than FHTs, patients of FHG physicians and the solo physicians are more likely to have financial difficulties and/or new 98, 102 immigrant status, and FHG patients are also more likely to have greater morbidity. Furthermore, the financial incentives in these fee-for-service models encourage one-problemper-visit strategies that are poorly suited for ensuring that vulnerable patients with multiple conditions receive evidence-based care. According to the Commonwealth Fund surveys, 103 Canada s healthcare system ranks 6th in overall performance on safety, quality, equity and efficiency measures among seven nations (Australia, Canada, Germany, the Netherlands, New Zealand, the United Kingdom, and the United States). While over 80% of Canadians report having a primary care provider, urgent access is poor, reducing the extent to which the primary health care system can improve outcomes. Coupled with sub-optimal access to care overall, the variety of primary care models in Ontario creates a range of contextual factors to consider when planning an audit and feedback intervention Existing audit and feedback programs in Ontario primary care Ontario lacks a coherent system for ongoing primary care performance measurement and feedback at the practice, organization, and system (community, regional, and provincial) levels 92 One previous study has assessed 8 Ontario physicians reactions to a 20-minute one-onone performance assessment presentation based on chart audits and patient questionnaires and found that physicians welcomed it. 104 Even though the data was garnered directly from charts, the participants expressed concerns about government involvement in the performance 14

30 improvement process. This may reflect the fact that there is relatively little systematic clinical audit in primary care in Ontario. 92 Until recently, this could be attributed in part to the limited use of electronic medical records (EMRs). With data relatively inaccessible, minimal efforts were put towards developing skillsets in population/panel management in primary care. Yet with the introduction of Ontario s Excellent Care For All act, it is anticipated that all primary care providers in patient enrolment models in the province (currently over 75% of family physicians) will have to develop and report upon progress for annual quality improvement plans by In addition, over the past decade, uptake of EMRs in primary care has substantially increased; the majority of primary care providers now use EMRs. Thus, clinical audit will likely become more prominent in the Ontario primary care setting as the government (and patients) seek greater accountability. To date, a few large-scale A&F programs have been implemented in Ontario. In 2010, the government launched a provincial attempt to develop a diabetes registry to facilitate performance feedback reports to primary care providers, as well as dissemination of patientmediated tools. However, the project proved too difficult to manage and was terminated in One factor that played a role in the cancellation of the project was that it was never developed in a manner that could integrate with EMRs in primary care. This illustrates how changes in the context can alter the effectiveness of A&F. More recently, Cancer Care Ontario developed reports for nearly all family physicians across the province regarding screening for colon cancer, cervical cancer, and breast cancer. After some paper-based reports were lost in the mail, Cancer Care Ontario developed a secure website to provide the feedback. No evaluation has been reported of this web-based initiative; anecdotally, acquiring access to the website is difficult and the data is not always accurate. To date, the website includes only data for colon cancer screening. Furthermore, there is no plan at present to embed the reports into EMR infrastructure. This illustrates the challenges for systematic A&F initiatives in Ontario with respect to managing privacy risks, obtaining comprehensive, accurate data, and feeding it back in a way that is actionable. 15

31 2 Objectives and approach 2.1 Overall aim and specific study questions In short, the overall aim is to optimize audit and feedback to improve quality in primary care. The goal is to determine how to best design and deliver performance feedback to family physicians regarding compliance with evidence-based chronic disease management recommendations to more reliably result in professional behaviour change that will benefit patients. This cannot be accomplished with a single study and so this dissertation asks a series of related questions: 1. What is the effect of audit and feedback interventions on quality of care and patient outcomes and what components of feedback act as effect modifiers? 2. Is the effectiveness of an audit and feedback intervention related to chronic disease management increased when accompanied by a goal-setting and action-planning worksheet? 3. What are the barriers and facilitators to improving quality of care for chronic disease management, as perceived by family physicians who have received performance feedback reports? Question 1 was examined using a systematic review, meta-analysis, and meta-regression. Question 2 was examined using a pragmatic, cluster-randomized trial. Question 3 was examined using qualitative methodology, specifically through participant interviews. The effect modifiers examined in the analyses of heterogeneity in the systematic review was informed by constructs from the A&F-relevant theories described above. These theories emphasize the potential for goal setting and action planning to enhance the effectiveness of feedback and one approach to operationalize goal setting and action planning was tested in the trial. Sub-questions addressed in each of the projects are described within the individual papers (Section 4, Section 5, Section 6). 16

32 Through these methods, this thesis aims to better understand how to use A&F as a KT strategy in primary care. Although the evidence-based medicine (EBM) movement is linked with the development of guidelines (and therefore with interventions aiming to implement guidelinebased recommendations), this thesis does not examine the strengths and weaknesses of EBM. I fully recognize that EBM as usually operationalized tends to miss some essential elements for optimal clinical practice, 105 especially in primary care where trustworthiness and applicability of the evidence is questioned. 106 I further acknowledge that the definition of good clinical performance is controversial and that, as noted above, disease-based guidelines do not address many important elements of clinical practice. As a result, there is a risk that audit of guidelinebased criteria may elevate the value of measurable parameters over those that are not readily measured. 107 As patients become increasingly complex, focus on implementation of diseasespecific guideline recommendations may have unintended consequences. Relatedly, A&F interventions can create perverse incentives, and recipients may attempt to game the system. 108 Furthermore, a flawed audit can result in inaccurate judgements about health service providers. 109, 110 Despite these risks, A&F is frequently implemented, suggesting a need to understand how it can best be used to improve clinical practice. An additional, related topic that is outside the scope of this thesis is the role of electronic medical records (EMR) in quality of care. There is great hope placed in the potential of EMR to create a more efficient system and increasing support in North America for implementation. 111 Improved ability to audit quality of care metrics and to provide recommendations accordingly is the basis for the potential seen in health information technology. Of note, the effects of EMR use for patients with diabetes are promising in terms of improving processes of care, but uncertain regarding improving patient outcomes. 112 By facilitating performance measurement, EMRs offer at least the potential for more accountable (if not higher quality) primary care in Canada

33 2.2 Approach Multiple methods As noted above, the research objectives are addressed using a variety of research methods, each chosen to address specific questions. Just as the questions are inter-related, the methods used to answer them are complementary. A sequential (yet overlapping) approach was used in which the first phase informed the second phase and so forth. Specifically, in the first phase, the systematic review estimated the effect of A&F interventions and indirectly examined a variety of effect modifiers, including goal setting and action planning through meta-regression. Systematic reviews represent an approach to knowledge synthesis that generate the K for KT, 114 and a proper foundation so that KT initiatives can be informed by all available research evidence. 115 In the second phase, a randomized trial was used to test a specific, promising approach to operationalizing goal setting and action planning. The intervention tested aimed to be readily scalable (i.e., easily implemented in other A&F initiatives) and so a pragmatic approach was taken in an attempt to maximize generalizability. 116 An experimental approach was used to facilitate causal inference, 117 given secular trends in the care of patients with diabetes and/or heart disease. 118, 119 To limit the risk for contamination, allocation was conducted at the level of the clinic (i.e., clusters of providers working together at the same location, and the patients they cared for were allocated as a group to one arm of the trial or the other). 120 In the third phase, the qualitative study examined how and why the recipients of the A&F intervention responded as they did through interviews of purposefully sampled participants. Potential advantages of conducting qualitative research alongside a quantitative analysis include the opportunity for insight regarding interpretation and usefulness of findings plus assessment of fidelity of the intervention. Although qualitative methodology has been inconsistently performed alongside complex quality improvement interventions, 121 there is reason to believe that it may be useful to help increase the likelihood of designing successful follow-up interventions. 122 The multiple methods resulted in a broad understanding of the challenges and opportunities for optimizing audit and feedback to improve quality of care. The approach taken and multiple methods used was informed by the UK Medical Research Council guidance on development of complex interventions. 17 The relevant evidence base was identified, explicit 18

34 theories developed regarding potential mechanism of action, and a rigorous evaluation was conducted, including an embedded process evaluation A pragmatic paradigm The first step in achieving the objectives laid out above is to reflect on the paradigm in which the research will proceed, based on a reflection of ontological (i.e., view of reality), epistemological (i.e., view of knowing), and axiological (i.e., view of value) positions. 123 In this case, I can reflect that as a practicing family physician with an interest in influencing the implementation of health services innovations, I take a viewpoint that places high significance on findings with practical application, and the research paradigm could be best described as pragmatic. Instead of rigid adherence to specific methodological choices, pragmatism encourages a moderate perspective, with the end-goal justifying the means. Attributed to Charles Sanders Pierce, the pragmatic maxim states: Consider what effects that might conceivably have practical bearings you conceive the objects of your conception to have. Then, your conception of those effects is the whole of your conception of the object. 124 Knowledge (and inference) is viewed as provisional. 125 Pragmatic theories are valued to the extent that they are applicable and effectively predict outcomes; the goal is to produce something useful and the value of a theory is dependent on how well it works to solve a problem. The application to clinical practice is fairly clear; the evidence-based medicine movement calls for physicians to apply up-to-date research results at the bedside, while accounting for the limitations in the evidence itself and striving for new, closer approximations to the truth. Often, evidence for a specific patient is not available or applicable. Pragmatists are willing to make necessary decisions based on, inference to the best explanation given the data at hand. 126 Pragmatism recognizes an external reality as well as the effect that perception and social interaction has on shaping that reality. In a pragmatic approach, there is no problem with asserting both that there is a single real world and that all individuals have their own unique interpretations of that world. 127 The first step in applying this paradigm to research methodology is determining the type of conclusions being sought. 128 Approaching a problem in this manner should lead to the 19

35 development of a refined research question and the nature of this question dictates the most useful research methodology. This perspective is concordant with the multiple methods used in this thesis, reflecting a belief that there are better (and worse) ways to conduct audit and feedback and that contextual factors including recipient characteristics moderate the effect in important ways. Importantly, pragmatism is related to, but separate from methodological choices that make clinical trials more or less pragmatic. 129, 130 The latter reflects not a philosophical approach, but methodological choices to improve real-world applicability of research findings. 20

36 3 Papers The next three sections are the manuscripts from the three projects conducted for this thesis. In the case of each manuscript, approval has been received to reproduce the papers in this thesis. The first paper (Section 4), entitled Audit and feedback: effects on professional practice and healthcare outcomes 131 is a systematic review, meta-analysis, and meta-regression of 140 RCTs. This was published in the Cochrane Database of Systematic Reviews in June Citation: Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O Brien MA, Johansen M, Grimshaw J, Oxman AD. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database of Systematic Reviews 2012, Issue 6. Art. No.: CD DOI: / CD pub3. The second paper (Section 5), entitled Feedback GAP: pragmatic, cluster-randomized trial of Goal setting and Action Plans to increase the effectiveness of audit and feedback interventions in primary care 132 was published in the journal Implementation Science in December Citation: Ivers NM, Tu K, Young J, Francis JJ, Barnsley J, Shah BR, Upshur RE, Moineddin R, Grimshaw JM, Zwarenstein M. Feedback GAP: pragmatic, cluster-randomized trial of Goal setting and Action Plans to increase the effectiveness of audit and feedback interventions in primary care. Implement Sci Dec 17;8:142. doi: / The third paper (Section 6), entitled My job is one patient at a time : perceived discordance between population-level quality targets and patient-centered care inhibits quality improvement 133 is a qualitative interview study of twelve family physician participants in the trial. Citation: Ivers N, Barnsley J, Upshur R, Tu K, Shah B, Grimshaw J, Zwarenstein M. My job is one patient at a time : perceived discordance between population-level quality targets and patient-centered care inhibits quality improvement Can Fam Physician Mar;60(3):

37 4 Paper 1: Audit and feedback: effects on professional practice and healthcare outcomes 4.1 Background Audit and feedback is widely used as a strategy to improve professional practice either on its own or a component of multifaceted quality improvement interventions. This is based on the belief that healthcare professionals are prompted to modify their practice when given performance feedback showing that their clinical practice is inconsistent with a desirable target. Despite its prevalence as a quality improvement strategy, there remains uncertainty regarding both the effectiveness of audit and feedback in improving healthcare practice and the characteristics of audit and feedback that lead to greater impact How the intervention might work Many theories exist (with multiple overlapping constructs) to further explain how feedback may lead to quality improvement (for a review of such theories, see Grol ). Briefly, individual behaviour change theories suggest that feedback may work in many ways, including (but not limited to) changing recipient awareness and beliefs about current practice and subsequent clinical consequences, changing perceived social norms, affecting selfefficacy, or by directing attention to a specific set of tasks (sub-goals). The observation that the effects of audit and feedback are greatest if baseline compliance is low supports the idea that audit and feedback is felt to be effective as a tool to improve practice because it may overcome health care providers limited ability to accurately self-assess. 32 Under this assumption, providers are thought to be inherently motivated to improve care, but lacking intention to change their current practices in large part because they are unaware of their suboptimal performance. In turn, they may be prompted to modify their practice if given feedback that their clinical practice was inconsistent with their peers or with accepted guidelines. 22

38 Nevertheless, even if intention to change behaviour is strong, the desired action may depend on multiple factors beyond the control of the health care provider. Organizational theories focused on quality improvement offer clues regarding potential important effect modifiers, including organizational culture with respect to quality improvement, and the actionability of feedback reports. 134 Van der Veer et al. 135 conducted a systematic review of the impact on quality of care of using medical registries to produce feedback reports to health care professionals. They analysed 53 studies of widely varying quality and considered both quantitative and qualitative data. Most of the studies featured multifaceted interventions. They noted that important effect modifiers seemed to be the quality of the data provided to recipients, the motivation and interest of recipients, and the organizational support for quality improvement. Some potentially important variables are difficult to operationalize in a trial and others have been tested with uncertain results. For instance, although perceived social and professional norms are considered important predictors of behaviour change, there is conflicting evidence regarding the role of peer-comparison in feedback. 44, 136, 137 In an attempt to further delineate how to most effectively design and deliver feedback interventions, Hysong 47 completed a re-analysis of the 2006 Cochrane review based on "Feedback Intervention Theory". 138 The results showed greater effectiveness with increasing frequency of the feedback, with written rather than verbal or graphical delivery and with feedback that included information about the correct solution. Similarly, Gardner and colleagues 139 conducted a reanalysis of the 2006 Cochrane review that applied the Control Theory of Carver and Scheier, 53 to test target-setting and action plans as effect-modifiers of feedback. Although the results of that re-analysis were inconclusive because very few studies explicitly described their use of targets or action plans, there is empirical evidence from non-health literature to suggest that goal-setting can increase the effectiveness of feedback, 140 especially if specific and measurable goals are used. However, the role of participant involvement in either target-setting or in feedback interventions remains uncertain. 141 Other empirical work from the psychology literature has demonstrated the value of action-plans with respect to improving the effectiveness of feedback. 142 Regardless of the feedback design, the nature of the clinical change that the feedback tries to encourage may play a role in the effectiveness of the intervention. Qualitative work indicates that it may be easier to comply with guidelines that aim to increase rather than decrease behaviours

39 4.1.2 Why it is important to do this review The aim of the current update is to investigate the effectiveness of audit and feedback to improve processes and outcomes of care and to examine factors that could influence the effectiveness of this intervention. Given the variability in results of the prior review and the inability to satisfactorily explain this based on intuitive factors, this review will attempt to examine multiple theory-informed feedback design characteristics. In so doing, we hope this review will clarify the effectiveness of audit and feedback in general and inform stakeholders regarding how to best employ feedback to change provider behaviours. 24

40 4.2 Objectives This review aims to assess the effects of audit and feedback on the practice of healthcare professionals and patient outcomes. We will address three primary questions: 1. Is audit and feedback effective for improving health provider performance and health care outcomes? 2. What are the key factors that explain variation in the effectiveness of audit and feedback? 3. How does the effectiveness of audit and feedback compare to other interventions? For question 1, we will consider the following comparisons: Comparison A. Audit and feedback alone or as the core/essential feature of a multifaceted intervention compared to usual care (includes comparisons B and C). Comparison B. Audit and feedback (alone) compared to usual care. Comparison C. Audit and feedback as the core/essential feature of a multifaceted intervention compared to usual care. As part of question 2, we will conduct a meta-regression on the studies in comparison A as well as consider the following comparisons: Comparison D. Head to head comparisons of different types of audit and feedback interventions (effect of changing the way that audit and feedback is designed or delivered). Comparison E. Audit and feedback as the core/essential feature of a multifaceted intervention compared to audit and feedback alone (effect of adding different cointerventions to audit and feedback). In the past review, we subjectively categorized both the intensity of the feedback intervention and the complexity of the targeted behaviour, but this method did not adequately predict feedback effectiveness in a manner that informed future intervention design. Therefore, to investigate the effectiveness of different ways of providing audit and feedback and other factors that might modify the effects of audit and feedback, studies in this review will be characterized according to a selection of variables considered to be both important (based on the 25

41 literature review and theories of behaviour change) and accessible in manuscripts (based on the experience of reviewers). Specifically, we will examine the effects of four ways of providing audit and feedback that might increase its effectiveness: Providing instruction for improvement with the feedback in the form of specific goals and/or action plans Providing verbal feedback in addition to written feedback Providing feedback from a senior or respected colleague, supervisor, employer, purchaser or professional standards review organization (compared to feedback provided by researchers) Providing more frequent feedback We will also examine additional factors not related to the intervention itself that might increase the effects of audit and feedback or its apparent effects: Lower baseline compliance Feedback requiring increasing current behaviors (compared to decreasing behaviors or changing the approach to a clinical problem) Audit and feedback targeting health professionals other than physician Higher risk of bias in the primary study There are many important factors that may predict effectiveness of audit and feedback; the basis for selecting the above factors to examine in a meta-regression and not including other potential effect modifiers is summarised in Appendix 1. (This appendix is not a comprehensive listing of all possible audit-and-feedback-design questions, but includes the key factors that we considered for inclusion in this update.) We recognize the importance of context with respect to the effectiveness of an intervention. In particular, the relative complexity of the targeted behaviour likely plays a role in the ability of feedback to increase guideline adherence. To investigate this issue, we conducted a limited number of exploratory sub-group analyses based on the target of the intervention. 26

42 For question 3, we will assess the following comparison: Comparison F. Audit and feedback alone or as the core/essential feature of a multifaceted intervention compared to other interventions 27

43 4.3 Methods Search methods We searched the Cochrane Central Register of Controlled Trials (CENTRAL) 2010, Issue 4, part of The Cochrane Library. including the Cochrane Effective Practice and Organisation of Care (EPOC) Group Specialised Register (searched 10 December 2010); MEDLINE, Ovid (1950 to November Week ) (searched 09 December 2010); EMBASE, Ovid (1980 to 2010 Week 48) (searched 09 December 2010); CINAHL, EBSCO (1981 to present) (searched 10 December 2010); Science Citation Index and Social Sciences Citation Index, ISI Web of Science (1975 to present) (searched 15 September 2011) Selection criteria Randomized trials of audit and feedback (defined as a summary of clinical performance over a specified period of time) that reported objectively measured health professional practice or patient outcomes. In the case of multifaceted interventions, only trials in which audit and feedback was considered the core, essential aspect of at least one intervention arm were included Data collection and analysis All data were abstracted by two independent reviewers. For the primary outcome(s) in each study, we calculated the median absolute risk difference (RD) (adjusted for baseline performance) of compliance with desired practice compliance for dichotomous outcomes and the median percent change relative to the control group for continuous outcomes. Across studies the median effect size was weighted by number of health professionals involved in each study. We investigated the following factors as possible explanations for the variation in the effectiveness 28

44 of interventions across comparisons: format of feedback, source of feedback, frequency of feedback, instructions for improvement, direction of change required, baseline performance, profession of recipient, and risk of bias within the trial itself. We also conducted exploratory analyses to assess the role of context and the targeted clinical behaviour. Quantitative (metaregression), visual, and qualitative analyses were undertaken to examine variation in effect size related to these factors Types of studies Randomised controlled trials (RCTs) Types of participants Healthcare professionals responsible for patient care. Healthcare professionals in postgraduate training were included, but studies involving only undergraduate students were not Types of interventions Audit and feedback, defined as 'any summary of clinical performance of health care over a specified period of time'. One may alternatively describe an audit and feedback intervention as 'clinical performance feedback'. The feedback may include recommendations for clinical action and may be delivered in a written, electronic or verbal format. Studies that focused on real-time feedback for procedural skills were excluded as were studies in which the feedback focused on performance on tests or simulated patient interactions. Studies that featured facilitated relay of communication regarding patient status or symptoms but that do not provide a summary of physician performance were also excluded. In general, even if the term 'feedback' was used in the manuscript, the study was excluded if the intervention would be best classified as 'facilitated relay' of patient specific clinical information or a 'reminder' 29

45 (especially when the intervention was at the point of care), or any other unique category in the EPOC classification of quality improvement interventions other than 'audit and feedback'. For this update, we only included interventions where we assessed audit and feedback to be a core or essential element. To this end, we categorized studies by the extent to which audit and feedback was the core component of the intervention into three groups: (i) audit and feedback alone; (ii) audit and feedback as a core, essential component of a multifaceted intervention; or (iii) audit and feedback as a component of a multifaceted intervention but not considered core and essential. In multifaceted interventions (which we defined as studies that utilized two or more interventions), we made the distinction between 'core' and 'not core' by considering whether the other components were likely to be used in the absence of audit and feedback, or whether the audit and feedback seemed to provide the foundation for the rest of the intervention. In cases where the audit and feedback was merely added to a multifaceted intervention that could easily be offered in its absence, the study would be classified as 'not core'. For comparisons C, D, E, and F, we used the EPOC classification scheme to identify the nature of the multifaceted interventions. 143 For example, the distinction between changing the nature of the audit and feedback intervention itself (comparison D) and adding an additional intervention along with the feedback (comparison E) is sometimes difficult, but all interventions that seemed to meet the criteria of the EPOC classification scheme were categorised accordingly. To illustrate, when a suggestion for improvement accompanies the feedback report, it may alternatively be viewed as a co-intervention (clinician education) or as an intrinsic feature of the feedback design (action plan). As with all other abstracted descriptive variables, this process was completed independently by two abstractors and discrepancies resolved through discussion, including other authors as needed Types of outcome measures We focused on objectively measured provider performance in a health care setting or patient health outcomes. We abstracted outcomes from the longest available follow up interval in the original publication, but we did not abstract data from separate articles or companion reports 30

46 wherein longer term follow up was reassessed. Studies that provided data only on cost were excluded as were studies that measured knowledge or performance in a test situation only Search methods for identification of studies The current search strategies differ from the strategies used in previous versions of this review. For this version we developed the MEDLINE search strategy based on common MESH and free text terms found in the MEDLINE records of 144 relevant studies (all MEDLINE indexed and included studies from the previous review versions and additional studies that we had identified as being, in addition to studies known to be eligible for inclusion, but not yet included, a total of 144 records. We found that 128 of the 144 records (89%) were picked up by the MEDLINE strategy. We then translated this strategy into the other databases using the appropriate controlled vocabulary as applicable. Full search strategies for all databases are included in Appendix Electronic searches We searched the following databases: Cochrane Central Register of Controlled Trials (CENTRAL) 2010, Issue 4, part of The Cochrane Library. including the Cochrane Effective Practice and Organisation of Care (EPOC) Group Specialised Register (searched 10 December 2010) MEDLINE, Ovid (1950 to November Week ) (searched 09 December 2010) EMBASE, Ovid (1980 to 2010 Week 48) (searched 09 December 2010) CINAHL, Ebsco (1981 to present) (searched 10 December 2010) Science Citation Index and Social Sciences Citation Index, ISI Web of Science (1975 to present) (searched 15 September 2011) Full search strategies for all databases are included in Appendix 1. 31

47 Searching other resources For this version of the review we reassessed whether each study from the previous review met the inclusion criteria. We searched the Science Citation Index and the Social Sciences Citation Index for studies citing all included studies in current review, in addition to selected studies from the Additional Reference list: 36, 39, 41, 47, 63, Reference lists of included articles were also reviewed and potentially relevant studies from the citation search and the reference lists were included as Studies Awaiting Classification. These will be assessed in a future update Data collection and analysis The following methods will be used in updating this review: Selection of studies Two reviewers (NI, GJ, SFl, or JY) independently screened the titles and abstracts and applied inclusion criteria; complete manuscripts were sought in the case of uncertainty and differences of opinion resolved through consensus. Conference abstracts were included if they provided sufficient data, a full report could be found or missing data could be obtained from the investigators. We categorized the extent to which audit and feedback was the 'core' component of the intervention as follows: Audit and feedback alone (included) Audit and feedback as a core, essential component, combined with other interventions categorized according to EPOC classification scheme (included) Audit and feedback as a component of a multifaceted intervention but not considered core and essential. (excluded) 32

48 Multifaceted interventions were defined as including two or more interventions. Where audit and feedback was not considered to be a core, essential component of the intervention, the study was excluded. In other words, this review included multifaceted interventions when the other components were judged to be unlikely to be used in the absence of audit and feedback, or are built around the audit and feedback, which provided the foundation for the rest of the intervention (rather than the audit and feedback being added to a multifaceted intervention that could easily be offered in its absence). This was assessed independently by two authors (NI, GJ, SFl, or JY); of all abstracts screened, only eight disagreements regarding inclusion were due to differences in the assessment of whether or not the article was 'core' audit and feedback. All disagreements were resolved by consensus Data extraction and management Data from included studies were abstracted independently by two review authors (NI, GJ, SF, or SFr). A revised version of the EPOC data collection checklist was used to collect information on study design, type of interventions compared, type of targeted behaviour, participants, setting, methods, outcomes, and results. Discrepancies between authors were resolved through discussion. Studies included in the previous review were reassessed due to changes in the data abstraction form and methods for this updated review. For articles included in the previous review, the new variables analyzed in this update (instruction for improvement and direction of change required) were abstracted by one author (NI). In all other cases, the variables have been double-abstracted. For numerical results, abstraction was performed by one author (NI) and double-checked by another author (GJ, SF, or SFr). 33

49 Assessment of risk of bias in included studies Two reviewers (GJ, NI, SFl, or SFr) independently assessed the risk of bias of each study and extracted data for newly identified studies using a revised data collection form; discrepancies were resolved by consensus with a third author as needed. The risk of bias for each main outcome in all studies included in the review were assessed according to the revised EPOC criteria. The degree of confidence in the estimate of effect across studies were assessed using GRADEpro and the GRADE approach An overall assessment of the risk of bias (high, moderate or low risk of bias) was assigned to each of the included studies using the approach suggested in the Chapter 8 of the Cochrane Handbook for Systematic Reviews of Interventions. 143 Studies with low risk of bias for all key domains or where it seems unlikely for bias to seriously alter the results were considered to have a low risk of bias. Studies where risk of bias in at least one domain was unclear or judged to have some bias that could plausibly raise doubts about the conclusions were considered to have an unclear risk of bias. Studies with a high risk of bias in at least one domain or judged to have serious bias that decreased the certainty of the conclusions were considered to have a high risk of bias. For the studies included in the previous review, one review author (NI) updated the risk of bias assessment using this approach. Any discrepancies between the conclusions regarding risk of bias using the new and the previous approach were discussed with other review authors and resolved through consensus Measures of treatment effect All outcomes were expressed as compliance with desired practice. Professional and patient outcomes were analyzed separately. For trials reporting summary and individual measures of performance, the summary measures were used. When several outcomes were reported in a trial we only extracted results for the variable(s) explicitly described as the primary outcome(s). When the primary outcome was not specified we took the variable(s) described in the sample size calculation as the primary outcome. When the primary outcome was still unclear 34

50 or when the manuscript described several primary outcomes, we calculated the median value across multiple outcomes. Since important baseline differences between intervention and control groups are frequently found in cluster-randomized trials, our primary analyses was based on estimates of effect that were adjusted for baseline differences. Therefore, only studies providing data on baseline performance were included in the statistical analysis. Baseline compliance, defined as compliance with desired practice (or with the targeted behaviours) prior to the intervention, was treated as a continuous variable ranging from zero to 100%, based on the median value of preintervention level of compliance in the audit and feedback group and control group. For dichotomous outcomes we calculated the adjusted risk difference (RD) as the difference in adherence after the intervention minus the difference before the intervention. A positive RD indicates that performance improved more in the audit and feedback group than in the control group (eg. an adjusted RD of 0.09 indicates an absolute improvement in compliance with targeted behaviours of 9%). For continuous outcomes we calculated adjusted change relative to the control group as the post-intervention difference in means minus the baseline difference in means divided by the baseline control group mean. As with the adjusted RD, a positive change indicates that performance improved more in the audit and feedback group than in the control group. This is a relative effect rather than an absolute effect; the effect size reflects the baseline performance as well as the change in performance and it is not bound between -100 and +100% Unit of analysis issues Cluster randomised trials Due to the nature of the intervention, we expected that most of the trials would be randomised by cluster. Under such circumstances it is necessary to adjust results from primary trials for clustering before they are included in a meta-analyses in order to avoid underestimating the standard error (SE) of the estimate of effect. As in the previous versions of this review, we have not abstracted the observed SEs, p-values, or confidence intervals for our statistical analysis, instead performing meta-regression using the number of health professionals as the basis for weighting. 35

51 Studies with more than two arms If more than one comparison from a study with more than two arms were eligible for the same comparison, we adjusted the number of health care professionals to avoid double counting. The adjustment was done by dividing the number of health care professionals in the shared arm approximately evenly among the comparisons Dealing with missing data Only studies reporting baseline data for primary outcomes were included in the statistical analysis because the previous review identified baseline performance as an important predictor of feedback effectiveness. Missing data regarding the characteristics of the studies or of the audit and feedback intervention were not imputed Assessment of heterogeneity We explored heterogeneity visually by preparing tables, bubble plots and box plots (displaying medians, inter-quartile ranges, and ranges) to explore the size of the observed effects in relationship to each of these variables. The size of the bubble for each comparison corresponds to the number of healthcare professionals who participated. We also plotted the lines from the weighted regression to aid the visual analysis of the bubble plots Data synthesis Across studies, the median effect size was weighted by the number of health professionals involved in the trial reported to ensure that very small trials did not contribute the same to the overall estimate as larger trials. If the number of health professionals was not reported, the number of practices/hospitals/communities was used instead. Thus, the summary statistics in the meta-analyses reported as weighted median adjusted RD or weighted median 36

52 adjusted change relative to baseline control are weighted by number of health professionals, while the results reported from individual studies are not. The primary analyses excluded studies at high risk of bias Subgroup analysis and investigation of heterogeneity Visual analyses were supplemented with meta-regression to examine how the size of the effect (adjusted RD) was related to the potential explanatory variables (listed below), weighted according to the number of health care professionals. We accounted for baseline differences in compliance by using adjusted estimates of effect to avoid the effect of potentially important baseline differences in compliance between groups. We conducted a multivariable linear regression using main effects only; baseline compliance was treated as a continuous explanatory variable and the others as categorical. For this analysis, we excluded studies with a high risk of bias. The analyses were conducted using the GLIMMIX procedure in SAS (Version 9.2. SAS Institute Inc., Cary, NC, USA), taking the dependency between comparisons from the same trial into account. P-values were based on the classical sandwich estimator. Each comparison was characterized relative to the other variables in the tables, looking at one potential explanatory variable at a time. We assessed the following potential sources of heterogeneity to explain variation in the results of the included studies: Format (verbal; written; both; unclear) Source (supervisor or senior colleague; professional standards review organization or representative of employer/purchaser; investigators; unclear) Frequency (weekly; monthly; less than monthly; one-time) Instruction for improvement (explicit measurable target or specific goal but no action plan; action plan with suggestions or advice given to help participants improve but no goal/target; both; neither) Direction of change required (increase current behaviour; decrease current behaviour; mix or unclear) Recipient (physician; other health professional) Risk of bias (high; unclear; low) 37

53 These variables were chosen from a lengthy list of plausible effect modifiers developed by the research team (see Table 1), after considering whether there was an a priori directional hypothesis and the likelihood that the variable would be reported in the literature. We hypothesized that audit and feedback with the following characteristics would be most effective: provided in both verbal and written format, from a supervisor or senior colleague, delivered more frequently than less, featuring both specific goals and action plans, aiming to increase rather than decrease behaviours, and received by non-physician providers. We also hypothesized that studies with low risk of bias would be associated with smaller effect sizes. 38

54 Table 1: Selected variables considered for inclusion in meta-regression analysis Variable Previous version Comments Decision for new version Intensity of AF In analysis Previous approach unhelpful Remove Complexity of behavior In analysis Not predictive Remove Seriousness of outcome In analysis Not predictive Remove Baseline compliance In analysis Predictive Keep Risk of bias In analysis Update based on revised Cochrane Handbook Keep requirements Peer comparison In analysis Not predictive Remove Close to time of decision New Mugford 1991 Not added making Quality of data, Motivation of recipients Judged to be difficult to abstract New van der Veer 2010 Based on perception of recipients, thus difficult to abstract Not added Organizational support/culture New van der Veer 2010, Hysong 2006 Not added Judged to be difficult to abstract Participative intervention New van der Veer 2010, Locke and Latham 2002 Not added Judged to be difficult to abstract Profession of recipient New Physicians behaviour is likely harder to change Add Direction of change New Carlsen Qualitative evidence that Add decreasing is harder. Correct solution information, goal-setting and action- plans New Locke and Latham 2002, Hysong 2009, Sniehotta 2009, Gardner 2010 Add Theory suggests these should help Tailoring of intervention after Descriptive Grimshaw Not added assessment of barriers Not feedback-specific. Clinical topic Descriptive No clear hypothesis to test Not added Setting Descriptive Axt-Adam Likely important, but no clear Not added hypothesis to test Frequency Part of Hysong 2006 found this to be associated with Keep intensity high performing groups Format (written or verbal) Part of Hysong 2009 Keep intensity Very important in recent reanalysis Source Part of Hysong 2006 and other qualitative work suggest Keep intensity that trust matters Recipient Part of Judged to be less important than other aspects Remove intensity related to intensity Setting (inpatient versus New Add outpatient) Inpatient feedback may be more effective given more resources and often higher acuity of target/problem In addition, we conducted two exploratory analyses to examine the importance of context and the relative complexity of the targeted behaviour on the likelihood that feedback would improve professional practice. We compared the effectiveness of feedback in outpatient (primary care or outpatient clinics) and hospital (inpatient, emergency room or hospital) settings. In addition, we considered common targets of feedback interventions, including: appropriate prescribing, test ordering (laboratory or radiology), and diabetes or cardiovascular disease management (two chronic clinical conditions with similar management and targets). We did not have any a priori 39

55 hypotheses for these analyses. However, the second analysis reflects two hypotheses that we tested in the previous update of this review: that the effectiveness of feedback would be greater for behaviours that are important but not complex (ie. prescribing) compared to more complex behaviours (ie. disease management) or compared to behaviours that clinicians might perceive as less important (ie. test ordering). For these analyses we compared the weighted median effect sizes and conducted a univariate meta-regression for studies reporting dichotomous outcomes. If we found potentially important and statistically significant differences, we included these explanatory factors in the full model for the meta-regression described above to assess the robustness of these exploratory findings Sensitivity analysis We performed sensitivity analyses by including studies with a high risk of bias. We also examined whether differences in the level of the unit of analysis (groups of professionals versus individual professionals versus patients) was a source of heterogeneity, since analyses conducted at different levels can result in different effect estimates. 40

56 4.4 Results 140 studies were included and analysed qualitatively for this review. In the main analyses, a total of 108 comparisons from 70 studies were included that compared any intervention in which audit and feedback was a core, essential component to usual care and evaluated effects on professional practice. After excluding studies at high risk of bias, there were 82 comparisons from 49 studies featuring dichotomous outcomes, and the weighted median adjusted RD was a 4.3% (interquartile range (IQR) 0.5 to 16%) absolute increase in health care professionals' compliance with desired practice. Across 26 comparisons from 21 studies with continuous outcomes, the weighted median adjusted percent change relative to control was 1.3% (IQR = 1.3 to 28.9%). For patient outcomes the weighted median RD was -0.4% (IQR -1.3 to 1.6%) for 12 comparisons from six studies reporting dichotomous outcomes and the weighted median percentage change was 17% (IQR 1.5 to 17%) for eight comparisons from five studies reporting continuous outcomes. Multivariate meta-regression indicated that feedback may be more effective when baseline performance is low, the source is a supervisor or colleague, it is provided more than once, it is delivered both verbally and written, and when it includes both explicit targets and an action plan. In addition the effect size varied based on the clinical behaviour targeted by the intervention Description of studies For this update we screened 3623 new studies and reviewed the full text of 282. The total number of studies included is 140. Of note, 53 new studies were added to this review since the previous update and 31 were removed from the previous version of the review as they no longer met inclusion criteria. See study flow diagram for details ( Figure 1). 41

57 5505 records identified through database searching 3623 records after duplicates CENTRAL (including EPOC register): 2168 MEDLINE: 1853 EMBASE: 895 CINAHL: titles and abstracts screened 282 full-text articles reviewed, including 118 from previous version of Cochrane review of audit and 140 studies included in review 111 studies directly tested audit and feedback versus usual care 142 of full-text articles excluded: 12 had no results or only reported costs (2 from previous review) 40 were not RCT or only had one group per arm (8 from previous review) 30 did not fit our definition of audit feedback (5 from previous review) 58 had audit and feedback not 82 comparisons from 45 dichotomous outcomes for professional practice included in Figure 1: Study flow diagram 42

58 All abstracted information is available upon request; the general characteristics of the included studies are described in 43

59 Table 2. The unit of allocation was a single health care provider in 51 studies (5056 total providers, median 56), groups of clusters of health care professionals (e.g. clinics, wards, hospitals, communities) in 88 studies (5267 total clusters, median 32), and in one study (24 providers, 1140 patients) the unit of allocation was not clear. 150 Twenty studies had four arms, 22 studies had three and the remaining 98 had two arms. 44

60 Table 2: Description of included trials (N=140) Study Characteristic Number Percent Intervention Characteristic Number Percent Publication Year Audit and Feedback alone Multifaceted intervention with AF as core feature with Case management or team change with Clinician education (not outreach) before with Educational outreach Country with Clinician reminders, including decision support USA with Patient intervention (eg. Self mgmt/reminders) UK or Ireland with Continuous quality improvement Canada with Financial incentives Australia or New Zealand Format Other verbal Unit of Allocation written Provider both Many Providers/Groups unclear Unclear Source Unit of Analysis supervisor/colleague Patient employer Provider investigators/unclear Many Providers/Groups Frequency Unclear weekly Risk of Bias monthly Low repeated less than monthly Unclear once only High Instructions for Improvement Number of Arms in Trial Goal-setting Two Action planning Three Both Four Neither Clinical Setting Direction of Change Required Outpatient Increase current behaviour Inpatient Decrease current behaviour Other/unclear Mix or unclear Medical Specialty (could include more than one) Targeted Health Professional (could include more than one) GP/Family physician Physician Internists Nurses Other Pharmacists Other Clinical Topic / Targeted Behaviour (could be more than one) Diabetes/Cardiovascular disease management Size of trial Median IQR Laboratory testing/radiology Providers (when providers Prescribing allocated) Groups (when many Other providers allocated) 45

61 4.4.2 Characteristics of setting and professionals Eighty trials were based in North America (69 in USA, 11 in Canada), 21 in the UK or Ireland, 10 in Australia or New Zealand, and 29 elsewhere. Only four studies were from lowand middle-income countries (two in Sudan, one in Thailand, and one in Laos). In 121 trials the targeted health professionals for the intervention were physicians. Five studies explicitly targeted pharmacists and 16 studies explicitly targeted nurses. The most common clinical specialty area was general or family practice, targeted in 84 trials. Ninety-four trials were in an outpatient setting, 36 were in inpatient settings, and in 10 studies the clinical setting was unclear Targeted behaviours There were 39 trials specifically aiming to improve appropriate prescribing and 31 specifically targeting laboratory or radiology test utilization. Thirty-four trials focused on management of patients with either cardiovascular disease or diabetes (two exemplar chronic conditions with common management strategies). The remaining trials varied widely across conditions and targeted behaviours Characteristics of interventions There were 49 studies in which audit and feedback was the only intervention, while audit and feedback was considered the core, essential component of a multifaceted intervention in 91 studies. The format of the feedback was clearly reported in 129 studies: 13 had verbal feedback, 84 had written feedback, and 32 had both. In the majority of studies (112), the source of the feedback was unclear or it was provided by the researchers who had no other relationship to the recipients. In 13 studies feedback was provided from a supervisor or senior colleague, and in 15 46

62 from a 'professional standards review organisation or representative of the employer or purchaser. The frequency of the feedback was weekly in 11 trials, monthly in 19 trials, repeated but less than monthly in 36, and once only in 68 trials. In 11 studies the feedback provided recipients with explicit, measurable goals and 41 studies included action plans or correct solution information with the feedback. The feedback had both these features in four studies and neither in 84 studies. In 57 studies, the feedback required recipients to increase current behaviours; in 29 they had to decrease current behaviours, and in 55 studies the feedback was judged to require a complex or uncertain change in behaviour Outcome measures There was large variation in outcome measures, and studies often reported multiple primary outcomes related to compliance with different aspects of a guideline. Most trials measured professional practice, such as prescribing or use of laboratory tests. Some trials reported both practice and patient outcomes such as smoking status or blood pressure. There was a mixture of dichotomous outcomes (for example the proportion compliance with guidelines or the proportion of patients with appropriate management) and continuous outcome measures (for example costs, number of laboratory tests, or number of prescriptions) across and within studies. Baseline performance was not reported in 10 studies (Balas 1998; Berman 1998; Curtis 2007; Everett 1983; Linn BS 1980; Lobach 1996; Robling 2002;Sandbaek 1999; Tierney 1986; Wones 1987) Risk of bias in included studies See Figure 2. Of the 140 trials, 45 (32%) had a low risk of bias, 70 (50%) had an unclear risk of bias, and 25 (18%) had a high risk of bias (Everett 1983;Gehlbach 1984; Sommers 1984; Boekeloo 1990; Buffington 1991; Gama 1992; Brown 1994; Winkens 1995; Baker 1997; Berman 1998; Sandbaek 1999; Kim 1999; Zwar 1999; Rust 1999; Batty 2001; Robling 47

63 2002; Claes 2005; Sondergaard 2006; Foster 2007; Wadland 2007; Charrier 2008; Curran 2008; Millard 2008; Schneider 2008; Canovas 2009). The most common sources of a high risk of bias related to lack of outcome blinding (e.g. when outcomes were reported by participating healthcare professionals) (12 trials), completeness of follow up (six trials) and similarity at baseline (10 trials). Clarity of reporting regarding the risk of bias variables was frequently inadequate. For example, the nature of the randomization sequence was unclear in 81 trials, outcome blinding was unclear in 55 trials, similarity at baseline was unclear in 43 trials, and risk of contamination was unclear in 42 trials. Randomisation was clearly concealed (or there was cluster randomisation) in 117 trials. There was adequate follow-up in 112 trials. Figure 2: Risk of bias graph: review authors judgements about each risk of bias item presented as percentages across all included studies Effects of interventions See Summary of findings Table 3. 48

64 Table 3: Summary of findings: Audit and feedback for health professionals Patient or population: Health care professionals Settings: Primary and secondary care Intervention: Audit and feedback with or without other interventions 1 Comparison: Usual care Outcomes Compliance with desired practice (dichotomous outcomes) Compliance with desired practice (continuous outcomes) Patient outcomes (dichotomous) Patient outcomes (continuous) Absolute improvement 2 Median 4.3% absolute increase in desired practice (IQR 0.5% to 16.0%) Median 1.3% improvement in desired practice (IQR 1.3% to 28.9%) Median percent change -0.4% (IQR -1.3% to 1.6%) Median percent change 17% Number of health professionals (studies) 82 comparisons from 49 studies clusters/groups of health providers (from 32 cluster trials) and 2053 health professionals (from 17 trials allocating individual providers). 26 comparisons from 21 studies. 661 clusters/groups of health providers (from 13 cluster trials) and 605 health professionals (from 8 trials allocating individual providers). 12 comparisons from 6 studies. 8 comparisons from 5 studies. Quality of the evidence (GRADE) moderate 4 moderate 4 low 5 low 5 Comments The effect appears to be larger when baseline performance is low, the source is a supervisor or senior colleague, delivered both verbally and written, provided more than once, aims to decrease current behaviours, targets prescribing, and includes both explicit targets and an action plan. (IQR 1.5 to 17%) GRADE Working Group grades of evidence: High quality: We are very confident that the true effect lies close to that of the estimate of the effect. Moderate quality: We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different. Low quality: Our confidence in the effect estimate is limited: The true effect may be substantially different from the estimate of the effect. Very low quality: We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect. 1 The effect of audit and feedback alone on professional practice was similar to audit and feedback as the core, essential feature in multifacetedinterventions. 2 The post-intervention risk differences are adjusted for pre-intervention differences between the comparison groups to account for baseline differences. The effect was weighted across studies by the number of health professionals involved in the study to ensure that small trials did not contribute as much to the estimate of effect as large trials. 3 Many studies had more than two arms and therefore contributed multiple comparisons of audit and feedback versus usual care. 4 We have downgraded the evidence from high to moderate because of inconsistency in the results that could not be fully explained. 5 We have downgraded the evidence from moderate to low because of the limited number of trials targeting patient outcomes as a primary outcome. 49

65 Comparison A. Any intervention in which audit and feedback is the single intervention or is the core, essential feature of a multifaceted intervention, compared to usual care A total of 171 comparisons from 109 studies were included in this comparison. Of these, 17 comparisons from 10 studies had no baseline data, and 21 comparisons from 14 studies were at high risk of bias. Twenty-five comparisons from 15 studies included patient outcomes as a primary outcome. Thus, 108 comparisons from 70 studies were included in the primary analyses assessing the effects of audit and feedback on professional practice Dichotomous measures of compliance with desired practice There were 124 total comparisons, of which 11 comparisons were removed due to lack of adequate baseline data. Of the 113 remaining comparisons, 15 had patient-oriented outcomes, leaving 98 comparisons from 62 studies. In the primary meta-analysis, a further 16 comparisons from 12 studies at high risk of bias were excluded, leaving 82 comparisons from 49 studies with dichotomous outcomes. These studies included 2310 clusters / groups of health providers (from 32 cluster trials), and 2053 health professionals (from 17 trials allocating individual providers). For these studies, the weighted median adjusted RD was a 4.3% increase in compliance with desired practice (interquartile range (IQR) 0.5 to 16%). The weighted median RD when studies with high risk of bias were included in the sensitivity analysis was also 4.3% (IQR 0.6 to 16%). The range in adjusted RDs for compliance with desired practice was wide: a 9% absolute decrease to a 70% increase in compliance. Of the 98 total comparisons, 27 had an adjusted RD of at least 10% and in 20 comparisons the adjusted RD was between 5% and 10%. For 50 comparisons the adjusted RD was small (ranging from -5% to 5%). Only one study reported a negative effect greater than 5%; an adjusted RD of -9% for appropriate prescribing of benzodiazepines (Batty 2001). This study had a high risk of bias due to imbalance at baseline. Three other studies had unusually large effect sizes. Foster 2007 reported a 45% increase in the utilization of peak flow in asthma patients. This study had a high risk of bias due to incomplete follow up. Gehlbach 1984 reported a 45% improvement in the use of generic prescriptions and this study also had a high risk of bias. Finally, Mayer 1998 showed a 70% increase in the 50

66 provision of skin cancer preventive advice among pharmacists, from a baseline performance of 0%. As in the previous version of this review, this study was excluded from the primary analysis because it differed from the others, as it aimed to initiate an entirely new clinical behaviour in the intervention group, rather than help providers to improve their performance in an area of known professional responsibility. There were 11 comparisons from seven studies with dichotomous outcomes that did not report baseline data (Balas 1998; Berman 1998; Curtis 2007;Lobach 1996; Robling 2002; Sandbaek 1999; Tierney 1986). The range of (unadjusted) RD seen in these studies was - 2.3% to 29.2%. The median unadjusted RD for these studies was 4% (IQR 1 to 7%) Continuous measures of compliance with desired practice There were 47 total comparisons, of which six were removed due to lack of adequate baseline data. Of the 41 remaining comparisons with continuous primary outcomes, 10 had patient-oriented outcomes, leaving 31 comparisons from 25 studies. The primary meta-analysis excluded a further five comparisons from four studies at high risk of bias leaving 26 comparisons from 21 studies with continuous outcomes. These studies included 661 groups of healthcare providers (from 13 cluster trials) and 605 healthcare professionals (from eight trials allocating individual providers). For these studies, the weighted median adjusted change relative to baseline control was a 1.3% increase in compliance with desired practice (IQR 1.3 to 23.2%). When studies at high risk of bias were included, the weighted median effect was 2.9% (IQR 1.3 to 26.1%). The adjusted percent change relative to control varied widely, from a 50% decrease in desired practice to a 139% increase in desired practice. Of the 31 total comparisons with continuous outcomes, 21 had an adjusted percent change relative to control of at least 10%. For eight comparisons the effect size was relatively small (-5% to 5%). Two comparisons had larger negative effects: one (Holm 1990) showed a 10% negative effect (relative increase in benzodiazepine/sedative medications); the other comparison (Cohen 1982), showed a 50% negative effect (an increase in laboratory test utilization), but actually reported a positive effect during the intervention period, which reversed after the intervention stopped. The trial (Wadland 51

67 2007) that reported a 139% positive effect (increase in smoking cessation referrals) had a high risk of bias. There were six comparisons from three studies with continuous outcomes that did not report baseline data (Everett 1983; Linn BS 1980; Wones 1987). The median effect seen in these studies was a 54% increase in desired practice (IQR 15.1 to 54%) Patient outcomes Fifteen studies (Mitchell 2005; Buffington 1991; Curran 2008; Fairbrother 1999; Gullion 1988; Hemminiki 1992; Hendryx 1998; Linn BS 1980; Lomas 1991;O'Connor 2009; Phillips 2005; Rantz 2001; Rust 1999; Svetkey 2009; Thomas 2007) reported patienttype outcomes as a primary outcome. One study (Linn BS 1980) did not have any baseline data, and two studies (Buffington 1991; Curran 2008) had high risk of bias, leaving 12 comparisons with dichotomous outcomes and eight comparisons with continuous outcomes for analysis. There was minimal discernable effect observed for patient outcomes with dichotomous outcomes, while a positive effect was noted in studies with continuous outcomes. Specifically, for dichotomous outcomes, the weighted median adjusted RD was a 0.4% decrease in desired outcomes (IQR -1.3 to 1.6%) and for continuous outcomes, the weighted median adjusted change relative to baseline control was a 17% improvement (IQR 1.5 to 17%) Investigation of heterogeneity The multivariable meta-regression analysis explored the role of five characteristics of the intervention (format, source, frequency, instructions for improvement, direction of change required), two characteristics of the recipients (baseline performance, profession), and one characteristic of the trial design (risk of bias) on heterogeneity in effect size. This was performed 52

68 on trials that had dichotomous outcomes and that compared audit and feedback as the only intervention or as the core, essential feature of a multi-faceted intervention versus usual care. Studies at high risk of bias were excluded, leaving 80 comparisons in this analysis with either unclear or low risk of bias. All five characteristics of the intervention were identified as significant in the model, as described in 53

69 Table 4, indicating that the format (p=0.02), source (p<0.001), frequency (p<0.001), instructions for improvement (p<0.001), and the direction of change required (p=0.007) each help explain variation in effects. Within these variables, relatively large differences in effect size were seen when comparing certain characteristics: presented in both verbal and written format versus only verbal (expected difference in adjusted RD = 8%); delivered by a supervisor or senior colleague versus the investigators (expected difference in adjusted RD = 11%); frequency of monthly versus once only (expected difference in adjusted RD = 7%); containing both an explicit, measurable target and a specific action plan versus neither (expected difference in adjusted RD = 5%); and requiring a decrease versus an increase of current behaviour to achieve a higher score (expected difference in adjusted RD = 6%). Risk of bias (p=0.679) and profession (physician vs non-physician) (p=0.561) were not associated with variation in effect size. Lower baseline performance was associated with greater effectiveness for the intervention (p = 0.007). To illustrate, the model predicts that recipients who achieved 25% of desired practice at baseline would have an expected adjusted RD of 9%, while those who achieved 75% of desired practice at baseline would have an expected adjusted RD of only 5%. See Figure 3 for a bubble plot of effect size by baseline performance. 54

70 55 Table 4: Assessment of Heterogeneity: results from meta-regression Characteristic*of*the*Feedback*or*Recipient*or*Trial* Effect* Format*of*feedback* p=0.020* Verbal' 3.38' Written' 9.50' Both'verbal'and'written' 11.23' Not'clear' 5.27' Source*of*feedback* p<0.001* A'supervisor'or'colleague' 16.50' A''professionals'standards'review'organization''or'employer' 2.37' The'investigators' 5.04' Not'clear' 5.48' Frequency*of*feedback* p<0.001* Frequent'(up'to'weekly)' 1.44' Moderate'(up'to'monthly)' 9.83' Infrequent'(less'than'monthly)' 4.78' Once'only' 2.56' Unclear;' 18.12' Instructions*for*improvement* p<0.001* Explicit,'measurable'target/goal,'but'no'action'plan' 2.52' Action'plan' 9.57' Both' 11.09' Neither' 6.20' Direction*of*change*required* p<0.001* Increase'current'behaviour' 4.34' Decrease'current'behaviour' 10.54' Change'behaviour'or'mix'or'unclear' 7.16' Baseline*performance* p=0.007* at'25%' 9.11' at'50%' 7.07' at'75%' 5.03' Profession*of*recipient* p=0.561* Physician' 7.90' Non\physician' 6.80' Risk*of*bias* p=0.679* Low'risk'of'bias' 7.68' Unclear'' 7.02' High'risk'of'bias' n/a'

71 Figure 3: Bubble plot, adjusted risk difference by baseline performance Examination of box plots for each of the explanatory variables primary analysis supported the statistical conclusions (see Figure 4, Figure 5, Figure 6, Figure 7, and Figure 8). For exploratory purposes, we also examined box plots for explanatory variables considering trials with continuous outcomes from Comparison A. This did not result in any qualitative differences in the assessment of heterogeneity. Finally, we examined the box plots for trials with dichotomous and continuous outcomes, respectively, for Comparison B (audit and feedback alone versus usual care) and then for Comparison C (audit and feedback as the core, essential feature of a multifaceted intervention versus usual care), separately. These analyses revealed 56

72 consistency in the direction of effects for the explanatory variables, supporting the initial conclusions. Figure 4: Box plot, comparing adjusted risk difference by format of feedback 57

73 Figure 5: Box plot, comparing adjusted risk difference by source of feedback 58

74 Figure 6: Box plot, comparing adjusted risk difference by frequency of feedback 59

75 Figure 7: Box plot, comparing adjusted risk difference by presence/extent of instructions for improvement 60

76 Figure 8: Box plot, comparing adjusted risk difference by direction of change required by the feedback Although the multifaceted studies appeared to have a larger median effect size, when comparing the mean estimate of effect for audit and feedback alone versus audit and feedback in a multifaceted intervention using a univariate analysis we found that the differences were not statistically significant for dichotomous outcomes (estimated absolute difference in adjusted RD = 3.3%; p=0.27). The similarity in estimated adjusted RD is illustrated in Figure 9. However, there was a significant difference when examining the studies with continuous outcomes (estimated absolute difference in adjusted percent change = 24%; p<0.0001). 61

77 Figure 9: Box plot, comparing adjusted risk difference for Comparison B (audit and feedback alone versus usual care) and Comparison C (multifaceted intervention featuring audit and feedback versus usual care) The sensitivity analysis adding level of analysis (patient versus provider versus cluster) to the model did not lead to any significant changes in the results. When studies with a high risk of bias were included in the model, the findings remained consistent, with two exceptions: format (written versus verbal versus both) no longer had a significant effect, and profession of recipient did, with non-physicians performing better than physicians. It was observed that the model-based estimated effect sizes increased when the high risk of bias studies were included, suggesting caution is needed when interpreting these results. Given that some of the strata within the model were quite small (e.g. only six comparisons from four studies assessed 'both' goals and action plans), such instability is not surprising. 62

78 Exploratory analyses Exploratory analyses were conducted to examine the importance of context and the complexity of the targeted behaviour on the likelihood that audit and feedback will improve professional practice. Although clinical setting (outpatient versus inpatient versus mixed, other or unclear) was marginally statistically significant in the multivariate meta-regression model (p=0.037), the estimated effects were similar across inpatient and outpatient settings (inpatient = 7.7; outpatient = 7.1; mixed, other or unclear = 3.0). When 'targeted behaviour' (prescribing versus laboratory or radiology utilization versus diabetes or cardiovascular disease management versus other) was added to the meta-regression model, it was statistically significant (p<0.0001), with estimated RD for prescribing (11.1) larger than diabetes or cardiovascular disease (5.9), laboratory or radiology testing (4.2), or other (4.7). In that model, the 'nature of change required' (increase current behaviour versus decrease versus other) was no longer statistically significant (p=0.525) and the estimates for some other variables changed (see Table 5: Exploratory analysis, meta regression with targeted behaviour). For prescribing, the median adjusted RD was 13.1% (IQR 3 to 17%) based on 26 comparisons with dichotomous outcomes at unclear or low risk of bias. For laboratory or radiology test utilization, the median adjusted RD was -0.1% (IQR -0.1 to 6.5%) based on three comparisons, and for trials focusing on the management of diabetes or cardiovascular disease, the median adjusted RD was 0.5% (IQR -0.5 to 3.4%) based on 14 comparisons. 63

79 Table 5: Exploratory analysis, meta regression with targeted behaviour Characteristic*of*the*Feedback*or*Recipient*or*Trial* Effect* * Type*of*professional*practice** * P*<*0.001* ' Diabetes/CVD'* 5.91* ' Laboratory'testing/radiology' 4.21* referrals'* ' Prescribing'* 11.11' ' Other'* 4.71* Format*of*feedback' * P*<*0.001* ' Verbal* 2.42' ' Written* 5.86' ' Both'verbal'and'written* 10.07* ' Not'clear* 7.60* Source*of*feedback** * P*<*0.001* ' ' A'supervisor'or'colleague* 13.71* ' A' professionals'standards'review' 2.44* organization 'or'employer' ' The'investigators' 4.95' ' Not'clear' 4.85' Frequency*of*feedback' ' P*=*0.002' ' Frequent'(up'to'weekly)' 3.09' ' Moderate'(up'to'monthly)' 9.58' ' Infrequent'(less'than'monthly)' 6.28' ' Once'only' 3.59' ' Unclear'' 9.89' Instructions*for*improvement*' ' P*<*0.001' ' Explicit,measurable'target/goal,'but' 2.84' no'action'plan' ' Action'plan' 9.30' ' Both' 7.18' ' Neither' 6.63' Direction*of*change*required' ' P*=*0.525' ' Increase'current'behaviour' 6.64' ' Decrease'current'behaviour' 7.13' ' Change'behaviour'or'mix'or'unclear' 5.70' Baseline*performance' ' P*=*0.002' ' at'25%' 8.72' ' at'50%' 6.75' ' at'75%' 4.77' Profession*of*recipient' ' P*=*0.059' ' Physician' 5.04' ' Non\physician' 7.94' Risk*of*bias' ' P*=*0.454' * Low'risk'of'bias' 5.88* * Unclear' 7.09* * High'risk'of'bias'(not'included'in' primary'analysis)' n/a* 64

80 Comparison B. Audit and feedback alone compared to no intervention A total of 82 comparisons from 65 studies were included in this comparison. Nine comparisons from six trials did not report baseline data and 13 comparisons from 10 trials assessed patient outcomes as a primary outcome, leaving 59 comparisons from 48 studies for the analyses. For studies with audit and feedback alone targeting professional practice with dichotomous outcomes, there were nine comparisons from seven studies excluded due to high risk of bias, leaving 32 comparisons from 26 studies for the primary analysis. These studies included 759 groups of health providers (from 12 cluster trials) and 1617 health professionals (from 14 trials allocating individual providers). The weighted median adjusted RD was 3.0% (IQR 1.8 to 7.7%). Including the studies at high risk of bias resulted in no change to the estimate of effect. For studies with audit and feedback alone targeting professional practice with continuous outcomes, there were five comparisons from four studies excluded due to high risk of bias, leaving 14 comparisons from 13 studies for the primary analysis. These studies included 348 groups of health providers (from eight cluster trials) and 494 health professionals (from five trials allocating individual providers). The weighted median adjusted RD was 1.3% (IQR 1.3 to 11.0%). Including the studies at high risk of bias studies in the sensitivity analysis also resulted in a weighted adjusted RD of 1.3% (IQR 1.3 to 20.1%). 65

81 Comparison C. Audit and feedback as the core feature of a multifaceted intervention compared to no intervention A total of 90 comparisons from 65 studies were included in this comparison. Seven comparisons from six trials did not report baseline data and 13 comparisons from nine trials assessed patient outcomes as a primary outcome, leaving 70 comparisons from 50 studies for the analyses. For studies with multifaceted interventions featuring audit and feedback targeting professional practice with dichotomous outcomes, there were seven comparisons from seven studies excluded due to high risk of bias, leaving 50 comparisons from 32 studies for the primary analysis. These studies included 1574 groups of health providers (from 26 cluster trials) and 480 health professionals (from seven trials allocating individual providers). The weighted median adjusted RD was 5.5% (IQR 0.4 to 16%). Including high risk of bias studies in the sensitivity analysis resulted in a revised weighted adjusted RD = 6.5% (IQR 0.5 to 16%). For studies with multifaceted interventions featuring audit and feedback targeting professional practice with continuous outcomes, there were 12 comparisons from 11 studies for the primary analysis. These studies included 317 groups of health providers (from 7 cluster trials) and 111 health professionals (from four trials allocating individual providers). The weighted median adjusted RD was 26.1% (IQR 12.7 to 26.1%). There were no studies in this group with high risk of bias. 66

82 Comparison D. Different ways of providing audit and feedback (head-to-head comparisons) Seventeen trials included 16 head-to-head comparisons of different ways of providing audit and feedback. For each comparison, we determined the adjusted RD or the adjusted relative percent change. This is reported below in addition to any statistical comparisons conducted by the authors of a particular study (e.g. odds ratios or p-values) to provide a standard measure of effect across all comparisons in this review. Peer comparison Søndergaard 2002 and Wones 1987 each found small differences when adding peer comparison data to the audit and feedback for asthma management (adjusted RD = 2%) or inpatient laboratory test utilization (adjusted change = 5%), respectively. Kiefe 2001 compared audit and feedback featuring a mean score of peers with feedback that featured an achievable benchmark (the mean score of the top 10% of peers). They found that the achievable benchmark group improved quality of care for diabetic patients (median adjusted RD = 3%, IQR = 2 to 4, range = 1 to 12). In particular, statistically significant increases were observed for influenza vaccination (OR 1.54, 95% CI 1.26 to 1.96), foot examination, (OR 1.33, 95% CI 1.05 to 1.69) and haemoglobin A1C measurement (OR 1.33, 95% CI 1.04 to 1.69), while cholesterol measurement (OR 1.20, 95% CI 0.95 to 1.51) and triglyceride measurement (OR 1.15, 95% CI 0.92 to 1.44) had non-statistically significant increases. In contrast, Schneider 2008 found that identifying top performers in feedback presented in a quality circle (i.e. learning collaborative) did not lead to improvements in management of asthma (adjusted RD = -5%, high risk of bias). Presentation of feedback and inclusion of additional information Mitchell 2005 found that feedback was slightly more effective for control of blood pressure if it presented information in a way that identified patients at higher risk, suggesting that action for such patients should be prioritized (adjusted RD = 2%; OR 1.72, 95% CI 1.09 to 2.70). (This is a patient outcome due to the role of patient-specific factors in achieving control of hypertension. Larger effects on professional practice outcomes might be expected.) 67

83 Two studies directly compared including a small amounts of extra information to not including that information. Buntinx 1993 added brief advice to typical feedback. They found similar effects for the quality of pap smears (adjusted RD = 1%; no statistical test reported for this comparison). Curran 2008 added Pareto and cause and effect charts to help recipients identify barriers and focus improvement efforts. They did not find a statistically significant difference in rates of methicillin-resistant S. aureus infections in hospital wards (adjusted change = 5%, high risk of bias, patient outcome; p=0.46). Two studies tested the type and amount of data used for the feedback reports. Gullion 1988 compared feedback regarding blood pressure laboratory values, and medications from chart audits to feedback regarding blood pressure and adherence to medication and lifestyle recommendations from patient surveys. They reported no differences in blood pressure control (adjusted RD = 2%, patient outcome; no p-value reported for this comparison). Herrin 2006 compared feedback based on administrative data to this plus additional, patient-specific clinical data from medical records. They also did not find a statistically significant difference in the proportion of adequate glucose control (adjusted RD = 1.9%, patient outcome; p=0.97). Source and delivery Four studies directly tested whether feedback should be delivered by mail (written) or inperson (verbally). Rubin 2001 compared written feedback delivered only to the hospital administration to the addition of verbal feedback at staff meetings. They did not find a difference in appropriateness of red blood cell transfusions (adjusted RD = -2%; no statistical test reported for this comparison). Sauaia 2000 found differences that were not statistically significant between verbal feedback in a large group setting by an expert cardiologist and written feedback for improving eight quality of care outcomes related to acute management of myocardial infarction (median adjusted RD = 7%; p-value for each outcome > 0.05). Batty 2001 compared similar interventions for in-hospital benzodiazepine prescriptions. The verbal presentation was more effective than the written feedback (adjusted RD = 24%, high risk of bias). Finally, Anderson 1994 found little or no difference when they compared feedback given to large groups as part of a CME program with and without sending individualized feedback reports to participants for prophylaxis of venous thromboembolism (adjusted RD = 0%; no p-value reported for this comparison). 68

84 Two studies directly tested the effects of who delivered the feedback. Ward 1996 compared audit and feedback delivered by a physician-peer with audit and feedback delivered by a nurse. They found that peer-physician feedback led to non-statistically significant improved management of diabetes (adjusted change = 12%; p-value reported as NS ). They also noted that the physician interviews were longer (25min versus 14min; p<0.001) and that there was a significant variation in effect size across the different physicians providing the outreach. Similarly, van den Hombergh 1999 found that mutual feedback by physician-peers (ie. each physician provides and receives feedback in turn) improved outcomes as measured by 33 indicators of practice management compared to unidirectional feedback by a non-physician (median adjusted RD = 5; no overall statistical test reported). Recipient Participation Two studies directly tested the role of recipient participation. Sommers 1984 found that participation in criteria setting prior to the feedback resulted in worse management of anaemia in hospitalised patients (adjusted RD = -21%, high risk of bias; OR = -3.36, p = 0.002). Conversely, Brady 1988 found that when resident physicians conducted a self audit at baseline, it led to improvements compared to simply receiving the data for mammographic screening rates (adjusted RD = 8%; no OR reported, p value reported as <0.05) but not to a statistically significant improvement for influenza vaccination rates (adjusted RD = 1.5; no OR reported, p = 0.17). 69

85 Comparison E. Audit and feedback combined with complementary interventions compared to audit and feedback alone Fifty-three comparisons from 43 trials were included. Below, the results of these comparisons are summarized within categories related to the 'type' of intervention that audit and feedback was combined with when comparing to audit and feedback alone. We acknowledge that some of the multifaceted interventions may fit into multiple categories, but only describe the findings from each trial once. Multi-arm studies may be described in multiple sections corresponding with the type of comparison. Due to the variation in outcome type (dichotomous, continuous, patient, provider) across the studies, we were unable to conduct quantitative metaanalyses, with the exception of trials comparing audit and feedback with educational outreach to audit and feedback alone (see below). For each comparison, we determined the adjusted RD or the adjusted percent change relative to baseline performance in the audit and feedback alone arm. This is reported below in addition to any statistical comparisons conducted by the authors of a particular study (e.g. odds ratios or p-values) to provide a standard measure of effect across all comparisons in this review. Audit and Feedback with Reminders compared to Audit and Feedback Alone Seven studies evaluated adding reminders to audit and feedback. Two of these aimed to reduce outpatient test-ordering. In a 2x2 factorial trial, Eccles 2001found that adding reminders to audit and feedback reduced x-ray utilization (adjusted change relative to baseline control = 46%; no p-value reported). In another 2x2 factorial trial Thomas 2006 found that feedback and reminders both significantly reduced blood test utilization and that the effect seemed to be additive, but not synergistic (adjusted change relative to baseline performance in the audit and feedback alone arm -2%; OR = 0.78, 95% CI for both versus OR = 0.87, 95% CI for reminders alone, no p-value reported). Two studies combined reminders with audit and feedback in an attempt to improve management of diabetes. Phillips 2005 found little or no differences in hemoglobin A1C, systolic blood pressure, and low-density lipoprotein cholesterol levels (median adjusted change relative to baseline performance in the audit and feedback alone arm = 2%; no p-value reported). Ziemer 2006 assessed clinical inertia in diabetes and found that the combination of reminders and 70

86 feedback had a greater effect on treatment intensification than feedback alone (adjusted RD = 7.25; no p-value reported). Tierney 1986 in a complex factorial trial with active controls found that reminders together with audit and feedback was more effective than feedback alone for provision of preventive services by internal medicine trainees (unadjusted RD = 8.0; no p-value reported). Baker 1997 found improvement in the management of chronic benzodiazepine prescriptions (median adjusted RD = 1.7, high risk of bias; no overall statistical test conducted). Finally, Boekeloo 1990 found a significant decline in the quality of cholesterol management in hospital when reminders were combined with feedback compared with feedback alone (median adjusted RD = -8, high risk of bias; no overall statistical test conducted). One trial, Bahrami 2004, compared audit and feedback with a computer decision support system to audit and feedback alone to improve the management of impacted molars; neither intervention produced a statistically significant improvement (adjusted RD = 6; no p-value presented for this comparison). Audit and Feedback with Educational Outreach compared to Audit and Feedback Alone We found 24 comparisons from 19 studies that compared audit and feedback alone to the combination of audit and feedback and educational outreach (also known as academic detailing). For the 15 studies with dichotomous outcomes focusing on professional practice, the weighted median adjusted RD for audit and feedback with outreach versus feedback alone was a 0.7% increase in desired practice (IQR -1.1 to 5.1%). For the four studies with continuous outcomes, the median adjusted percent change was 27% (IQR 0 to 40.5%). The PINCER trial (Avery 2010) had a median adjusted RD = 1.6 across three outcomes related to safe prescribing. However, in their multivariable model they found that educational outreach by a pharmacist reduced unsafe prescribing practices by GPs compared to feedback alone for the primary outcomes of NSAID (non-steriodal anti-inflammatory drug) use without PPI (proton-pump-inhibitor) (OR 0.58, 95% CI 0.38, 0.89), beta-blocker use in asthmatics (OR 0.73, 95% CI 0.58, 0.91), and ACE (angiotensin-converting enzyme) or diuretic use without electrolyte measurements (OR 0.51, 95% CI 0.34, 0.78). The educational outreach in Moher 2001 focused on showing primary care providers how to utilize the feedback reports to develop 71

87 and implement systematic patient recall systems. This resulted in an improvement (adjusted RD = 22%; p=0.002) in the proportion of patients with adequate assessment of cardiovascular risk factors, but no differences in actual treatment. Ward 1996 also found that outreach led to small but statistically significant improvements in diabetes care compared to postal feedback alone (adjusted change for relative to baseline performance in the audit and feedback alone arm = 35%; p<0.001). Two 2x2 factorial studies in Sudan both found small effects on inappropriate antibiotic prescribing with academic detailing compared to audit and feedback alone (Awad adjusted change relative to baseline performance in the audit and feedback alone arm = 29%, p<0.001; Eltayeb adjusted RD = 9.2, no p-value reported). Six studies comparing educational outreach plus feedback to audit and feedback alone had mixed findings. McClellan 2004 found small, but potentially clinically meaningful improvements in the management of dialysis by adding a multifaceted intervention including educational outreach to feedback (difference in mean urea reduction ratio: p=0.002), but no statistically significant improvement in the primary outcome (proportion of patients with urea reduction ratio > 65%: adjusted RD = 0.70; p=0.8). Rask 2001 found that outreach improved diabetes care for only one of six professional outcomes (median adjusted RD = 9.5, high risk of bias; no overall p-value reported) but not in any of three patient outcomes (median adjusted RD = 0, high risk of bias; no p-value reported for this comparison). Siriwardena 2002 found that only two of seven outcomes related to immunization rates improved (median adjusted RD = 5, no overall p-value reported). Kinsinger 1998 combined audit and feedback with educational outreach aiming to help primary care providers improve office systems to increase breast cancer screening rates. The intervention did improve the office systems and found an increase in the proportion of patients discussing mammograms (adjusted RD = 4.75; p=0.01), but not a statistically significant difference in actual mammography rates (p=0.56) compared to feedback alone. Likewise, Mold 2008 found that academic detailing led to increased implementation of a variety of quality improvement processes in primary care (e.g. standardized protocols), but these efforts translated into a statistically significant improvement in only 1 of 6 preventive services measured (median adjusted RD = 8, no overall p-value reported). Finally, Ornstein 2004 found statistically significant improvement in only two of 21 outcomes related to preventive 72

88 cardiovascular care in the primary care setting and a difference in overall improvement that was not statistically significant (adjusted RD = 5.5, p>0.2). Opinion leaders were explicitly identified to provide the educational outreach in three studies. Soumerai 1998 found improvements in two of four outcomes related to management of acute myocardial infarction (median adjusted RD = 8.5; no overall p-value presented). Laskshminarayan 2010 found significant improvement in two of 10 outcomes related to management of acute ischemic strokes in hospital, but no overall effect (median adjusted RD = 4; p-value reported as non-significant). Guadagnoli 2000 found no differences for breast cancer treatment (adjusted RD = -2; no p-value reported). The final six studies found no statistically significant effects when adding educational outreach to audit and feedback for the following outcomes: a global quality score incorporating screening, diagnosis, and management in primary care (Borgiel 1999: adjusted RD = -0.2; no p- value reported); prescribing statins and anticoagulants for high cardiovascular risk patients in primary care (Naughton 2007: median adjusted RD = -0.5; no overall p-value reported); antibiotic prescribing in primary care (Naughton 2009: adjusted change relative to baseline performance in the audit and feedback alone arm = 0%; p=0.33); and management of urinary incontinence by nurses in primary care (Cheater 2006: median adjusted RD = -3.6; no overall p- value reported). Rantz 2001 also found no effect of outreach on nursing home care (median adjusted RD = -1.1; no overall p-value reported), although a subgroup analysis showed that those who actively participated in the outreach did seem to improve. Audit and Feedback plus other Educational interventions compared Audit and Feedback Alone Four studies tested the combination of small group education with audit and feedback compared to audit and feedback alone. Herbert 2004 compared combining feedback with problem-based learning groups in primary care to feedback alone and found the combination had a greater effect on appropriate use of antihypertensives (adjusted RD = 7.4; p-value not reported). Also in primary care, Verstappen 2004 compared groups that focused on identifying gaps and developing quality improvement plans for decreasing total laboratory tests ordered to feedback alone and found the groups to be more effective (adjusted change relative to baseline performance in the audit and feedback alone arm = 9%, p=0.005). However, in the hospital 73

89 setting Kritchevsky 2008 found that adding a quality improvement collaborative to feedback alone did not improve the utilization of antibiotics within one hour prior to surgery (adjusted RD = -3.6; ARR -3.8, 95% CI to 6.2). Likewise, Filardo 2009 found that education regarding continuous quality improvement had no statistically significant impact on hospital based quality indicators for pneumonia or heart failure compared to feedback alone (median adjusted RD = - 1.7; p=0.47), although the authors reported that this finding may be due to poor participation in the intervention group. Hayes 2001 performed a study comparing written feedback with feedback enhanced by the participation of a trained physician, quality improvement tools, and a project liaison for anticoagulant management of venous thrombosis. The multifaceted intervention did not have a statistically significant effect on the quality of care for venous thrombosis (median adjusted RD = 1; p-values >0.2 for each of five process outcomes). Hayes 2002 conducted a very similar trial targeting heart failure, again finding no statistically significant effect (median adjusted RD = -1; no p-values reported). These studies did not seem to meet strict definitions for either educational outreach or opinion leaders, but did have many similar aspects. The effect of adding a seminar to audit and feedback was tested in three studies. Both Eltayeb 2005 (adjusted RD = 7.1; pre-post change for seminar+feedback: 11.6, 95% CI 6.6 to 16.7 versus pre-post change for feedback alone: 3.8, 95% CI -1.2 to 8.8) and Awad 2006 (adjusted change relative to baseline control = 26%; p<0.001) found that adding seminars to audit and feedback reduced inappropriate prescribing of antibiotics in Sudan. Robling 2002 found minimal difference in compliance with guidelines for MRI of the lumbar spine or knee (unadjusted RD = 4, high risk of bias; no p-value reported). Finally, three studies tested written educational materials. Everett 1983 found that combining written education regarding costs with feedback regarding laboratory use seemed to decreased test utilization compared to audit and feedback alone (unadjusted difference = 22.3%, high risk of bias; no p-value reported). Marton 1985 also found that offering a manual outlining laboratory costs reduced laboratory test utilization compared to feedback alone (adjusted change relative to baseline performance in the audit and feedback alone arm = 33%; no p-value reported). Conversely, Hershey 1988 found no significant effect on prescription rates when attaching to feedback a newsletter outlining advantages, disadvantages, and indications of 74

90 treatment options (adjusted change relative to baseline performance in the audit and feedback alone arm = 8%; no p-value reported). Audit and Feedback with Case Management or Organizational interventions compared to Audit and Feedback Alone Four trials compared audit and feedback with team changes or case management-type interventions to audit and feedback alone. Moher 2001 compared mailed feedback to feedback plus a nurse recall system in a three arm study. The nurse recall system improved the proportion of patients with adequate assessment of cardiovascular risk factors compared to feedback alone (adjusted RD = 33%; ARR = 33 (95% CI 19-46). However, this difference was not reflected in clinical outcomes, such as blood pressure or cholesterol. Similarly, Herrin 2006 found that adding a diabetes resource nurse resulted in minimal changes in glucose control when compared to two different types of feedback alone (adjusted RD = 3.1%, 1.2%; all comparisons reported with p-values >0.1). Using a more intensive intervention, Svetkey 2009 tested the addition of chronic disease group visits and case management to a feedback intervention but found little or no additional effect at 18 months for mean systolic blood pressure (adjusted change relative to baseline performance in the audit and feedback alone arm = 1%; no p-values reported). One study added a telephone follow-up to audit and feedback targeting pneumococcal vaccine coverage (Quinley 2004). This was an administrative task that encouraged use of the feedback reports and required no clinical expertise and the intervention resulted in little or no difference in vaccine use across the two sub-groups of physicians analysed (median adjusted RD = 0.97; p-values 0.07, 0.09). Audit and Feedback with Financial Incentives compared to Audit and Feedback Alone Two studies compared audit and feedback to audit and feedback plus incentives. Fairbrother 1999 had three arms that compared audit and feedback alone to audit and feedback plus an one-off "financial bonus" based on up-to-date coverage for four immunisations, and audit and feedback plus "enhanced fee for service" (five dollars for each vaccine administered within 30 days of its due date). Rates of immunisation improved from 29% to 54% 75

91 coverage in the bonus group after eight months (adjusted RD: 12.7%; no p-value comparing bonus group to feedback alone). The enhanced fee-for-service group decreased performance relative to feedback alone (adjusted RD -8.3%; no p-value for this comparison). A separate study (Hillman 1999) found that adding incentives to audit and feedback did not improve the implementation of pediatric preventative care guidelines (adjusted RD -5.4%, no p-value reported). Audit and Feedback with Patient-mediated interventions compared to Audit and Feedback Alone Five trials compared audit and feedback plus patient educational materials with audit and feedback alone and only one showed a positive effect in favour of adding patient education to audit and feedback. Mainous 2000 was a four-arm study that found adding patient educational pamphlets to audit and feedback had little or no influence on antibiotic prescribing for respiratory infections (adjusted RD = 0%; no p-value reported for this comparison). Similarly, Schectman 2003 found that patient pamphlets and videos did not improve management of low back pain compared to feedback alone, probably because it was poorly adopted (raw data not reported, patient intervention described as not effective). Buffington 1991 found that mailed patient reminders resulted in little or no difference from weekly feedback alone for influenza vaccination rates (adjusted RD = 1%, high risk of bias; p-value reported as 'not significant'). O'Connor 2009 found that mailed information with reminders to patients with diabetes did not increase the effectiveness of a feedback intervention for control of hemoglobin A1C (adjusted change relative to baseline performance in the audit and feedback alone arm = - 1%; no p-value for this comparison). Weitzman 2009 found that the addition of patient reminders to feedback using both a letter and phone-call to urge comprehensive follow up resulted in improved control of diabetes based on achieving glucose, cholesterol and blood pressure targets (median adjusted RD = 4.4%; OR = 2.4, p<0.01). 76

92 Comparison F. Other interventions compared to audit and feedback Twenty two comparisons from 20 trials were included in this comparison. Below, the results of these comparisons are summarized within categories related to the 'type' of intervention that audit and feedback was combined with when comparing to audit and feedback alone. We acknowledge that some of the multifaceted interventions may fit into multiple categories, but only describe the findings from each trial once. Multi-arm studies may be described in multiple sections corresponding with the type of comparison. Due to the variation in outcome type (dichotomous, continuous, patient, provider) across the studies, we were unable to conduct quantitative meta-analyses. For each comparison, we determined the adjusted RD or the adjusted percent change relative to baseline performance in the audit and feedback arm. This is reported below in addition to any statistical comparisons conducted by the authors of a particular study (e.g. odds ratios or p-values) to provide a standard measure of effect across all comparisons in this review. Reminders compared to Audit and Feedback Audit and feedback was compared to reminders in eight studies. Eccles 2001 found that educational reminders appended to radiology reports were more effective than twice yearly feedback to general practitioners for reducing overall radiology requests (median adjusted change relative to baseline performance in audit and feedback arm 15%; pre-post difference in rate for reminders = 1.57, 95% CI 0.6 to 2.5 and pre-post difference for feedback = 0, no p-value for this comparison). Tierney 1986 also found that reminders were superior to monthly feedback to medical residents for improving delivery of a variety of preventive services (unadjusted RD 4.5%, no p-value reported). In Thomas 2006, feedback led to greater reductions in the number of laboratory tests ordered compared to reminders although the model-based analyses suggested similar effects (adjusted change relative to baseline performance in audit and feedback arm = 12%; OR for feedback = 0.87, 95% CI 0.81 to 0.94, OR for reminders = 0.89, 95% CI 0.83 to 0.93). In Ziemer 2006 feedback was more effective than reminders for reducing clinical inertia in diabetes, measured as the proportion of visits with action taken to improve glucose control (adjusted RD = 6%; p<0.01). Finally, Boekeloo 1990 found that audit and feedback was superior to reminders 77

93 for inpatient cholesterol management (median adjusted RD = 15%, high risk of bias; no p-value for this comparison). Grady 1997found little or no difference between the interventions in rate of mammography referral (adjusted RD = -1%; p-value reported as not significant) and Phillips 2005 found minimal difference in management of diabetes (adjusted RD = -0.1%; no p-value for this comparison). Bahrami 2004, compared audit and feedback to a computer decision support system to improve the management of impacted molars. Neither intervention was shown to be effective (adjusted RD = 2%; no p-value reported for this comparison). Educational Outreach compared to Audit and Feedback Lomas 1991 compared audit and feedback to the use of local opinion leaders to implement guidelines for the management of women with a previous caesarean section in a high quality study. The opinion leader group increased the proportion of women offered a trial of labour and the audit and feedback group did not (unadjusted RD = 17.9%; p=0.002). Cheater 2006 found somewhat favourable effect for audit and feedback compared to the educational outreach arm, but their models revealed no evidence for either arm in the management of urinary incontinence by nurses situated in family practices and intervention (median adjusted RD = - 3.9%; ARR = -2.3%, 95% CI -6.3 to 1.7 for feedback versus ARR = 0.9%, 95% CI -3.3 to 5.1 for outreach). Other Educational interventions compared to Audit and Feedback Two studies directly compared seminars to audit and feedback. Robling 2002 did not find a statistically significant difference between feedback and a seminar in appropriateness of MRI requests of the lumbar spine or knee, (unadjusted RD = 12%, high risk of bias; concordance = 67%, 95% CI 52 to 81% for feedback versus 79%, 95% CI 66 to 92% for seminar, no p-value reported). Holm 1990 found that a seminar was more effective than audit and feedback for reducing benzodiazepine prescriptions (adjusted change relative to baseline performance in the audit and feedback arm = 22%; p=0.03). 78

94 Herbert 2004 found that practice-based small group learning similarly effective as postal audit and feedback amongst family physicians for increasing appropriate use of antihypertensives (adjusted RD = 0.8; no p-value reported for this comparison). Finally, two studies tested printed educational materials. Everett 1983 found that printed materials regarding costs of laboratory tests did not lead to changes in laboratory test utilization, but audit and feedback actually increased utilization (unadjusted RD = -12.9%, high risk of bias; no p-value reported). However, Marton 1985 found that neither a manual outlining costs nor feedback every two weeks on laboratory expenditures significantly reduced laboratory test utilization (adjusted change relative to baseline performance in the audit and feedback arm = 6%; p-value reported as non-significant). Case Management or Organizational interventions compared to Audit and Feedback When Svetkey 2009 compared chronic disease group visits and case management to audit and feedback, no effect was found for either intervention on systolic blood pressure at 18 months (adjusted change relative to baseline performance in the audit and feedback arm = -1%; effect for feedback = 0.3 mm Hg, p=0.81 and effect for case management = -0.2 mm Hg, p=0.89). Claes 2005 did not find a difference between feedback and either point-of-care testing or rapid clinical decision support from the laboratory for keeping patients within target INR (International Normalized Ratio) for their oral anticoagulation, although all interventions seemed to be effective (adjusted change relative to baseline performance in the audit and feedback arm = 4% for both comparisons; p=0.13 for difference across all arms). Financial Incentives compared to Audit and Feedback Martin 1980 compared incentives to audit and feedback to reduce test-ordering in hospitals. Incentives were less effective than audit and feedback at reducing test ordering (adjusted change relative to baseline performance in the audit and feedback arm = -41%; p-value reported as <0.05). 79

95 Patient mediated interventions compared to Audit and Feedback Three studies directly compared patient-mediated interventions with provider-directed audit and feedback; none found a statistically significant difference in outcomes. Mainous 2000 compared patient educational pamphlets to feedback, finding little or no difference between groups in antibiotic prescribing rates (adjusted RD = 2%, no p-value reported for this comparison). Schectman 2003 did not find a statistically significant effect of patient pamphlets and videos on the management of low back pain. The details of the results of this group compared to the feedback group were not reported. Finally, one study (O'Connor 2009) compared a patient intervention featuring a postal letter to each patient summarizing their diabetes-related risk factors and offering suggestions for improvement with a physician intervention featuring audit and feedback plus reminders. No improvement in hemoglobin A1C level was found (adjusted change relative to baseline performance in the audit and feedback arm = -1%; no p-value for this comparison). 80

96 4.5 Discussion Summary of main results Audit and feedback can be a useful intervention to improve health professionals' compliance with desired practice. The median adjusted RD of compliance with desired practice was a 4.3% absolute increase in desired practice (IQR 0.5 to 16%) when considering any trial in which audit and feedback was considered the core, essential aspect of the intervention, compared to no audit and feedback. For continuous variables, we found that the adjusted percent change relative to the performance of the control group at baseline was a 1.3% increase in compliance with desired practice (IQR 1.3 to 23.2%). Although the median effect may be perceived as relatively small, the 75th percentile effect size is much larger (16% absolute improvement in health professionals compliance with desired behaviour), suggesting that audit and feedback, when optimally-designed and used in the right context, can play an important role in improving professional practice. There are a number of plausible explanations why some interventions were more effective than others and we tested some of the hypothesized variables in a meta-regression. As in the previous versions of this review, we found that baseline performance was associated (inversely) with the effectiveness of audit and feedback. The meta-regression provides indirect evidence that five feedback characteristics are also associated with the effectiveness of audit and feedback interventions. Specifically, our findings indicate that feedback will be most effective when provided from a source that is a 'supervisor or senior colleague', and delivered at least 'monthly', in both a 'verbal and written' format, aiming to decrease rather than increase provider behaviours, and offers instructions with 'both explicit goals and a specific action plan'. However, the ability to make firm conclusions from the analysis of heterogeneity is hindered both by the indirect nature of the comparisons and by the non-specific nature of the components of those variables. For instance, while it appears that verbal feedback is the least effective format, such 'verbal' feedback could have been a lecture to a large group or a one-to-one discussion. Likewise, while it appears 81

97 that a 'supervisor or colleague' is the most effective source, this finding may depend on whether or not the colleague is a respected opinion leader. In addition, the difference in effect between interventions aiming to decrease or increase behaviours vanished when the targeted behaviour was analysed in the exploratory analysis. Therefore, the results of our meta-regression should be interpreted cautiously. Seventeen studies provided direct, randomised comparisons of different ways of providing audit and feedback; only four of these trials were published after Based on these comparisons and also based on indirect comparisons across studies it is difficult to determine what other features of audit and feedback have an important impact on its effectiveness. For example, we found conflicting evidence regarding the role of peer comparisons. Kiefe 2001indicated that comparing to the top 10% of peers might be an improvement over comparing to the mean, but Schneider 2008 found that identifying top performers in the context of a quality circle did not increase the effectiveness of feedback. The difference may reflect the role of explicit goal / target setting in determining the reaction to feedback. 51, 53 Active participation in goal-setting may also play an important role. 151 Although there are theoretical reasons why some forms of audit and feedback might be more effective than others, there remains a need to operationalize and directly compare different approaches to improving the design and delivery of audit and feedback. For now, decisions about when to provide audit and feedback must largely be guided by pragmatic considerations and hypotheses based on a priori theory. In addition to the design of the intervention itself it is likely that the characteristics of the context and the recipients might influence the effectiveness of feedback. In addition, feedback might also be best suited for changing specific types of behaviours; for example, more complex targeted behaviours might be harder to change by providing feedback. When we attempted in the previous version of this review to include the complexity of the targeted behaviour as a variable in our meta-regression, we did not find a statistically significant association between the complexity of the targeted behaviour and the effectiveness of feedback, possibly because it was difficult to reliably assess complexity. In this review, we conducted an exploratory analysis for a small number of targeted behaviours (prescribing, test ordering, and management of diabetes or cardiovascular disease) chosen because they were frequently targeted in feedback trials. We found a relatively large effect for prescribing (median adjusted RD 13.1%) compared to test ordering (-0.1%) and 82

98 management of diabetes or cardiovascular disease (0.5%). A plausible explanation for this difference is that prescribing is typically not a complex behaviour and may be perceived as important, whereas test ordering may be perceived as less important (and might be more complex) and disease management is typically more complex. However, within the diabetes and cardiovascular disease subgroup there was great variation in the targeted behaviours. This is also true of the prescribing and the test ordering subgroups. In some trials the intention was to increase prescribing, test ordering or referrals (addressing under-use), while in others the goal was to reduce utilization (addressing over-use). It is important that future trials consider carefully the intended target of the intervention and precisely describe the targeted behaviours, ideally including an assessment of their complexity and perceived importance. Although our analysis suggests that audit and feedback might be highly effective for improving prescribing (and less effective for test ordering or disease management), this was an exploratory analysis and there remains a great deal of uncertainty regarding which clinical or behavioural targets would be most appropriate for audit and feedback. The previous version of this review investigated the impact of audit and feedback when used as part of a multifaceted intervention, finding little evidence of enhanced effectiveness, consistent with other reviews that have concluded that multifaceted interventions are not necessarily more effective than single strategies. 37, 65, 152 In this review, we found that when audit and feedback was combined with other interventions, the effect size of the intervention tended to be larger than when audit and feedback was used alone. This difference was statistically significant for studies with continuous outcomes but not with dichotomous outcomes. The results were also inconsistent with respect to suggesting which combinations of interventions might be most effective. Thus, the added costs of multifaceted interventions need to be weighed against the uncertainty of whether a multifaceted intervention is likely to produce a greater effect. When and how to best combine feedback with other interventions warrants systematic investigation, ideally through a series of comparative trials. 83

99 4.5.2 Overall completeness and applicability of evidence Although the variation in effect size is noteworthy and requires further study, the consistency of median effect size found in this review compared to the previous review, despite changes in methodology, is of interest. While the best way to design and deliver feedback remains uncertain, this review provides greater certainty about its likely effect compared to usual care across a variety of clinical situations. Given the large number of RCTs included in this review and the stability in effect size observed over time, we believe it is unlikely that missing or new trials of audit and feedback versus usual care would substantially alter the estimated median effect of audit and feedback on professional practice. Thus, future trials should aim to determine the best way to deliver audit and feedback in head-to-head trials rather than comparing audit and feedback to usual care Quality of the evidence In most of the included studies, the method of allocation was not clearly indicated in the published report. Although lack of allocation concealment can result in overestimates of effect, 153 the importance of this criterion in trials where a group of healthcare professionals is randomised at one point in time is not established. In this review we have given cluster randomised trials the benefit of the doubt and assumed that there was adequate concealment of allocation for these studies. Nonetheless, we judged only 32% of the included studies to have a low risk of bias. This compares favourably to the previous version of this review where only 20% of the included trials had a low risk of bias. However, we judged 18% of the studies included in this review to have a high risk of bias, while in the previous review only 12% were deemed high risk of bias. The lower proportion of studies with unclear risk of bias may represent improved reporting over time. As with the previous review, we found no association between overall risk of bias (low versus unclear) and the estimate of effect. 84

100 4.5.4 Potential biases in the review process In this review, our inclusion criteria required that at least one arm of the trial use audit and feedback as the core, essential feature of the intervention. This was necessary to avoid including trials of multifaceted interventions where feedback was included but where the main effects of the intervention were unlikely to be due to feedback. If some effective multifaceted interventions were inappropriately excluded, this would create a conservative bias (and viceversa). Although application of this criterion depended on judgments made by the reviewers, only eight disagreements occurred between independent reviewers of 282 full-text manuscripts reviewed and all were resolved easily through discussion. Furthermore, the similarity in the estimate of effectiveness for multifaceted interventions featuring audit and feedback between this review (adjusted RD 5.5%) and the previous review (adjusted RD 5.7%) supports the notion that the operationalization of this criterion did not substantively bias the results. In earlier reviews of this topic we considered printed educational materials to have little 154, 155 or no effect on changing professional practice based on information available at the time. However, recent reviews 37, 156 found that printed educational materials have a small (but potentially important) effect. By abstracting printed materials as usual care for many studies, we may have created a conservative bias for studies comparing feedback to printed materials, but an overestimation of the effect attributed to audit and feedback in studies where feedback plus printed materials are compared to no intervention. In most studies educational materials were distributed to all groups, thus meeting a pragmatic definition of usual care. One possible reason for our finding that few studies featured patient outcomes is that we only abstracted primary outcomes and many studies provided patient outcomes as secondary outcomes. This assessment would have been easier to make if more studies clearly stated their primary outcome in general and if more studies had planned to have a patient level outcome as the primary outcome. However, since most studies reporting patient outcomes as secondary outcomes are likely to be under powered to detect a difference in patient outcomes, this is unlikely to have affected our finding that improvements in patient outcomes were at best small. The reason for this is that impacts on patient outcomes depend on the combined effectiveness of feedback on professional practice and the effectiveness of the clinical intervention (delivered as a result of the changes in professional practice). Since the effectiveness of feedback is typically 85

101 small or moderate (e.g. a 4.3% absolute improvement) and the effectiveness of targeted clinical interventions (changes in practice) is typically moderate, feedback can only be expected to have a small effect on patient outcomes in most circumstances. Thus large trials are needed to reliably measure the impacts of feedback on patient outcomes. As illustrated in Table 1, there are many possible factors related to feedback design that could potentially predict effectiveness. It is certainly possible that we neglected to abstract some important design factors, especially organizational and contextual characteristics. We limited the exploration of such factors for pragmatic reasons (based on feasibility of abstraction) and to limit risk of spurious findings. We chose to focus on comparisons where it was possible to calculate an adjusted risk difference and adjusted change relative to the baseline control. The adjustments were based on pre-intervention measurements of the outcome in the audit and feedback group. We excluded from the quantitative analyses studies without baseline data because of previous evidence that baseline performance is associated with effectiveness of audit and feedback. Since many studies included small numbers of healthcare professionals, baseline differences were common and unadjusted estimates of effect often differed from the adjusted estimates. We weighted the analyses by the number of health professional involved in each trial. Trials that did not report the number of health professionals involved were weighted by the number of practices/hospitals/communities involved in the trial; this typically occurred when the unit of allocation was a cluster of providers (e.g. practice, hospital, or community) rather than a single provider. This approach may have led to some larger studies with many participants but relatively few clusters being assigned a weight that did not reflect the actual size of the trial Agreements and disagreements with other studies or reviews The previous update of this Cochrane review found similar estimates of effect for audit and feedback on professional practices. It also found that greater "intensity" of feedback was associated with greater effect. However, the assessment of "intensity" simultaneously captured numerous variables and was therefore difficult to operationalize as it could not discern which components were most important. In this review, we tested five specific characteristics of 86

102 feedback design in a meta-regression in an attempt to identify important active ingredients of audit and feedback. The sources of feedback associated with the lowest effect size were 'professionals standards review organisation' and 'representative of the employer or purchaser'. This fits well with previous qualitative work comparing high and low performing hospitals finding that feedback with a punitive tone seems to be less effective. 63 Also of note was the stability in effect across clinical setting and profession of recipient, although the latter finding may be due to the paucity of trials with interventions directed to non-physicians. Our finding that risk of bias was not associated with effect size is consistent with the previous version of this review. In both cases, this may be explained by suboptimal reporting, resulting in many risk of bias domains judged to be 'unclear'. The findings of this review regarding format and source were consistent with a reanalysis of a previous version of the review 47 and should also be considered in light of the Feedback Intervention Theory, 46 which suggests that feedback that directs attention towards acceptable and familiar tasks (as opposed to those that generate emotional responses or cause deep self-reflection) seem most likely to lead to improvement. Our results regarding actionplanning are also consistent with the re-analysis of the previous version of the review informed by the Feedback Intervention Theory. 47 However, a separate re-analysis of the previous version of this review aiming to test the hypothesis regarding goal-setting and action-planning found too few studies to reach any conclusions. 146 Although we hypothesized based on Carlsen that feedback aiming to increase provider behaviours would be more effective than feedback aiming to decrease behaviours we found that the opposite was true. This suggests that stated preferences with respect to quality improvement interventions should be empirically tested. In this review, we also found evidence that the targeted behaviour may be associated with the effectiveness of feedback. In particular, we found that feedback aiming to change prescribing habits may be more effective than feedback aiming to improve chronic disease management. A recent review of audit and feedback given to general practitioners regarding diabetes management 157 included 10 studies with great heterogeneity in outcomes. The authors were unable to conclude which diabetes process measures should be targeted by future interventions and more work is clearly needed in this area. 87

103 Previous reviews have looked at factors associated with the effectiveness of audit and feedback and we recognized from the outset that there are far more plausible factors that could alter the effectiveness of audit and feedback than we could test in this review. Mugford and colleagues 40 identified 36 published studies of information feedback which they defined as the use of comparative information from statistical systems. These authors distinguished passive from active feedback where passive feedback was the provision of unsolicited information and active feedback engaged the interest of the clinician. They also assessed the impact of the recipient of the information, the format of the information and the timing of the feedback. Studies were included if their design used either a historical or a concurrent control group for comparison. The authors concluded that information feedback was most likely to influence clinical practice if the information was presented close to the time of decision-making and the clinicians had previously agreed to review their practice. The results of this review do not support or refute these conclusions. Axt-Adam and colleagues 144 reviewed 67 published papers of interventions (26 studies of feedback) designed to influence the ordering of diagnostic laboratory tests. They reported factors that could be important included the message, the provider of the feedback, the addressee, the timeliness and the vehicle. They concluded that there was considerable variation among different studies and that this variation could be explained in part by the extent, the timing, the frequency, and the availability of comparative information related to peers. They also felt that the practice setting was an important factor. Our findings support many of these conclusions. Buntinx and colleagues (Buntinx 1993) conducted a systematic review of 26 studies of feedback and reminders to improve diagnostic and preventive care practices in primary care. They categorised the information provision that occurred after or during the target performance as feedback whereas information provision that occurred before the target performance was called reminders. Ten of the 26 studies used randomised designs but the quality of the included trials was not reported. The authors concluded that both feedback and reminders might reduce the use of diagnostic tests and improve the delivery of preventive care services. However, they also reported that it was not clear how feedback or reminders work, especially the use of peer group comparisons. Balas and colleagues 36 reviewed the effectiveness of peer-comparison feedback profiles in changing practice patterns. They located 12 eligible trials and concluded that profiling had a statistically significant but minimally important effect. 88

104 4.5.6 Implications for practice Audit and feedback can be effective in improving professional practice. The effects are generally small to moderate and vary based on the way the intervention is designed and delivered. As with any quality improvement strategy, efforts to change provider practice should be targeted at behaviours for which there is evidence between processes and patient outcomes. The results of this review suggest that feedback may be more effective when baseline performance is low, when the source is a supervisor or senior colleague, when it is provided more than once, when it is provided both verbally and written, and when it includes both measurable targets and an action plan. In addition the effect size varies based on the clinical behaviour targeted by the intervention. Although the quality of evidence for these findings is low, it is sensible to provide measurable targets and an action plan when this is practical, since this is unlikely to entail additional costs or harms. On the other hand, pragmatic consideration needs to be given to additional costs associated with providing feedback more frequently, providing both verbal and written feedback, and using a supervisor or colleague to provide feedback, since these features may entail additional costs while the benefit is not certain. The finding related to decreasing provider behaviours may suggest that feedback could be useful in situations where there is a desire to curb over-utilization, keeping in mind that the source of the feedback should preferably be a senior colleague rather than the payor. Audit is commonly used to improve accountability, either in the context of governance or as a feature of ongoing quality improvement efforts. The findings of this review suggest that it may be possible to increase the effect of feeding back the results of such audits on professional practice through careful attention to the way the feedback is designed and delivered. Those planning new interventions aiming to change practice should consider audit and feedback alongside other interventions and weigh the potential benefits against the potential challenges with respect to cost and/or logistics. 89

105 4.5.7 Implications for research There are two main research audiences for this review: those who wish to implement and rigorously evaluate the effectiveness of a local audit and feedback intervention and those who wish to examine the underlying cognitions and behavioural control mechanisms that may explain how to best design and deliver these interventions. Like other reviews of quality improvement interventions, we have found limited progress over time in the knowledge of when and how to best conduct audit and feedback interventions. 65, 152, 158 This suggests an opportunity for improved collaboration between the 'applied' scientists aiming to improve local quality of care and 'basic' scientists aiming to produce generalizable knowledge. In particular, each new audit and feedback intervention may provide an opportunity to incorporate evaluations of different ways of designing and/or delivering the feedback to explore how to optimize this intervention in routine practice settings. To build upon the current evidence base, the field would benefit from more attention to four areas: improved reporting and methods; explicit use of theory, empirical evidence, and logic to develop hypotheses and to design the intervention and comparison arms; a focus on professional practices for which there is compelling evidence of patient benefits with clearly defined primary outcomes; and more head-to-head trials (e.g. comparing different ways of providing feedback). At a minimum, to contribute to the literature, trials need to be well designed and clearly reported. 159 Better reporting of study methods, targeted behaviours, characteristics of participants, and the context are needed. 160 A clear, thorough description of the intervention, ideally with illustrative examples would be useful. Primary outcomes should be important and clearly specified. The results should be adjusted for baseline differences, which are common in cluster randomised trials, and the analysis should take account of the unit of allocation. Furthermore, trials need to be large enough to detect small effects (especially for changes in patient outcomes), when those effects are considered important. The field would likely benefit if investigators explicitly built upon knowledge generated from prior trials, systematic reviews, and relevant theory to design audit and feedback interventions. In addition to some of the psychology literature referred to in the background section, the education and the organizational/management literature suggest how the design and delivery of feedback might be optimized to improve performance (see, for example, Shute 90

106 ). Well-designed, mixed methods process evaluations embedded within trials can be useful to explore and provide insights into the complex dynamics underlying the variable effectiveness of audit and feedback. In particular, researchers should examine hypotheses regarding how their audit and feedback intervention will be acted upon in practice. Finally, although there have been more trials over time directly comparing different ways of conducting feedback interventions, there is a continued need to emphasize this type of headto-head trial. The cumulated evidence suggests that further two-arm trials comparing feedback to usual care are likely to be of limited value. The focus should shift from whether audit and feedback works better than usual care to discerning ways to optimize the effectiveness of audit and feedback interventions for particular contexts or clinical practices. The utility of future updates of this review will depend on the availability of new, well-designed (and well-reported) trials and on our ability to recognize, abstract, and analyze important explanatory factors Conclusions Audit and feedback generally leads to small but potentially important improvements in professional practice. The effectiveness of audit and feedback seems to depend on baseline performance and how the feedback is provided. Future studies of audit and feedback should directly compare different ways of providing feedback. 91

107 5 Paper 2: Feedback GAP: pragmatic, clusterrandomized trial of Goal setting and Action Plans to increase the effectiveness of audit and feedback interventions in primary care. 5.1 Background Audit and feedback is often the foundation of quality improvement projects aiming to close the gap between ideal and actual practice. Audit and feedback is known to improve quality of care but there is variability in the magnitude of effect observed. 131 This variability may be attributed to the nature of the targeted behaviour, the context, and the characteristics of the recipient, as well as to the design of the audit and feedback intervention itself'. 161 The Cochrane review of audit and feedback found that feedback is more effective when sent more than once, delivered by a supervisor or senior colleague in both verbal and written formats, and when it includes both explicit targets and an action plan. 162 However, these conclusions are based on indirect comparisons from meta-regressions and are thus less reliable than those that would be generated directly from head-to-head trials of different approaches to providing audit and feedback. Further there is little information to guide operationalization of these factors. 39 For example, few audit and feedback trials explicitly describe goal setting or action planning as part of the intervention, and those trials that did appeared to deliver this component of the intervention in various ways. 146, 163 Although action planning is a familiar activity in clinical practice, the plan can more effectively lead to behaviour change if it includes two key elements: an if statement (specifying contextual factors that will trigger the action) and a then statement (specifying precisely the action to be taken). 164 In the context of feedback and goals, implementation intention-based action plans could increase goal-directed behaviours possibly by increasing both self-efficacy (confidence in ones ability to perform an action effectively) and goal-commitment (degree to which the person is determined to achieve the goal). 142 For patients with diabetes, audit and feedback is known to modestly improve processes of care, as well as blood pressure and glycemic control. 157, 165 In this trial, we aimed to build on the extant knowledge regarding audit and feedback by asking not whether it can improve care, but 92

108 whether it could be modified to be more effective to improve processes of care related to patients with chronic disease, including diabetes and ischemic heart disease (IHD). Our hypothesis was that feedback reports delivered to family physicians regarding care patterns for patients with chronic disease would draw attention to a discrepancy between actual and desired quality of care (e.g., fewer patients than expected with blood pressure (BP) recently measured) and that a worksheet accompanying the feedback to facilitate goal-setting and action-planning would increase the likelihood that family physicians would act to improve quality of care (e.g., identify and more aggressively treat patients with BP above target). 93

109 5.2 Methods Study design This was a two-arm, pragmatic cluster-trial conducted in primary care. To reduce the risk of contamination, randomisation was at the level of the primary care clinic. Each physician in the intervention clinics received feedback accompanied by a goal-setting and action-planning worksheet, while each physician in the clinics allocated to usual care received feedback unaccompanied by the worksheet. Feedback reports addressed guideline-based quality indicators for patients in their practice with diabetes and/or IHD. Such patients are at elevated risk of cardiovascular events, especially if they have a history of both conditions 3 and guidelines recommend similar processes of care as well as control of blood pressure (BP) and cholesterol to reduce this risk. 166, 167 The trial was pragmatic in that it sought to determine if the intervention could be effective under usual circumstances: the goal-setting and action-planning worksheet was designed to be readily scalable and was delivered with minimal supports; the usual feedback arm was not standardized with respect to co-interventions; patient-level outcomes were assessed from databases with unobtrusive measurement of compliance; and analysis was by intention-totreat. 168 The protocol has been previously published 169 and is summarized below. This study received approval from the Research Ethics Office at Sunnybrook Health Sciences Centre ( ) and registered at ClinicalTrials.gov (NCT ) Setting In the province of Ontario, patients with chronic conditions such as diabetes and stable ischemic heart disease are generally managed in primary care, mostly by family physicians. There is no co-pay for doctor visits or laboratory tests for Ontarians, but medications are covered by the provincial drug plan only for the elderly and those on social assistance. The majority of 94

110 primary care providers in the province work in groups and are paid by a mix of capitation and fee for service Participants and data collection Participants were family physicians throughout Ontario who signed data-sharing agreements with the Electronic Medical Record Administrative data Linked Database (EMRALD), held at the Institute for Clinical Evaluative Sciences (ICES). EMRALD has developed mechanisms to extract, securely transfer, and de-identify the EMR data for analysis at ICES, maintaining strict standards for confidentiality. 170 Family physicians were originally invited to participate in EMRALD through convenience sampling of EMR users and all EMRALD contributors consented to this study. Physicians with less than one year of experience using their EMR or with less than 100 active adult patients enrolled in their practice were excluded. Included patients were over age 18 at the start of the trial, were enrolled with their family physician throughout the study, and had diabetes and/or IHD. Only patients with one or more visits at least one year prior to the trial to were included to ensure that enough data existed in the EMR to assess quality of care and ensure that providers were not audited for transient or new patients. EMRALD has validated algorithms to identify patients with diabetes and IHD, 171, 172 which do not require special data input by physicians Allocation Practices were allocated using minimization (conducted by the study analyst using the free software, MINIM 173 ) to achieve balance on baseline values of the primary outcomes and on the number of eligible patients in each cluster. 162 Using the baseline data for each cluster, these variables were classified as high or low using the median value as the cut-point. After recruitment was completed, practices were allocated simultaneously to ensure low risk of bias related to allocation concealment, as per Cochrane Effective Practice and Organization of Care Group criteria

111 5.2.5 Intervention The intervention was developed through an iterative process and piloted with family physicians, as described previously. 169 Each physician received a package (Table'6) by courier every six months for two years featuring feedback reports describing the aggregate percentage of patients with diabetes and/or IHD meeting quality targets, along with explanatory documents and self-reflection surveys to be completed for continuing medical education credits (CME). For each disease condition, the report fit on one page and for every quality target, the aggregate performance achieved by the participating physician was compared to the score achieved by the top 10% of participating physician performers. 136 See Appendix 2 for prototype feedback reports. The frequency was limited to twice yearly for two years due to capacity of the research team, and the reports included only aggregate data (no patient-specific information) due to potential privacy risks of sending patient-specific information. Table 6: Intervention components and description Feedback reports Explanatory document Suggestions document Continuing medical education form One page focusing on patients with diabetes and one on patients with heart disease. Aggregate rather than patient-specific data provided. Performance compared to top 10% of peers, presented in tables and bar graphs. One page description of how data was generated, including possible limitations. One page with clinical recommendations (ie. managing muscle aches for patients taking statins) and generic quality improvement strategies (ie. work with administrative staff to encourage patients to have periodic visits only for chronic disease management) Two-page self-reflection survey required by the College of Family Physicians to earn continuing medical education credits related to practice audits. Physicians randomized to the feedback plus worksheet arm also received a one-page worksheet appended to the feedback along with the standard CME survey. The worksheet was designed to facilitate goal-setting and implementation intention-based action-plans, using the if and then formulation explained above. 142 See Appendix 2 for prototype of worksheet. Participants were asked to submit the worksheet along with the CME surveys in order to process their CME credits. 96

112 Prior to the second cycle of feedback reports, the College of Family Physicians of Canada implemented the use of standardized forms to earn continuing medical education credits for practice audits. Therefore, the CME surveys changed during the trial. The original surveys asked participants what they learned about diabetes and IHD care, intention to change practice, and asked about potential barriers to change. The revised surveys asked participants to make a decision about your practice, to declare what will you have to do to integrate these decisions, and to continue to reassess the practice change and make further plans to improve. Given the nature of the intervention, blinding of physicians was not possible, but they were not aware of the exact nature of the intervention being tested Outcomes Outcomes were monitored using validated processes to analyze data collected from participants EMRs. There were two patient / disease-level primary outcomes and one professional practice / process-level primary outcome. The patient-level primary outcomes were the patients most recent LDL and systolic BP values, if tested within 24 or 12 months, respectively. The process-level primary outcome was a composite process score indicating whether patients received prescriptions and tests in accordance with relevant guidelines Patients received a composite process score with a maximum of 6, as outlined in Table'7, which we multiplied by 100 to report the score as a percentage. Secondary outcomes were chosen because they were thought to reflect the targets of action by family physicians receiving the feedback. Each of the items in the composite process score was assessed, plus glycemic control (haemoglobin A1c level), the proportion meeting targets recommended in guidelines for LDL (<2mmol/L), and BP (<130/80 mmhg in diabetes and <140/90 mmhg in IHD), and prescriptions rates for insulin and beta blockers. While not every patient requires each prescription or investigation, we anticipated balance between groups in the proportion of eligible patients. Therefore, increases in aggregate proportions of processes performed indicate general intensification of treatment for patients with these conditions. 97

113 Table 7: Composite process score calculated for each patient as primary process outcome. Quality indicator (for each patient receives a score) Diabetes (maximum score = 6) IHD (maximum score = 6) Both Diabetes+IHD (multiply by 6/7 for max score =6) BP test in 6M X X X A1C test in 6M X X FBG test in 24M X LDL test in 12M X X X ACR test in 12M X X Rx ASA X X Rx Statin X X X Rx ACE/ARB X X X IHD=ischemic heart disease, BP = blood pressure, A1C = haemoglobin A1c, FBG = fasting blood glucose, LDL = low density lipoprotein cholesterol, ACR = urinary albumin-to-creatinine ratio, ASA = aspirin, ACE/ARB = angiotensin-modifying agent Analysis Primary analysis was performed on an intention-to-treat basis, using patient level variables, combining patients with diabetes and/or IHD. Since the intervention was directed to the physician, final analysis was limited to patients with diabetes and/or IHD who were enrolled in their physician s practice throughout the trial. We used linear mixed models (SAS MIXED procedure) for continuous variables to estimate the mean difference in each outcome between arms, together with their 95% confidence interval (CI). For dichotomous variables, we used generalized estimating equation models (SAS GENMOD procedure) and estimated relative risk and the 95% confidence interval using log-binomial regression. 181 We used generalized linear models to examine random variation for each outcome at both physician and practice levels (SAS GLIMMIX procedure). The clustering of patients within physicians was accounted for using random effect models and if the practice-level was also significant we included both levels when assessing outcomes. We ran a model for each outcome with and without adjustment for baseline values of the dependent variable. The adjustment for baseline values of the dependent variable was carried out by specifying the pre-intervention measure of the outcome as a covariate. We planned this approach a priori because although there were thousands of patients, we were not confident that allocation of only 14 clinics would result in adequate balance at 162, 182 baseline. 98

114 Planned sub-group analyses were performed on patients with only IHD, only diabetes, or both, to assess the same outcome variables. We hypothesized that patients with only IHD would have lower quality of care scores at baseline and greater potential for improvement during the intervention because locally it has received relatively less attention with respect to quality improvement initiatives and because identifying patients in the EMR with IHD is more difficult than identifying patients with DM due to the lack of relevant laboratory results or specific medications. A planned per-protocol analysis assessed whether full completion of the intervention worksheet resulted in improved outcomes in general and for the specific clinical topics in participants goal statements. All analyses were carried out using the SAS Version 9.2 statistical program (SAS Institute, Cary, NC, USA). The number of participating practices and eligible physicians providing data to EMRALD determined the sample size. With 54 physicians from 14 practices initially providing data and consenting to this trial and a presumed intra-cluster correlation of 0.05, we estimated 80% power to find an absolute difference in LDL of 0.32mmol/L and in systolic BP of 7mmHg, based on pilot data

115 5.3 Results Cluster and patient flow are described in Figure'10. Just prior to allocation, one physician stopped practicing. Thus, the study began with 4617 patients cared for by 53 physicians from 14 clinics. At baseline (August 2010), 22 physicians cared for 2157 patients with diabetes and/or IHD in the feedback plus worksheet arm and 31 physicians cared for 2460 patients were in the usual feedback arm. During the two-year trial two physicians from each arm were lost to follow up. Of the 562 patients lost to follow up, 175 belonged to these four physicians, and 166 changed physicians during the study. Average age of patients lost to follow up was 68 years and 44% were female, reflecting the underlying distribution of those allocated. 100

116 14 clinics Allocated'to'intervention' (feedback'+'worksheet)' 7'clinics' 22'physicians' 2157'patients' Allocated'to'usual'care' (feedback'alone)' 7'clinics' 31'physicians' 2460'patients' Lost'to'follow'up' 0'clinics' 2'physicians'' (1'died,'1'retired)' 325'patients' Lost'to'follow'up' 0'clinics' 2'physicians' (1'withdrew,'1'retired)' 237'patients' Analyzed' 7'clinics' 20'physicians' 1832'patients' Analyzed' 7'clinics' 29'physicians' 2223'patients' Figure 10: Cluster-Patient flow diagram. Comparability at baseline Cluster and participant characteristics are described in Table 8 and 101

117 Table 9, respectively. The median number of included patients per clinic was 234, (interquartile range (IQR) ) and the median per physician was 81 (IQR, ). Half of the practices (7/14, 50%) were located in urban settings. About half of the physicians were female (25/53, 47%). Physicians had been in practice for a median of 19 years (IQR, 9-24) and had been using their EMR for a median of 7 years (IQR 6-7). Intervention physicians were more likely to be male, with more years experience, and located in rural settings. They also tended to have smaller practices overall but with more eligible patients. Patients averaged 65.6 years and 44.4% were female. Most included patients (3435, 74.4%) had diabetes, the remainder had IHD (25.6%); one-eighth (12.4%) had both diabetes and IHD. The mean weight of included patients was 89 kilograms (standard deviation [SD] = 26, 501 missing). Mean BP was 130/74 (systolic SD = 17, diastolic SD = 11, 467 missing), with 55.1% achieving target, and mean LDL was 2.31 (SD = 0.9, 674 missing), with 67.7% achieving target. The mean process composite score was 72 (SD = 26). Values for these variables and other process measures were similar across groups, except for greater proportion of patients in the feedback plus worksheet arm group than the feedback alone arm with a recent BP test (85% versus 74%) and HbA1c test (79% versus 69%). Table 8: Baseline characteristics of clinics, and family physicians * Feedback*plus*worksheet* Feedback*alone* Clinic*characteristics*(N*=*14)* 7' 7' Total*patients*(median,*IQR)* 3121'(2258\2671)' 4432'(3488\8201)' Eligible**patients*(median,*IQR)* 205'(189\389)' 257'(211\499)' Location*(n,*%*rural)* 4'(57%)' 2'(29%)' Physician*characteristics*(N*=*53)* 22' 31' Roster*size*(median,*IQR)* 1010'(596\1416)' 1291'(884\1696)' Sex*(n,*%*female)* 6'(27%)' 19'(61%)' Years*in*practice*(median,*IQR)* 23'(16\25)' 15'(7\22)' Years*using*EMR*(median,*IQR)* 7'(6.7\7.9)' 6'(6.2\6.6)' Eligible**patients*(median,*IQR)* 77'(49' '120)' 179'(62' '216)' *'Eligible'patients'are'those'meeting'inclusion'criteria'with'diabetes'and/or'heart'disease.'' IQR'='interquartile'range,'%'='proportion' 102

118 Table 9: Baseline characteristics of patients Feedback plus Feedback ICC worksheet alone Patients (total = 4617) IHD only 545 (25%) 637 (26%) - DM only 1329 (62%) 1534 (62%) - IHD and DM 283 (13%) 289 (12%) - Age (mean years ±sd) 67 ±14 65 ±14 - Female 906 (42%) 1251 (46%) - WT (mean kg ±sd) 88 ±22 90 ± Systolic BP (mean mmhg ±sd) 130 ± ± Diastolic BP (mean mmhg ±sd) 73 ±11 75 ± LDL (mean mmol/l ±sd) 2.3 ± ± HbA1c^ (mean % ±sd) 7.4 ± ± LDL at target~ 665 (31%) 826 (34%) BP at target~ 1019 (47%) 1054 (43%) BP test in 6M 1839 (85%) 1831 (74%) A1C test in 6M^ 1267 (79%) 1250 (69%) FBG test in 24M 1904 (88%) 2134 (87%) LDL test in 12M 1574 (73%) 1770 (72%) ACR test in 12M^ 1044 (65%) 1232 (68%) Rx ASA* 602 (73%) 657 (71%) Rx Statin 1463 (68%) 1757 (71%) Rx ACE/ARB^ 1153 (71%) 1243 (68%) Rx Beta blocker* 497 (60%) 562 (61%) Rx Insulin^ 339 (21%) 391 (21%) Composite process score (mean ±sd) 73 ±25 71 ± ICC = Intra-cluster correlation reported at physician-level, ^ = Calculated for patients with diabetes, * Calculated for patients with IHD, ~ = Target BP 130/80 for diabetes but 140/90 for IHD and Target LDL < 2, IHD = ischemic heart disease, BP = blood pressure, A1C = haemoglobin A1c, FBG = fasting blood glucose, LDL = low density lipoprotein cholesterol, ACR = urinary albumin-to-creatinine ratio, ASA = aspirin, ACE/ARB = angiotensin-modifying agent, kg = kilogram, sd = standard deviation. 103

119 Intention-to-treat analyses There were 4055 patients cared for by 49 physicians from 14 clinics in the final analysis. The primary and secondary outcomes at the end of the trial are summarized in Table'10. No clinically or statistically significant differences were observed across groups in the primary outcomes in both the adjusted and unadjusted models. Mean systolic BP was 128 in both arms (adjusted mean difference = -0.05; 95% CI -2.11, 2.02) and diastolic BP was 72 in the feedback plus worksheet arm and 73 in the feedback alone arm (adjusted mean difference = -0.72; 95% CI -2.18, 0.75). LDL was 2.1 in the feedback plus worksheet arm and 2.0 in the feedback alone arm (adjusted mean difference = 0.04; 95% CI -0.02, 0.10). The mean composite score was 72 in the feedback plus worksheet arm and 70 in the feedback alone arm [adjusted mean difference = 1.76; 95% CI -1.4, 4.9]. Mean haemoglobin A1c in patients with diabetes was lower in the feedback plus worksheet arm (7.2% versus 7.4%; adjusted mean difference -0.2; 95% CI -0.3, -0.1). A greater proportion of patients in the feedback plus worksheet arm had their BP tested within six months (81% versus 69%, adjusted relative risk [arr] = 1.2; 95% CI 1.1, 1.3) and more achieved target BP (53% versus 46%, arr 1.2; 95% CI ). No other outcomes were significantly different across the arms. 104

120 Table 10: Outcomes for patients receiving feedback plus the goal-setting and actionplanning worksheet versus feedback alone Continuous Feedback Feedback Model-based differences in quality of care outcomes outcomes +Worksheet Alone MD 95% Missing Adjusted 95% Missing (n=1832) (n=2223) CI MD CI Systolic BP 128 ± ± , , Diastolic BP 72 ±11 73 ± , , LDL 2.1 ± ± , , HbA1c%^ 7.2 ± ± , , Composite 72 ±26 70 ± , , 0 score Dichotomous outcomes RR 95% CI Adjusted RR 95% CI LDL at target 866 (47%) 1119 (50%) , , 1.06 BP at target 965 (53%) 1023 (46%) , , 1.29 BP test in 6M 1491 (81%) 1541 (69%) , , 1.26 A1C test in 6M^ 947 (71%) 1067 (65%) , , 1.21 FBG test in 24M 1596 (87%) 1918 (86%) , , 1.08 LDL test in 12M 1217 (66%) 1480 (67%) , , 1.12 ACR test in 12M^ 851 (64%) 1060 (65%) , , 1.20 Rx ASA* 552 (75%) 615 (73%) , , 1.03 Rx Statin 1309 (72%) 1686 (76%) , , 1.00 Rx ACE/ARB^ 647 (83%) 679 (84%) , ,

121 Rx Beta blocker* 440 (59%) 514 (61%) , , 1.03 Rx Insulin^ 351 (26%) 426 (26%) , , 1.04 All models adjusted for clustering. Adjusted models also controlled for baseline values of dependent variable. Legend: ^Analysis restricted to patients with diabetes, *Analysis restricted to patients with IHD, ~ Target BP 130/80 for diabetes and 140/90 for IHD and Target LDL<2 MD=mean difference, RR=relative risk, IHD=ischemic heart disease, BP=blood pressure, A1C=haemoglobin A1c, FBG=fasting blood glucose, LDL=low density lipoprotein cholesterol, ACR=albumin-to-creatinine ratio, ASA=aspirin, ACE/ARB=angiotensin-modifying agent, Rx=active prescription, CI=confidence interval, M=months After adjusting for variation at the level of the physicians, there was no further significant variation at the level of the clinic so that all models included a random variable only for the physician. Patients with no recent values for BP or LDL did not differ across arms with respect to sex, or proportion with diabetes and/or IHD. Those with missing values for BP were also similar with respect to age, but patients with missing values for LDL from the feedback plus worksheet arm tended to be older than those in the feedback alone arm (68 years versus 63 years; p = 0.001). LDL values and diastolic BP decreased slightly in both study arms over time, while systolic BP decreased only in the feedback plus worksheet arm (Table 11). Haemoglobin A1c also decreased in the feedback plus worksheet arm, but increased slightly in the feedback only arm. The proportion of patients with BP and LDL at target increased in both arms during the study period. The proportion with an LDL test within 12 months and the proportion with a statin prescribed also improved over time in both arms, but the proportion with BP measured within 6 months decreased in both arms and the proportion prescribed an anti-hypertensive (beta blocker or angiotensin-modifying agent) did not change over time except for a small increase in angiotensin-modifying agents in the feedback alone arm. Insulin prescribing increased in both arms over time, but testing of haemoglobin A1c and fasting blood glucose decreased. 106

122 Table 11: Change in outcomes over time by intervention arm, adjusted for effects of clustering * Feedback*+Worksheet* Feedback*Alone* Continuous* Mean* 95%*CI* Missing& Mean* 95%*CI* Missing& outcomes* difference* difference* Systolic*BP* \1.83' \0.68,'\ 257$ \0.96' \1.99,' 708$ 2.98' 0.07' Diastolic*BP* \1.30' \0.62,'\ 257$ \1.74' \1.08,'\ 708$ 1.98' 2.41' LDL* \0.24' \0.19,'\ 549$ \0.26' \0.32,'\ 647$ 0.30' 0.21' HbA1c%^* \0.14' \0.03,'\ 374$ 0.12' 0.02,' 604$ 0.25' 0.22' Composite*score* \3.09' \1.49,'\ ' \1.81' \3.31,'\ ' 4.69' 0.31' Dichotomous* Relative'Risk' 95%'CI' ' Relative'Risk' 95%'CI' ' outcomes* LDL*at*target* 1.33' 1.22,' ' 1.32' 1.23,' ' 1.44' 1.43' BP*at*target* 1.08' 1.01,' ' 1.08' 1.01,' ' 1.16' 1.15' BP*test*in*6M* 0.95' 0.92,' ' 0.93' 0.88,' ' 0.98' 0.99' A1C*test*in*6M^* 0.88' 0.80,' ' 0.94' 0.88,' ' 0.97' 1.00' FBG*test*in*24M* 0.98' 0.96,' ' 0.99' 0.97,' ' 1.00' 1.01' LDL*test*in*12M* 1.12' 1.06,' ' 1.10' 1.04,' ' 1.18' 1.16' ACR*test*in*12M^* 0.92' 0.88,' ' 0.95' 0.88,' ' 0.97' 1.01' Rx*ASA** 1.00' 0.97,' ' 1.03' 0.99,' ' 1.03' 1.09' Rx*Statin* 1.02' 1.00,' 1.04' ' 1.06' 1.03,' 1.09' ' 107

123 Rx*ACE/ARB^* 1.01' 0.98,' ' 1.03' 1.01,' ' 1.04' 1.06' Rx*Beta*blocker** 0.99' 0.97,' ' 0.99' 0.96,' ' 1.02' 1.03' Rx*Insulin^* 1.24' 1.16,' 1.33' ' 1.22' 1.17,' 1.28' ' ^Analysis'restricted'to'patients'with'diabetes,'*Analysis'restricted'to'patients'with'IHD,'~'Target'BP' 130/80'for'diabetes'and'140/90'for'IHD'and'Target'LDL<2'' IHD=ischemic'heart'disease,'BP=blood'pressure,'A1C=haemoglobin'A1c,'FBG=fasting'blood'glucose,' LDL=low'density'lipoprotein'cholesterol,'ACR=albumin\to\creatinine'ratio,'ASA=aspirin,' ACE/ARB=angiotensin\modifying'agent,'Rx=active'prescription,'CI=confidence'interval,'M=months' Results of the planned sub-group analyses for patients with only diabetes, only IHD, or both conditions are described in (Table 13, Table 14). The results indicate that patients with IHD are treated more aggressively than patients with only diabetes and that patients with both conditions are treated most aggressively. Mean BP and LDL values were lowest and the composite process score was highest amongst patients with diabetes and IHD. 108

124 Table 12: Outcomes for patients with diabetes but not IHD, adjusted for effects of clustering Continuous Feedback Feedback Model-based differences in quality of care outcomes outcomes +Worksheet Alone MD 95% Missing Adjusted 95% Missing (n=1049) (n=1329) CI MD CI Systolic BP 129 ± ± , , Diastolic BP 73 ±10 75 ± , , LDL 2.2 ± ± , , HbA1c 7.1 ± ± , , Composite 69 ±28 67 ± , , 0 score Dichotomous outcomes RR 95% CI Adjusted RR 95% CI LDL at target 456 (43%) 622 (47%) , , 1.04 BP at target 397 (38%) 434 (33%) , , 1.41 BP test in 6M 838 (80%) 916 (69%) , , 1.23 A1C test in 6M 742 (71%) 856 (64%) , , 1.21 FBG test in 24M 932 (89%) 1169 (88%) , , 1.07 LDL test in 12M 704 (67%) 903 (68%) , , 1.10 ACR test in 12M 650 (62%) 839 (63%) , , 1.21 Rx ASA 599 (57%) 620 (47%) , , 1.06 Rx Statin 683 (65%) 918 (69%) , , 1.00 Rx ACE/ARB 727 (69%) 898 (68%) , , 1.00 Rx Insulin 251 (24%) 333 (25%) , , 1.04 Adjusted models controlled for baseline values of dependent variable. 109

125 Legend: ^ = Analysis restricted to patients with diabetes, *Analysis restricted to patients with IHD, ~ = Target BP 130/80 for diabetes and 140/90 for IHD and Target LDL<2 Acronyms: RR = relative risk, IHD = ischemic heart disease, BP = blood pressure, A1C = haemoglobin A1c, FBG = fasting blood glucose, LDL = low density lipoprotein cholesterol, ACR = albumin-to-creatinine ratio, ASA = aspirin, ACE/ARB = angiotensin-modifying agent, Rx = active prescription, CI = confidence interval, M = months Table 13: Outcomes for patients with IHD but not diabetes, adjusted for effects of clustering Continuous Feedback Feedback Model-based differences in quality of care outcomes outcomes +Worksheet Alone MD 95% Missing Adjusted 95% Missing (n=467) (n=560) CI MD CI Systolic BP 127 ± ± , , Diastolic BP 71 ±11 71 ± , , LDL 2.0 ± ± , , Composite 73 ±25 71 ± , , 0 score Dichotomous outcomes RR 95% CI Adjusted RR 95% CI LDL at target 218 (47%) 276 (49%) , , 1.16 BP at target 330 (71%) 362 (65%) , , 1.19 BP test in 6M 372 (80%) 364 (65%) , , 1.42 FBG test in 24M 284 (61%) 322 (57%) , , 1.14 LDL test in 12M 99 (21%) 149 (27%) , , 1.26 Rx ASA 366 (78%) 472 (84%) , , 1.02 Rx Statin 329 (70%) 399 (71%) , , 1.00 Rx Beta blocker 249 (53%) 332 (59%) , , 1.05 Rx ACE/ARB 218 (47%) 276 (49%) , , 1.06 Adjusted models also controlled for baseline values of dependent variable. Legend: ^Analysis restricted to patients with diabetes, *Analysis restricted to patients with IHD, ~ Target BP 130/80 for diabetes and 140/90 for IHD and Target LDL<2 MD=mean difference, RR=relative risk, IHD=ischemic heart disease, BP=blood pressure, A1C=haemoglobin A1c, FBG=fasting blood glucose, LDL=low density lipoprotein cholesterol, ACR=albumin-to-creatinine ratio, ASA=aspirin, ACE/ARB=angiotensin-modifying agent, Rx=active prescription, CI=confidence interval, M=months 110

126 Table 14: Outcomes for patients with both DM and IHD, adjusted for effects of clustering Continuous Feedback Feedback Model-based differences in quality of care outcomes outcomes +Worksheet Alone MD 95% CI Missing Adjusted 95% CI Missing (n=249) (n=255) MD Systolic BP 127 ± ± , , Diastolic BP 69 ±11 71 ± , , LDL 2.1 ± ± , , HbA1c 7.3 ± ± , , Composite score 79 ±20 77 ± , , Dichotomous RR 95% CI Adjusted 95% CI outcomes RR LDL at target 153 (61%) 167 (65%) , , 1.08 BP at target 184 (74%) 169 (66%) , , 1.28 BP test in 6M 222 (89%) 196 (79%) , , 1.24 A1C test in 6M 180 (72%) 175 (69%) , , 1.30 FBG test in 24M 232 (93%) 235 (92%) , , 1.09 LDL test in 12M 184 (74%) 191 (75%) , , 1.10 ACR test in 12M 179 (72%) 185 (73%) , , 1.20 Rx ASA 198 (80%) 187 (73%) , , 1.07 Rx Statin 203 (82%) 223 (87%) , , 1.04 Rx ACE/ARB 209 (84%) 215 (84%) , , 1.03 Rx Beta blocker 172 (69%) 161 (63%) , , 1.07 Rx Insulin 91 (37%) 77 (30%) , , 1.10 All models adjusted for clustering. Adjusted models also controlled for baseline values of dependent variable. 111

127 Legend: ^Analysis restricted to patients with diabetes, *Analysis restricted to patients with IHD, ~ Target BP 130/80 for diabetes and 140/90 for IHD and Target LDL<2 MD=mean difference, RR=relative risk, IHD=ischemic heart disease, BP=blood pressure, A1C=haemoglobin A1c, FBG=fasting blood glucose, LDL=low density lipoprotein cholesterol, ACR=albumin-to-creatinine ratio, ASA=aspirin, ACE/ARB=angiotensin-modifying agent, Rx=active prescription, CI=confidence interval, M=months Per-protocol analyses Half of physicians in the intervention arm completed the worksheet (10/22, 45%); of those, half responded only once. Restricting analysis for the primary outcomes to the 10 physicians who completed the intervention worksheet indicated non-statistically significant improvement in the feedback plus worksheet arm for systolic BP (126 versus 128, adjusted mean difference -0.8; 95% CI -3.1, 1.5) and for the composite score (73 versus 70, adjusted mean difference 2.6; 95% CI -2.0, 7.2), but a slight decrease in LDL (2.1 versus 2.0, adjusted mean difference 0.1; 95% CI 0.0, 0.2). The most common goals set by participants in the worksheet for their patients with diabetes were for achievement of target BP (i.e., increase the proportion of patients meeting BP target, 3 times) and increasing testing rates of urinary albumin-to-creatinine ratio (3 times). The most common goals for IHD were for achievement of target LDL (3 times) and increasing prescription rates of ASA (5 times). Patients belonging to the 10 participants who completed the worksheets more often achieved BP targets but the effect was not statistically significant (52% versus 46%, arr = 1.11; 95% CI 0.97, 1.27). Testing for urinary albumin was similar between arms (66% versus 65%, arr = 1.00; 95% CI 0.86, 1.17), as was achievement of LDL targets (48% versus 50%, arr = 0.99; 95% CI 0.90, 1.09). Prescribing rates of ASA was higher amongst participants who completed the worksheets, but the model-based difference was not statistically significant, as baseline rates of ASA prescribing were also higher in this group (67% versus 57%, arr = 0.99; 95% CI 0.95, 1.04). 112

128 5.4 Discussion We found no difference in the primary outcomes of blood pressure and cholesterol levels and no difference in the composite process score when providers were given feedback plus a goal-setting and action-planning worksheet compared to feedback alone. While it was a secondary outcome and should be interpreted cautiously, the intervention did result in improved glycemic control to a similar extent as many other complex QI strategies for diabetes. 21 We also observed that blood pressure and cholesterol improved in both arms, as well as half of the process outcomes, which emphasizes the importance of controlled studies when testing strategies aiming to improve quality of care. It remains plausible that completing a goal-setting and action-planning exercise could enhance the effectiveness of feedback. 131 Unfortunately, only 10 out of 22 physicians in the intervention arm completed the intervention worksheet and only 5 of these completed it more than once. Poor compliance (with minimal supports) limited our ability to test the effects of action planning in this pragmatic trial. Half of the goals set by active participants were behavioural (what will I do) and half were outcome-oriented (what would I like to happen as the result of what I do). For actions plans to be most effective, they must very specifically relate to behavioural goals, not outcome goals. 21 Thus, it would seem that half of participants who completed the worksheet did so ineffectively and there is a need to explore how to make actionplanning activities more salient and usable. More active, practice-based supports may be needed to implement the development of goals and action plans. For instance, one RCT found that feedback reports plus structured peer interactions in which goals and action plans for improvement were discussed was more effective than feedback alone. 61 A recent Canadian cross-sectional study showed that data management support for patient identification and recall plus assistance by allied health providers with standardized testing and prescribing was associated with improved quality in primary care. 73 To result in behaviour change, those receiving feedback must (i) be dissatisfied with their performance, (ii) meet a threshold level of self-efficacy for improvement, and (iii) be committed to the goal. 51, 164 In our separately reported embedded qualitative evaluation, 133 we found that self-efficacy was low, as many practices lacked the necessary quality improvement infrastructure 113

129 to take action. For instance, no one was responsible for searching the EMR to identify patients who may require reassessment. Our qualitative work also found that participants were not highly committed to achieving the targets described in the feedback reports. There was uncertainty regarding the impact on patient outcomes of achieving targets perceived to be aggressive. Many expressed concern that practice-level quality improvement efforts would be at odds with their attempts to achieve patient-centered care. It would appear that although Canadian family physicians generally agree with and accept guideline-based best-practice targets for diabetes and IHD, 183 achievement of best-practice targets for chronic disease management was not perceived as urgent compared to other tasks, especially given the relatively high quality of care already achieved. Yet even if mean performance was acceptable, many patients stand to benefit from improved processes of care. In such settings, to increase goal commitment it may be necessary to first address limited self-efficacy by providing more active supports for quality improvement, 184 as there is evidence that self-efficacy influences goal commitment. 51 Recognizing that feedback alone is sometimes not enough to change provider behaviours, further improvements have been sought by pairing feedback with intensive co-interventions, such as academic detailing 65 or practice facilitation. 185 In the Cochrane review, pairing educational outreach with audit and feedback was found to increase desired professional behaviour. 162 However, these intensive interventions are costly and more cost-efficient approaches may exist. The particular nature of the feedback intervention used in this study may have played an important role in the poor uptake of the intervention worksheet. Qualitative work conducted in the Veterans Affairs health system in the USA also indicates that high performing healthcare organizations tend to deliver feedback with more actionable information. 63 It is possible that the participants did not feel that achieving higher scores on their feedback was achievable because only aggregate data was provided. 186 It is also possible that concerns regarding data validity allowed participants to resolve any cognitive dissonance arising from the results provided in the feedback without needing to commit resources to improvement. 41 For instance, the explanatory notes accompanying the reports described that if relevant tests were conducted by specialists but were not received into the EMR in a standardized format they would not be included in the feedback. In addition, the presence of multiple competing priorities is known to mitigate achievement of particular guideline recommendations, 187 and this may be highly appropriate for patients with significant symptoms from concomitant illness (e.g., severe depression, cancer) or reduced life expectancy. The Cochrane review indicated that feedback was less effective when 114

130 targeting many indicators reflecting chronic disease management than when targeting a specific behaviour, and feedback intervention theory suggests that feedback should direct attention to a specific task in order to most reliably change behaviour. 46 It has also been observed that feedback may be more effective when participants choose standards, 163 and when presented by senior colleagues, 162 whereas externally generated feedback focusing on a multitude of guideline-based best practices were used in this study. Therefore, the feedback may have been more salient - and the intervention worksheet may have had greater impact - if it focused on quality indicators chosen by participants to be high priority, and delivered by carefully selected opinion leaders 188 with clear and readily achievable tasks for improved scores. Some limitations in this study warrant further discussion. The lack of a pure control group limits our ability to comment on the impact of this particular audit and feedback intervention on quality of care. However, our approach was necessary because participants expected something in return for contributing data. Furthermore, feedback is known to work for these outcomes 165 such that our interest was in determining whether a simple enhancement could increase feedback effectiveness. Although we used minimization to achieve baseline balance successfully for the primary outcomes, differences in cluster-level characteristics remained. In addition, the study analyst had access to the allocation list; this non-blinding could theoretically create bias, but the same validated algorithms were used to assess outcomes for each study group. We also acknowledge the potential for measurement bias as investigations were counted only if results were available in the EMR and tests performed by specialists may be missed. For outcomes related to investigations and treatments, data in EMRALD compare well with (and often out-perform) administrative databases. 189 While the trial should balance reasons for misclassification or missing data, we observed that those without a recent LDL test tended to be older in the intervention arm. We took a pragmatic approach to intervention delivery, limiting the number of reminders and supports to mimic expected conditions if the intervention were to be widely implemented. This may explain the limited completion rate of the worksheet and raises a question about the role of pragmatic health services trials when evaluating new interventions. In this case, more data regarding how to support implementation would have been useful prior to embarking on a trial with this type of design. We did embed a qualitative evaluation to explore this, but participants focused on the usefulness of the particular audit and feedback intervention used in this trial rather than the goal-setting and action-planning worksheets. 183 It is also 115

131 important to note that the secondary, per-protocol analyses are at risk of bias in favour of the intervention. In addition, a number of factors may have limited our ability in this study to find differences between intervention arms. First, while this study did include thousands of patients, they were clustered within only 14 clinics and the intra-cluster correlations for disease-level primary outcomes were larger than expected from pilot data. Second, physicians voluntarily provide data to EMRALD and many participating clinics were involved in other quality improvement interventions. Thus, these clinics may be more innovative and may also be achieving a higher level of evidence-based care than most other primary care providers, potentially decreasing both generalizability and the likelihood of finding an effect in this study. Furthermore, we observed that the composite process score was highest amongst patients with both diabetes and IHD indicating that participating providers appropriately intensified monitoring and management in patients at greatest risk and suggesting the possibility that further gains may be limited by a ceiling effect. Third, risk for type 2 error is exacerbated in trials comparing similar interventions (ie., head-to-head trials) where anticipated effect sizes are not expected to be large. Finally, differences between groups may also have been more difficult to identify if goal-setting and action-planning aspects of the intervention were duplicated by the revised CME surveys mandated by the College of Family Physicians. During the trial, 52% of physicians (14/27) in the usual feedback group and 54% (12/22) in the feedback plus worksheet arm completed and returned at least one CME survey. Unlike the first iteration of CME surveys, which asked participants what have you learned, the revised surveys explicitly asked participants to make a decision about their practice and to identify how to integrate the decision into practice. Though less specific or directive than the intervention worksheets, these questions may have similarly prompted participants to set goals and develop action plans. In conclusion, we found no effect of adding a theory-informed goal-setting and actionplanning worksheet to an audit and feedback intervention. Unfortunately, passive dissemination of this worksheet led to inadequate engagement with the intervention. In the context of primary care practices with minimal quality improvement infrastructure, CME credit alone may not be enough incentive to encourage engagement in goal-setting or action planning activities. To maximize the impact of audit and feedback and to ensure that QI in primary care is prioritized, relevant stakeholders, including professional colleges, associations, and health system payers, should consider the need for further supports to carry out practice-based QI. 116

132 6 Paper 3: My job is one patient at a time : perceived discordance between population-level quality targets and patient-centered care inhibits quality improvement 6.1 Background Audit and feedback, defined as a summary of the clinical performance of healthcare provider(s) over a specified period of time, 131 is a widely used quality improvement (QI) strategy. 116 A recently updated Cochrane review, with 140 randomized trials of audit and feedback conducted across many clinical conditions and settings, found that it increases provider compliance with desired practice by 4.3% (median; interquartile range 0.5% to 16%). 131 Pressures to increase accountability in primary care often result in audit and feedback initiatives led by agencies external to a family practice. However, previous research found that general practitioners perceived external quality programs to be an imposition, while internal QI was perceived to be a professional obligation. 190 In this study, we sought to understand the perceived usefulness of externally generated feedback amongst family physicians and perceived barriers and facilitators to using audit and feedback to improve processes of care and patient outcomes. We also explored how to optimize the design of audit and feedback interventions to be most actionable for family physicians. 6.2 Methods This qualitative study was embedded within a pragmatic cluster-randomized trial in which all family physician participants received feedback reports. The protocol for the overall project has been previously reported. 169 The study was approved by the Sunnybrook Research Ethics Board. 117

133 6.2.1 Setting and Context: Ontario has a single-payer system, in which there is no access fee for physician visits or hospitalizations. Pharmaceutical costs are covered for inpatients, patients on social assistance, and those over age 65. Over the last decade, Ontario has implemented substantial primary care reforms, with the majority of family physicians moving from independent, fee-for-service models to group-based models with partial capitation-based payment, requiring physicians to roster patients to their practice. 102 All participants in this study roster their patients and also benefit from funds from the Ministry of Health to support allied healthcare providers in their clinics in models conceptually similar to the patient-centered medical home. 191 The Ministry of Health has identified improvements in diabetes care as a key priority. In 2010, the Ontario Ministry of Health provided family physicians with reports summarizing the proportion of patients with diabetes receiving guideline-concordant care 192 and new provincial legislation 193 indicates that similar initiatives for other conditions will be developed in an attempt to improve accountability. However, most Ontario primary care practices had not experienced recurrent clinical audits or performance feedback at the time of this study Participants: All fifty-four family physicians in the overall study contributed data from their electronic medical records (EMRs) to the Electronic Medical Record Administrative data Linked Database (EMRALD). The patient records in EMRALD are de-identified and patients with diabetes can be identified using validated algorithms. 194 In this study, EMR records of patients with diabetes were mined for the presence of guideline-recommended processes and/or treatments. 118

134 6.2.3 Intervention: Participating family physicians received feedback reports detailing their scores for nine different evidence-based quality targets (Figure 11). An achievable benchmark of care representing the performance of the top 10% of peers was calculated for each quality indicator, based on research indicating that this comparator led to greater improvements in care than mean performance. 53 These EMRALD feedback reports described only the overall proportions of patients meeting targets; no patient-specific data are included. The reports were delivered to each physician at their practice by courier from the investigators in envelopes marked confidential, along with reflection worksheets that could be completed for continuing medical education credits. Participants, like all other family physicians in the province, also received a feedback report from the Ontario Ministry of Health regarding their patients with diabetes, which did include patient-specific data (i.e., detailing whether or not each given patient was overdue for a given test) but only for three process measures (Figure 12). At the time of this study, participants had each received one report from both EMRALD and from the Ontario Ministry of Health. This provided an opportunity to determine whether and how family physicians used the feedback reports and to explore preferences regarding design. 119

135 A1C 7.0 % A1C tested in 6M Practice Profile, Diabetes (Type 2) 45% 45% 75% 75% BP < 130/80 20% 40% BP tested in 6M 55% 75% Rx ACE / ARB 50% 70% LDL % 80% LDL tested in 12M 25% 65% Rx Statin 60% 70% ACR tested in 12M PHYSICIAN ID#: #### 65% Your Practice Top 10% 85% Approximately %% of your rostered adult patients have diabetes, and %% of these patients also have ischemic heart disease Overall in this study, %% of rostered adult patients have diabetes, and %% of these patients also have ischemic heart disease Your diabetic patients are ## years old on average and are %% male. All diabetic patients in the study avg. ## yrs and are %% male. Targets Your Practice Top 10% A1C 7.0 % 45% 75% "Top 10%" = weighted average for the 10% of physicians A1C tested in 6M 45% 75% who had the highest results in each category BP < 130/80 20% 40% BP tested in 6M 55% 75% (This data is based on your most recent data upload: MONTH,YEAR) Rx ACE / ARB 50% 70% LDL % 80% LDL tested in 12M 25% 65% Rx Statin 60% 70% ACR tested in 12M 65% 85% ACR = albumin creatinine ratio (test for microalbuminuria) Figure 11: Study feedback report provided to participating family physicians with aggregated proportion of patients meeting nine quality indicators 120

136 Figure 12: Ontario Ministry of Health feedback report with patient-specific data provided to all family physicians across the province for three process measures Data Collection Semi-structured, in-depth, individual interviews were conducted by a single interviewer (NI) between October and December To inform question formation and sequencing, pilot interviews were conducted prior to recruitment. The interview guide started with questions to build rapport and define the study context. After this, open-ended questions were asked about the feedback reports to elicit themes in a non-threatening fashion. Probing questions followed to 121

137 pursue areas of particular interest and/or issues brought up by the participant. The guide was iteratively adapted as interviews were conducted. The questions in the interview guide were informed by the clinical and research experience of the multi-disciplinary team of investigators and by relevant behavioural and psychological theories, 43, 46, 51, 63, 123, 195, 196 which informed probing questions about likely barriers to the use of feedback in the clinical setting and moderating factors in design of the feedback. Interviews were conducted at the time and place of the participants choosing and were recorded using a transcription service to produce verbatim electronic transcripts. We used stratified purposeful sampling, 197 selecting participants with those features reported as relevant in previous studies, seeking informational rather than probabilistic representativeness. For instance, guideline adherence and quality of care may be related to years in practice 198 and physician gender, 199 so variety was sought in those factors. Additionally, we identified participants with various levels of baseline performance across all indicators, because this is an important predictor of feedback effectiveness. 131 After themes were established based on the analysis of the first round of interviews (see below), we used snowball sampling to seek out participants who might challenge our early findings. Specifically, we asked participants to recommend potential interviewees who were either highly involved in QI or who were particularly disinclined to participate in QI. Early findings were explored with these participants to search for disconfirming evidence and to crystalize interpretations. Although we sought variation in certain characteristics, the sample was similar in many other ways. All participants were EMR-using family physicians that worked in team-based practices with access to allied health care providers, and all consented to receiving feedback reports as part of the overall project. Given the targeted nature of the questions, we expected that saturation would be accomplished with approximately twelve interviews. 200 To account for time away from patient care, we provided a $75 honorarium. 122

138 6.2.5 Analysis We used the framework approach, 201 aiming to accurately reflect the original accounts of the participants through the use of inductive techniques, directed by the a priori goals and objectives for the project. We tracked the identification of themes along with dates of interpretations to provide an audit trail documenting the analysis. 202 NVivo software was used to assist with the data analysis. We established an initial index of themes based on a priori defined issues of interest and combined these with a data-driven coding framework developed after analyzing the first three interviews. For the next four interviews, two members of the research team independently identified key findings arising from the data. The results were then discussed with a third investigator to gain consensus on key initial findings. We pursued multiple coding in this way to provide reassurance that all possible themes were given consideration. 94 In keeping with the constant comparison method, we revised the coding framework and the interview guide as the data collection proceeded. For example, we found that participants focused on the nature of the performance targets and their professional role and self-efficacy with respect to quality improvement, rather than the specifics regarding how the feedback is designed or delivered. Thus, the final coding framework incorporated few of the a priori topics. After seven interviews, consensus was reached regarding the descriptive codes. In the following interviews we sought disconfirming evidence to both clarify the findings and ensure saturation. Findings were coalesced into themes and then organized within four topic areas. To illustrate, the topic personal barriers or facilitators had three major sub-themes: competing priorities, perceived roles of patient and provider, and QI interest and expectations. Other topic areas included initial response and reaction; organizational barriers or facilitators; and feedback design preferences. As per the framework approach, key findings from each interview were placed into a matrix for each topic area (available upon request). Each participant had their own row with key quotes or summaries listed under the relevant sub-theme. This facilitated the identification of central themes and patterns across participants, within and across the related topics. After multiple readings and discussion amongst the entire team of investigators the results were ultimately grouped into three major topics 1) usefulness of feedback for systematic chronic 123

139 disease management, 2) reported barriers to QI efforts in response to feedback, 3) preference for intervention design to support QI. 6.3 Results Participant characteristics Data saturation was reached after twelve interviews. Interviews lasted a median of 50 minutes (range 37 to 70 minutes). The participants varied with respect to sex, years experience, location, and practice size (see Table 15). Three participants were particularly high performers; two others had relatively low proportion of diabetes patients meeting targets. Snowball sampling led to inclusion of one participant who was highly skeptical of the benefits of any practice-based innovations and two participants highly engaged in practice-based QI activities. Table 15: Characteristics of twelve participants selected for interviews and of fifty-four potential participants Characteristic Interview Participants Trial Participants Sex Male 8 (67%) 30 (56%) Female 4 (33%) 24 (44%) Years in Practice (25%) 14 (26%) (33%) 18 (33%) > 25 5 (42%) 22 (41%) Location Rural 7 (58%) 26 (48%) Urban 5 (42%) 28 (52%) Practice Size < 600 patients 2 (17%) 15 (28%) patients 4 (33%) 18 (33%) >1000 patients 6 (50%) 21 (39%) 124

140 6.3.2 Usefulness of the feedback for systematic chronic disease management None of the participants reported that they found the feedback particularly useful. Participants commonly reported that they intended to improve performance by being more mindful of the relevant targets during patient encounters. However, no participants reported using the feedback to set specific goals for improvement or action plans for reaching these goals. Even when prompted, most participants could not envision ways for the practice to facilitate proactive chronic disease management (i.e., as in the chronic care model 203 ). A few proposed the concept of developing disease-based patient-registries to check the data and then contacting those requiring action and/or using reminders in the EMR. However, none had actually followed through on these ideas during the 8-12 week interval between receiving the feedback report and participating in the interview. Box 2: Usefulness of the feedback reports for QI It s like getting a D and really, we re all type A Do you know what I do with bad report cards? I filed it away until you came today because I didn t want to look at it again. (Interview 11) It wanted me to sign that I was going to improve x, x, and x, over the next period of time and so on. I just said, no way, I am not buying into this guilt trip. I tried and I m not perfect. I m going to continue to try but I m not going to be burdened with extra guilt. (Interview 9) When I was seeing my diabetic patients, I spent more time, you know, making sure I was paying attention, you know, to the things I m supposed to pay attention to, which I thought I did, but obviously could improve upon (Interview 4) Barriers to QI efforts in response to feedback The most commonly reported barrier to using the feedback for QI related to concerns regarding the validity of the data used to generate the reports and the ability to leverage EMR for QI. Providers didn t trust the data even though (or possibly because) it came from their own 125

141 EMR. To act upon reports providing only aggregate data and not individual patient names, physicians had to manually identify patients for whom action was needed. However, few were motivated and/or skilled enough to check the data or to generate lists by conducting EMR-based searches. Box 3: Barrier: Participants challenges leveraging the EMR for action A number of my patients with these diseases aren t captured in the EMR (Interview 5) I'm the only one that learned how to create my own searches so I guess I have a greater appreciation for what programs to do and as a result also probably I'm among the most motivated for having thorough information in the patient profile 'cause I know what it can get me. (Interview 10) The problem is going back to try to verify that is really time consuming. It just takes a lot of work to go and try to hunt down if that is really true and what is going on (Interview 8) Another important barrier to using the feedback reports for QI related to the tension between standardized targets for populations and patient-centered care. Many endorsed a desire to practice in a patient-centered fashion and felt that population-level targets or QI initiatives were in conflict with this ideal. They expressed professional pride in judiciously applying targets and guidelines and worried that standardizing care would result in disease-oriented rather than person-oriented decisions. Although the performance targets were based on well-established guidelines with high-level evidence, participants described concerns with the measurement of their performance based on these targets. Participants focused on patients that should be excluded because targets would not be appropriate (e.g. elderly patients with multi-morbidity). A few participants expressed uncertainty about whether their (potential) efforts toward QI in response to feedback would translate into meaningful differences for their practice or for their patients. Concerns were also raised about being judged on outcomes beyond the physician s control. 126

142 Box 4: Barrier: Tension between population-level targets and individualized clinical decisions It talks about whole populations as opposed to the one individual and I think my approach to this job is the one person at a time (Interview 2) We have to look at the whole person... What do we stand for as family doctors, you know? We are there to walk, to make life s journey as medical professionals with our patients and we, I think are going - I m tearing up - We are doing them a big disservice by buying into this it is just short-sighted I want to explain myself I guess and say, Listen, my numbers aren t that good, but I m a good doctor. (Interview 3) So, I mean this tells us what we as physicians should do as technicians. It doesn t tell us what we as physicians should do as motivators I think this is great, I just think it s incomplete (Interview 4) The last thing I want is to have one of my older patients become hypoglycemic, fall, break their hip because they re on some (drug) which is really totally inappropriate, because some doctors try to meet an unrealistic target. So we have to be really careful of the clinical practice of medicine is different from the guideline practice of medicine. It s really important not to get caught up on guideline-itis just treat your patients properly (Interview 5) They are all different, they all have their own financial supports and home supports and degrees of motivation, and it is a real tricky task. We are kind of stick handling on what is the most important thing and you are trying to practice patient-centered medicine too. So you are trying to be they come to you and they are really worried about their daughter and, Okay, we ll talk about your daughter another day, let s talk about your LDL (Interview 8) A third key barrier related to the challenges of priority setting in primary care. Participants expressed a sense of being overwhelmed and unable to fully balance demands on their time at the clinical, organizational, and personal levels. From a clinical perspective, the frequent presence of acute issues that interfere with chronic disease management was expressed as an inevitable problem. In such cases, the management of chronic disease was deemed to be clinically important, but other patient-problems were the priority. A few of the participants more inclined towards QI discussed setting priorities at the practice level, noting other ongoing projects or programs that limited their ability to galvanize support or direct attention toward addressing the gaps identified by the externally-produced feedback. 127

143 Box 5: Barrier: Challenges with priority setting in primary care How much time do you want your doctor devoting to that because the more time I m devoting to my computer extractions the less time I am to phoning the patients or bringing the patients in and seeing them to meet these targets. You can t have your cake and eat it too. (Interview 1) Really, the time the five of us physicians here can get together, we re dealing with contract issues and leaky roofs, paint on the wall, contract signing, nursing crisis, new staffing. The five of us are not at the point where we re dealing with clinical stuff other than, you know, the corridor consult. (Interview 6) For a while we ve been trying to, at our business meetings, to set aside time to have a little data or a reminder about how to use one thing or another [in the EMR] and then we haven t been having that, it hasn t been showing up on the agenda. (Interview 10) Preferences for intervention design to support QI The participants varied with respect to their stated preferences for other aspects of the intervention design. Despite being EMR users, most preferred paper-based reports; participants reported that they might not view feedback available on a website. In terms of frequency, the participants wanted enough time to improve the outcomes prior to the next report. However, a few participants foresaw a near future when reports for various diseases would provide more information than they could deeply reflect upon. To make the reports manageable, some requested only summary information, with the capacity to access more details as their time or interest-level increased. Many suggested an emphasis on high-risk patients who were overdue for visits or who would clearly benefit from additional or more intensive care. 128

144 Box 6: Desire to focus feedback on higher-risk patients Visits within the last two years or the last six months, so that you can see, ok these are the patients who are not coming in. (Interview 3) So if you had A1c greater than 8 and I saw how many patients or what percentage of my diabetic population was over 8 then you d probably catch my attention and I d be searching through you re beyond this number, how far above blood pressure are you? (Interview 2) You know what I would really like is the number of people, if there was and this is a very different statistic to get but the number some way of showing a change has occurred over time from A1c s somewhere in the 8 s to the 7 s or some number from the 9 s to the 8 s... I wouldn t want more people getting worse, that is for sure. (Interview 8) There s the group I would target despite what else is going on socially in their lives, most of the time. You know, if their A1c is over 8, in my mind, those are people I would target but there s the subtler differences of the LDL of 2.1 and 2.2 who, often, there s going to be things at a higher priority. (Interview 6) Even participants inclined towards QI felt it was not the family doctor s role (or that it would add too much work) to initiate changes to practice-based processes. Participants expressed a desire for complementary interventions that would provide support to take action, including both technical assistance for managing clinical information and administrative assistance to determine and implement the recommended QI activities. Many welcomed the idea of a follow up phone call by a supportive colleague to review the report and discuss explanations for the results and strategies for improving patient care. Box 7: Desire for additional resources to manage chronic disease initiatives There probably should be like an information management person affiliated with every group-type practice. You need like an epidemiologist onsite whose job it is to generate information like this, provide feedback to patients and providers about how they are doing, and to track the data. (Interview 8) We hired someone a couple half days a week to help us mine data, but we discovered a lot more work associated with it, because once we mine the data and we need to communicate to our patients, then we need someone to call the patients as well. (Interview 12) It becomes so overwhelming unless there are staff resources available to do that for me, to manage the data, mine the data, and then dedicated time to sit down with my team, with my nurse practitioner, with my pharmacist, with my dietician and just talk about it. And maybe the mental health worker and say, okay, tell me what the data look like, tell me where we re not doing so well how can you help me to do better, how can you help me to get better marks? (Interview 7) 129

145 6.4 Discussion Increased awareness of suboptimal performance usually resulted in the intention to try harder to do more during each patient visit, rather than work smarter by implementing point of care reminders or initiating systems to identify and contact patients for reassessment. Participants reported that they welcomed the feedback, yet the reports often generated strong emotive responses where-in participants defended their position and their profession. Such findings help to explain the small to moderate effects generally observed in randomized trials of audit and feedback. 131 As noted in previous work in primary care, we found that family physicians struggle with integrating QI concepts into their practice. Previous studies in primary care also found that systematic implementation of QI occurs slowly Similar to previous primary care studies, 198 participants in this study reported discordance between patientcentered ideals (tailored, specific care) and quality improvement interventions (systematic approach, population-level metrics). Our finding that not all quality targets or guideline recommendations are regarded as equally important by family physicians for measuring primary care performance echoes earlier work with Ontario family physicians. 207 The salience of feedback may be increased if targets reflect the priorities of the family physician. 208 For example, more holistic measures of quality in primary care are available that include indicators for access and patient-centeredness 209 and may be more fitting with the person (rather than disease) focus subscribed to by family physicians. 106 Ideally, all six Institute of Medicine quality domains would be covered: safety, effectiveness, patient centeredness, timeliness, efficiency, and equity. 2 The family physicians were challenged by multiple competing priorities. Goal-conflict has been shown to be a significant predictor of whether professionals follow through on accepted guideline recommendations 187, 210 and consideration of competing demands provides insight into what is commonly deemed clinical inertia. 211 One way to partially address legitimate competing priorities may be to heed our participants requests that feedback be tailored to focus on manageable numbers of higher-risk patients needing (semi-) urgent action. 137, 212 One previous trial found that feedback reports stratified by risk levels was only marginally more successful for improving management of hypertension

146 Just as patients cannot focus on chronic disease management when they have unstable shelter, providers cannot focus on QI when their office resources are not established or available. For Patient-Centered Medical Homes to achieve their potential for providing communityoriented primary care, 213 they require the human and technological resources to practice population-health type management. 214, 215 In our study, most physicians were unable to personally leverage their EMR data to facilitate QI. Indeed, most EMRs in use in primary care in the U.S. or in Canada are not yet truly functional for QI. 216, 217 Another qualitative study in Ontario found that even multidisciplinary primary care teams failed to take action upon receiving feedback reports due to a lack of performance management skill development. 218 While this expertise develops, external supports may be needed to leverage available data to identify important gaps in care and to work with primary care providers to identify changes that would help them to achieve their goals. Our findings also suggest that those designing audit and feedback interventions need to think carefully about precise behaviours they want the feedback to provoke and be sure that the intervention provides support to operationalize those behaviours. Participants in our study were relatively early-adopters of EMR and most practices were also involved throughout the study in extraneous quality improvement initiatives. If these physicians and practices did not act upon the feedback due to discomfort with the targets or due to lack of resources, other primary care providers may be even less likely to act upon externally generated feedback. Nevertheless, we acknowledge that transferability of findings from this qualitative study of purposively sampled Ontario-based family physicians working in EMRusing multidisciplinary primary care practices is uncertain. It is also possible that repeated exposure to feedback over time could lead to different responses amongst the participants. In conclusion, we found that family physicians did not readily act upon the feedback reports for a number of reasons (Table 16). For QI champions, this was generally due to competing organizational-level priorities; these participants knew what would be necessary but were busy implementing other initiatives. The rest of the participants struggled with patient-level (and personal-level) priority setting and focused on potential flaws in the data and/or targets used in the feedback. Such participants perceived minimal utility in knowing the aggregate proportion of patients reaching guideline-based targets, believing that patients are unique, requiring tailored, patient-centered care. 131

147 Table 16: Selected barriers and suggested areas for future research when conducting audit and feedback for quality improvement in primary care Barriers identified Discordance between patient-centered ideals and quality improvement goals Competing priorities and goal-conflict Lack of technical expertise or human resources dedicated to quality improvement Areas for future research Holistic measures of quality covering all domains effectiveness measures must have patient-level data Provide data regarding areas of high priority and focus on improvement for higher risk patients External support to manage data and support quality improvement activities, while developing capacity in-house For audit and feedback interventions to lead to changes in the behavior of family physicians, it is necessary for the content of the feedback to align with the patient-centered priorities of the family physician. Leveraging feedback to proactively identify and contact highrisk patients who might benefit from clinical assessment was considered desirable, but did not often occur due to a lack of QI infrastructure. This includes both technical expertise and dedicated human resources committed to QI. Therefore, it is necessary to carefully consider the abilities and resources of the primary care practice; if adequate QI infrastructure does not exist, co-interventions should be delivered with the feedback to facilitate systematic, sustainable changes. Further research should methodically test how to best combine and implement cointerventions with audit and feedback. 132

148 7 Overall thesis discussion 7.1 Summary of findings In this thesis, key effect modifiers with respect to the design and delivery of A&F were identified through a systematic review and meta-regression (Section 4). Next, a scalable method to operationalize one such intervention component was tested in a cluster-randomized trial (Section 5). Finally, the roles of contextual barriers and facilitators as well as the particular behaviour changes targeted by A&F were explored through qualitative interviews (Section 6). Below, the main findings from each of these studies and the contribution to the literature are summarized. This is followed by an assessment of methodological strengths and limitations. Finally, the application of these findings to routine clinical audits (e.g., conducted by primary care clinicians for quality improvement in their practice) and to system-level A&F initiatives (e.g., conducted by policy makers to achieve dual aims of accountability and effectiveness) is discussed. Throughout the discussion, I describe lessons learned and identify opportunities for further study that might lead to more effective A&F specifically and KT in general Effectiveness of audit and feedback is partially explained by how it is designed and delivered The meta-analysis findings (Table 3) indicate that the median effect size for A&F is a 4.3% absolute improvement in desired behaviour by health professionals (IQR 0.5%-16%) for dichotomous outcomes. For continuous outcomes, the weighted median adjusted change relative to baseline control was a 1.3% increase in compliance with desired practice (IQR 1.3% to 23.2%). If the characteristics differentiating successful A&F interventions (e.g., those in the upper quartile, achieving >16% absolute improvement) from unsuccessful A&F interventions (e.g., those in the lowest quartile, achieving <1% absolute improvement) could be ascertained 133

149 and used to inform future KT initiatives, important benefits on a population-level could be achieved. Determining how to optimize the delivery of A&F to maximize the return on investment is critical to advancing the field of KT science. Building on the 2006 Cochrane review of A&F, which suggested that feedback intensity was associated with effect size, we used meta-regression to identify individual variables posited to make feedback more or less effective. The results indicated that feedback is more effective when presented both in-person and in writing than when using only one modality and when the source is a supervisor or respected colleague rather than when it comes from an investigator/unknown source. The meta-regression also indicated that feedback was least effective when the source was a regulatory body, in keeping with qualitative findings that feedback perceived to have potential punitive consequences appeared less effective than a supportive approach. 48, 63 Feedback intervention theory 46 suggests that this may be because such initiatives could provoke distress and potentially distract attention from the specific task requiring change. In addition, the meta-regression indicated that repeated feedback cycles led to greater improvements than once-only feedback. This may be partly because recipients are more likely to perceive the data as relevant and accurate when it is delivered closer in time to their own performance. 70 The meta-regression also found that goals and action plans enhance the effectiveness of A&F, building upon evidence that goals make feedback more salient 51, 53 and correct solution information can help focus recipients attention more productively on the desired task. 138 However, of 140 trials in the review, only 15 studies (11%) included explicit goals and only 45 (32%) included clear action plans for improvement as components of the A&F intervention. There remains a need to directly compare different approaches operationalizing these findings. For instance, the evidence is mixed regarding whether and how to involve participants in setting 56, 57, 151 goals or targets during an audit. Like the previous Cochrane review of A&F, this update confirmed the inverse relationship between baseline performance and effectiveness size. However, unlike the previous version and in contrast to the findings of other reviews 155 we found that multifaceted interventions tended to achieve greater effect sizes than interventions with A&F alone. This is best exemplified in the comparison of A&F alone versus A&F plus educational outreach: for

150 studies with dichotomous outcomes, there was a weighted median adjusted 1% absolute increase in desired practice (IQR -1.1% to 5.1%); for 4 studies with continuous outcomes, the weighted median adjusted change relative to baseline control was 27% (IQR 0% to 40.5%). However, cost effectiveness was not examined and must be carefully considered when designing multifaceted interventions an additional 5% improvement in processes of care may not be an acceptable return on investment to sustain expensive initiatives like educational outreach. Of course, this will depend on the nature of the targeted behaviour, the context, and the costs involved. In this respect, it might be helpful to incorporate cost effectiveness into power calculations when planning A&F or other KT trials. 219 In summary, the systematic review, meta-analysis, and meta-regression provided new information for those developing KT interventions featuring A&F. In addition to considering the targeting behaviour change and contextual barriers and facilitators, the variation in effect size observed in trials testing A&F can be partially explained by how it is designed and delivered Intervention components must be implemented in a manner that fits the context Cognitive psychologists propose that receipt of feedback highlighting a discrepancy between expected and actual performance results creates an uncomfortable state described as cognitive dissonance. 54 Recipients of A&F may attempt to reduce the discrepancy by planning to improve 53 or by undermining the saliency of the discrepancy by discounting the data (or the goals targeted) as invalid. 41 To be most effective, A&F interventions should emphasize the ease of carrying out processes of care that will improve performance (and thereby reduce cognitive dissonance). In primary care, providers often manage multiple competing goals with their patients; 210 desirable prioritization of these goals may be enabled if A&F successfully re-directs attention toward specific tasks. Based on this understanding of the steps leading to a recipient s behavioural response we developed an intervention that we believed would guide recipients to carry out quality improvement activities. 135

151 A worksheet was developed to guide A&F recipients to set specific goals and develop specific action plans. The worksheet was developed with input from behavioural psychologists and experts from continuing professional development and knowledge translation. Recipients were expected to self-select targets for improvement based on the data in the feedback reports, in the hope that this would solidify goal-commitment, building on evidence from one study that this may lead to greater impact. 151 Action and coping plans were organized using implementation intentions, wherein common clinical scenarios were described with if statements, and blank space where recipients were expected to fill out the intended behavioural response ( then ). Implementation intention statements have been observed to increase goal-commitment and increase goal-directed behaviours. 164 The final format was iteratively refined through informal user testing with family physicians to ensure the worksheet could be understood and properly completed. Our goal was to determine if the effectiveness of large-scale, external A&F could be enhanced by inclusion of this type of worksheet. In keeping with the pragmatic approach, in an effort to improve external validity we simply disseminated the worksheet embedded within the A&F materials without additional supports. Specifically, the worksheet was placed in the A&F package along with standardized surveys required to earn continuing medical education credits from the College of Family Physicians of Canada. Opportunities to receive feedback and to use the process to earn credit were popular reasons for family physicians to sign up with the program. Therefore, the standard surveys were provided to participants in both arms. Operationalized in this manner and in this setting, the worksheet did not appreciably enhance the effectiveness of the A&F intervention with respect to improving processes of care. We discovered that the standardized surveys and the worksheets were not consistently completed. The qualitative evaluation uncovered that the focus of the A&F was not aligned with the priorities of the recipients. The family physicians valued chronic disease management and found the feedback interesting, but felt that stringent adherence to population-level targets was not concordant with delivery of patient-centered care. In addition, the aggregate-level data provided was not actionable enough to shift attention toward specific tasks; many of the family physicians would have appreciated information that identified patients overdue for reassessment. 136

152 It remains plausible that other, more intensive approaches to supporting goal setting and action planning by recipients of A&F may facilitate improved quality of care. This might take the form of reminders, external supports, or local champions. The extent of further support required may be dependent on the recipient s self-efficacy for accomplishing the goals. It is possible that less support may be adequate in cases where the A&F includes actionable content for less complex targeted behaviours, in keeping with the tentative findings in the systematic review of greater effect sizes with simpler targeted behaviours. The extent of further support required would also depend on the alignment between the priorities of the recipient (and their organization) and the targeted behaviour. Organizational readiness for change may play a role, 72 especially when a systematic, team-based response to the feedback is desired. In cases where the feedback is attempting to raise the organizational priority-level and/or recipient s goalcommitment for a targeted behaviour, co-interventions may be needed to justify the benefits of shifting capacity and attention in this manner Capacity for accomplishing the targeted behaviour change must be clearly understood The most striking finding from the qualitative study was that many family physicians do not endorse systematic approaches to population or panel management as a core part of their role. Many family physicians view themselves as defenders of customization (i.e., patientcentered care) in the face of inappropriate pressures for standardization (i.e., guideline implementation). In some respects, this is an inspired perspective: frequently recommended best practices are not appropriate for individual patients. In some cases, careful audits have found that the Quality Chasm may not be as large as previously feared in primary care, once patient preferences are considered. 220 However, there are many patients who would be likely to benefit from guideline-recommended processes of care and (of equal importance) whose preferences would be to receive such care. The gap between ideal and actual care encompasses under-use as well as over-use, and is attributable to factors at the patient, provider, and system-level that interact in complex ways. 137

153 For instance, the lack of system-level infrastructure to identify patients with modifiable risk factors may engender a sense of futility amongst providers. Previous investigators have shown that both self-efficacy 221 with regard to improvement and also goal commitment, 140 as well as personality factors, will predispose the recipient of feedback to a particular reaction. 184 This has implications for the design of the intervention in that the optimal nature of the behaviour(s) targeted by the feedback may vary by recipient and their setting. The qualitative study found that in the Ontario primary care setting, competing patient, professional, and organizational priorities may have limited some participants goal commitment and that inadequate quality improvement infrastructure and training may have limited some participants self-efficacy. Baseline data regarding these and other recipient characteristics (e.g., internal versus external locus of control / sensitivity to subjective norms) as well as relevant organizational characteristics such as presence of local champions who could support colleagues in reacting to data could have permitted opportunities to tailor the A&F intervention accordingly. Our hope in developing the A&F intervention was that the data would drive participants to consider more efficient approaches to improve the delivery of standardized processes of care. We envisioned that (in the best case scenario) systematic techniques commonly employed in the Chronic Care Model, including patient reminders and computerized point-of-care provider reminders would be implemented in an effort to improve scores. 94 Furthermore, we hypothesized that the goal-setting and action-planning worksheet would facilitate recipients to develop this type of coordinated, systematic response. However, we observed that practices had no slack organizational capacity for action, 71 concordant with previous observations about the primary care system in Ontario. In this light, it is not surprising that interview participants were more likely to endorse an intention to try harder rather than develop novel, standardized approaches. This may help to explain tentative findings in the systematic review suggesting that less complex targeted behaviours might be more amenable to change with A&F. It is possible that over time, as quality improvement infrastructure is developed in primary care in Ontario (and if the perceived role of family physicians shift to feature more emphasis on population/panel management), similar A&F may lead to greater improvements in processes of care through greater ability to respond in a systematic manner. 138

154 7.2 Strengths and limitations Methodological strengths and weaknesses for each paper of the thesis were summarized previously. Therefore, in this section, I will consider the strengths and weaknesses of the overall approach and then focus on some noteworthy issues arising through these projects that may inform future research. The growing pressure for accountability combined with the increasing availability of data seems to be creating an environment where A&F is increasingly utilized in healthcare systems around the world. This thesis did not examine the reasons for these trends, or the potential problems they might bring. I took this context as a given and aimed to determine how A&F may be optimized. I aimed to build upon extant knowledge to inform those implementing A&F initiatives with useful information that might improve their likelihood of successfully using this KT strategy to improve practice. The overall approach emphasized problem solving, rather than problematizing - theory use rather than theory building. When theory is used to inform the design of interventions and analyses, it provides a framework for understanding observed effects. Such theories need not be formal; they may combine relevant constructs from various fields, 54 with judicious use of common sense, as long as the hypotheses are made explicit. 222 Informing each phase of the dissertation by relevant theory facilitated an understanding of the results within the broader literature. The focus initially was on behavioural theory (e.g., goal-setting theory, selfregulation/control theory, feedback intervention theory). However, while these theories and others can help identify determinants of behaviour change in order to craft a tailored KT intervention, 223 they offer less direction for identifying important levers in the system or contextual inputs necessary to ensure the wheels are greased. 42 Other integrative metatheoretical frameworks provide a structure for considering how micro, meso, and macro-level constructs may play a role in determining the success of a KT intervention. For instance, the Consolidated Framework for Implementation Research categorizes relevant constructs in five domains: intervention characteristics, outer setting, inner setting, individual characteristics, and implementation processes

155 While A&F may play an important role by elevating the tension for change and/or raising the relative priority of certain tasks, many studies emphasize that change is more likely when there is leadership engagement and available resources to support implementation. 71 In addition, constructively building upon relationships amongst providers in a team can be associated with improved quality. 225 The relative lack of attention in the trial paid to local organizational or structural barriers to improvement in chronic disease management at each participating primary care practice reflected the desire to develop an easy to implement, pragmatic, solution that could improve the effectiveness of the intervention. This trade-off between a highly tailored intervention and a more general, easily scalable intervention corresponds to trade-offs made between a flexible clinical approach that might focus on the end-users with highest needs and a more standardized public health approach that might aim to shift the entire population curve. Relatedly, the perceived discordance between the patient-specific focus of clinicians and the population-health focus of the intervention was highlighted in the qualitative evaluation. With this approach and these goals in mind, a major strength of the thesis is the multiple methods used, with each methodology chosen to address a specific objective, allowing for a multi-faceted examination of the issues related to optimizing the effect of A&F. The findings of each project informed the next and each provided novel insights. The systematic review was conducted following methodological guidelines in the Cochrane Handbook, 226 adapted where appropriate in accordance with the Cochrane Effective Practice and Organization of Care group recommendations. The trial was cluster-randomized to limit risk of contamination, applied appropriate techniques to account for clustering, and was reported in accordance with the CONSORT extension for cluster trials. In addition, the protocol for the trial and the embedded qualitative study were published. 169 Development and testing of the worksheet was based on two hypotheses. First, derived from the systematic review was the hypothesis that there are better and worse ways to implement A&F (e.g., feedback that features goal setting and action plans are more likely to be effective). Second, derived from the review of the non-rct literature in an effort to understand the mechanism of action of feedback and to identify actionable approaches to facilitate behaviour change in response to feedback was the hypothesis that a goal-setting and action-planning worksheet featuring implementation intentions was a promising solution. Such a worksheet was particularly attractive in a pragmatic cluster randomized trial because of its ability to be easily 140

156 incorporated in large-scale applications of external A&F initiatives. This highlights the importance of making a distinction between external audit programs and internal/clinical audit. External A&F is generally conducted by persons unknown to the recipient and generally attempts to implement guidelines and achieve goals set by outsiders. Clinical audits generally refer to a more organic process, usually led by internal champions. As described in the implications section below (Section 4.3), lessons learned from the projects in this thesis are only partially applicable to clinical audits since much of the literature (especially RCTs) evaluates external A&F. The systematic review identified a number of potential approaches that might enhance the effectiveness of A&F, as well as a number of gaps in the literature. The choice of the specific intervention tested in the trial was drawn from these findings, but was inevitably subjective and value-laden. This dissertation does not address the philosophical question regarding how to weigh the importance and/or utility of different types of questions that might be asked about A&F (i.e., what are the sociological/historical/political reasons for increasing application of A&F?). In addition to these caveats about the general approach, noteworthy weaknesses of the individual thesis projects that I believe warrant further discussion but were not adequately covered within the papers are explored below Systematic review, meta-analysis and meta-regression In any systematic review, validity of results derived through meta-analysis is contingent upon the primary studies included. The outcomes must be measured reliably and reported fully and all relevant studies should be included. In systematic reviews of quality improvement interventions risk of publication bias is difficult to mitigate, given the lack of standardized search terms. In addition, selective outcome reporting bias in this literature is a particular risk because many outcomes are potentially relevant and often many are measured, raising the risk for type 1 error in the primary literature. Furthermore, a high degree of heterogeneity is common in reviews of quality improvement interventions. For those developing new interventions, individual trials performed on relevant targeted behaviours in familiar settings might have greater applicability 141

157 than results from a meta-analysis. However, since effect sizes from a single published study are rarely replicated, 227 systematic reviews may provide a more accurate representation of the distribution of likely effect sizes to be achieved in future studies. Furthermore, systematic reviews can provide the power to explore sources of variation in the effect size, can identify when a result is unlikely to change with further study, 228 and can also illustrate areas where further research is needed. The review of RCTs of A&F interventions had variation in the nature of the intervention, the targeted behaviour, the recipient, the healthcare context, and the outcomes. Audit and feedback may be considered a complex intervention; 229 A&F-based interventions may contain a mix of components that can interact (synergistically or antagonistically) between themselves and with the context. We explored heterogeneity by assessing a series of factors chosen a priori based on existence of a directional hypothesis (i.e., a theory about how it might alter the effectiveness of A&F) as well as practical considerations regarding the likelihood that the variable of interest would be reported in the included trials. The exploration of heterogeneity through meta-regression provided useful insights, but was limited in that it focused only on trials with dichotomous outcomes and did not assess for interactions. The challenge of identifying factors that differentiate more and less successful A&F interventions was exacerbated by poor reporting of both intervention components and contextual factors in the primary trials. In addition, the multitude of outcomes evaluated in the included trials resulted in substantial heterogeneity. Some investigators may have concluded that the extent of heterogeneity precluded meta-analysis. 226 Proceeding with statistical analysis required a foundational assumption that regardless of the variety of settings, recipients, and targeted behaviours, the potential behavioural response to A&F to conduct a task falls within a predictable range. The consistency of effects observed across trials supports the decision to proceed with quantitative analysis, and the meta-regression was used to quantify the variation explained by factors of interest. However, while the regression of median effect sizes weighted by sample size provided useful information, the approach was suboptimal due to poor reporting and analysis of the cluster trials and the model results were unstable. It is plausible that other approaches may be better suited to determining inter-relationships of mechanism of action and 230, 231 contextual factors in syntheses of complex interventions (e.g. realist reviews). 142

158 Another issue that limits faith in the conclusions regarding intervention components relates to construct validity. As noted above, practical considerations played an important role in determining the variables for inclusion. The previous update of the review already extracted format (verbal; written; both; unclear) and source (supervisor or senior colleague; professional standards review organisation or representative of employer/ purchaser; investigators; unclear) as part of the approach to determining intensity. However, the definition of format doesn t clarify to whom the feedback is delivered and verbal feedback to a large group may be reasonably deemed less intensive than written feedback with relevant, reliable data for each recipient. In addition, the role of source may actually be a function of two related issues of trust in the data and credibility and this may vary by the nature of data required and the condition examined. This illustrates how features of the audit (not just the feedback) require further examination. Relatedly (and highly pertinent to the rest of the thesis), the rules applied to extract the variable instruction for improvement (explicit measurable target or specific goal but no action plan; action plan with suggestions or advice given to help participants improve but no goal/target; both; neither) varied slightly from the definitions used in the psychology of behaviour change literature. 18 For instance, distinction was not made between prompting general goals for achieving outcomes and specific behavioural goal setting with details regarding frequency, intensity, duration, and context. With respect to action plans, tips for improving performance may work differently than asking recipients to identify likely barriers and developing plans to overcome these (i.e., coping plans). 142 In a recent consensus development exercise, even experts in psychology had difficulty coming to agreement in defining different types of goal setting and action planning techniques. 21 Therefore, while I believe that the analytic approach offered an improvement from the previous systematic reviews of A&F in extracting features of intensity, issues related to construct validity may have created noise in the analysis and risk bias in the interpretations. In general, the conclusions from the meta-regression should be considered as having similar strength as those from observational studies

159 7.2.2 Development and evaluation of the goal setting and action planning intervention As noted above, one possible explanation for the lack of difference between study arms is that the strategies utilized in the worksheet for goal setting and action planning (e.g., implementation intentions) were not different enough from the standard continuing medical education surveys provided in both arms. However, another possibility is that we sub-optimally operationalized the behaviour change techniques featured in the worksheet. For example, prior studies involving coping plans ask recipients to identify situations where they may encounter barriers to accomplishing the targeted behaviour (e.g., if I m at the bar and my friend offers a cigarette) and to determine goal-directed responses (e.g., instead of smoking I will chew some gum). In the worksheet developed for the trial, we pre-filled the if part of the implementation intentions statements, which may have minimized their effect. In contrast, we asked the participants to specify their goal and sometimes the response was not very specific; it is possible that more guidance on behavioural goals could have allowed the action plans to become more specific and more effective. Further work is needed to better understand how to operationalize goal setting and action planning for chronic disease management, where multiple competing demands exist and where multiple actions are needed to manage the condition, as opposed to situations where goals may be accomplished with a single behaviour change. The Medical Research Council Framework for development and evaluation of complex interventions recommends four (non-linear) stages: development (i.e., identifying evidence and theory for intervention design), piloting (i.e., testing procedures), evaluation (i.e., understanding change processes and assessing effectiveness) and implementation (i.e., long-term monitoring). 233 In this thesis, while the theoretical basis informing the intervention tested in the trial was explicit, development and piloting phases could have featured a more formal approach to iterative design. 234 The design of the actual paper-based report was developed iteratively with team members, but no formal usability testing was performed with intended users. Doing so may have resulted in a more realistic understanding of how the intervention would be used prior to moving ahead to evaluation in a trial. 14 It is possible that this would have elucidated barriers experienced by the family physicians in enacting the desired behavioural response and the team 144

160 may have been able to use such information to optimize the intervention. If such an iterative process was based on a careful characterization of the targeted behaviours (separately from the factors influencing the behaviour), the levers that may require pressure could be identified. 42 For example, I could have provided reminders to the recipients to complete the forms and I also could have resent participants their completed forms to prompt ongoing reflection. It is possible that a more explicit understanding of the desired behaviour changes could have resulted in a more effective KT intervention. In retrospect, while the intervention was directed at individual physicians, I was hoping that the A&F would lead to implementation of systematic approaches to chronic disease management. This type of response requires multiple members of the practice (e.g., physicians, nurses, administrators) working together. By giving feedback marked confidential to individual physicians, I may have inadvertently encouraged the try harder approach observed, rather than the work smarter response that was desired. To facilitate the desired type of behaviour change, I could have helped to arrange team meetings at each clinic to discuss the A&F, potentially by identifying and coaching a local champion or by conducting conference calls. With some (asynchronous) guidance, this champion may then have been able to marshal the resources needed to act upon the A&F by, for instance, instituting medical directives or implementing recall systems for patients with chronic disease overdue for reassessment. In the qualitative evaluation, although the original intention was to focus on the A&F intervention and the worksheet, participants consistently redirected the conversation to broader issues related to quality improvement. The focus by interview participants on the targeted behaviours and contextual factors stands in contrast to the more thorough examination in the systematic review of factors related to the design and delivery of the feedback (e.g., format, frequency, source, etc.). In this way, the qualitative evaluation helped to explain the findings of my cluster trial and also suggested additional effect modifiers to explore in future updates of the systematic review. 145

161 7.3 Implications Lack of Progress in the field The lack of clarity and depth in the description of implementation strategies within the published literature precludes replication in both research and practice. 235 Consistent effect sizes have been observed over multiple versions of the Cochrane review of A&F, as illustrated in the table below. Table 17: Consistent effect sizes observed multiple versions of Cochrane review of A&F Year of Review 2003 (search up to January 2001) 2006 (search up to January 2004) 2012 (search up to December 2010) Effect Size 47 studies with dichotomous outcomes: 7% (IQR: 2 11) median absolute increase in compliance with intended professional behaviours or processes 49 studies with dichotomous outcomes: 5% (IQR: 3 11) median absolute increase in compliance with intended professional behaviours or processes 62 studies with dichotomous outcomes: 4% (IQR: 1 16) weighted median absolute increase in compliance with intended professional behaviours or processes Conclusion Audit and feedback can be effective in improving professional practice. When it is effective, the effects are generally small to moderate. The absolute effects of audit and feedback are more likely to be larger when baseline adherence to recommended practice is low. 38 Audit and feedback can be effective in improving professional practice. The effects are generally small to moderate. The absolute effects are likely to be larger when baseline adherence to recommended practice is low and intensity of audit and feedback is high. 35 Audit and feedback generally leads to small but potentially important improvements in professional practice. The effectiveness of audit and feedback seems to depend on baseline performance and how the feedback is provided. Future studies of audit and feedback should directly compare different ways of providing feedback. 131 This suggests the possibility that A&F trial lists have failed to cumulatively learn from previous studies (or from systematic reviews). Rather, it would appear that the norm for those testing audit and feedback interventions is to re-invent the wheel, repeating rather than learning from and contributing to extant knowledge. 236 The opportunity cost for patients, providers, and health systems of continuing in the current manner is large. This dissertation illustrates how, prior to conducting A&F, the nature of the behaviour change required to improve performance should be elucidated in order to develop clear action 146

162 plans that can be enacted in the study setting. Theory and evidence should be made explicit with respect to proposed design choices when manipulating the design of individual intervention components and the expected mechanisms of action should be justified to aid interpretation. A secondary analysis of the systematic review of A&F trials indicated that only 9% of studies used theory to inform the intervention design. 237 As a result, A&F initiatives are infrequently launched with explicit program theories that describe how the intervention is expected to lead to changes in care processes, slowing the identification of active ingredients as well as key contextual inputs. 238 In general, A&F is becoming more widespread in health care and there is emerging evidence that the way it is designed and delivered can substantially alter its ability to change behaviours of providers and improve quality of care. However, instead of moving the science forward, most trials might be better characterized as designed based on the principle of ISLAGIATT (it seemed like a good idea at the time) without explicitly building on previous research or extant theory. 239 Given the many important aspects of A&F to be tested, the potential for significant impact, and because definitive trials to evaluate A&F interventions can take a great deal of resources, there is a need for greater coordination to determine how to best design and deliver A&F. In addition, after more than 140 trials, there may no longer be equipoise regarding whether A&F tends to improve professional practice (at least for common clinical scenarios and contexts), suggesting that typical, local trials of A&F versus usual care may be less ethical than multi-arm trials with head-to-head comparisons of different strategies for implementing A&F and/or other interventions Implications for further A&F research Despite the growth in use of performance feedback, there is a large gap between what we are currently doing and what needs to be done to improve outcomes for our patients. 240 Given the importance of audit and feedback as a key component of many QI interventions, there is a need to identify opportunities to sequentially and systematically test 147

163 various approaches to the design and development of A&F. Advancing cumulative knowledge regarding A&F and other QI strategies will proceed slowly by continuing to conduct uncoordinated trials of A&F versus usual care with subsequent periodic analyses across studies to explore effect modifiers. Compared to this haphazard approach, with components indirectly compared using meta-regression, researchers might achieve greater confidence in causal inference regarding how to develop more effective A&F through a limited number of multi-arm trials with direct, head-to-head comparisons testing different, promising approaches for intervention design. This proposed shift in direction for QI trials parallels the movement to limit placebo-controlled trials of clinical interventions and to increase focus on comparative effectiveness research. 241 A&F researchers could more efficiently pursue this comparative effectiveness research agenda for KT by partnering with those routinely conducting audits to test variations of intervention components that might increase effectiveness. 242 Administrators of quality improvement initiatives involving A&F may be keen to participate in head-to-head trials comparing different A&F designs because, in this case, all providers receive an intervention, while decision makers learn about how their programs may be improved. Thus, settings where valid data for audit and mechanisms for feedback are already established can become implementation research laboratories, enabling relatively rapid testing of multiple intervention components as well as exploration regarding potential effect modifiers. In general, prior to directing further research effort towards new trials of A&F versus usual care, investigators should consider how the proposed study could add to present empirical knowledge. At a minimum, there needs to be good justification for considering particular intervention components and a clear hypothesis regarding how the components will alter the effectiveness of A&F. A&F interventions consist of multiple components, each requiring attention during the design stage. Since the inclusion of all desirable components and designing each component in the optimal fashion is rarely possible, intervention components may be considered as levers to be manipulated within setting-specific constraints. Once a prototype intervention is developed, usability testing should be carried out to assess whether the program theory applies: do recipients respond as anticipated and do the effect modifiers moderate this response in expected ways? Advances in the field depend on further 148

164 development of generalizable principles regarding how A&F can be optimized. Therefore, it is essential that those developing A&F are able to successfully operationalize intervention components through a systematic, iterative process prior to testing the intervention in a trial. When the intervention is ready, meaning that usability testing has tentatively confirmed the program theory, formal evaluation with head-to-head comparisons in a trial should be pursued. For example, if one were interested in testing whether A&F is more effective when delivered by opinion leaders, 68 it would be essential to first establish that effective opinion leaders could be identified and activated in a replicable manner. Otherwise, if a trial testing whether opinion leaders enhance the effectiveness of A&F found no difference, then readers might be unable to establish whether this was due to how the intervention was implemented (suggesting further trials are needed to answer the question) or due to true lack of effectiveness (suggesting other approaches should be pursued). In contrast, if pilot work established that the approach to identifying opinion leaders resulted in finding credible and trusted professionals and the approach to training them resulted in consistent provision of constructive feedback, one could argue that no further trials testing opinion leaders plus A&F would be needed (at least not for the targeted behaviour in that trial). This example illustrates the importance of developing a program theory and how usability studies can and should be used to establish the foundation for causal inference. When considering how to optimize A&F, the combination of multiple intervention components that play a role in the recipient s response (each with multiple potential designs), plus contextual variables that could act as effect modifiers, results in a long list of potential questions to be asked. To enhance the effectiveness of A&F, it will be important to identify highpriority research questions, keeping in mind that A&F may be moderated by contextual factors (e.g., size of the team responsible for outcomes of interest and organizational resources required); recipient characteristics (e.g., goal-commitment, self-efficacy); intervention components (e.g., social pressure, credibility of source, sign or framing of the content); cointerventions (e.g., patient oriented, practice oriented, provider oriented); and the targeted behaviour (e.g., complexity of change, fit with workflow). Therefore, new methodological approaches will be needed to efficiently rule out less promising intervention components and/or strategies for delivering those components. For example, with A&F, one could vary the content related to the message framing to be positive or negative and one also could vary the frequency 149

165 and the mode of delivery. Testing one combination of these factors that seems to work best in usability studies is potentially efficient, but results will relate only to the package. There could be further opportunities to optimize by removing unhelpful components and/or by comparing additional options for presenting the information. One option worth pursuing is the Multiphase Optimization Strategy (MOST), using fractional factorial study designs. 243 These study designs offer a highly efficient approach to simultaneously compare multiple components of design and/or delivery and may be less biased than post-hoc theorizing about active ingredients and mechanisms of action. 244 While usability studies and/or MOST will delay the onset of the confirmatory trial, it seems more likely to result in studies that will both be effective and contribute more to the literature. Another promising approach is the sequential, multiple assignment randomized trial (SMART), using response-adaptive interventions to examine how to tailor interventions to sub-groups over time. 243 For instance, it is unclear whether those who do not respond to the first cycle of A&F should receive A&F with content that is revised in some fashion in the second cycle. In a SMART trial, those not responding to the first cycle would be randomized to receive standard versus revised A&F in the second cycle. Thus, the SMART approach offers an opportunity to embed multiple formal analyses regarding how to optimize and tailor the intervention over time according to recipient characteristics. 245 This type of approach could be pursued in situations when the behavioural response to the A&F intervention can be rapidly measured, when there is reason to believe that characteristics of A&F recipients may moderate that response and when the intervention can be rapidly modified in reasons to measured variables. To my knowledge, no previous trials of A&F have incorporated MOST or SMART approaches. In primary care, further work is required to elucidate the metrics that properly reflect the priorities of the family physician and new methods are needed to more comprehensively measure the roles of the primary care provider. For instance, given the challenges presented by multimorbidity, there is increasing interest in promoting minimally disruptive medicine an approach that involves the patient in setting priorities and seeks to minimize the burden of medical care. 246 It is not yet known how to best measure the primary care patient burden in chronic illness management and it is uncertain how or if such data could be used to encourage behaviour changes that decrease burden. This would likely require further development of methods that involve feedback of patient-reported outcomes, as there is relatively less experience 150

166 in that area. 247 In addition, development of standardized approaches to implementing cointerventions that increase primary care providers skills in quality improvement may be useful because skill development and resultant self-efficacy may increase the likelihood that feedback 140, 248 will lead to improved performance. Furthermore, if the goal of the activity is to identify outlier providers or to increase attention to patients at high-risk (i.e. those with greatest illness burden), it is important to recognize that such goals may be achieved without substantial changes in the population-level mean for the outcomes of interest. 212 In these instances, the A&F initiative may have been highly successful, but the usual approaches to evaluate them (i.e., comparing overall mean values of the outcome across study arms) will not reject the null hypothesis. New analytical methods may be needed to evaluate the impact of A&F interventions on outliers (at both provider and patient levels), possibly by focusing on changes in the distribution of outcomes, rather than only looking at mean effects Implications for conducting audits Managerial knowledge and skill in applying metrics has not kept up with organizations ability to create them 249 Effective clinical audit has been previously described as including the following elements: 40, 135, 250 : Part of a structured programme with a local lead; Topics based on high risk, high volume, or high cost problems or on national clinical audits, national service frameworks, or guidelines; Based on high quality data; Standards derived from good quality guidelines; Delivered to those with the power to act; Including action plans to overcome local barriers to change; and 151

167 Repeated to find out whether improvements in care have been implemented as a result of clinical audit and sustain service improvements once the audit cycle has been completed This dissertation adds to the literature by providing a clear evidence base for a number of the recommendations above, including the use of action plans and repeated cycles. It also supports the use of a local lead to deliver written and/or verbal feedback, as opposed to instances when feedback is delivered in written format only and from sources deemed to have potential for punitive action. The role of the targeted behaviour was also illustrated, but in primary care, it is important to consider more than just individual guidelines. Targets may be more salient when they are perceived to fit with the comprehensive, person-centred goals of primary care and reflect the prevalence of multi-morbidity in day-to-day practice. In his primer on clinical audit in primary care written in 1980, Shaw emphasized that, General practice is concerned more with people than with diagnoses, disease, and life-threatening events. 251 Likewise, in his 1991 review on the effects of feedback on clinical practice, Mugford 40 emphasized a focus on indicators of interest to the recipients. Therefore, it is possible that greater buy-in and a more robust response to A&F in primary care could be achieved if developers of quality indicators and those conducting audits were able to focus on metrics that better capture the nature of primary care. For example, disease-specific guidelines (and quality improvement initiatives developed to implement these guidelines) generally try to emphasize standardization through systematic approaches to chronic disease management. If the patient (and family physician) prioritize maintenance of function over biological markers of risk, judicious application of specific guideline recommendations may be necessary. 252 If family physicians continue to view their professional role as one that focuses on customization rather than standardization, audits might focus on issues that facilitate this in a productive manner, including measures of patient centeredness in chronic illness care. 253 In such instances, it would be important to make explicit how to improve and to ensure that recommendations are evidence-based. A recent Cochrane review of interventions to promote patient centered care featured multifaceted interventions and successful interventions seemed to focus on post-intentional barriers. Therefore, it is uncertain whether A&F alone would be sufficient to improve this type of outcome

168 While domains of quality other than clinical effectiveness (such as patient-centeredness) deserve further attention, there is an expected population-level benefit through implementation of disease-specific guidelines that is best achieved through standardized approaches. For example, over 10 years of implementation efforts emphasizing the importance of blood pressure screening and treatment targets, the Canadian Hypertension Education Program observed a decrease in the burden of hypertension-related morbidity and mortality in Canada. 255 Though causality cannot be attributed with any certainty, this accomplishment creates a model for other disease-focused entities. Yet the resulting multiple implementation teams and quality improvement initiatives could create challenges for primary care providers due to potentially overwhelming amounts of data presented, as well as the risk for inconsistent recommendations. 4 Therefore, a major challenge for ensuring success of A&F initiatives in primary care is to achieve a balance between the priorities of health systems and of providers (and their patients); when these are not aligned, health system payers must either invest in additional resources or otherwise convince providers that shifting attention will make life better for patients (without making life worse for the providers). In addition, it would seem there is a need to consider whether training programs for family physicians should emphasize competencies related to systematic approaches to management of their patient panels. Those conducting clinical audit programs should identify their specific program goals and the behaviour changes needed to accomplish these goals. Aside from the general desire to increase accountability, program goals can likely be classified into two main categories: to identify outliers (quality assurance / safety perspective) or to improve mean performance (quality improvement / promotion perspective). 36 Since the targeted behaviour changes may be different, different designs of A&F may be needed. Both goals are worthwhile and the appropriate approach may depend how the recipient perceives the targeted behaviour. For instance, family physicians may perceive achievement of target blood pressure for a given patient as less important than avoiding dangerous blood pressure readings; theory suggests that feedback with a negative sign would be more effective to address the latter. 48 Feedback with a positive sign may be more likely to result in improvement when the physician recipient is driven to be the best for the given metric and the reverse is true when the recipient is motivated by risk avoidance. 256 This illustrates how careful attention to the program goals and description of how and why the A&F was designed to accomplish those goals is needed to further develop recommendations that 153

169 effectively connect the two. It also fits with Ilgen s 1979 model of the effects of feedback on recipients, which emphasized the role of individual differences and suggested an opportunity for greater impact if feedback messaging is tailored accordingly. 70 This could explain the tendency to see greater effect sizes when respected colleagues verbally deliver feedback, as these persons likely tailor the content to some extent for each recipient. In Ontario, the varied primary care contexts and the lack of quality improvement infrastructure must be considered. Even in relatively well-resourced primary care clinics, there is a need to invest in the capacity of the providers to incorporate A&F into standard workflows and to effectively respond to the results in a way that will benefit patients. Family physicians paid through fee-for-service without systems in place to collect data or to analyze it are unlikely to conduct effective audit cycles on their own without major policy changes that substantially alter their incentives. Further, if incentive structures were to change to allow for more time committed to panel management, the result (at least temporarily) could be reduced access for patients. Thus, in settings without any slack resources that can be shifted to quality improvement activities, A&F initiatives will only be accomplished through support from external services. Such services must offer more than just data collection and provision. Ideally, the feedback should come from a respected source, provided in a non-judgemental manner, with an opportunity for sensemaking 257 and support to implement realistic action plans. Primary care practices that mostly earn their income through capitation (or salary) should have incentives aligned for panel management, including conducting audit cycles. However, the breadth of possible topics, and the lack of quality improvement know-how, may have stymied development of this aspect the professional role. Here again there is a potential role for external support services in such practices to help providers maximize their existing resources and learn to capitalize on existing infrastructure (including use of data from EMR) in order to conduct effective audit cycles. In addition, support is needed to focus quality improvement activities on high-priority topics (i.e., aspects of care identified as important for ensuring health system sustainability). This could take the form of local consensus building, taking account of barriers and facilitators to improvement for given metrics and/or policy levers. Yet while financial incentives are an intuitively appealing strategy to focus efforts, evidence indicates that they do not consistently enhance the effectiveness of A&F, especially when goal commitment is low. 184 Goal commitment is more likely to be present, and therefore the desired response to A&F more likely to be achieved, when 154

170 the targeted behaviour change fits within the primary care provider s workflow and does not increase burden for the patient. 246 External supports are also needed to conduct A&F in Ontario anytime a comprehensive assessment of healthcare utilization by the practice population is desired. In particular, comprehensive data that could link practice rosters and EMR-based data to administrative health databases is available through ICES. Therefore, ICES has an opportunity to play an important role in the development of broad A&F initiatives in the province. Yet privacy rules and regulations governing the use of ICES data present a major challenge to undertake this role. Two previous A&F trials have been published using ICES data. Both used Zelen s design, 258 and in both studies less than one-third of participants randomized consented to follow-up. In one study, three rounds of feedback were provided over six months, resulting in improved antibiotic prescribing by shifting to preferred antibiotic choices. 259 In the other study, a similar intervention resulted in no change in benzodiazepine prescribing. 260 The noteworthy variation in effectiveness of these two similar programs is likely due to differences in the complexity of targeted behaviour changes, especially considering that in the antibiotic option the alternative behaviour was relatively easy to conduct. In addition, due to privacy limitations, the feedback reports were aggregate/practice-level data (as in my trial). This may be sufficient to influence new behaviours that occur periodically, where alternative options are available (e.g., antibiotic prescriptions). However, to encourage changes to previous decisions made for patients receiving chronic treatment, especially when alternative options are difficult to implement (e.g., benzodiazepine prescriptions), patient-level data with specific suggestions would be more actionable. Accomplishing this may require changes in privacy regulations (or changes to how they are interpreted) to facilitate use of administrative databases to support clinical care. For instance, although a number of restrictions apply, Ontario s Personal Health Information Protection Act states that patient consent is not necessary when personal health information is relayed for health care purposes and/or for planning health care services by the same health information custodian who collected the data. 261 However, Ontario s Privacy Commissioner cautions that the health information custodians should not share disaggregated personal health information with other health care providers solely for the purposes of planning health services (rather than delivering specific health care for that individual). 262 This could limit the potential for capitalizing on the 155

171 large data holdings available at ICES and elsewhere for cumulating data from multiple sources for a comprehensive A&F initiative. 7.4 Conclusions This thesis was initiated with the intention of examining how to optimize A&F interventions for primary care. The first objective in pursuit of this goal, to determine the effectiveness of A&F and identify effect modifiers, was accomplished through a systematic review that included 140 RCTs. Building on these findings as well as behavioural theory, I developed a worksheet that could be appended to A&F interventions to facilitate goal setting and action planning and then tested its effectiveness in a cluster trial. In general, A&F with the worksheet did not result in improved processes of care for patients with chronic disease compared to A&F alone. The embedded qualitative evaluation helped to explain these findings, highlighting the key role of contextual factors, including the targeted behaviours, the recipients, and the organizational infrastructure in the primary care settings. Through these projects, I was able to contribute to the literature regarding how to develop A&F and also to identify gaps in the literature where advances in KT science are needed. There is evidence that the effectiveness of A&F to some extent will depend on how it is designed and delivered. Following best practices outlined in this thesis should result in more effective interventions. Further evaluation regarding how to present feedback in ways that make it more actionable offers a potentially cost-effective approach to increase the yield from these interventions. However, the effect sizes from small tweaks in design may be small compared to the gains from better understanding the capacity of recipients to accomplish the targeted behaviour change and from usability testing to ensure the intervention actually facilitates the desired response. Overall, this thesis calls for a more thoughtful approach to the design and evaluation of A&F interventions. To summarize, developers of A&F should be able to answer the following questions: 156

172 1. What is the targeted behaviour change for the A&F recipient? Exactly who is responsible for the intended behaviour and when, and where, and how should it be implemented? 2. What options are available to vary the key components of the A&F intervention that are expected to encourage the intended change(s)? 3. What are the characteristics of recipients that might affect the response to A&F? 4. What are the characteristics of the context that might affect the response to A&F? The choices made in designing the A&F should flow from the answers to these questions. The targeted behaviour change must be achievable by those who will be receiving the A&F, but determining this requires an assessment of personal and organizational barriers. The audit process must be perceived as valid and delivered from a trusted source, features that require an understanding of the recipients perspectives. Whenever possible, the data should reflect recent performance with updated data presented over time, goals set for the target behaviour should be as specific as possible and should align with personal and organizational priorities, and a clear action plan for improvement should be made available. Prior to providing feedback on a particular metric, there should be an assessment of the capability of recipients to improve, and if necessary, co-interventions should be developed accordingly. Of course, the ability to incorporate all these components in an effective way will vary based on contextual constraints and expectations regarding cost effectiveness. Where the goal of A&F is local quality improvement, best practices should be applied in the design and a quasi-experimental design and analysis should be sufficient, since the expected effect size of A&F is well established. Where the goal is to contribute generalizable knowledge, research funders (and journal editors) should compel those designing A&F interventions to consider the above questions and to conduct trials that have a clear potential to lead to more effective A&F interventions in future. 157

173 8 References 1. Woolf S, Johnson R. The break-even point: when medical advances are less important than improving the fidelity with which they are delivered. Ann Fam Med. 2005;3(6): Crossing the Quality Chasm: A New Health System for the 21st Century: The National Academies Press; Haffner SM, Lehto S, Ronnemaa T, Pyorala K, Laakso M. Mortality from coronary heart disease in subjects with type 2 diabetes and in nondiabetic subjects with and without prior myocardial infarction. N Engl J Med. Jul ;339(4): Tobe SW, Stone JA, Brouwers M, et al. Harmonization of guidelines for the prevention and treatment of cardiovascular disease: the C-CHANGE Initiative. CMAJ. Oct ;183(15):E Narayan KM, Benjamin E, Gregg EW, Norris SL, Engelgau MM. Diabetes translation research: where are we and where do we want to be? Ann Intern Med. Jun ;140(11): Newby LK, Allen LaPointe NM, Chen AY, et al. Long-Term Adherence to Evidence- Based Secondary Prevention Therapies in Coronary Artery Disease. Circulation. 2006;113(2): Farahani P, Levine M. Goal attainment for multiple cardiovascular risk factors in community-based clinical practice (a Canadian experience). J Eval Clin Pract. Feb 2009;15(1): Yan RT, Yan AT, Tan M, et al. Underuse of evidence-based treatment partly explains the worse clinical outcome in diabetic patients with acute coronary syndromes. Am Heart J. Oct 2006;152(4): CIHR. More About Knowledge Translation at CIHR. 2013; Accessed March 30, Graham ID, Logan J, Harrison MB, et al. Lost in knowledge translation: time for a map? J Contin Educ Health Prof. Winter 2006;26(1): Straus SE, Tetroe J, Graham I. Defining knowledge translation. Canadian Medical Association Journal. 2009;181(3-4): Cabana MD, Rand CS, Powe NR, et al. Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA. Oct ;282(15): Shaw B, Cheater F, Baker R, et al. Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2005(3):CD

174 14. Shojania KG, Grimshaw JM. Evidence-based quality improvement: the state of the science. Health Aff (Millwood). Jan-Feb 2005;24(1): Grol RP, Bosch MC, Hulscher ME, Eccles MP, Wensing M. Planning and studying improvement in patient care: the use of theoretical perspectives. Milbank Q. 2007;85(1): Eccles M, Grimshaw J, Walker A, Johnston M, Pitts N. Changing the behavior of healthcare professionals: the use of theory in promoting the uptake of research findings. J Clin Epidemiol. Feb 2005;58(2): Craig P, Dieppe P, Macintyre S, et al. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a Abraham C, Michie S. A taxonomy of behavior change techniques used in interventions. Health Psychol. May 2008;27(3): Tricco AC, Ivers NM, Grimshaw JM, et al. Effectiveness of quality improvement strategies on the management of diabetes: a systematic review and meta-analysis. Lancet. Jun ;379(9833): Brouwers MC, Garcia K, Makarski J, Daraz L, Evidence Expert P, Team KTfCCiCPR. The landscape of knowledge translation interventions in cancer control: what do we know and where to next? A review of systematic reviews. Implement Sci. 2011;6: Michie S, Richardson M, Johnston M, et al. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med. Aug 2013;46(1): McKibbon KA, Lokker C, Wilczynski NL, et al. A cross-sectional study of the number and frequency of terms used to refer to knowledge translation in a body of health literature in 2006: a Tower of Babel? Implement Sci. 2010;5: Michie S, Johnston M, Francis J, Hardeman W, Eccles M. From Theory to Intervention: Mapping Theoretically Derived Behavioural Determinants to Behaviour Change Techniques. Applied Psychology. 2008;57(4): Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translation of research findings. Implement Sci. 2012;7: Cochrane Effective Practice and Organisation of Care Group. Our Reviews. Accessed March 30, Andrea B. Audit: how to do it in practice. BMJ. 2008; Schön DA. The Reflective Practitioner: How Professionals Think in Action: Basic Books;

175 28. Adams AS, Soumerai SB, Lomas J, Ross-Degnan D. Evidence of self-report bias in assessing adherence to guidelines. Int J Qual Health Care. Jun 1999;11(3): Davis D, O'Brien MA, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA. Sep ;282(9): Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA. Sep ;274(9): Regher G. Contributions from Learning Theory: Self-Assessment, Self-Direction, and Other Myths: Implications for the Self-Regulating Professional. Paper presented at: CME Congress; May 30, 2008; Vancouver. 32. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. Sep ;296(9): Donabedian A. Evaluating the quality of medical care Milbank Q. 2005;83(4): Principles for Best Practice in Clinical Audit Radcliffe Medical Jamtvedt G, Young JM, Kristoffersen DT, O'Brien MA, Oxman AD. Does telling people what they have been doing change what they do? A systematic review of the effects of audit and feedback. Qual Saf Health Care. Dec 2006;15(6): Balas EA, Boren SA, Brown GD, Ewigman BG, Mitchell JA, Perkoff GT. Effect of physician profiling on utilization. Meta-analysis of randomized clinical trials. J Gen Intern Med. Oct 1996;11(10): Grimshaw JM, Thomas RE, MacLennan G, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. Feb 2004;8(6):iii-iv, Jamtvedt G, Young JM, Kristoffersen DT, Thomson O'Brien MA, Oxman AD. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2003(3):CD Foy R, Eccles MP, Jamtvedt G, Young J, Grimshaw JM, Baker R. What do we know about how to do audit and feedback? Pitfalls in applying evidence from a systematic review. BMC Health Serv Res. 2005;5: Mugford M, Banfield P, O'Hanlon M. Effects of feedback of information on clinical practice: a review. BMJ. Aug ;303(6799):

176 41. van der Veer SN, de Keizer NF, Ravelli AC, Tenkink S, Jager KJ. Improving quality of care. A systematic review on how medical registries provide information feedback to health care providers. Int J Med Inform. May 2010;79(5): Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011;6: Shute VJ. Focus on Formative Feedback. Review of Educational Research. 2008;78(1): Wones RG. Failure of low-cost audits with feedback to reduce laboratory test utilization. Med Care. Jan 1987;25(1): Sondergaard J, Andersen M, Vach K, Kragstrup J, Maclure M, Gram LF. Detailed postal feedback about prescribing to asthma patients combined with a guideline statement showed no impact: a randomised controlled trial. Eur J Clin Pharmacol. May 2002;58(2): Kluger AN, DeNisi A. The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull. 1996;119(2): Hysong SJ. Meta-analysis: audit and feedback features impact effectiveness on care quality. Med Care. Mar 2009;47(3): Kluger AN, Van Dijk D. Feedback, the various tasks of the doctor, and the feedforward alternative. Med Educ. Dec 2010;44(12): Bickman L, Riemer M, Breda C, Kelley SD. CFITA System to Provide a Continuous Quality Improvement Infrastructure Through Organizational Responsiveness, Measurement, Training, and Feedback. Report on Emotional & Behavioral Disorders in Youth. 2006;6(4):pp , Sargeant JM, Mann KV, van der Vleuten CP, Metsemakers JF. Reflection: a link between receiving and using assessment feedback. Adv Health Sci Educ Theory Pract. Aug 2009;14(3): Locke EA, Latham GP. Building a practically useful theory of goal setting and task motivation. A 35-year odyssey. Am Psychol. Sep 2002;57(9): Gollwitzer PM, Fujita K, Oettingen G. Planning and the Implementation of Goals. Handbook of self-regulation : research, theory, and applications. 2004: Carver CS, Scheier MF. Control theory: a useful conceptual framework for personalitysocial, clinical, and health psychology. Psychol Bull. Jul 1982;92(1): Brehaut JC, Eva KW. Building theories of knowledge translation interventions: use the entire menu of constructs. Implement Sci. 2012;7:

177 55. Bandura A. Self-efficacy. Toronto: Academic Press; Brady WJ, Hissa DC, McConnell M, Wones RG. Should physicians perform their own quality assurance audits? J Gen Intern Med. Nov-Dec 1988;3(6): Sommers LS, Sholtz R, Shepherd RM, Starkweather DB. Physician involvement in quality assurance. Med Care. Dec 1984;22(12): Levy PE, Williams JR. The Social Context of Performance Appraisal: A Review and Framework for the Future. Journal of Management. 2004;30(6): Linderbaum BA, Levy PE. The Development and Validation of the Feedback Orientation Scale (FOS). Journal of Management. 2010;36(6): van den Hombergh P, Grol R, van den Hoogen HJ, van den Bosch WJ. Practice visits as a tool in quality improvement: mutual visits and feedback by peers compared with visits and feedback by non-physician observers. Qual Health Care. Sep 1999;8(3): Verstappen WH, van der Weijden T, Dubois WI, et al. Improving test ordering in primary care: the added value of a small-group quality improvement strategy compared with classic feedback only. Ann Fam Med. Nov-Dec 2004;2(6): Steelman LA, Levy PE, Snell AF. The Feedback Environment Scale: Construct Definition, Measurement, and Validation. Educational and Psychological Measurement. 2004;64(1): Hysong SJ, Best RG, Pugh JA. Audit and feedback and clinical practice guideline adherence: making feedback actionable. Implement Sci. 2006;1: Gist ME, Mitchell TB. Self-Efficacy: a Theoretical Analysis of its Determinants and Malleability. Academy of Management Review 1992;17(2): O'Brien MA, Rogers S, Jamtvedt G, et al. Educational outreach visits: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2007(4):CD Davies P, Walker AE, Grimshaw JM. A systematic review of the use of theory in the design of guideline dissemination and implementation strategies and interpretation of the results of rigorous evaluations. Implement Sci. 2010;5: Grilli R, Lomas J. Evaluating the message: the relationship between compliance rate and the subject of a practice guideline. Med Care. Mar 1994;32(3): Rogers EM. Diffusion of Innovations. New York: Free Press; Murray E, Treweek S, Pope C, et al. Normalisation process theory: a framework for developing, evaluating and implementing complex interventions. BMC Med. 2010;8:

178 70. Ilgen DR, Fisher CD, Taylor MS. Consequences of individual feedback on behavior in organizations. Journal of Applied Psychology. 1979;64(4): Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4): Attieh R, Gagnon MP, Estabrooks CA, et al. Organizational readiness for knowledge translation in chronic care: a review of theoretical components. Implement Sci. 2013;8: Beaulieu M-D, Haggerty J, Tousignant P, et al. Characteristics of primary care practices associated with high quality of care. Canadian Medical Association Journal. 2013;185(12):E590-E Starfield B, Lemke KW, Bernhardt T, Foldes SS, Forrest CB, Weiner JP. Comorbidity: Implications for the Importance of Primary Care in Case Management. The Annals of Family Medicine. 2003;1(1): Austin PC, Tu JV, Ko DT, Alter DA. Factors associated with the use of evidence-based therapies after discharge among elderly patients with myocardial infarction. CMAJ. Oct ;179(9): Shah BR, Hux JE, Laupacis A, Zinman B, Booth GL. Use of vascular risk-modifying medications for diabetic patients differs between physician specialties. Diabet Med. Oct 2006;23(10): Sinnott C, Mc Hugh S, Browne J, Bradley C. GPs perspectives on the management of patients with multimorbidity: systematic review and synthesis of qualitative research. BMJ Open. 2013;3(9). 78. Carlsen B, Glenton C, Pope C. Thou shalt versus thou shalt not: a meta-synthesis of GPs' attitudes to clinical practice guidelines. Br J Gen Pract. Dec 2007;57(545): Upshur RE, Tracy S. Chronicity and complexity: is what's good for the diseases always good for the patients? Can Fam Physician. Dec 2008;54(12): Ostbye T, Yarnall KS, Krause KM, Pollak KI, Gradison M, Michener JL. Is there time for management of patients with chronic diseases in primary care? Ann Fam Med. May- Jun 2005;3(3): Starfield B. Is primary care essential? Lancet. Oct ;344(8930): Starfield B. Point: the changing nature of disease: implications for health services. Med Care. Nov 2011;49(11): Mitchinson AR, Kerr EA, Krein SL. Management of chronic noncancer pain by VA primary care providers: when is pain control a priority? Am J Manag Care. Feb 2008;14(2):

179 84. Stewart M. Towards a global definition of patient centred care. BMJ. Feb ;322(7284): CFPC. Four Principles of Family Medicine. Accessed March 30, Boyd CM, Darer J, Boult C, Fried LP, Boult L, Wu AW. Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance. JAMA. Aug ;294(6): Finance OMo. Commission on the reform of Ontario s public services. Accessed March 30, Wagner EH, Austin BT, Davis C, Hindmarsh M, Schaefer J, Bonomi A. Improving chronic illness care: translating evidence into action. Health Aff (Millwood). Nov-Dec 2001;20(6): Ruth M, Caroline W, Desmond ON. Retrospective analysis of attitudes to ageing in the Economist: apocalyptic demography for opinion formers. BMJ. 2009; Bajcar JM, Wang L, Moineddin R, Nie JX, Tracy CS, Upshur RE. From pharmacotherapy to pharmaco-prevention: trends in prescribing to older adults in Ontario, Canada, BMC Fam Pract. 2010;11: Denton FT, Spencer BG. Chronic health conditions: changing prevalence in an aging population and some implications for the delivery of health care services. Can J Aging. Mar 2010;29(1): Hutchison B, Glazier R. Ontario's primary care reforms have transformed the local care landscape, but a plan is needed for ongoing improvement. Health Aff (Millwood). Apr 2013;32(4): de Bruin SR, Versnel N, Lemmens LC, et al. Comprehensive care programs for patients with multiple chronic conditions: a systematic literature review. Health Policy. Oct 2012;107(2-3): Coleman K, Austin BT, Brach C, Wagner EH. Evidence on the Chronic Care Model in the new millennium. Health Aff (Millwood). Jan-Feb 2009;28(1): Reid RJ, Coleman K, Johnson EA, et al. The Group Health medical home at year two: cost savings, higher patient satisfaction, and less burnout for providers. Health Aff (Millwood). May 2010;29(5): Fishman PA, Johnson EA, Coleman K, et al. Impact on seniors of the patient-centered medical home: evidence from a pilot study. Gerontologist. Oct 2012;52(5): Rosenthal MB, Friedberg MW, Singer SJ, Eastman D, Li Z, Schneider EC. Effect of a multipayer patient-centered medical home on health care utilization and quality: the 164

180 Rhode Island chronic care sustainability initiative pilot program. JAMA Intern Med. Nov ;173(20): Glazier RH, Zagorski BM, Rayner J. Comparison of Primary Care Models in Ontario by Demographics, Case Mic and Eemrgency Department Use. Toronto: Institute for Clinical Evaluative Sciences; Russell GM, Dahrouge S, Hogg W, Geneau R, Muldoon L, Tuna M. Managing Chronic Disease in Ontario Primary Care: The Impact of Organizational Factors. The Annals of Family Medicine. 2009;7(4): Hutchison B, Levesque J-F, Strumpf E, Coyle N. Primary Health Care in Canada: Systems in Motion. Milbank Quarterly. 2011;89(2): Canada S. Canadian Survey of Experiences with Primary Health Care (CSE-PHC) , Glazier RH, Kopp A, Schultz SE, Kiran T, Henry DA. All the right intentions but few of the desired results: lessons on access to primary care from Ontario's patient enrolment models. Healthc Q. 2012;15(3): Schoen C, Osborn R, Squires D, Doty M, Pierson R, Applebaum S. New 2011 survey of patients with complex care needs in eleven countries finds that care is often poorly coordinated. Health Aff (Millwood). Dec 2011;30(12): Rowan MS, Hogg W, Martin C, Vilis E. Family physicians' reactions to performance assessment feedback. Can Fam Physician. Dec 2006;52(12): Hay MC, Weisner TS, Subramanian S, Duan N, Niedzinski EJ, Kravitz RL. Harnessing experience: exploring the gap between evidence-based medicine and clinical practice. J Eval Clin Pract. Oct 2008;14(5): Tracy CS, Dantas GC, Upshur RE. Evidence-based medicine in primary care: qualitative study of family physicians. BMC Fam Pract. May ;4: Parkerton PH, Smith DG, Belin TR, Feldbau GA. Physician performance assessment: nonequivalence of primary care measures. Med Care. Sep 2003;41(9): Hofer TP, Hayward RA, Greenfield S, Wagner EH, Kaplan SH, Manning WG. The unreliability of individual physician "report cards" for assessing the costs and quality of care of a chronic disease. JAMA. Jun ;281(22): van Gestel YR, Lemmens VE, Lingsma HF, de Hingh IH, Rutten HJ, Coebergh JW. The hospital standardized mortality ratio fallacy: a narrative review. Med Care. Aug 2012;50(8): Salem-Schatz S, Moore G, Rucker M, Pearson SD. The case for case-mix adjustment in practice profiling. When good apples look bad. JAMA. Sep ;272(11):

181 111. Balfour DC, 3rd, Evans S, Januska J, et al. Health information technology--results from a roundtable discussion. J Manag Care Pharm. Jan-Feb 2009;15(1 Suppl A): O'Connor PJ, Crain AL, Rush WA, Sperl-Hillen JM, Gutenkauf JJ, Duncan JE. Impact of an electronic medical record on diabetes quality of care. Ann Fam Med. Jul-Aug 2005;3(4): Johnston S, Dahrouge S, Hogg W. Gauging to gain: primary care performance measurement. Can Fam Physician. Sep 2008;54(9): , Brouwers M, Stacey D, O'Connor A. Knowledge creation: synthesis, tools and products. CMAJ. Feb ;182(2):E Grimshaw JM, Santesso N, Cumpston M, Mayhew A, McGowan J. Knowledge for knowledge translation: the role of the Cochrane Collaboration. J Contin Educ Health Prof. Winter 2006;26(1): Oxman AD, Lombard C, Treweek S, Gagnier JJ, Maclure M, Zwarenstein M. A pragmatic resolution. J Clin Epidemiol. May 2009;62(5): Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin; Ane Cecilie D, Lars JV, Tom Ivar N, Kristian M, Rune W. Secular decline in mortality from coronary heart disease in adults with diabetes mellitus: cohort study. BMJ. 2008; Saaddine JB, Cadwell B, Gregg EW, et al. Improvements in diabetes processes of care and intermediate outcomes: United States, Ann Intern Med. Apr ;144(7): Eccles M, Grimshaw J, Campbell M, Ramsay C. Research designs for studies evaluating the effectiveness of change and improvement strategies. Quality and Safety in Health Care. 2003;12(1): Simon L, Claire G, Andrew DO. Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study. BMJ. 2009; Tripp-Reimer T, Doebbeling B. Evidence-based quality improvement: the state of the science. Worldviews Evid Based Nurs. 2004;1:Suppl 1:S Sandelowski M. Combining qualitative and quantitative sampling, data collection, and analysis techniques in mixed-method studies. Res Nurs Health. Jun 2000;23(3): Peirce CS. Collected Papers of Charles Sanders Peirce. Vol 1 6. Cambridge, MA: Harvard University Press;

182 125. Upshur RE. Seven characteristics of medical evidence. J Eval Clin Pract. May 2000;6(2): Upshur R. Certainty, probability and abduction: why we should look to C.S. Peirce rather than Godel for a theory of clinical reasoning. J Eval Clin Pract. Aug 1997;3(3): Morgan DL. Paradigms Lost and Pragmatism Regained: Methodological Implications of Combining Qualitative and Quantitative Methods. Journal of Mixed Methods Research. 2007;1(1): Kuzel A. Sampling in Qualitative Inquiry. In: Crabtree B, Miller W, eds. Doing Qualitative Research. Research Methods for Primary Care. Vol 3. California: Sage Publications Inc; 1992: Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis. Aug 1967;20(8): Zwarenstein M, Oxman A, Pragmatic Trials in Health Care S. Why are so few randomized trials useful, and what can we do about it? J Clin Epidemiol. Nov 2006;59(11): Ivers N, Jamtvedt G, Flottorp S, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;6:CD Ivers N, Tu K, Young J, et al. Feedback GAP: pragmatic, cluster-randomized trial of goal setting and action plans to increase the effectiveness of audit and feedback interventions in primary care. Implementation Science. 2013;8(1): Ivers N, Barnsley J, Upshur R, et al. 'My job is one patient at a time': perceived discordance between population-level quality targets and patient-centered care inhibits quality improvement. Can Fam Physician Hysong S, Best R, Pugh J. Audit and feedback and clinical practice guideline adherence: making feedback actionable. Implement Sci. 2006;1: van der Veer S, de Keizer N, Ravelli A, Tenkink S, Jager K. Improving quality of care. A systematic review on how medical registries provide information feedback to health care providers. Int J Med Inform. 2010;79(5): Kiefe C, Allison J, Williams O, Person S, Weaver M, Weissman N. Improving quality improvement using achievable benchmarks for physician feedback: a randomized controlled trial. JAMA. 2001;285(22): Sondergaard J, Andersen M, Kragstrup J, Hansen P, Freng Gram L. Why has postal prescriber feedback no substantial impact on general practitioners' prescribing practice? A qualitative study. Eur J Clin Pharmacol. May 2002;58(2):

183 138. Kluger A, DeNisi A. The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull. 1996;119(2): Gardner B, Whittington C, McAteer J, Eccles M, Michie S. Using theory to synthesise evidence from behavior change interventions: the example of audit and feedback. Soc Sci Med. 2010;70(10): Locke E, Latham G. Building a practically useful theory of goal setting and task motivation. A 35-year odyssey. Am Psychol. 2002;57(9): Nasser M, Oxman A, Paulson E, Fedorowicz Z. Local consensus processes: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews. 2007(1) Sniehotta F. Towards a theory of intentional behavior change: plans, planning, and selfregulation. Br J Health Psychol. 2009;14(Pt 2): Cochrane Effective Practice and Organisation of Care Group (EPOC). Cochrane Effective Practice and Organisation of Care Review Group Data Collection Checklist. 2002; Accessed March 30, Axt-Adam P, van der Wouden JC, van der Does E. Influencing behavior of physicians ordering laboratory tests: a literature study. Med Care. Sep 1993;31(9): Foy R, MacLennan G, Grimshaw J, Penney G, Campbell M, Grol R. Attributes of clinical recommendations that influence change in practice following audit and feedback. J Clin Epidemiol. Jul 2002;55(7): Gardner B, Whittington C, McAteer J, Eccles MP, Michie S. Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback. Soc Sci Med. May 2010;70(10): Guyatt GH, Oxman AD, Kunz R, et al. What is "quality of evidence" and why is it important to clinicians? BMJ. May ;336(7651): Schünemann HJ, Oxman AD, Vist GE, et al. Chapter 12 Interpreting results and drawing conclusions. In: J.P.T. H, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. Chichester, UK: Wiley-Blackwell; The GRADE Working Group. GRADE handbook for grading quality of evidence and strength of recommendation. 3.2:Available from Everett GD, deblois CS, Chang PF, Holets T. Effect of cost education, cost audits, and faculty chart review on the use of laboratory services. Arch Intern Med. May 1983;143(5):

184 151. Medical audit in general practice. I: Effects on doctors' clinical behaviour for common childhood conditions. North of England Study of Standards and Performance in General Practice. BMJ. Jun ;304(6840): Forsetlund L, Bjorndal A, Rashidian A, et al. Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2009(2):CD Odgaard-Jensen J, Vist GE, Timmer A, et al. Randomisation to protect against selection bias in healthcare trials [protocol] Cochrane Database of Systematic Review. 2011(4):Art. No.: MR Freemantle N, Harvey EL, Wolf F, Grimshaw JM, Grilli R, Bero LA. Printed educational materials: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2000(2):CD Grimshaw JM, Shirran L, Thomas R, et al. Changing provider behavior: an overview of systematic reviews of interventions. Med Care. Aug 2001;39(8 Suppl 2):II Farmer AP, Legare F, Turcot L, et al. Printed educational materials: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2008(3):CD Guldberg TL, Lauritzen T, Kristensen JK, Vedsted P. The effect of feedback to general practitioners on quality of care for people with type 2 diabetes. A systematic review of the literature. BMC Fam Pract. 2009;10: Doumit G, Gattellari M, Grimshaw J, O'Brien MA. Local opinion leaders: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2007;24(1):CD Simera I, Moher D, Hirst A, Hoey J, Schulz KF, Altman DG. Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network. BMC Med. 2010;8: Davidoff F, Batalden P, Stevens D, Ogrinc G, Mooney SE. Publication guidelines for quality improvement studies in health care: evolution of the SQUIRE project. BMJ :00: ; Ilgen D, Fisher C, Taylor M. Consequences of individual feedback on behavior in organizations. J Appl Psychol. 1979;64(4): Ivers N, Halperin I, Barnsley J, et al. Allocation techniques for balance at baseline in cluster randomized trials: a methodological review. Trials. 2012;13: practice AMaig. I: Effects on doctors' clinical behavior for common childhood conditions. North of England study of standards and performance in general practice. BMJ. 1992;304(6840):

185 164. Gollwitzer P, Fujita K, Oettingen G. Planning and the implementation of goals. Handbook of self-regulation: research, theory, and applications. 2004: Hermans M, Elisaf M, Michel G, et al. Benchmarking is associated with improved quality of care in type 2 diabetes: the OPTIMISE randomized, controlled trial. Diabetes Care. 2013;36(11): Anderson T, Gregoire J, Hegele R, et al update of the Canadian cardiovascular society guidelines for the diagnosis and treatment of dyslipidemia for the prevention of cardiovascular disease in the adult. Can J Cardiol. 2013;29(2): Hackam D, Quinn R, Ravani P, et al. The 2013 Canadian hypertension education program recommendations for blood pressure measurement, diagnosis, assessment of risk, prevention, and treatment of hypertension. Can J Cardiol. 2013;29(5): Thorpe K, Zwarenstein M, Oxman A, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. CMAJ. 2009;180(10):E Ivers N, Tu K, Francis J, et al. Feedback GAP: study protocol for a cluster-randomized trial of goal setting and action plans to increase the effectiveness of audit and feedback interventions in primary care. Implement Sci. 2010;5: Privacy code: Protecting personal health information at ICES: Institute for Clinical Evaluative Sciences; Ivers N, Pylypenko B, Tu K. Identifying patients with ischemic heart disease in an electronic medical record. J Prim Care Community Health. 2011;2(1): Tu K, Manuel D, Lam K, Kavanagh D, Mitiku T, Guo H. Diabetics can be identified in an electronic medical record using laboratory tests and prescriptions. J Clin Epidemiol. 2011;64(4): Minim: allocation by minimisation in clinical trials. Accessed March 30, Suggested risk of bias criteria for EPOC reviews:. 20bias%20criteria%20for%20EPOC%20reviews.pdf. Accessed March 30, Bhattacharyya O, Estey E, Cheng A, Association CD. Update on the Canadian diabetes association 2008 clinical practice guidelines. Can Fam Physician. 2009;55(1): Campbell N, Kwong M, Program CHE Canadian hypertension education program recommendations: an annual update. Can Fam Physician. 2010;56(7): Holbrook A, Thabane L, Keshavjee K, et al. Individualized electronic decision support and reminders to improve diabetes care in the community: COMPETE II randomized trial. CMAJ. 2009;181(1-2):

186 178. Cooper A, Skinner J, Nherera L, et al.: London: National Collaborating Centre for Primary Care and Royal College of General Practitioners; National Collaborating Centre for Chronic Conditions. Accessed March 30, Genest J, McPherson R, Frohlich J, et al Canadian cardiovascular society/canadian guidelines for the diagnosis and treatment of dyslipidemia and prevention of cardiovascular disease in the adult recommendations. Can J Cardiol. 2009;25(10): McNutt L, Wu C, Xue X, Hafner J. Estimating the relative risk in cohort studies and clinical trials of common outcomes. Am J Epidemiol. 2003;157(10): Glynn R, Brookhart M, Stedman M, Avorn J, Solomon D. Design of cluster-randomized trials of quality improvement interventions aimed at medical care providers. Med Care. 2007;45(10 Supl 2):S Burge F, Bower K, Putnam W, Cox J. Quality indicators for cardiovascular primary care. Can J Cardiol. 2007;23(5): Hysong S, Simpson K, Pietz K, SoRelle R, Broussard Smitham K, Petersen L. Financial incentives and physician commitment to guideline-recommended hypertension management. Am J Manag Care. 2012;18(10):e Baskerville N, Liddy C, Hogg W. Systematic review and meta-analysis of practice facilitation within primary care settings. Ann Fam Med. 2012;10(1): Maroney B, Buckley M. Does research in performance appraisal influence the practice of performance appraisal. Public Pers Manage. 1992;21(2): Presseau J, Francis J, Campbell N, Sniehotta F. Goal conflict, goal facilitation, and health professionals' provision of physical activity advice in primary care: an exploratory prospective study. Implement Sci. 2011;6: Flodgren G, Parmelli E, Doumit G, et al. Local opinion leaders: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2011;8(8):CD Tu K, Mitiku T, Ivers N, et al. How comprehensive is EMR data? Evaluation of an Electronic Medical Record Administrative data Linked Database (EMRALD). Am J Manage Care Strandberg EL, Ovhed I, Hakansson A, Troein M. The meaning of quality work from the general practitioner's perspective: an interview study. BMC Fam Pract. 2006;7: Rosser WW, Colwill JM, Kasperski J, Wilson L. Progress of Ontario's Family Health Team model: a patient-centered medical home. Ann Fam Med. Mar-Apr 2011;9(2):

187 192. Ontario Ministry of Long Term Care. Stand up to diabetes - diabetes testing report. September 29, 2009;; p.html. Accessed August 8, Ontario Ministry of Long Term Care. The excellent care for all act. June 23,2010; Accessed June 6, Tu K, Manuel D, Lam K, Kavanagh D, Mitiku TF, Guo H. Diabetics can be identified in an electronic medical record using laboratory tests and prescriptions. J Clin Epidemiol. Apr 2011;64(4): Michie S, Johnston M, Abraham C, et al. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care. Feb 2005;14(1): Sapyta J, Riemer M, Bickman L. Feedback to clinicians: theory, research, and practice. J Clin Psychol. Feb 2005;61(2): Doroodchi H, Abdolrasulnia M, Foster JA, et al. Knowledge and attitudes of primary care physicians in the management of patients at risk for cardiovascular events. BMC Fam Pract. 2008;9: Berthold HK, Gouni-Berthold I, Bestehorn KP, Bohm M, Krone W. Physician gender is associated with the quality of type 2 diabetes care. J Intern Med. Oct 2008;264(4): Kuzel AJ. Doing Qualitative Research. Research Methods for Primary Care. Vol 3. California: Sage Publications; Crabtree BF, Miller WL. Doing Qualitative Research. 2nd ed. California: SAGE Publications, Inc.; Wolf ZR. Exploring the audit trail for qualitative investigations. Nurse Educ. Jul-Aug 2003;28(4): Barbour RS. Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ. May ;322(7294): Wilkinson EK, McColl A, Exworthy M, et al. Reactions to the use of evidence-based performance indicators in primary care: a qualitative study. Qual Health Care. Sep 2000;9(3): Elwyn G, Hocking P, Burtonwood A, Harry K, Turner A. Learning to plan? A critical fiction about the facilitation of professional and practice development plans in primary care. J Interprof Care. Nov 2002;16(4):

188 205. Marshall M, Sheaff R, Rogers A, et al. A qualitative study of the cultural changes in primary care organisations needed to implement clinical governance. Br J Gen Pract. Aug 2002;52(481): Tomlin Z, Humphrey C, Rogers S. General practitioners' perceptions of effective health care. BMJ. 1999;318(7197): Barnsley J, Berta W, Cockerill R, MacPhail J, Vayda E. Identifying performance indicators for family practice: assessing levels of consensus. Can Fam Physician. May 2005;51: Exworthy M, Wilkinson EK, McColl A, et al. The role of performance indicators in changing the autonomy of the general practice profession in the UK. Soc Sci Med. Apr 2003;56(7): Quality in family practice: Book of tools. Levitt C, Hilts L (editors). Hamilton: McMaster Innovation Press; Presseau J, Sniehotta FF, Francis JJ, Campbell NC. Multiple goals and time constraints: perceived impact on physicians' performance of evidence-based behaviours. Implement Sci. 2009;4: Parchman ML, Pugh JA, Romero RL, Bowers KW. Competing demands or clinical inertia: the case of elevated glycosylated hemoglobin. Ann Fam Med. May-Jun 2007;5(3): Mitchell E, Sullivan F, Grimshaw JM, Donnan PT, Watt G. Improving management of hypertension in general practice: a randomised controlled trial of feedback derived from electronic patient data. Br J Gen Pract. Feb 2005;55(511): Communities of solution: the Folsom Report revisited. Ann Fam Med. May- Jun;10(3): McLaughlin AS, Schulte DM, Sharp G. Toward a collaborative care delivery model: the role of population health improvement organizations in the patient-centered medical home. Popul Health Manag. Oct 2010;13(5): Gottlieb LM. Learning from Alma Ata: the medical home and comprehensive primary health care. J Am Board Fam Med. May-Jun 2009;22(3): Blumenthal D, Tavenner M. The "meaningful use" regulation for electronic health records. N Engl J Med. Aug ;363(6): Dermer M, Morgan M. Certification of primary care electronic medical records: lessons learned from Canada. J Healthc Inf Manag. Summer 2010;24(3): Johnston S, Green M, Thille P, et al. Performance feedback: an exploratory study to examine the acceptability and impact for interdisciplinary primary care teams. BMC Fam Pract. 2011;12:

189 219. Hollingworth W, McKell-Redwood D, Hampson L, Metcalfe C. Cost-utility analysis conducted alongside randomized controlled trials: are economic end points considered in sample size calculations and does it matter? Clin Trials. Feb 2013;10(1): Srivastava A, Hudson M, Hamoud I, Cavalcante J, Pai C, Kaatz S. Examining warfarin underutilization rates in patients with atrial fibrillation: Detailed chart review essential to capture contraindications to warfarin therapy. Thromb J. 2008;6: Bandura AC, D. Self-evaluative and self-efficacy mechanisms governing the motivational effects of goal systems. Journal of Personality and Social Psychology. 1983;45(5): Oxman AD, Fretheim A, Flottorp S. The OFF theory of research utilization. J Clin Epidemiol. Feb 2005;58(2): ; discussion French SD, Green SE, O'Connor DA, et al. Developing theory-informed behaviour change interventions to implement evidence into practice: a systematic approach using the Theoretical Domains Framework. Implement Sci. 2012;7: Damschroder LJ, Lowery JC. Evaluation of a large-scale weight management program using the consolidated framework for implementation research (CFIR). Implement Sci. 2013;8: Safran DG, Miller W, Beckman H. Organizational dimensions of relationship-centered care. Theory, evidence, and practice. J Gen Intern Med. Jan 2006;21 Suppl 1:S Cochrane Handbook for Systematic Reviews of Interventions. March 2011 ed: Higgins, J.P.T., Green, S (editors). The Cochrane Collaboration; Ioannidis JPA. Why Most Published Research Findings Are False. PLoS Med 2005;2(8):e Lau J, Antman EM, Jimenez-Silva J, Kupelnick B, Mosteller F, Chalmers TC. Cumulative meta-analysis of therapeutic trials for myocardial infarction. N Engl J Med. Jul ;327(4): Noyes J, Gough D, Lewin S, et al. A research and development agenda for systematic reviews that ask complex questions about complex interventions. J Clin Epidemiol. Nov 2013;66(11): Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review--a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. Jul 2005;10 Suppl 1: Anderson LM, Oliver SR, Michie S, Rehfuess E, Noyes J, Shemilt I. Investigating complexity in systematic reviews of interventions by using a spectrum of methods. J Clin Epidemiol. Nov 2013;66(11):

190 232. Pavel SR, Natasha F, Jeff MW, et al. Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ :43:51; Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a Li AC, Kannry JL, Kushniruk A, et al. Integrating usability testing and think-aloud protocol analysis with "near-live" clinical simulations in evaluating clinical decision support. Int J Med Inform. Nov 2012;81(11): Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8: Auerbach AD, Landefeld CS, Shojania KG. The tension between needing to improve care and knowing how to do it. N Engl J Med. Aug ;357(6): Colquhoun HL, Brehaut JC, Sales A, et al. A systematic review of the use of theory in randomized controlled trials of audit and feedback. Implement Sci. 2013;8: Dixon-Woods M, Redwood S, Leslie M, Minion J, Martin GP, Coleman JJ. Improving quality and safety of care using "technovigilance": an ethnographic case study of secondary use of data from an electronic prescribing and decision support system. Milbank Q. Sep 2013;91(3): Sales A, Smith J, Curran G, Kochevar L. Models, strategies, and tools. Theory in implementing evidence-based findings into health care practice. J Gen Intern Med. Feb 2006;21 Suppl 2:S Dimick JB, Hendren SK. Hospital Report Cards: Necessary but Not Sufficient? JAMA Surg. Dec Chokshi DA, Avorn J, Kesselheim AS. Designing comparative effectiveness research on prescription drugs: lessons from the clinical trial literature. Health Aff (Millwood). Oct 2010;29(10): Glynn RJ, Brookhart MA, Stedman M, Avorn J, Solomon DH. Design of clusterrandomized trials of quality improvement interventions aimed at medical care providers. Med Care. Oct 2007;45(10 Supl 2):S Collins LM, Baker TB, Mermelstein RJ, et al. The multiphase optimization strategy for engineering effective tobacco use interventions. Ann Behav Med. Apr 2007;41(2): Collins LM, Murphy SA, Nair VN, Strecher VJ. A strategy for optimizing and evaluating behavioral interventions. Ann Behav Med. Aug 2005;30(1):

191 245. Murphy SA. Optimal dynamic treatment regimes. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 2003;65(2): May C, Montori VM, Mair FS. We need minimally disruptive medicine. BMJ. 2009;339:b Madan A, Borckardt JJ, Connell A, et al. Routine assessment of patient-reported outcomes in behavioral health: room for improvement. Qual Manag Health Care. Jan- Mar 2010;19(1): Bonetti D, Johnston M, Pitts NB, et al. Knowledge may not be the best target for strategies to influence evidence-based practice: using psychological models to understand RCT effects. Int J Behav Med. 2009;16(3): Johnson RD. Executives Ignore Valuable Employee Actions that They Can t Measure December Benjamin A. Audit: how to do it in practice. BMJ :00: ;336(7655): Shaw CD. Aspects of audit. 3: Audit in British general practice. Br Med J. 1980;280(6228): Fried TR, Tinetti ME, Iannone L. Primary care clinicians' experiences with treatment decision making for older persons with multiple conditions. Arch Intern Med. Jan ;171(1): Schmittdiel J, Mosen DM, Glasgow RE, Hibbard J, Remmers C, Bellows J. Patient Assessment of Chronic Illness Care (PACIC) and improved patient-centered outcomes for chronic conditions. J Gen Intern Med. Jan 2008;23(1): Dwamena F, Holmes-Rovner M, Gaulden CM, et al. Interventions for providers to promote a patient-centred approach in clinical consultations. Cochrane Database Syst Rev. 2012;12:CD McAlister FA, Feldman RD, Wyard K, Brant R, Campbell NR. The impact of the Canadian Hypertension Education Programme in its first decade. Eur Heart J. Jun 2009;30(12): Higgins ET. Beyond pleasure and pain. Am Psychol. Dec 1997;52(12): Simpson KM, Porter K, McConnell ES, et al. Tool for evaluating research implementation challenges: a sense-making protocol for addressing implementation challenges in complex research settings. Implement Sci. 2013;8: Torgerson DJ, Roland M. What is Zelen's design? BMJ. Feb ;316(7131):

192 259. Hux JE, Melady MP, DeBoer D. Confidential prescriber feedback and education to improve antibiotic use in primary care: a controlled trial. CMAJ. Aug ;161(4): Pimlott NJ, Hux JE, Wilson LM, Kahan M, Li C, Rosser WW. Educating physicians to reduce benzodiazepine use by elderly patients: a randomized controlled trial. CMAJ. Apr ;168(7): Service Ontario. Personal Health Information Protection Act. From January 1, 2013; A Guide to the personal health information protection act. March 30,

193 9 Appendices 9.1 Appendix Electronic Search Strategies CENTRAL #1 MeSH descriptor Clinical Audit, this term only #2 MeSH descriptor Medical Audit, this term only #3 MeSH descriptor Nursing Audit, this term only #4 MeSH descriptor Dental Audit, this term only #5 MeSH descriptor Management Audit, this term only #6 MeSH descriptor Benchmarking, this term only #7 MeSH descriptor Commission on Professional and Hospital Activities, this term only #8 MeSH descriptor Feedback, this term only #9 MeSH descriptor Feedback, Psychological, this term only #10 MeSH descriptor Utilization Review, this term only #11 MeSH descriptor Drug Utilization Review, this term only #12 MeSH descriptor Concurrent Review, this term only #13 MeSH descriptor Peer Review, Health Care, this term only #14 (audit or audits or auditing or feedback or benchmark*):ti,ab #15 (review NEAR/3 record* or chart NEXT review or practice NEXT data or hospital* NEXT data):ti,ab #16 (#1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7 OR #8 OR #9 OR #10 OR #11 OR #12 OR #13 OR #14 OR #15) #17 MeSH descriptor Health Personnel explode all trees #18 MeSH descriptor Hospitals explode all trees #19 MeSH descriptor Professional Practice explode all trees #20 MeSH descriptor Family Practice, this term only #21 MeSH descriptor Professional Competence, this term only #22 MeSH descriptor Clinical Competence, this term only #23 MeSH descriptor Physician's Practice Patterns, this term only #24 MeSH descriptor Nurse's Practice Patterns, this term only #25 MeSH descriptor Dentist's Practice Patterns, this term only #26 MeSH descriptor Quality Assurance, Health Care, this term only #27 MeSH descriptor Quality of Health Care, this term only #28 (health* NEXT personnel or "health care personnel" or physician* or doctor* or clinician* or nurse* or provider* or practitioner* or resident* or professional* or nursing or clinical) NEAR/3 (skill or skills or behaviour or behavior or competence):ti,ab #29 (clinical or medical or dental or private or general or family or professional or hospital*) NEXT practice*:ti,ab #30 (practice NEAR/2 pattern*):ti,ab #31 quality NEXT (assurance or improvement or control):ti,ab #32 (health* or care) NEAR/2 quality:ti,ab #33 performance:ti,ab #34 (influenc* NEAR/3 behaviour* or influenc* NEAR/3 behavior* or chang* NEAR/3 behaviour* or chang* NEAR/3 behavior*):ti,ab #35 (#17 OR #18 OR #19 OR #20 OR #21 OR #22 OR #23 OR #24 OR #25 OR #26 OR #27 OR #28 OR #29 OR #30 OR #31 OR #32 OR #33 OR #34) #36 (#16 AND #35) 178

194 #37 audit* NEAR/3 feedback:ti,ab #38 (#36 OR #37) MEDLINE 1. (audit* adj3 feedback).tw. 2. Clinical Audit/ 3. Medical Audit/ 4. Nursing Audit/ 5. Dental Audit/ 6. Management Audit/ 7. Benchmarking/ 8. "Commission on Professional and Hospital Activities"/ 9. Feedback/ 10. Feedback, Psychological/ 11. Utilization Review/ 12. Drug Utilization Review/ 13. Concurrent Review/ 14. Peer Review, Health Care/ 15. (audit or audits or auditing).tw. 16. feedback.tw. 17. (review adj3 record?).tw. 18. chart review.tw. 19. (practice data or hospital* data).tw. 20. benchmark*.tw. 21. or/ exp Health Personnel/ 23. exp Hospitals/ 24. exp Professional Practice/ 25. Family Practice/ 26. Professional Competence/ 27. Clinical Competence/ 28. Physician's Practice Patterns/ 29. Nurse's Practice Patterns/ 30. Dentist's Practice Patterns/ 31. Quality Assurance, Health Care/ 32. Quality of Health Care/ 33. ((health* personnel or health care personnel or physician? or doctor? or clinician? or nurse? or provider? or practitioner? or resident? or professional? or nursing or clinical) adj3 (skill or skills or behaviour or behavior or competence)).tw. 34. ((clinical or medical or dental or private or general or family or professional or hospital?) adj practice?).tw. 35. (practice pattern? or pattern of practice).tw. 36. (quality adj (assurance or improvement or control)).tw. 37. (health care quality or healthcare quality or quality of healthcare or quality of health care or quality of care).tw. 38. performance.tw. 39. ((influenc* or chang*) adj3 (behaviour* or behavior*)).tw. 40. or/ randomized controlled trial.pt. 42. controlled clinical trial.pt. 43. (randomi* or randomly).tw. 44. or/ Animals/ 46. Humans/ not (45 and 46) not and 40 and and or

195 52. (2005* or 2006* or 2007* or 2008* or 2009* or 2010*).ed,ep,yr and 52 EMBASE 1. (audit* adj3 feedback).tw. 2. Medical Audit/ 3. Feedback System/ 4. Negative Feedback/ 5. Positive Feedback/ 6. "Utilization Review"/ 7. "Medical Record Review"/ 8. (audit or audits or auditing).tw. 9. feedback.tw. 10. (review adj3 record?).tw. 11. chart review.tw. 12. (practice data or hospital* data).tw. 13. benchmark*.tw. 14. or/ exp Health Care Personnel/ 16. exp Hospital/ 17. exp Professional Practice/ 18. Professional Competence/ 19. Nursing Competence/ 20. Clinical Competence/ 21. Health Care Quality/ 22. Quality Control/ 23. ((health* personnel or health care personnel or physician? or doctor? or clinician? or nurse? or provider? or practitioner? or resident? or professional? or nursing or clinical) adj3 (skill or skills or behaviour or behavior or competence)).tw. 24. ((clinical or medical or dental or private or general or family or professional or hospital?) adj practice?).tw. 25. (practice pattern? or pattern of practice).tw. 26. (quality adj (assurance or improvement or control)).tw. 27. (health care quality or healthcare quality or quality of healthcare or quality of health care or quality of care).tw. 28. performance.tw. 29. ((influenc* or chang*) adj3 (behaviour* or behavior*)).tw. 30. or/ Randomized Controlled Trial/ 32. (randomi* or randomly).tw. 33. or/ Nonhuman/ not and 30 and and or not medlinex00ae.cr *.em and 40 CINAHL S47 S46 - Limiters - Exclude MEDLINE records S46 S44 or S45 S45 S42 and S43 S44 S13 and S36 and S42 S43 TI ( audit* and feedback ) or AB ( audit* and feedback ) S42 S37 or S38 or S39 or S40 or S41 S41 TI ( ( randomi* or randomly ) ) or AB ( ( randomi* or randomly ) ) S40 (MH "Simple Random Sample") 180

196 S39 S38 S37 S36 (MH "Random Sample") (MH "Random Assignment") (MH "Clinical Trials") S14 or S15 or S16 or S17 or S18 or S19 or S20 or S21 or S22 or S23 or S24 or S25 or S26 or S27 or S28 or S29 or S30 or S31 or S32 or S33 or S34 or S35 S35 TI ( influenc* N3 behaviour* or influenc* N3 behavior* or chang* N3 behaviour* or chang* N3 behavior* ) or AB ( influenc* N3 behaviour* or influenc* N3 behavior* or chang* N3 behaviour* or chang* N3 behavior* ) S34 S33 S32 S31 S30 S29 S28 S27 S26 TI performance or AB performance TI ( "health care quality" or "healthcare quality" or quality W1 healthcare or quality W2 care ) or AB ( "health care quality" or "healthcare quality" or quality W1 healthcare or quality W2 care ) TI ( quality W0 assurance or quality W0 improvement or quality W0 control ) or AB ( quality W0 assurance or quality W0 improvement or quality W0 control ) TI practice N1 pattern* or AB practice N1 pattern* TI ( clinical W0 practice* or medical W0 practice* or dental W0 practice* or private W0 practice* or general W0 practice* or family W0 practice* or professional W0 practice* or hospital* W0 practice* ) or AB ( clinical W0 practice* or medical W0 practice* or dental W0 practice* or private W0 practice* or general W0 practice* or family W0 practice* or professional W0 practice* or hospital* W0 practice* ) TI ( "health personnel" N3 competence or "healthcare personnel" N3 competence or "health care personnel" N3 competence or physician N3 competence or physicians N3 competence or doctor N3 competence or doctors N3 competence or clinician N3 competence or clinicians N3 competence or nurse N3 competence or nurses N3 competence or provider N3 competence or providers N3 competence or practitioner N3 competence or practitioners N3 competence or resident N3 competence or residents N3 competence or professional N3 competence or professionals N3 competence or nursing N3 competence or clinical N3 competence ) or AB ( "health personnel" N3 competence or "healthcare personnel" N3 competence or "health care personnel" N3 competence or physician N3 competence or physicians N3 competence or doctor N3 competence or doctors N3 competence or clinician N3 competence or clinicians N3 competence or nurse N3 competence or nurses N3 competence or provider N3 competence or providers N3 competence or practitioner N3 competence or practitioners N3 competence or resident N3 competence or residents N3 competence or professional N3 competence or professionals N3 competence or nursing N3 competence or clinical N3 competence ) TI ( "health personnel" N3 behavior or "healthcare personnel" N3 behavior or "health care personnel" N3 behavior or physician N3 behavior or physicians N3 behavior or doctor N3 behavior or doctors N3 behavior or clinician N3 behavior or clinicians N3 behavior or nurse N3 behavior or nurses N3 behavior or provider N3 behavior or providers N3 behavior or practitioner N3 behavior or practitioners N3 behavior or resident N3 behavior or residents N3 behavior or professional N3 behavior or professionals N3 behavior or nursing N3 behavior or clinical N3 behavior ) or AB ( "health personnel" N3 behavior or "healthcare personnel" N3 behavior or "health care personnel" N3 behavior or physician N3 behavior or physicians N3 behavior or doctor N3 behavior or doctors N3 behavior or clinician N3 behavior or clinicians N3 behavior or nurse N3 behavior or nurses N3 behavior or provider N3 behavior or providers N3 behavior or practitioner N3 behavior or practitioners N3 behavior or resident N3 behavior or residents N3 behavior or professional N3 behavior or professionals N3 behavior or nursing N3 behavior or clinical N3 behavior ) TI ( "health personnel" N3 behaviour or "healthcare personnel" N3 behaviour or "health care personnel" N3 behaviour or physician N3 behaviour or physicians N3 behaviour or doctor N3 behaviour or doctors N3 behaviour or clinician N3 behaviour or clinicians N3 behaviour or nurse N3 behaviour or nurses N3 behaviour or provider N3 behaviour or providers N3 behaviour or practitioner N3 behaviour or practitioners N3 behaviour or resident N3 behaviour or residents N3 behaviour or professional N3 behaviour or professionals N3 behaviour or nursing N3 behaviour or clinical N3 behaviour ) or AB ( "health personnel" N3 behaviour or "healthcare personnel" N3 behaviour or "health care personnel" N3 behaviour or physician N3 behaviour or physicians N3 behaviour or doctor N3 behaviour or doctors N3 behaviour or clinician N3 behaviour or clinicians N3 behaviour or nurse N3 behaviour or nurses N3 behaviour or provider N3 behaviour or providers N3 behaviour or practitioner N3 behaviour or practitioners N3 behaviour or resident N3 behaviour or residents N3 behaviour or professional N3 behaviour or professionals N3 behaviour or nursing N3 behaviour or clinical N3 behaviour ) TI ( "health personnel" N3 skills or "healthcare personnel" N3 skills or "health care personnel" N3 skills or physician N3 skills or physicians N3 skills or doctor N3 skills or doctors N3 skills or clinician N3 skills or 181

197 clinicians N3 skills or nurse N3 skills or nurses N3 skills or provider N3 skills or providers N3 skills or practitioner N3 skills or practitioners N3 skills or resident N3 skills or residents N3 skills or professional N3 skills or professionals N3 skills or nursing N3 skills or clinical N3 skills ) or AB ( "health personnel" N3 skills or "healthcare personnel" N3 skills or "health care personnel" N3 skills or physician N3 skills or physicians N3 skills or doctor N3 skills or doctors N3 skills or clinician N3 skills or clinicians N3 skills or nurse N3 skills or nurses N3 skills or provider N3 skills or providers N3 skills or practitioner N3 skills or practitioners N3 skills or resident N3 skills or residents N3 skills or professional N3 skills or professionals N3 skills or nursing N3 skills or clinical N3 skills ) S25 TI ( "health personnel" N3 skill or "healthcare personnel" N3 skill or "health care personnel" N3 skill or physician N3 skill or physicians N3 skill or doctor N3 skill or doctors N3 skill or clinician N3 skill or clinicians N3 skill or nurse N3 skill or nurses N3 skill or provider N3 skill or providers N3 skill or practitioner N3 skill or practitioners N3 skill or resident N3 skill or residents N3 skill or professional N3 skill or professionals N3 skill or nursing N3 skill or clinical N3 skill ) or AB ( "health personnel" N3 skill or "healthcare personnel" N3 skill or "health care personnel" N3 skill or physician N3 skill or physicians N3 skill or doctor N3 skill or doctors N3 skill or clinician N3 skill or clinicians N3 skill or nurse N3 skill or nurses N3 skill or provider N3 skill or providers N3 skill or practitioner N3 skill or practitioners N3 skill or resident N3 skill or residents N3 skill or professional N3 skill or professionals N3 skill or nursing N3 skill or clinical N3 skill ) S24 (MH "Quality of Nursing Care") S23 (MH "Quality of Health Care") S22 (MH "Quality Assurance") S21 (MH "Prescribing Patterns") S20 (MH "Practice Patterns") S19 (MH "Nursing Skills") S18 (MH "Clinical Competence") S17 (MH "Professional Competence") S16 (MH "Professional Practice+") S15 (MH "Hospitals+") S14 (MH "Health Personnel+") S13 S1 or S2 or S3 or S4 or S5 or S6 or S7 or S8 or S9 or S10 or S11 or S12 S12 TI benchmark* or AB benchmark* S11 TI hospital* W0 data or AB hospital* W0 data S10 TI "practice data" or AB "practice data" S9 TI "chart review" or AB "chart review" S8 TI review N3 record* or AB review N3 record* S7 TI feedback or AB feedback S6 TI ( audit or audits or auditing or feedback ) or AB ( audit or audits or auditing or feedback ) S5 (MH "Utilization Review") S4 (MH "Feedback") S3 (MH "Benchmarking") S2 (MH "Nursing Audit") S1 (MH "Audit") List of Included studies Anderson 1994 Anderson FA Jr, Wheeler HB, Goldberg RJ, Hosmer DW, Forcier A, Patwardhan NA. Changing clinical practice. Prospective study of the impact of continuing medical education and quality assurance programs on use of prophylaxis for venous thromboembolism. Archives of Internal Medicine 1994;154: Avery

198 Anthony J Avery, Sarah Rodgers, Judith A Cantrill, Sarah Armstrong, Matthew Boyd, Kathrin Cresswell, Martin Eden, Rachel Elliott, Matthew Franklin, Julia Hippisley-Cox, Rachel Howard, Denise Kendrick, Caroline J Morris, Scott A Murray, Robin J Prescott, Koen Putman, Glen Swanwick, Lorna Tuersley, Tom Turner, Yana Vinogradova, Aziz Sheikh. PINCER trial: a cluster randomised trial comparing the effectiveness and cost-effectiveness of a pharmacist-led IT-based intervention with simple feedback in reducing rates of clinically important errors in medicines management in general practices. In: A report for the Department of Health Patient Safety Research Portfolio Cantrill J.. In: International Journal of Pharmacy Practice Conference:5. Awad 2006 Awad AI, Eltayeb IB, Baraka OZ, Cano-Garcinuno A, az-vazquez C, Carvajal-Uruena I, Praena-Crespo M, Gatti- Vinoly A, Garcia-Guerra I. Changing antibiotics prescribing practices in health centers of Khartoum State, Sudan; Group education on asthma for children and caregivers: a randomized, controlled trial addressing effects on morbidity and quality of life. European Journal of Clinical Pharmacology 2006;62: Bahrami 2004 Bahrami M, Deery C, Clarkson JE, Pitts NB, Johnston M, Ricketts I, MacLennan G, Nugent ZJ, Tilley C, Bonetti D, Ramsay C. Effectiveness of strategies to disseminate and implement clinical guidelines for the management of impacted and unerupted third molars in primary dental care, a cluster randomised controlled trial. British Dental Journal 2004;197(11): Baker 1997 Baker R, Farooqui A, Tait C, Walsh S. Randomised controlled trial of reminders to enhance the impact of audit in general practice on management of patients who use benzodiazepines. Quality in Health Care 1997;6:14-8. Baker 2003 Baker R, Falconer J, Lambert PC. Randomized controlled trial of the effectiveness of feedback in improving test ordering in general practice. Scandinavian Journal of Primary Health Care 2003;21: Baker 2003A Baker R, Fraser RC, Stone M, Lambert P, Stevenson K, Shiels C. Randomised controlled trial of the impact of guidelines, prioritised review criteria and feedback on implementation of recommendations for angina and asthma.. British Journal of General Practice 2003;53: Balas 1998 Balas E, Boren SA, Hicks LL, Chonko AM, Stephenson K. Effect of linking practice data to published evidence: A randomized controlled trial of clinical direct reports. Medical Care 1998;36: Batty 2001 Batty G, Oborne CA, Hooper R, Jackson S. Investigation of intervention strategies to increase the appropriate use of antithrombotics in elderly hospital inpatients with atrial fibrillation. Journal of Clinical Governance 2001;9: Beck 2005 Beck CA, Richard H, Tu JV, Pilote L. Administrative Data Feedback for Effective Cardiac Treatment: AFFECT, a cluster randomized trial. JAMA 2005;294: Bentz 2007 Bentz CJ, Bayley KB, Bonin KE, Fleming L, Hollis JF, Hunt JS, LeBlanc B, McAfee T, Payne N, Siemienczuk J. Provider feedback to improve 5A's tobacco cessation in primary care: a cluster randomized clinical trial. Nicotine and tobacco research 2007;9(3): Berman 1998 Berman MF, Simon AE. The effect of a drug and supply cost feedback system on the use of intraoperative resources by anesthesiologists. Anesthesia and Analgesia 1998;86: Blais 2008 Blais R, Laurier C, Paré M. Effect of feedback letters to physicians and pharmacists on the appropriate use of medication in the treatment of asthma. Journal of Asthma 2008;45(3): Boekeloo 1990 Boekeloo BO, Becker DM, Levine DM, Belitsos PC, Pearson TA. Strategies for increasing house staff management of cholesterol with inpatients. American Journal of Preventive Medicine 1990;6(Suppl 2):51-9. Bonevski 1999 Bonevski B, Sanson-Fisher RW, Campbell E, Carruthers A, Reid ALA, Ireland M. Randomized controlled trial of a computer strategy to increase general practitioner preventive care. Preventive Medicine 1999;29: Borgiel 1999 Borgiel AEM, Williams JI, Davis DA, Dunn EV, Hobbs N, Hutchison B, et al. Evaluating the effectiveness of 2 educational interventions on family practice. Canadian Medical Association 1999;8: Brady

199 Brady WJ, Hissa DC, McConnell M, Wones RG. Should physicians perform their own quality assurance audits? Journal of General Internal Medicine 1988;3: Bregnhoj 2009 Bregnhøj L, Thirstrup S, Kristensen MB, Bjerrum L, Sonne J. Combined intervention programme reduces inappropriate prescribing in elderly patients exposed to polypharmacy in primary care. European journal of clinical pharmacology 2009;65(2): Brown 1994 Brown LF, Keily PA, Spencer AJ. Evaluation of a continuing education intervention "Periodontics in General Practice". Community Dentistry and Oral Epidemiology 1994;22: Buffington 1991 Buffington J, Bell KM, LaForce FM. A target-based model for increasing influenza immunizations in private practice. Journal of General Internal Medicine 1991;6: Buntinx 1993 Buntinx F, Knottnerus JA, Crebolder HF, Seegers T, Essed GG, Schouten H. Does feedback improve the quality of cervical smears? A randomized controlled trial. British Journal of General Practice 1993;43: Canovas 2009 Cánovas JJ, Hernández PJ, Botella JJ. Effectiveness of internal quality assurance programmes in improving clinical practice and reducing costs. Journal of evaluation in clinical practice 2009;15(5): Charrier 2008 Charrier L, Allochis MC, Cavallo MR, Gregori D, Cavallo F, Zotti CM. Integrated audit as a means to implement unit protocols: a randomized and controlled study. Journal of evaluation in clinical practice. 2008;14(5): Chassin 1986 Chassin MR, McCue SM. A randomized trial of medical quality assurance. Improving physicians' use of pelvimetry. JAMA 1986;256: Cheater 2006 Cheater FM, Baker R, Reddish S, Spiers N, Wailoo A, Gillies C, Robertson N, Cawood C. Cluster randomized controlled trial of the effectiveness of audit and feedback and educational outreach on improving nursing practice and patient outcomes. Medical care 2006;44(6): Claes 2005 Claes N, Buntinx F, Vijgen J, Arnout J, Vermylen J, Fieuws S et al. The Belgian Improvement Study on Oral Anticoagulation Therapy: a randomized clinical trial. Eur.Heart J 2005;26: Cline 2007 Cline D, Ayala C, Caskie D, Ferrario C. Patient Specific Feedback Increases Referral of Hypertensive Emergency Department Patients: A Randomized Controlled Trial. Academic Emergency Medicine 2007;14(5):S117. Cohen 1982 Cohen DI, Jones P, Littenberg B, Neuhauser D. Does cost information availability reduce physician test usage? A randomized clinical trial with unexpected findings. Medical Care 1982;20: Curran 2008 Curran E, Harper P, Loveday H, Gilmour H, Jones S, Benneyan J, Hood J, Pratt R. Results of a multicentre randomised controlled trial of statistical process control charts and structured diagnostic tools to reduce wardacquired meticillin-resistant Staphylococcus aureus: the CHART Project. Journal of hospital infection 2008;70(2): Curtis 2005 Curtis JR, Olivieri J, Allison JJ, Gaffo A, Juarez L, Kovac SH, Person S, Saag KG. A group randomized trial to improve safe use of nonsteroidal anti-inflammatory drugs. American journal of managed care 2005;11(9): Curtis 2007 Curtis JR, Westfall AO, Allison J, Becker A, Melton ME, Freeman A, Kiefe CI, MacArthur M, Ockershausen T, Stewart E, Weissman N, Saag KG. Challenges in improving the quality of osteoporosis care for long-term glucocorticoid users: a prospective randomized trial. Archives of internal medicine 2007;167(6): De Almeida Neto 2000 Neto ACDA, Benrimoj SI, Kavanagh DJ, Boakes RA. A pharmacy based protocol and training program for nonprescription analgesics. Journal of Social and Administrative Pharmacy 2000;17(3): Eccles 2001 Eccles M, Steen N, Grimshaw J, Thomas L, McNamee P, Soutter J, et al. Effect of audit and feedback, and reminder messages on primary-care radiology referrals: a randomised trial. Lancet 2001;357(9266): Eltayeb

200 Eltayeb IB, Awad AI, Mohamed-Salih MS, Daffa-Alla MA, Ahmed MB, Ogail MA et al. Changing the prescribing patterns of sexually transmitted infections in the White Nile Region of Sudan. Sexually transmitted infections 2005;81: Everett 1983 Everett GD, deblois CS, Chang PF, Holets T. Effect of cost education, cost audits, and faculty chart review on the use of laboratory services. Archives of Internal Medicine 1983;143: Fairbrother 1999 Fairbrother G, Hanson KL, Friedman S, Butts GC. The impact of physician bonuses, enhanced fees, and feedback on childhood immunization coverage rates. American Journal of Public Health 1999;89(2): Ferguson 2003 Ferguson TB, Peterson ED, Coombs LP, Eiken MC, Carey ML, Grover FL, et al. Use of contiouous quality improvement to increase use of process measures in patients undergoing coronary artery bypass graft surgery. JAMA 2003;290(1): Filardo 2009 Filardo G, Nicewander D, Herrin J, Edwards J, Galimbertti P, Tietze M et al. A hospital-randomized controlled trial of a formal quality improvement educational program in rural and small community Texas hospitals: one year results. International journal of quality in health care 2009;21: Foster 2007 Foster JM, Hoskins G, Smith B, Lee AJ, Price D, Pinnock H. Practice development plans to improve the primary care management of acute asthma: randomised controlled trial. BMC family practice 2007;8:23. Foy 2004 Foy R, Penney GC, Grimshaw JM, Ramsay CR, Walker AE, MacLennan G, Stearns SC, McKenzie L, Glasier A. A randomised controlled trial of a tailored multifaceted strategy to promote implementation of a clinical guideline on induced abortion care. BJOG 2004;111(7): Frijiling 2002 Frijling BD, Lobo CM, Hulscher MEJL, Akkarmans RP, Braspenning JCC, Prins A, et al. Multifaceted support to improve clinical decision making in diabetes care: a randomized controlled trial in general practice.. Diabetic Medicine 2002;19: Frijiling 2003 Frijling BD, Lobo CM, Hulscher MEJL, Akkarmans RP, van Drenth BB, Prins A, van der Wouden JC, Grol RPTM.. Intensive support to improve clinical decision making in cardiovascular care: a randomised controlled trial in general practice.. Qual Saf Health Care 2003;12: Gama 1992 Gama R, Nightingale PG, Broughton PM, Peters M, Ratcliffe JG, Bradby GV, Berg J.. Modifying the request behaviour of clinicians. Journal of clinical pathology 1992;45(8): Gehlbach 1984 Gehlbach SH, Wilkinson WE, Hammond WE, Clapp NE, Finn AL, Taylor WJ, et al. Improving drug prescribing in a primary care practice. Medical Care 1984;22: Goff 2003 Goff DC, Gu L, Cantley LK, Sheedy DJ, Cohen SJ. Quality of care for secondary prevention for patients with coronary heart disease: Results of the hastening the effective application of research through technology (HEART) trial.. American Heart Journal 2003;146(6): Grady 1997 Grady KE, Lemkau JP, Lee NR, Caddell C. Enhancing mammography referral in primary care. Preventive Medicine 1997;26: Guadagnoli 2000 Guadagnoli E, Soumerai SB, Gurwitz JH, Borbas C, Shapiro CL, Weeks JC, et al. Improving discussion of surgical treatment options for patients with breast cancer: local medical opinion leaders versus audit and performance feedback. Breast Cancer Research and Treatment 2000;61(2): Gullion 1988 Gullion DS, Tschann JM, Adamson TE, Coates TJ. Management of hypertension in private practice: a randomized controlled trial in continuing medical education. The Journal of Continuing Education in the Health Professions 1988;8: Hayes 2001 Hayes R, Bratzler D, Armour B, Moore l. Comparison of an enhanced versus written feedback model on the management of Medicare inpatients with venous thrombosis. Joint Commission Journal on Quality Improvement 2001;27(3):

201 Hayes 2002 Hayes RP, Baker DW, Luthi JC, Baggett RL, McClellan W, FitzGerald D et al. The effect of external feedback on the management of medicare inpatients with congestive heart failure. American journal of medical quality 2002;17: Heller 2001 Heller RF, DEste C, Lim LL, OConnel RL, Powell H. Randomised controlled trial to change hospital management of unstable angina. Medical Journal of Australia 2001;175(5): Hemminiki 1992 Hemminiki E, Teperi J, Tuominen K. Need for and influence or feedback from the Finnish birth register to data providers. Quality Assurance in Health Care 1992;4(2): Hendryx 1998 Hendryx MS, Fieselmann JF, Bock MJ, Wakefield DS, Helms CM, Bentler SE. Outreach education to improve quality of rural icu care. American Journal of Respiratory Critical Care Medicine 1998;158: Herbert 2004 Herbert CP, Wright JM, Maclure M, Wakefield J, Dormuth C, Brett-MacLean P et al. Better Prescribing Project: a randomized controlled trial of the impact of case-based educational modules and personal prescribing feedback on prescribing for hypertension in primary care. Family practice 2004;21: Herrin 2006 Herrin J, Nicewander DA, Hollander PA, Couch CE, Winter FD, Haydar ZR, Warren SS, Ballard DJ. Effectiveness of diabetes resource nurse case management and physician profiling in a fee-for-service setting: a cluster randomized trial. Proceedings (Baylor University Medical Centre) 2006;19(2): Hershey 1986 Hershey CO, Porter DK, Breslau D, Cohen DI. Influence of simple computerized feedback on prescription charges in an ambulatory clinic. A randomized clinical trial. Medical Care 1986;24: Hershey 1988 Hershey CO, Goldberg HI, Cohen DI. The effect of computerized feedback coupled with a newsletter upon outpatient prescribing charges. A randomized controlled trial. Medical Care 1988;26(1): [MEDLINE: ] Hillman 1998 * Hillman AL, Ripley K, Goldfarb N, Nuamah I, Weiner J, Lusk E. Physician financial incentives and feedback: Failure to increase cancer screening in medicaid managed care. American Journal of Public Health 1998;88(11): Hillman 1999 Hillman AL, Ripley K, Goldfarb N, Weiner J, Nuamah I, Lusk E. The use of physician financial incentives and feedback to improve pediatric preventive care in Medicaid care. Pediatrics 1999;104(4): Holm 1990 Holm M. Intervention against long-term use if hypnotics/sedatives in general practice. Scandinavian Journal of Primary Health Care 1990;8: Hux 1999 Hux JE, Melady MP, DeBoer D. Confidential prescriber feedback and education to improve antibiotic use in primary care: a controlled trial. Canadian Medical Association 1999;161: Kahan 2009 Kahan NR, Kahan E, Waitman DA, Kitai E, Chintz DP. The tools of an evidence-based culture: implementing clinical-practice guidelines in an Israeli HMO. Academic medicine 2009;84(9): Kerry 2000 Kerry S, Oakeshott P, Dundas D, Williams J. Influence of postal distribution of the royal college of radiologists guidelines, together with feedback on radiological referral rates, on x-ray referrals from general practice: a randomized controlled trial. Family Practice 2000;17(1): Kiefe 2001 Kiefe CI, Allison JJ, Williams OD, Person SD, Weaver MT, Weissman NW. Improving quality improvement using achievable benchmarks for physician feedback: a randomized controlled trial. JAMA 2001;285(22): Kim 1999 Kim CS, Kristopaitis RJ, Stone E, Pelter M, Sandhu M, Weingarten SR. Physician education and report cards: do they make the grade? Results from a randomized controlled trial. The American Journal of Medicine 1999;107: Kinsinger

202 Kinsinger LS, Harris R, Qaqish B, Strecher V, Kaluzny A. Using an office system intervention to increase breast cancer screening. JGIM 1998;13: Kogan 2003 Kogan JR, Reynolds EE, Shea JA. Effectiveness of report cards based on chart audits of residents adherence to practice guidelines on practice performance: a randomized controlled trial. Teaching and Learning in Medicine 2003;15(1): Kritchevsky 2008 Kritchevsky SB, Braun BI, Bush AJ, Bozikis MR, Kusek L, Burke JP, Wong ES, Jernigan J, Davis CC, Simmons B; TRAPE Study Group. The effect of a quality improvement collaborative to improve antimicrobial prophylaxis in surgical patients: a randomized trial. Annals of Internal Medicine 2008;149(7):472-80, W Lagerløv 2000 Lagerløv P, Loeb M, Andrew M, Hjortdahl P. Improving doctors' prescribing behaviour through reflection on guidelines and prescription feedback: a randomised controlled study. Quality in health care 2000;9: Laskshminarayan 2010 Lakshminarayan K, Borbas C, McLaughlin B, Morris NE, Vazquez G, Luepker RV et al. A cluster-randomized trial to improve stroke care in hospitals. Neurology 2010;74: Linn BS 1980 Linn BS. Continuing medical education. Impact on emergency room burn care. JAMA 1980;244: Lobach 1996 Lobach DF. Electronically distributed computer-generated feedback enhances the use of a computarized practice guidelines. In: Proceedings/AMIA Annual Fall symposium. 1996: Lomas 1991 Lomas J, Enkin M, Anderson GM, Hannah WJ, Vayda E, Singer J. Opinion leaders vs audit and feedback to implement practice guidelines. Delivery after previous cesarean section. JAMA 1991;265: Mainous 2000 Mainous AG, Hueston WJ, Love MM, Evans ME, Finger R. An evaluation of statewide strategies to reduce antibiotic overuse. Family Medicine 2000;32(1):22-9. Martin 1980 Martin AR, Wolf MA, Thibodeau LA, Dzau V, Braunwald E. A trial of two strategies to modify the test-ordering behavior of medical residents. New England Journal of Medicine 1980;303: Marton 1985 Marton KI, Tul V, Sox HC Jr. Modifying test-ordering behavior in the outpatient medical clinic. A controlled trial of two educational interventions. Archives of Internal Medicine 1985;145: Mayer 1998 Mayer JA, Eckhardt L, Stepanski BM, Sallis JF, Elder JP, Slymen DJ, et al. Promoting skin cancer prevention counseling. American Journal for Public Health 1998;88(7): McAlister 1986 McAlister NH, Covvey HD, Tong C, Lee A, Wigle ED. Randomised controlled trial of computer assisted management of hypertension in primary care. BMJ 1986;293: McCartney 1997 McCartney P, Macdowall W, Thorogood M. A randomised controlled trial of feedback to general, practitioners of their prophylactic aspirin prescribing. BMJ 1997;315:35-6. McClellan 2003 McClellan WM, Millman L, Presley R, Couzins J, Flanders WD. Improved diabetes care by primary care physicians: results of a group-randomized evaluation of the Medicare Health Care Quality Improvement Program (HCQIP). Journal of clinical epidemiology 2003;56: McClellan 2004 McClellan WM, Hodgin E, Pastan S, McAdams L, Soucie M. A randomized evaluation of two health care quality improvement program (HCQIP) interventions to improve the adequacy of hemodialysis care of ESRD patients: feedback alone versus intensive intervention. Journal of the American Society of Nephrology 2004;15: McConnell 1982 McCollell TS, Cushing AH, Healy JL, McIlvenna PA, Skipper BJ. Physician behavior modification using claims data: tetracycline for upper respiratory infection. The Western Journal of Medicine 1982;137(5): Millard 2008 Millard FB, Thistlethwaite J, Spagnolo C, Kennedy RL, Baune BT. Dementia diagnosis: A pilot randomised controlled trial of education and IT audit to assess change in GP dementia documentation. Australian journal of primary health 2008;14:

203 Mitchell 2005 Mitchell E, Sullivan F, Grimshaw JM, Donnan PT, Watt G. Improving management of hypertension in general practice: a randomised controlled trial of feedback derived from electronic patient data. British journal of general practice 2005;55(511): Moher 2001 Moher M, Yudkin P, Wright L, Turner R. Cluster randomised controlled trial to compare three methods of promoting secondary prevention of coronary hearth disease in primary care. BMJ 2001;322(7298):1338. Mold 2008 Mold JW, Aspy CA, Nagykaldi Z; Oklahoma Physicians Resource/Research Network. Implementation of evidencebased preventive services delivery processes in primary care: an Oklahoma Physicians Resource/Research Network (OKPRN) study. Journal of the American board of family medicine 2008;21(4): Naughton 2007 Naughton C, Feely J, Bennett K. A clustered randomized trial of the effects of feedback using academic detailing compared to postal bulletin on prescribing of preventative cardiovascular therapy. Family practice 2007;24(5): Naughton 2009 Naughton C, Feely J, Bennett K. A RCT evaluating the effectiveness and cost-effectiveness of academic detailing versus postal prescribing feedback in changing GP antibiotic prescribing. Journal of evaluation in clinical practice 2009;15(5): Nilsson 2001 Nilsson G, Hjemdal P, Hassler A, Vitols S, Wallen NH, Krakau I. Feedback on prescribing rate combined with problem-oriented pharmacotherapy education as a model to improve prescribing among general practitioners. European Journal of Clinical Pharmacology 2001;56(11): Norton 1985 Norton PG, Dempsey LJ. Self-audit: its effect on quality of care. Journal of Family Practice 1985;21: O'Connell 1999 O Connell DL, Henry D, Tomlins R. Randomised controlled trial of effect of feedback on general practitioners prescribing in Australia. BMJ 1999;318: O'Connor 2009 O'Connor PJ, Sperl-Hillen J, Johnson PE, Rush WA, Crain AL. Customized feedback to patients and providers failed to improve safety or quality of diabetes care: a randomized trial. Diabetes care 2009;32(7): Ornstein 2004 Ornstein S, Jenkins RG, Nietert PJ, Feifer C, Roylance LF, Nemeth L, Corley S, Dickerson L, Bradford WD, Litvin C. A multimethod quality improvement intervention to improve preventive cardiovascular care: a cluster randomized trial. Annals of internal medicine 2004;141(7): Palmer 1985 Palmer RH, Louis TA, Hsu LN, Peterson HF, Rothrock JK, Strain R, et al. A randomized controlled trial of quality assurance in sixteen ambulatory care practices. Medical Care 1985;23: Phillips 2005 Phillips LS, Ziemer DC, Doyle JP, Barnes CS, Kolm P, Branch WT et al. An endocrinologist-supported intervention aimed at providers improves diabetes management in a primary care site: improving primary care of African Americans with diabetes (IPCAAD) 7. Diabetes care 2005;28: Pimlott 2003 Pimlott NJG, Hux JE, Wilson LM, Kahan M, Li C, Rosser WW. Educating physicians to reduce benzodiazepine use by elderly patients: a randomized controlled trial. CMAJ 2003;168: Quinley 2004 Quinley JC, Shih A. Improving physician coverage of pneumcoccal vaccine: a randomized trial of telephone intervention. Journal of Community Health 2004;29: Raasch 2000 Raasch BA, Hays R, Buettner PG. An educational intervention to improve diagnosis and management of suspicious skin lesions. The Journal of Continuing Education in the Health Professions 2000;20: Rantz 2001 Rantz MJ, Popejoy L, Petroski GF, Madsen RW, Mehr DR, Zwygart-Stauffacher M, et al. Randomized clinical trial of quality improvement intervention in nursing homes. The Gerontologist 2001;41(4): Rask 2001 Rask K, Kohler SA, Wells KJ, Williams JA, Diamond CC. Performance improvement interventions to improve delivery of screening services in diabetes care. Journal of Clinical Outcomes Management 2001;8:

204 Robling 2002 Robling MR, Houston HL, Kinnersley P, Hourihan MD, Cohen DR, Hale J, et al. General practitioners' use of magnetic resonance imaging: an open randomized trial comparing telephone and written requests and an open randomized controlled trial of different methods of local guideline dissemination. Clinical Radiology 2002;57(5): Ruangkanchanasetr 1993 Ruangkanchanastr S. Laboratory investigation utilization in pediatric out-patient department ramathibodi hospital. Journal of the Medical Association of Thailand 1993;76: Rubin 2001 Rubin GL, Schofield WN, Dean MG, Shakeshaft AP. Appropriateness of red blood cell transfusions in major urban hospitals and effectiveness of an intervention. The Medical journal of Australia 2001;175: Rust 1999 Rust CT, Sisk FA, Kuo AR, Smith J, Miller R, Sullivan KM. Impact of resident feedback on immunization outcomes. Archives of Pediatrics & Adolescent Medicine 1999;153: Sandbaek 1999 Sandbaek A, Kragstrup J. Randomized controlled trial of the effect of medical audit on aids prevention in general practice. Family Practice 1999;16: Sauaia 2000 Sauaia A, Ralston D, Schluter WW, Marciniak TA, Havranek EP, Dunn TR. Influencing care in acute myocardial infarction: a randomized trial comparing 2 types of intervention. American Journal of Medical Quality 2000;15: Schectman 1995 Schectman JM, Kanwal NK, Schroth WS, Elinsky EG. The effect of an education and feedback intervention on group-model and network-model health maintenance organization physician prescribing behavior. Medical Care 1995;33: Schectman 2003 Schectman JM, Schroth WS, Verme D, Voss JD. Randomized controlled trial of education and feedback for implementation of guidelines for acute low back pain.. Journal of General Internal Medicine 2003;18: Schneider 2008 Schneider A, Wensing M, Biessecker K, Quinzler R, Kaufmann-Kolle P, Szecsenyi J. Impact of quality circles for improvement of asthma care: results of a randomized controlled trial. Journal of evaluation in clinical practice 2008;14(2): Scholes 2006 Scholes D, Grothaus L, McClure J, Reid R, Fishman P, Sisk C, Lindenbaum JE, Green B, Grafton J, Thompson RS. A randomized trial of strategies to increase chlamydia screening in young women. Preventive medicine 2006;43(4): Sinclair 1982 Sinclair C, Frankel M. The effect of quality assurance activities on the quality of mental health services. QRB 1982;8(7):7-15. Siriwardena 2002 Siriwardena AN, Rashid A, Johnson MRD, Dewey ME. Cluster randomised controlled trial of an educational outreach visit to improve influenza and pneumococcal immunisation rates in primary care. The British Journal of General Practice 2002;52: Smith 1998 Smith DK, Shaw RW, Slack J, Marteau TM. Training obstetricians and midwives to present screening tests evaluation of two brief interventions. Prenatal Diagnosis 1995;15: Socolar 1998 Socolar RR, Raines B, Chen-Mok M, Runyan DK, Green C, Paterno S. Intervention to improve physician documentation and knowledge of child sexual abuse: a randomized, controlled trial. Pediatrics 1998;101(5): Solomon 2004 Solomon DH, Katz JN, La Tourette AM, Coblyn JS. Multifaceted intervention to improve rheumatologists' management of glucocorticoid-induced osteoporosis: a randomized controlled trial. Arthritis and rheumatism 2004;51(3): Sommers 1984 Sommers LS, Sholtz R, Shepherd RM, Starkweather DB. Physician involvement in quality assurance. Medical Care 1984;22: Sondergaard

205 Søndergaard J, Hansen DG, Aarslev P, Munck AP. A multifaceted intervention according to the Audit Project Odense method improved secondary prevention of ischemic heart disease: a randomised controlled trial. Family practice 2006;23(2): Soumerai 1998 Soumerai SB, McLaughlin TJ, Gurwitz JH, Guadagnoli E, Hauptman PJ, Borbas C, et al. Effect of local medical opinion leaders on quality of care for acute myocardial infarction: a randomized controlled trial. JAMA 1998;279(17): Svetkey 2009 Svetkey LP, Pollak KI, Yancy WS Jr, Dolor RJ, Batch BC, Samsa G, Matchar DB, Lin PH. Hypertension improvement project: randomized trial of quality improvement for physicians and lifestyle modification for patients. Hypertension 2009;54(6): Søndergaard 2002 Søndergaard J, Adersen M, Vach K, Kragstrup J, Maclure M, Gram LF. Detailed postal feedback about prescribing to asthma patients combined with a guideline statement showed no impact: a randomised controlled trial. European Journal of Clinical Pharmacology 2002;58: Søndergaard 2003 Søndergaard J, Andersen M, Støvring H, Kragstrup J.. Mailed prescribed feedback in addition to a clinical guideline has no impact: a randomised, controlled trial.. Scandinavian Journal of Primary Health Care 2003;21: Thomas 2006 Thomas RE, Croal BL, Ramsay C, Eccles M, Grimshaw J. Effect of enhanced feedback and brief educational reminder messages on laboratory test requesting in primary care: a cluster randomised trial. Lancet 2006;367: Thomas 2007 Thomas KG, Thomas MR, Stroebel RJ, McDonald FS, Hanson GJ, Naessens JM et al. Use of a registry-generated audit, feedback, and patient reminder intervention in an internal medicine resident clinic--a randomized trial. Journal of general internal medicine 2007;22: Tierney 1986 Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physician performance versus immediate reminders to perform preventive care. Effects on physician compliance. Medical Care 1986;24(8): Tu 2009 Tu JV, Donovan LR, Lee DS, Wang JT, Austin PC, Alter DA, Ko DT. Effectiveness of public report cards for improving the quality of cardiac care: the EFFECT study: a randomized trial. JAMA 2009;302(21): van den Hombergh 1999 Hombergh Pvd, Grol R, Hoogen HJMvd, Bosch WJHMvd. Practice visits as a tool in quality improvement: mutual visits and feedback by peers compared with visits and feedback by non-physician observers. Quality in Health Care 1999;8: van der Weijden 1999 van der Weijden T, Grol RP, Knottinerus JA. Feasibility of a national cholestrol guideline in daily practice. A randomized controlled trial in 20 general practices. International Journal for Quality in Health Care 1999;11(2): Veninga 1999 Lundborg CS, Wahlström, Oke T, Tomson G, Diwan V. Influencing prescribing for urinary tract infection and asthma in primary care in Sweden: a randomized controlled trial of an interactive educational intervention. Journal of Clinical Epidemiology 1999;52(8): Veninga CCM, Denig P, Zwaagstra R, Haaijer-Ruskamp FM. Improving drug treatment in general practice. Journal of Clinical Epidemiology 2000;53: * Veninga CCM, Lagerløv P, Wahlstöm R, Muskova M, Denig P, Berkhof J, et al. Evaluating an educational intervention to improve the treatment of asthma in four European countries. American Journal of Respiratory and Critical Care Medicine 1999;160: Veninga N. Improving prescribing in general practice. Thesis, Rijksuiversiteit Groningen Verstappen 2003 Verstappen WHJM, van der Weijden T, Sijbrandij J, Smeele J, Hermsen J, Grimshaw J, et al. Effect of a practicebased strategy on test ordering performance of primary care physicians. JAMA 2003;289: Verstappen 2004 Verstappen WHJM, van der Weijden T, Dubois WI, Smeele I, Hermsen J, Tan FES et al. Improving test ordering in primary care: the added value of a small-group quality improvement strategy compared with classic feedback only. Annals of Family Medicine 2004;2:

206 Vingerhoets 2001 Vingerhoets B, Wensing M, Grol R. Feedback of patients ' evaluations of general practice care: a randomised trial. Quality in Health Care 2001;10: Wadland 2007 Wadland WC, Holtrop JS, Weismantel D, Pathak PK, Fadel H, Powell J. Practice-based referrals to a tobacco cessation quit line: assessing the impact of comparative feedback vs general reminders. Annals of family medicine 2007;5(2): Wahlström 2003 Wahlström R, Kounnavong S, Sisounthone B, Phanyanouvong A, Southammavong T, Eriksson B, et al. Effectiveness of feedback for improving case management of malaria, diarrhoea and pneumonia - a randomized controlled trial at provincial hospitals in Lao PDR. Tropical Medicine and International Health 2003;8: Ward 1996 Ward A, Kamien M, Mansfield F, Fatovich B. Educational feedback in management of diabetes in general practice. Education for General Practice 1996;7: Weitzman 2009 Weitzman S, Greenfield S, Billimek J, Hava T, Schvartzman P, Yehiel E et al. Improving combined diabetes outcomes by adding a simple patient intervention to physician feedback: a cluster randomized trial. Israeli medical association journal 2009;11: Winickoff 1984 Winickoff RN, Coltin KL, Morgan MM, Buxbaum RC, Barnett GO. Improving physician performance through peer comparison feedback. Medical Care 1984;22: Winkens 1995 Winkens RA, Pop P, Bugter-Maessen AM, Grol RP, Kester AD, Beusmands GH, et al. Randomised controlled trial of routine individual feedback to improve rationality and reduce numbers of test requests. Lancet 1995;345: Wones 1987 Wones RG. Failure of low-cost audits with feedback to reduce laboratory test utilization. Medical Care 1987;25: Ziemer 2006 Ziemer DC, Doyle JP, Barnes CS, Branch WTJ, Cook CB, El Kebbi IM et al. An intervention to overcome clinical inertia and improve diabetes mellitus control in a primary care setting: Improving Primary Care of African Americans with Diabetes (IPCAAD) 8. Archives of internal medicine2006(166): Zwar 1999 Zwar N, Wolk J, Gordon J, Fisher RS, Kehoe L. Influencing antibiotic prescribing in general practice: a trial of prescriber feedback and management guidelines. Family Practice 1999;16(5): List of excluded studies Aittasalo 2006 Aittasalo M, Miilunpalo S, Kukkonen-Harjula K, Pasanen M. A randomized intervention of physical activity promotion and patient self-monitoring in primary health care. Prev.Med 2006;42:40-6. Allard 2001 Allard J, Hébert R, Rioux M, Asselin J, Voyer L. Efficacy of a clinical medication review on the number of potentially inappropriate prescriptions prescribed for community-dwelling elderly people. CMAJ 2001;164: Allison 2005 Allison JJ, Kiefe CI, Wall T, Casebeer L, Ray MN, Spettell CM et al. Multicomponent Internet continuing medical education to promote chlamydia screening.. Am.J.Prev.Med 2005;28: Althabe 2008 Althabe F, Buekens P, Bergel E, Belizan JM, Campbell MK, Moss N, HartwellT, Wright LL. A behavioral intervention to improve obstetrical care. New England Journal of Medicine 2008;358(18): Anderson 1996 Anderson JF, McEwan KL, Hrudey WP. Effectiveness of notification and group education in modifying prescribing of regulated analgesics. CMAJ 1996;154:31-9. Anonymous I

207 Anonymous. North of England study of standards and performence in general practice. University of Newcastle, report nr 40 and report nr 50. Aspy 2008 Aspy CB, Enright M, Halstead L, Mold JW, Oklahoma Physicians Resource/Research Network. Improving mammography screening using best practices and practice enhancement assistants: an Oklahoma Physicians Resource/Research Network (OKPRN) study. Journal of the American Board of Family Medicine: JABFM 2008;21(4): Ballard 2002 Ballard DJ, Nicewander D, Skinner C. Health care provider quality improvement organization medicare data sharing: A diabetes quality improvement initiative. In: Proceedings of AMIA Symposium. 2002:22-5. Belcher 1990 Belcher DV. Implementing preventive services success and failure in an outpatient trial. Archives of Internal Medicine 1990;150: Bertoni 2009 Bertoni AG, Bonds DE, Chen H, Hogan P, Crago L, Rosenberger E, Barham AH, Clinch CR, Goff DCJ. Impact of a multifaceted intervention on cholesterol management in primary care practices: guideline adherence for heart health randomized trial. Archives of Internal Medicine 2009;169(7): Berwick 1986 Berwick DM, Coltin KL. Feedback reduces test use in a health maintenance organization. JAMA 1986;255(11): [MEDLINE: ] Billi 1987 Billi JE, Hejna GF, Wolf FM, Shapiri LR, Stross JK. The effects of a cost-education program on hospital charges. Journal of General Internal Medicine 1987;2: Bindels 2003 Bindels R, Hasman A, Kester AD, Talmon JL, De Clercq PA, Winkens RA. The efficacy of an automated feedback system for general practitioners. Inform Prim Care 2003;11: Bischoff 2000 Bischoff WE, Reynolds TM, Sessler CN, Edmond MB, Wenzel RP. Handwashing compliance by health care workers: The impact of introducing an accessible, alcohol-based hand antiseptic. Arch.Intern.Med 2000;160: Bonds 2009 Bonds DE, Hogan PE, Bertoni AG, Chen H, Clinch CR, Hiott AE, Rosenberger EL, Goff DC. A multifaceted intervention to improve blood pressure control: The Guideline Adherence for Heart Health (GLAD) study. American Heart Journal 2009;157(2): Bonetti 2005 Bonetti D, Eccles M, Johnston M, Steen N, Grimshaw J, Baker R et al. Guiding the design and selection of interventions to influence the implementation of evidence-based practice: an experimental simulation of a complex intervention trial. Soc.Sci.Med 2005;60: Brand 2005 Brand S, rter M, Sitta P, van Calker D, Menke R, Heindl A et al. Data supporting quality circle management of inpatient depression treatment. Nervenarzt 2005;76: Britton 1991 Britton ML,.Lurvey PL. Impact of medication profile review on prescribing in a general medicine clinic. Am.J.Hosp.Pharm 1991;48: Brown 1988 Brown RL. Evaluation of a continuing medical education program for primary care physcicians on the management of alcoholism. Journal of Medical Education 1988;63: Buekens 1993 Buekens P, Boutsen M, Kittel F, Vandenbussche P, Dramaix M. Does awareness of rates of obstetric interventions change practice? BMJ 1993;306:623. Bunting 2004 Bunting PS,.Van Walraven C. Effect of a controlled feedback intervention on laboratory test ordering by community physicians. Clin.Chem 2004;50: Campbell 2006 Campbell E, Walsh RA, Sanson-Fisher R, Burrows S, Stojanovski E. A group randomised trial of two methods for disseminating a smoking cessation programme to public antenatal clinics: effects on patient outcomes. Tobacco Control 2006;15(2):

208 Carney 1992 Carney PA, Dietrich AJ, Keller A, Landgraf J, O'Connor GT. Tools, teamwork and tenacity: An office system for cancer prevention. The Journal of Family Practice 1992;35: Chin 2007 Chin MH, Drum ML, Guillen M, Rimington A, Levie JR, Kirchhoff AC et al. Improving and sustaining diabetes care in community health centers with the health disparities collaboratives. Med.Care 2007;45: Chowdhury 2007 Chowdhury AKA, Khan OF, Matin MA, Begum K, Galib MA. Effect of standard treatment guidelines with or without prescription audit on prescribing for acute respiratory tract infection (ARI) and diarrhoea in some thana health complexes (THCs) of Bangladesh. Bangladesh Med.Res.Counc.Bull 2007;33: Cleveringa 2008 Cleveringa FGW, Gorter KJ, Van Donk MD, Rutten GEHM. Combined task delegation, computerized decision support, and feedback improve cardiovascular risk for type 2 diabetic patients. Diabetes Care 2008;31(12): Cohen 1996 Cohen MM, Rose DK, Yee DA. Changing anesthesiologists' practice patterns. Can it be done? Anesthesiology 1996;85: Colón-Emeric 2007 Colón-Emeric CS, Lyles KW, House P, Levine DA, Schenck AP, Allison J, Gorospe J, Fermazin M, Oliver K, Curtis JR, Weissman N, Xie A, Saag KG. Randomized trial to improve fracture prevention in nursing home residents. The American journal of medicine 2007;120(10): Cope 1986 Cope DW, Linn LS, Leake BD, Barrett PA. Modification of residents' behavior by preceptor feedback of patient satisfaction. J.Gen.Intern.Med 1986;1: Cranney 1999 Cranney M, Barton S, Walley T. Addressing barriers to change: an RCT of practice-based education to improve the management of hypertension in the elderly. The British journal of general practice 1999;49: Crits-Christoph 2010 Crits-Christoph P, Ring-Kurtz S, McClure B, Temes C, Kulaga A, Gallop R et al. A randomized controlled study of a web-based performance improvement system for substance abuse treatment providers. J.Subst.Abuse Treat 2010;38: Crotty 2004 Crotty M, Whitehead C, Rowett D, Halbert J, Weller D, Finucane P et al. An outreach intervention to implement evidence based practice in residential care: a randomized controlled trial. BMC Health Serv Res 2004;4:6. Curtis 2009 Curtis JR, Nielsen E, Treece P, Downey L, Dotolo D, Shannon S et al. In: Crit.Care Med. 2009; Conference:A388. Del Mar 1998 Del Mar CB, Lowe JB, Adkins P, Arnold E, Baade P. Improving general practitioner clinical records with a quality assurance minimal intervention. British Journal of General Practice 1998;48: Denton 2001 Denton GD, Smith J, Faust J, Holmboe E. Comparing the efficacy of staff versus housestaff instruction in an intervention to improve hypertension management. Academic Medicine 2001;76: De Silva 1994 De Silva M, Abrahanson G. Does medical audit change practice? Transfucion Science 1994;15:277. Dickinson 1981 Dickinson JC, Warshaw GA, Gehlbach SH, Bobula JA, Muhlbaier LH, Parkerson GR Jr. Improving hypertension control: impact of computer feedback and physician education. Medical Care 1981;19: Doherty 2006 Doherty SR, Jones PD. Use of an 'evidence-based implementation' strategy to implement evidence-based care of asthma into rural district hospital emergency departments. Rural and Remote Health 2006;6:529. Doherty 2007 Doherty SR, Jones PD, Davis L, Ryan NJV, Treeve V. Evidence-based implementation of adult asthma guidelines in the emergency department: a controlled trial. Emergency Medicine Australasia 2007;19(1):31-8. Dranitsaris 1995 Dranitsaris G, Warr D, Puodziunas A. A randomized trial of the effects of pharmacist intervention on the cost of antiemetic therapy with ondansetron. Supportive Care in Cancer 1995;3: Dulko

209 Dulko D, Hertz E, Julien J, Beck S, Mooney K. Implementation of cancer pain guidelines by acute care nurse practitioners using an audit and feedback strategy. J.Am.Acad.Nurse Pract 2010;22: Echouffo-Tcheugui 2009 Echouffo-Tcheugui JB, Simmons RK, Williams KM, Barling RS, Prevost AT, Kinmonth AL et al. The ADDITION- Cambridge trial protocol: a cluster- randomised controlled trial of screening for type 2 diabetes and intensive treatment for screen-detected patients. BMC Public Health 2009;9:136. Elnicki 1998 Elnicki DM, Layne RD, Ogden PE, Morris DK. Oral versus written feedback in medical clinic. J.Gen.Intern.Med 1998;13: Everett Everett GD, deblois CS, Chang PF, Holets T. Effect of cost education, cost audits, and faculty chart review on the use of laboratory services. Archives of Internal Medicine 1983;143: Fallowfield 2002 Fallowfield L, Jenkins V, Farewell V, Saul J, Duffy A, Eves R. Efficacy of a Cancer Research UK communication skills training model for oncologists: a randomised controlled trial. Lancet 2002;359(9307): Feder 1995 Feder G, Griffiths C, Highton C, Eldridge S, Spence M, Southgate L. Do clinical guidelines intorduced with practice based education improve care of asthmatic and dibetic patients? A randomised controlled trial in general practices in east London. BMJ 1995;311: Ferreira 2005 Ferreira MR, Dolan NC, Fitzgibbon ML, Davis TC, Gorby N, Ladewski L, Liu D, Rademaker AW, Medio F, Schmitt BP, Bennett CL. Health care provider-directed intervention to increase colorectal cancer screening among veterans: Results of a randomized controlled trial. Journal of Clinical Oncology 2005;23(7): Fick 2004 Fick DM, Maclean JR, Rodriguez NA, Short L, Heuvel RV, Waller JL et al. A randomized study to decrease the use of potentially inappropriate medications among community-dwelling older adults in a southeastern managed care organization. Am.J.Manag.Care 2004;10: Fihn 2004 Fihn DS, McDonell MB, Diehr P, Anderson SM, Bradley KA, Au DH, et al. Effects of sustained audit/feedback on self-reported health status of primary care patients. The American Journal of Medicine 2004;116: Finkelstein 2001 Finkelstein JA, Davis RL, Dowell SF, Metlay JP, Soumerai SB, Rifas-Shiman SL, et al. Reducing antibiotic use in children: a randomized trial in 12 practices. Pediatrics 2001;108(1):1-7. Finkelstein 2008 Finkelstein JA, Huang SS, Kleinman K, Rifas-Shiman SL, Stille CJ, Daniel J, Schiff N, Steingard R, Soumerai SB, Ross-Degnan D, Goldmann D, Platt R. Impact of a 16-community trial to promote judicious antibiotic use in Massachusetts. Pediatrics 2008;121(1):e Frame 1994 Frame PS, Zimmer JG, Werth PL, Hall WJ, Eberly SW. Computer-based vs manual health maintenance tracking. A controlled trial. Arch.Fam.Med 1994;3: Freeborn 1997 Freeborn DK, Shye D, Mullooly JP, Eraker S, Romeo J. Primary care physicians' use of lumbar spine imaging tests: effects of guidelines and practice pattern feedback. J.Gen.Intern.Med 1997;12: Fretheim 2006 Fretheim A, Oxman AD, Håvelsrud K, Treweek S, Kristoffersen DT, Bjørndal A. Rational prescribing in primary care (RaPP): a cluster randomized trial of a tailored intervention. PLoS medicine 2006;3(6):e134. Furniss 2000 Furniss L, burns A, Craig SKL, Scobie S, Cooke J, Farafher B. Effects of a pharmacists medication review in nursing homes. British Journal of Psychiatry 2000;176: Ganz 2005 Ganz PA, Farmer MM, Belman MJ, Garcia CA, Streja L, Dietrich AJ et al. Results of a randomized controlled trial to increase colorectal cancer screening in a managed care health plan. Cancer 2005;104: Garrouste-Org 2010 Garrouste-Orgeas, Tabah A, Schwebel C, Vesin A, Soufir L, Adrie C. A Multifaceted Program for Improving Quality of Care in Critically Ill Patients: The IATROREF Study (Part 3). In: Intensive Care Med. 2010:S89. Gask

210 Gask L, Goldberg D, Boardman J, Craig T, Goddard C, Jones O, et al. Training general practitioners to teach psychiatric interviewing skills: an evaluation of group training. Medical Education 1991;25(5): [MEDLINE: ] Gerbert 1988 Gerbert B, Maguire B, Badner V, Greenspan D, Greenspan J, Barnes D, et al. Changing dentists' knowledge, attitudes and behaviours relating to AIDS: a controlled educational intervention. JADA 1988;116: Goderis 2010 Goderis G, Borgermans L, Grol R, Van Den Broeke C, Boland B, Verbeke G et al. Start improving the quality of care for people with type 2 diabetes through a general practice support program: a cluster randomized trial. Diabetes Res.Clin.Pract 2010;88: Goldberg 1980 Goldberg DP, Steele JJ, Smith C, Spivey L. Training family doctors to recognise psychiatric illness with increased accuracy. Lancet 1980;6: Goldberg 1998 Goldberg HI, Wagner EH, Fihn SD, Martin DP, Horowitz CR, Christensen DB, et al. A randomized controlled trial of CQI teams and academic detailing: can they alter compliance with guidelines? Journal on Quality Improvement 1998;24(3): Grimshaw 1998 * Grimshaw J. Evaluation of four quality assurance initiatives to improve out-patient referrals from general practice to hospital. Thesis, University of Aberdeen Gunn 2003 Gunn J, Southern D, Chondros P, Thomson P, Robertson K. Guidelines for assessing postnatal problems: introducing evidence-based guidelines in Australian general practice. Family Practice 2003;20: Hall 2001 Hall L, Eccles M, Barton R, Steen N, Campbell M. Is untargeted outreach visiting in primary care effective? A pragmatic randomized controlled trial. Journal of Public Health Medicine 2001;23: Hampshire 1999 Hampshire A, Blair M, Crown N, Avery A, Williams I. Action research: a useful method of promoting change in primary care? Family Practice 1999;16(3): Hanlon 1996 Hanlon JT, Weinberger M, Samsa GP, Kenneth E, Uttech KM, Lewis IK, et al. A randomized controlled trial of a clinical pharmacists intervention to improve inappropriate prescribing in elderly outpatients with polypharmacy. The American Journal of Medicine 1996;100(4): Harbarth 2002 Harbarth S, Pittet D, Grady L, Zawacki A, Potter-Bynoe G, Samore MH et al. Interventional study to evaluate the impact of an alcohol-based hand gel in improving hand hygiene compliance. The pediatric infectious disease journal 2002;21: Harewood 2008 Harewood GC, Murray F, Winder S, Patchett S. Evaluation of formal feedback on endoscopic competence among trainees: the EFFECT trial. Irish Journal of Medical Science 2008;177(3): Hargraves 1996 Hargraves JL, Palmer RH, Orav EJ, Wright EA. Practice characteristics and performance of primary care practitioners. Medical Care 1996;34(9): Harris 2005 Harris SB, Leiter LA, Webster-Bogaert S, Van DM, O'Neill C. Teleconferenced educational detailing: diabetes education for primary care physicians. J.Contin.Educ.Health Prof 2005;25(87-97). Hartlaub 1993 Hartlaub PP, Barrett PH, Marine WM, Murphy JR. Evaluation of an intervention to change benzodiazepineprescribing behavior in a prepaid group practice setting. Am.J.Prev.Med 1993;9: Henderson 1979 Henderson D, D Alessandri R, Westfall B, Moore R, Smith R, Scobbo, et al. Hospital cost containment: a little knowledge helps. Clinical Research 1979;27:297A. Hetlevik 1998 Hetlevik I, Holmen J, Krüger Ø, Kristensen P, Iversen H. Implementing clinical guidelines in the treatment of hypertension in general practice. Blood Pressure 1998;7: * Hetlevik I, Holmen J, Krüger Ø. Implementing clinical guidelines in the treatment of hypertension in general practice. Scandinavian Journal of Primary Health Care 1999;17:

211 Hinchey 2010 Hinchey JA, Shephard T, Tonn ST, Ruthazer R, Hermann RC, Selker HP et al. The Stroke Practice Improvement Network: a quasiexperimental trial of a multifaceted intervention to improve quality. J STROKE CEREBROVASC.DIS 2010;19: Hirsch 2002 Hirsch IB, Goldberg HI, Ellsworth A, Evans TC, Herter CD, Ramsey SD et al. A multifaceted intervention in support of diabetes treatment guidelines: a cont trial. Diabetes Res.Clin.Pract 2002;58: Hogg 2008 Hogg W, Lemelin J, Graham ID, Grimshaw J, Martin C, Moore L et al. Improving prevention in primary care: evaluating the effectiveness of outreach facilitation. Fam. Pract. 2008;25:40-8. Holleman 1996 Holleman DR,.Simel DL. Effectiveness of automatic diagnostic test result feedback on outpatient laboratory and radiology testing in veterans. A controlled trial. Med.Care 1996;34: Horbar 2004 Horbar JD, Carpenter JH, Buzas J, Soll RF, Suresh G, Bracken MB et al. Collaborative quality improvement to promote evidence based surfactant for preterm infants: a cluster randomised trial. BMJ 2004;329:1004. Horowitz 1996 Horowitz CR, Goldberg HI, Martin DP, Wagner EH, Fihn SD, Christensen DB, et al. Conducting a randomized controlled trial of CQI and academic detailing to implement clinical guidelines. Journal on Quality Improvement 1996;22(11): Howe 1996 Howe A. Detecting psychological distress: can general practitioners improve their performance? British Journal of General Practice 1996;46: Hulscher 1997 Hulscher ME, van Drenth BB, van der Wouden JC, Mokkink HG, Van Weel C, Grol RP. Changing preventive practice: a controlled trial on the effects of outreach visits to organise prevention of cardiovascular disease. Quality in health care 1997;6: Ilag 2003 Ilag LL, Martin CL, Tabaei BP, Isaman DJ, Burke R, Greene DA et al. Improving diabetes processes of care in managed care. Diabetes Care 2003;26: Jaen 2010 Jaen CR, Crabtree BF, Palmer RF, Ferrer RL, Nutting PA, Miller WL et al. Methods for evaluating practice change toward a patient-centered medical home. Annals of Family Medicine 2010;8(Suppl 1):S9-20. Jans 2001 Jans MP, Schellevis FG, Le Coq EM, Bezemer PD, van Eijk JT. Health outcomes of asthma and COPD patients: the evaluation of a project to implement guidelines in general practice. Int.J.Qual.Health Care 2001;13: Johansen 1997 Johansen A. Using audit to improve senior officer training. Postgraduate Medical Journal 1997;73: Johnson 1976 Johnson RE, Campbell WH, Azevedo DJ, Christensen DB. Studying the impact of patient drug profile in an HMO. Medical Care 1976;15: Jones 1996 Jones HE, Cleave B, Zinman B, Szalai JP, Nichol HL, Hoffman BR.. Efficacy of feedback from quarterly laboratory comparison in maintaining quality of a hospital capillary blood glucose monitoring program. Diabetes Care 1996;19(2): Kafuko 1999 Kafuko JM, Zirabamuzaale, Bagena D. Rational drug use in rural health units of Uganda:effect of national standard treatment guidelines on rational drug use. In: 1st International Conference on Improving Use of Medications Katz 2004 Katz DA, Muehlenbruch DR, Brown RL, Fiore MC, Baker TB.. Effectiveness of implementing the agency for healthcare research and quality smoking cessation clinical practice guideline: A randomized, controlled trial.. Journal of the national cancer institute 2004;96: Kerse 1999 Kerse NM, Flicker L, Jolley D, Arroll B, Young D. Improving the health behaviours of elderly people: randomised controlled trial of a general practice education programme. BMJ 1999;319: Kinney

212 Kinney ED, Kennedy J, Cook CA, Freedman JA, Lane KA, Hui SL. A randomized trial of two quality improvement strategies implemented in a statewide public community-based, long-term care program. Med.Care 2003;41: Kirwin 2010 Kirwin JL, Cunningham RJ, Sequist TD. Pharmacist recommendations to improve the quality of diabetes care: a randomized controlled trial. J Manage Care Pharm 2010;16: Kroenke 1990 Kroenke K, Pinholt EM. Reducing polypharmacy in the elderly. Journal of American Geriatrics Society 1990;38:31-6. Kuilboer 2006 Kuilboer MM, van Wijk MAM, Mosseveld M, van der Does E, de Jongste JC, Overbeek SE et al. Computed critiquing integrated into daily clinical practice affects physicians' behavior--a randomized clinical trial with AsthmaCritic. Methods Inf.Med 2006;45: Lafata 2007 Lafata JE, Gunter MJ, Hsu J, Kaatz S, Krajenta R, Platt R, Schultz L, Selby JV, Simon SR, Simpkins J, Soumerai SB, Uratsu C. Academic detailing to improve laboratory testing among outpatient medication users. Medical Care 2007;45(10): Lassen 1992 Lassen LC,.Kristensen FB. Peer comparison feedback to achieve rational and economical drug therapy in general practice: a controlled intervention study. Scand.J.Prim.Health Care 1992;10: Lemelin 2001 Lemelin J, Hogg W, Baskerville N. Evidence to action: a tailored multifaceted approach to changing family physician practice patterns and improving preventive care. CMAJ 2001;164: Lenderink 2010 Lenderink AF, Spreeuwers D, van der Klink JJL, van Dijk FJH. Information and feedback to improve occupational physicians' reporting of occupational diseases: a randomised controlled trial. Int.Arch.Occup.Environ.Health 2010;83: Leviton 1999 Leviton LC, Goldenberg RL, Baker CS, Schwartz RM, Freda MC, Fish LJ, et al. Methods to encourage the use of antenatal corticosteroid therapy for fetal maturation. JAMA 1999;281(1): Linn 1980 Linn LS, Yager J. The effect of screening, sensitization, and feedback on notation of depression. Journal of Medical Education 1980;55: Luders 2010 Luders S, Schrader J, Schmieder RE, Smolka W, Wegscheider K, Bestehorn K. Improvement of hypertension management by structured physician education and feedback system: cluster randomized trial. Eur J Cardiovasc Prev Rehabil 2010;17: Lundborg 1999 Lundborg CS, Wahlström R, Oke T, Tomson G, Diwan VK. Influencing prescribing for urinary tract infection and asthma in primary care in Sweden: a randomized controlled trial of an interactive educational intervention. J.Clin.Epidemiol 1999;52: MacCosbe 1985 MacCosbe PE, Gartenberg G. Modifying empiric antibiotic prescribing: Experience with one strategy in a medical residency program. Hospital Formulary 1985;20: MacGowan 1996 MacGowan AP, Feeney R, Brown I, Mcculloch S, Reeves D, Lovering A. Routine feedback to GPs who request microbiological tests is effective. BMJ 1996;312:1481. Madridejos-Mora 2004 Madridejos-Mora R, Amado-Guirado E, Pérez-Rodríguez MT. Effectiveness of the combination of feedback and educational recommendations for improving drug prescription in general practice. Med.Care 2004;42: Mandel 1985 Mandel I, Franks P, Dickinson J. Improving physician compliance with preventive medicine guidelines. The Journal of Family Practice 1985;21(3): [MEDLINE: ] Manfredi 1998 Manfredi C, Czaja R, Freels S, Trubitt M, Warnecke R, Lacey L. Improving cancer screening in physcians practices serving low-income and minority populations. Archives of Family Medicine 1998;7: Manheim

213 Manheim LM, Feinglass J, Hughes R, Martin GJ, Conrad K, Hughes EF. Training house officers to be cost conscious. Effects of an educational intervention on charges and length of stay. Medical Care 1990;28: Manning 1986 Manning PR, Lee PV, Clintworth WA, Denson TA, Oppenheimer PR, Gilman NJ. Changing prescribing practices through individual continuing education. JAMA 1986;256: Martin 2007 Martin WE, Miller SC, Welch LC, Burrill J. Improving access to hospice: the Physician Feedback and Reminders to Improve Access to Hospice (PFRIAH) study. Medicine and Health, Rhode Island 2007;90(12): Mayefsky 1993 Mayefsky JH, Foye HR. Use of a chart audit: teaching well child care to paediatric house officers. Medical Education 1993;27: Mazzuca 1988 Mazzuca SA, Vinicor F, Cohen SJ, Norton JA, Fineberg NS, Fineberg SE. The Diabetes Education study: a controlled trial of the effects of intensive instructions of internal medicine residents on the management of diabetes mellitus. Journal of General Internal Medicine 1988;3:1-8. McDermott 2003 McDermott R, Tulip F, Schmidt B, Sinha A.. Sustaining better diabetes care in remote indigenous Australian communities.. BMJ 2003;327: McDonel 1997 McDonel EC, Bond GR, Salyers M, Fekete D, Chen A, McGrew JH, et al. Implementing assertive community treatment programs in rural settings. Administration and Policy in Mental Health 1997;25(2): McPhee 1989 * Fordham D, McPhee SJ, Bird JA, Rodnick JE, Detmer WM. The cancer prevention reminder system. Clinical Computing 1990;7(5): McPhee SJ, Bird JA, Jenkins CNH, Fordham D. Promoting cancer screening. A randomized controlled trial of three interventions. Archives of Internal Medicine 1989;149: Meehan 2001 Meehan TP, Weingarten SR, Holmboe ES, Mathur D, Wang Y, Petrillo MK et al. A statewide initiative to improve the care of hospitalized pneumonia patients: The Connecticut Pneumonia Pathway Project. Am.J.Med 2001;111: Mertz 2010 Mertz D, Dafoe N, Walter SD, Brazil K, Loeb M. Effect of a multifaceted intervention on adherence to hand hygiene among healthcare workers: A cluster-randomized trial. Infect.Control Hosp.Epidemiol 2010;31: Metlay 2007 Metlay JP, Camargo CAJ, MacKenzie T, McCulloch C, Maselli J, Levin SK, Kersey A, Gonzales R, IMPAACT Investigators. Cluster-randomized trial to improve antibiotic use for adults with acute respiratory infections treated in emergency departments. Annals of Emergency Medicine 2007;50(3): Meyer 1991 Meyer TJ, Van Kooten D, Marsh S, Prochazka AV. Reduction of polypharmacy by feedback to clinicians. Journal of General Internal Medicine 1991;6: Moongtui 2000 Moongtui W, Gauthier DK, Turner JG. Using peer feedback to improve handwashing and glove usage among Thai health care workers. American Journal of Infection Control 2000;28: Mourad 2010 Mourad S, Hermens RPMG, Nelen WLDM, Grol RPTM, Kremer JAM. In: Hum.Reprod Conference:i258. Munroe 1997 Munroe WP, Kunz K, Dalmandy-Israel C, Potter L, Schonfeld WH. Economic evaluation of pharmacist involving in disease management in a community pharmacy setting. Clinical Therapeutics 1997;19(1): Myers 2004 Myers RE, Turner B, Weinberg D, Hyslop T, Hauck WW, Brigham T et al. Impact of a physician-oriented intervention on follow-up in colorectal cancer screening. Prev.Med 2004;38: Nattinger 1989 Nattinger AB, Panzer RJ, Janus J. Improving the utilization of screening mammography in primary care practices. Archives of Internal Medicine 1989;149(9): [MEDLINE: ] Nicolas 1996 Nicolas F, Raimondeau J, Blanloeil Y, Le Conte P, Villers D, Touze MD. Cost analysis, Consensus Conference, medical audit. Journal D'Economie Medicale 1996;14:

214 North of England1992 North of England Study of Standards and Performance in General Practice. Medical audit in general practice. I: Effects on doctors' clinical behaviour for common childhood conditions. BMJ 1992;304(6840): [MEDLINE: ] North of England Study of Standards and Performance in General Practice. Medical audit in general practice. II: Effects on health of patients with common childhood conditions. BMJ 1992;304(6840): [MEDLINE: ] Nyman 1995 Nyman JA, Akhtar MR, Feldman R. Does publishing the parameters that trigger review of Medicare claims change provider behavior? Results of the parameter release study. Med.Care 1995;33: O'Connor 1996 O'Connor GT, Plume SK, Olmstead EM, Morton JR, Maloney CT, Nugent WC et al. A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. The Northern New England Cardiovascular Disease Study Group. JAMA 1996;275: O'Connor 2005 O'Connor PJ, Desai J, Solberg LI, Reger LA, Crain AL, Asche SE et al. Randomized trial of quality improvement intervention to improve diabetes care in primary care settings. Diabetes Care 2005;28: Ogwal-Okeng 2001 Ogwal-Okeng JW, Anokbonggo WW, Birungi H. Prescribing audit with feedback intervention in six regional hospitals and Mulagi referral/teaching hospitals. In: International conference on Improving use of Medicines Ornstein 2010 Ornstein S, Nemeth LS, Jenkins RG, Nietert PJ. Colorectal cancer screening in primary care: translating research into practice. Med. Care 2010;48: Ottolini 1998 Ottolini MC, Greenberg L. Development and evaluation of a CD-ROM computer program to teach residents telephone management. Pediatrics 1998;101(3):E2. Overbeek 2010 Overbeek LI, Hermens RP, Van Krieken JH, Adang EM, Casparie M, Nagengast FM et al. Electronic reminders for pathologists promote recognition of patients at risk for Lynch syndrome: Cluster-randomised controlled trial. Virchows Arch 2010;456: Papa 1999 Papa FJ, Aldrich D, Schumacker RE. The effects of immediate online feedback upon diagnostic performance. Academic medicine 1999;74:S16-S18. Patel 2010 Patel L, Glasscoe C, Dixon C, Dyer K, Southern KW. In: Pediatr.Pulmonol Conference:441. Payne 1978 Payne BC, Lyons TF, Neuhaus E, Kolton M, Dwarshius L. Method of Evaluating and Improving Ambulatory Medical Care. Final Report and Executive Summary. Final report 30. NTIS 1978;Order Number: PB Pearson 2001 Pearson SD, Kleefield SF, Soukop JR, Cook EF, Lee TH. Critical pathways intervention to reduce length of hospital stay. The American Journal of Medicine 2001;110: Performance 2006 Using performance feedback to change provider behaviour. Performance improvement advisor 2006;10(5):58-9. Peters-Klimm 2008 Peters-Klimm F, Muller-Tasch T, Remppis A, Szecsenyi J, Schellberg D. Improved guideline adherence to pharmacotherapy of chronic systolic heart failure in general practice--results from a cluster-randomized controlled trial of implementation of a clinical practice guideline. Journal of Evaluation in Clinical Practice 2008;14(5): Peters-Klimm 2009 Peters-Klimm F, Campbell S, Muller-Tasch T, Schellberg D, Gelbrich G, Herzog W, Szecsenyi J. Primary carebased multifaceted, interdisciplinary medical educational intervention for patients with systolic heart failure: lessons learned from a cluster randomised controlled trial. Trials [Electronic Resource] 2009;10:68. Pfirter 2010 Pfirter G, Lapertosa S, Villagra M, Caporale JE, Gonzalez C, Clark C et al. In: Diabetologia Conference:S109. Pit

215 Pit SW, Byles JE, Henry DA, Holt L, Hansen V, Bowman DA. A Quality Use of Medicines program for general practitioners and older people: a cluster randomised controlled trial. Medical Journal of Australia 2007;187(1): Pugh 1989 Pugh JA, Frazier LM, DeLong E, Wallace AG, Ellenbogen P, Linfors E. Effect of daily charge feedback on inpatient charges and physician knowledge and behavior. Arch.Intern.Med 1989;149: Putnam 1985 Putnam RW, Curry L. Impact of patient care appraisal on physicians behaviour in the office setting. Canadian Medical Association Journal 1985;132: Putnam W, Curry L. Primary care appraisal in the ambulatory setting: effectiveness of a continuing medical education tool. 419 RCT-C in register. Quilitch 1975 Quilitch HR. A comparison of three staff-management procedures. J.Appl.Behav.Anal 1975;8: Raisch 1999 Raisch DW,.Sleath BL. Using feedback letters to influence the use of antiulcer agents in a Medicaid program. J.Gen.Intern.Med 1999;14: Rascati 1996 Rascati KL, Okano GJ, Burch C. Evaluation of physician intervention letters. Med.Care 1996;34: Reid 1977 Reid RA, Lantz KH.. Physician profiles in training the graduate internist. Journal of Medical Education 1977;52: Restuccia 1982 Restuccia JD. The effect of concurrent feedback in reducing inappropriate hospital utilization. Medical Care 1982;20(1): [MEDLINE: ] Reuther 2010 Reuther L, Schultz-Larsen P, Damsgaard J, Gilsa HD, Munck A, Sondergaard J et al. In: Basic Clin Pharmacol Toxicol Conference. 2010:542. Rhew 1999 Rhew DC, Glassman PA, Goetz MB. Improving pneumococcal vaccine rates. Nurse protocols versus clinical reminders. J.Gen.Intern.Med 1999;14: Rollman 2002 Rollman BL, Hanusa BH, Lowe HJ, Gilbert T, Kapoor WN, Schulberg HC. A randomized trial using computerized decision support to improve treatment of major depression in primary care. Journal of General internal Medicine 2002;17: Roski 1998 Roski J. Changing practice patterns as a result of implementing the Agency for Health Care Policy and Research guidelines in 20 primary care clinics. Tobacco Control 1998;7 Suppl:S19-20; discussion S24-5. Rubenstein 1989 Rubenstein LV, Calkin DR, Young RT, Cleary PD, Fink A, Kosecoff J, et al. Improving patient function: a randomized trial of functional disability screening. Annals of Internal Medicine 1989;111(10): Rubenstein 1999 Rubenstein LV, Jackson-Tricke M, Unützer J, Miranda J, Minnium K, Pearson ML, et al. Evidence-based care for depression in managed primary care practices. Health Affairs 1999;18(5): Sanazaro 1978 Sanazaro PJ, Worth RM. Concurrent quality assurance in hospital care. Report of a study by Private Initiative in PSRO. The New England Journal of Medicine 1978;298: Seers 2004 Seers K, Crichton N, Carroll D, Richards S, Saunders T. Evidence-based postoperative pain management in nursing: is a randomized-controlled trial the most appropriate design? J.Nurs.Manag. 2004;12: Shaughnessy 1991 Shaughnessy AF, DAmico F, Nickel RO. Improving prescription-writing skills in a family practice residency. DICP, The Annals of Pharmacotherapy 1991;25: Simon 2000 Simon GE, VonKorff M, Rutter C, Wagner E. Randomised trial in monitoring, feedback, and management of care by telephone to improve treatment of depression in primary care. BMJ 2000;320: Simunovic

216 Simunovic M, Coates A, Goldsmith CH, Thabane L, Reeson D, Smith A et al. The cluster-randomized Quality Initiative in Rectal Cancer trial: evaluating a quality-improvement strategy in surgery. CMAJ 2010;182: Smeele 1998 Smeele IJM, Grol RPTM, van Schayck CP. Implementation of CARA standards in general practice: a randomised camparative study into the effectivity of a monitoring-feedback system and advancement of expertise on general practitioners. Ned.Tijdschr.Geneeskd 1998;142:1422. [Other: ] Smith 1995 Smith D, Christensen DB, Stergachis A, Holmes G. A randomized controlled trial of a drug use review intervention for sedative hypnotic medications. Prenatal Diagnosis 1998;15: Spector 1989 Spector WD, Drugovich ML. Reforming nursing home quality regulation. Impact on cited deficiencies and nursing home outcomes. Medical Care 1989;27(8): [MEDLINE: ] Steele 1989 Steele MA, Bess DT, Franse VL, Graber SE. Cost effectiveness of two interventions for reducing outpatient prescribing costs. DICP: The Annals of Pharmacotherapy 1989;23(6): Stewart 2005 Stewart M, Marshall JN, stbye T, Feightner JW, Brown JB, Harris S et al. Effectiveness of case-based on-line learning of evidence-based practice guidelines. Fam.Med 2005;37: Strasser 2008 Strasser DC, Falconer JA, Stevens AB, Uomoto JM, Herrin J, Bowen SE, Burridge AB. Team training and stroke rehabilitation outcomes: a cluster randomized trial. Archives of Physical Medicine and Rehabilitation 2008;89(1):10-5. Strikwerda 1994 Strikwerda P, Bootsma-de Langen AM, Berghuis F, Meyboom-de Jong B. Drug therapy in a nursing home; favorable effect of feedback by the pharmacist on family physician's prescribing behavior. Ned.Tijdschr.Geneeskd 1994;138: Sundaram 2009 Sundaram V, Lazzeroni LC, Douglass LR, Sanders GD, Tempio P, Owens DK. A randomized trial of computerbased reminders and audit and feedback to improve HIV screening in a primary care setting. International journal of STD & AIDS 2009;20: Szczepura 1994 Szczepura A, Wilmot J, Davies C, Fletcher J. Effectiveness and cost of different strategies for information feedback in general practice. The British Journal of General Practice 1994;44(378): [MEDLINE: ] Taylor 1997 Taylor EA, Thomas G, Cantrill JA. Changes in prescribing following a pharmicists-led audit of ulcer-healing therapy in general practice. The Pharmaceutical Journal 1997;259:R6. The SUPPORT 1995 The SUPPORT Principal Investigators. A controlled trial to improve care for seriously ill hospitalized patients. JAMA 1995;274: Thompson 2000 Thompson RS, Rivara FP, Thompson DC, Barlow WE, Sugg NK, Maiuro RD, et al. Identification and management of domestic violence: a randomized trial. American Journal of Preventive Medicine 2000;19(4): van Bruggen 2008 van Bruggen R, Gorter KJ, Stolk RP, Verhoeven RP, Rutten GEHM. Implementation of locally adapted guidelines on type 2 diabetes. Family Practice 2008;25(6): van der Sanden 2005 van der Sanden WJM, Mettes DG, Plasschaert AJM, Grol RPTM, Mulder J, Verdonschot EH. Effectiveness of clinical practice guideline implementation on lower third molar management in improving clinical decision-making: a randomized controlled trial. Eur.J.Oral Sci 2005;113(349-54). Van DWT 1998 Van DWT, Grol RPTM, Schouten BJ, Knottnerus JA. Barriers to working according to cholesterol guidelines: A randomized controlled trial on implementation of national guidelines in 20 general practices. European Journal of Public Health 1998;8: Velikova 2004 Velikova G, Booth L, Smith AB, Brown PM, Lynch P, Brown JM, et al. Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial. Journal of Clinical Oncology 2004;22:

217 Verstappen 2004 b Verstappen WHJM, van Merode F, Grimshaw J, Dubois WI, Grol RPTM, van der Weijden T. Comparing cost effects of two quality strategies to improve test ordering in primary care: a randomized trial. Int.J.Qual.Health Care 2004;16: Vinicor 1987 Vinicor F, Cohen SJ, Mazzuca SA, Moorman N, Wheeler M, Kuebler T, et al.. DIABEDS: a randomized trial of the effects of physician and/or patient education on diabetes patient outcomes. Journal of Chronic Diseases 1987;40: Walsh 2007 Walsh M, Laptook A, Kazzi SN, Engle WA, Yao Q, Rasmussen M et al. A cluster-randomized trial of benchmarking and multimodal quality improvement to improve rates of survival free of bronchopulmonary dysplasia for infants with birth weights of less than 1250 grams. Pediatrics 2007;119: Watkins 1981 Watkins CJ. Medical audit in general practice - fact or fantasy? The Journal of the Royal College of General Practitioners 1981;31: Weingarten 2000 Weingarten SR, Kim CS, Stone EG, Kristopaitis RJ, Pelter M, Sandhu M. Can peer-comparison feedback improve patient functional status? The American Journal of Managed Care 2000;6:35-9. Wells 2000 Wells KB, Sherbourne C, Schoenbaum M, Duan N, Meredith L, Unutzer J, et al. Impact of disseminating quality improvement programs for depression in managed primary care. JAMA 2000;283(2): Welschen 2004 Welschen I, Kuyvenhoven MM, Hoes AW, Verheij TJ. Effectiveness of a multiple intervention to reduce antibiotic prescribing for respiratory tract symptoms in primary care: randomised controlled trial. BMJ 2004;329:431. White 1995 White P, Atherton A, Hewett G, Howells K. Using information from asthma patients: a trial of information feedback in primary care. BMJ 1995;311: Wing 1987 Wing D, Duff HJ. Impact of a therapeutic drug monitoring program for digoxin. Archives of Internal Medicine 1987;147: Wing 1987 (II) Wing DS, Duff HJ. Evaluation of a therapeutic drug monitoring program frotheophylline in a teaching hospital. Drug Intelligence and Clinical Pharmacy 1987;21: Winickoff 1985 Winickoff RN, Wilner S, Neisuler R, Barnett GO. Limitations of provider interventions in hypertension quality assurance. American Journal of Public Health 1985;75:43-6. Winkens 1992 Winkens RA, Pop P, Grol RP, Kester AD, Knottnerus JA. Effect of feedback on test ordering behaviour of general practitioners. BMJ 1992;304: Winkens 1997 Winkens RAG, Knottnerus JA, Kester ADM, Grol RPTM, Pop P. Fitting a routine health-care activity into a randomized trial: an experiment possible without informed consent? Clinical Epidemiology 1997;50(4): Yano 2008 Yano EM, Rubenstein LV, Farmer MM, Chernof BA, Mittman BS, Lanto AB, Simon BF, Lee ML, Sherman SE. Targeting primary care referrals to smoking cessation clinics does not improve quit rates: Implementing evidencebased interventions into practice. Health Services Research 2008;43(5 P1): Young 2002 Young JM, DEste C, Ward JE.. Improving family physicians' Use of evidence-based smoking cessation strategies: A cluster randomization trial. Preventive Medicine 2002;35(6): Young JM, Ward JE. Randomised trial of intensive academic detailing to promote opportunistic recruitment of women to cervical screening by general practitioners.. Australian and New Zealand Journal of Public Health 2003;27: Zermansky 2002 Zermansky AG, Petty DR, Raynor DK, Lowe CJ, Freemantle N, Vail A. Clinical medication review by a pharmacist of patients on repeat prescriptions in general practice: a randomised controlled trial. Health Technol Assess 2002;6:1-86. Zoutman

218 Zoutman D,.Douglas F.. In: Am.J.Infect.Control Conference:E14-E15. Studies awaiting classification Bond 2011 Bond TC, Patel PR, Krisher J, Sauls L, Deane J, Strott K et al. A Group-Randomized Evaluation of a Quality Improvement Intervention to Improve Influenza Vaccination Rates in Dialysis Centers. American Journal of Kidney Diseases 2011;57: Daley 2011 Daley M, Shepard DS, Tompkins C, Dunigan R, Reif S, Perloff J et al. Randomized Trial of Enhanced Profiling in Substance Abuse Treatment. Administration and Policy in Mental Health and Mental Health Services Research 2011;38: Finkelstein 2005 Finkelstein JA, Lozano P, Fuhlbrigge AL, Carey VJ, Inui TS, Soumerai SB et al. Practice-level effects of interventions to improve asthma care in primary care settings: The pediatric asthma care patient outcomes research team. Health Services Research 2005;40: Guldberg 2011 Guldberg TL, Vedsted P, Kristensen JK, Lauritzen T. Improved quality of Type 2 diabetes care following electronic feedback of treatment status to general practitioners: a cluster randomized controlled trial. Diabetic Medicine 2011;28: Ivers 2010 Ivers NM, Tu K, Francis J, Barnsley J, Shah B, Upshur R et al. Feedback GAP: study protocol for a clusterrandomized trial of goal setting and action plans to increase the effectiveness of audit and feedback interventions in primary care. Implementation Science 2010;5:98. Labarene 2007 Labarere J, Bosson JL, Sevestre MA, Sellier E, Richaud C, Legagneux A. Intervention targeted at nurses to improve venous thrombloprophylaxis. International Journal for Quality in Health Care 2007;19(301-8). LaPointe 2006 LaPointe NMA, Delong ER, Chen A, Hammill BG, Muhlbaier LH, Califf RM et al. Multifaceted intervention to promote beta-blocker use in heart failure. American Heart Journal 2006;151: Lopez-Picazo 2011 Lopez-Picazo JJ, Ruiz JC, Sanchez JF, Ariza A, Aguilera B. A Randomized Trial of the Effectiveness and Efficiency of Interventions to Reduce Potential Drug Interactions in Primary Care. American Journal of Medical Quality 2011;26: Mourad 2011 Mourad SM, Hermens RPMG, Liefers J, Akkermans RP, Zielhuis GA, Adang E et al. A multi-faceted strategy to improve the use of national fertility guidelines; a cluster-randomized controlled trial. Human Reproduction 2011;26: Palmer 1996 Palmer RH, Louis TA, Peterson HF, Rothrock JK, Strain R, Wright EA. What makes quality assurance effective? Results from a randomized, controlled trial in 16 primary care group practices. Medical Care 1996;34:SS29-SS39. Sequist 2010 Sequist TD, Fitzmaurice GM, Marshall R, Shaykevich S, Marston A, Safran DG et al. Cultural Competency Training and Performance Reports to Improve Diabetes Care for Black Patients A Cluster Randomized, Controlled Trial. Annals of Internal Medicine 2010;152:40-W10. Smeets 2010 Smeets HM, de Wit NJ, Zuithoff NPA, van Dijk PCM, van der Lee APM, Hoes AW. A Health Insurance Company- Initiated Multifaceted Intervention for Optimizing Acid-Suppressing Drug Prescriptions in Primary Care A Randomized Controlled Trial. Archives of Internal Medicine 2010;170: Williams 2011 Williams JB, Delong ER, Peterson ED, Dokholyan RS, Ou FS, Ferguson TB. Secondary Prevention After Coronary Artery Bypass Graft Surgery Findings of a National Randomized Controlled Trial and Sustained Society-Led Incorporation Into Practice. Circulation 2011;123: ' 203

219 9.2 Appendix Goal-setting and Action-plan Worksheet for Intervention Arm 1. Describe a goal that you will achieve within the next 6 months for your diabetic patients and for your IHD patients. Your goal must be challenging but achievable. Be very specific. [Phrase your goal as follows: I will improve BP) (choose one of the outcomes in the practice profile e.g. % at target to the goal of (state a target for your efforts e.g. by 20 percentage points) ] For Diabetes, I will improve: To the goal of: For Ischemic heart disease (IHD), I will improve: To the goal of: 2. Complete the following statements by describing a specific action you will take to help you achieve your goal: To identify on an ongoing basis the specific patients in my practice who are not meeting targets, I will: 204

220 If a patient with Diabetes and/or IHD comes to clinic (for any reason) and is not meeting targets, I will: If I am too busy during an office visit to address all aspects of managing the patient s diabetes and/or IHD, I will: If I m not making progress with respect to implementing my plan for achieving my goals, I will: In signing below, I confirm my commitment to achieve this goal and my intention to carry out this action plan. Signature Date Prototype of feedback report that all participants received 205

221 PHYSICIAN ID#: x Approximately 12% of your rostered adult patients have diabetes, and 30% of these patients also have ischemic heart disease Overall in this study, 7% of rostered adult patients have diabetes, and 19% of these patients also have ischemic heart disease Your diabetic patients are 68 years old on average and are 57% male. All diabetic patients in the study average 63 years and are 55% male. Targets Your Top 10% Practice A1C 7.0 % 62% 67% A1C test in 6M 81% 91% BP < 130/80 48% 72% BP test in 6M 86% 98% Rx ACE / ARB 77% 88% LDL % 55% LDL test in 12M 55% 80% Rx Statin 72% 83% ACR test in 12M 84% 85% "Top 10%" = the score achieved by 10% of physicians with the best score for each target. (This data is based on your most recent EMR data upload, May,2010) ACR = urinary albumin creatinine ratio (microalbumin) 206

222 PHYSICIAN ID#: x Approximately 9% of your rostered adult patients have ischemic heart disease, and 41% of these patients also have diabetes Overall in this study, 5% of rostered adult patients have ischemic heart disease, and 28% of these patients also have diabetes Your IHD patients are 71 years old on average and are 74% male. All IHD patients in the study average 70 years and are 65% male. Targets Your Practice Top 10% Rx ASA 33% 69% BP < 140/90 75% 89% BP test in 6M 83% 95% Rx ACE / ARB 72% 85% Rx B-Blocker 57% 72% LDL % 56% LDL test 12M 58% 91% Rx Statin 77% 89% "Top 10%" = the score achieved by 10% of physicians with the best score for each target. (This data is based on your most recent EMR data upload, May,2010) 9.3 Appendix Original CME surveys 207