-Preliminary findings- Audits in hospital care: Which factors determine the effectiveness of audits? A realist review of the literature Lisanne Hut University Medical Centre Groningen, Centre of Expertise on Quality and Safety, the Netherlands EHMA Conference Porto 2016 Prof. dr. R.O.B. Gans Prof. dr. ir. C.T.B. Ahaus Dr. G.A. Welker
PhD research Clinical audit can be an effective instrument in improving the quality of local healthcare 1,2 However, its benefits have not rigorously been tested 1. Which determinants influence the effectiveness of audits? 2. Are these determinants also of influence on clinical audits? 3. What are the effects of clinical audit on the quality of hospital care? 1 Ivers, N., Jamtvedt, G., Flottorp, S., Young, J. M., Odgaard-Jensen, J., French, S. D., et al. (2012). Audit and feedback: Effects on professional practice and healthcare outcomes. The Cochrane Database of Systematic Reviews, 6, CD000259 2 Scrivener, R., Morrell, C., Baker, R., Redsell, S., Shaw, E., Stevenson, S., et al. (2002). Principles for best practice in clinical audit. Abingdon, UK: Radcliffe Medical Press Ltd 40
Practical example: Clinical audits Based on practical experiences of residents Many reports of falls Clinical audit conducted on fall-risk screening Action plans Re-audit after 6 months 41
Overview Introduction Methods Theory and steps of data extraction Questions & discussion 42 Overview Introduction Methods Theory Questions & Discussion
Audits in healthcare settings Purpose External audit Internal audit Clinical audit To demonstrate compliance with external criteria (certification; accreditation) To demonstrate compliance with regulations, often as preparation for external audit To improve clinical practice by examining care against criteria and implement improvements Performed by External auditors Internal auditors, often quality staff Healthcare professionals 43 Overview Introduction Methods Theory Questions & Discussion
Effectiveness Variety in effects of external audits and audit and feedback in improving healthcare quality 3-6 Effectiveness of quality improvement strategies seems to be influenced by Continuous improvement focus Bottom-up initiative Active involvement of healthcare professionals 3 Brubakk, K., et al. (2015). BMC Health Services Research, 15: 208. doi: 10.1186/s12913-015-0933-x 4 Flodgren, G., et al. (2011). The Cochrane Database of Systematic Reviews, 11. doi: 10.1002/14651858.CD008992.pub2 5 Greenfield, D., & Braithwaite, J. (2008). International Journal for Quality in Health Care, 20:3. doi: 10.1093/intqhc/mzn005 6 Ivers, N., et al. (2012). The Cochrane Database of Systematic Reviews, 13: 6. doi: 10.1002/14651858.CD000259.pub3 44 Overview Introduction Methods Theory Questions & Discussion
Conceptual model Audit context (C) Mechanisms of the audit (M) Audit outcomes (O) 1. Organisational culture 2. Other contextual factors 1. Continuous improvement 2. Bottom-up 3. Active involvement of professionals 4. Other mechanisms Improved quality of healthcare 45 Overview Introduction Methods Theory Questions & Discussion
Research objectives To identify determinants of the effectiveness of audits. To investigate whether a continuous improvement focus, bottom-up initiative and the active involvement of healthcare professionals contribute to the effectiveness of audits To identify other relevant determinants of effective audits 46 Overview Introduction Methods Theory Questions & Discussion
Realist review: Context, Mechanisms and Outcomes Don t ask does it work or not? Rather, investigate: why does it work and in what circumstances? The same intervention mechanism may have different outcomes in different contexts 47 Overview Introduction Methods Theory Questions & Discussion
Methodological steps Scope Search for evidence Quality appraisal of the evidence Data extraction and coding Synthesize findings Dissemenation 48 Overview Introduction Methods Theory Questions & Discussion
Search strategy Quality and safety studies in healthcare management and biomedical field MEDLINE Embase, PsychINFO, Academic Search Premier, Business Source Premier, EmeraldInsight, Cochrane Library and Web of Science 49 Overview Introduction Methods Theory Questions & Discussion
PRISMA flowdiagram 50 Overview Introduction Methods Theory Questions & Discussion
Quality Appraisal and Data Extraction Grading of Recommendations Assessment, Development, and Evaluation (GRADE) scale Data extraction form Country Study setting and design Unit of analysis CMO-elements Main findings Recommendations 51 Overview Introduction Methods Theory Questions & Discussion
Methodological steps Scope Search for evidence Quality appraisal of the evidence Data extraction and coding Synthesize findings Dissemination 52 Overview Introduction Methods Theory Questions & Discussion
Steps of data extraction 1 Mapping the outcomes 2 Explaining the outcomes 3 CMO-configurations 53 Overview Introduction Methods Theory Questions & Discussion
Step 1: Mapping the outcomes Audit context (C) Mechanisms of the audit Audit outcomes (O) (M) Study A Supportive culture for quality improvements Collaboration between stakeholders Improved quality of healthcare 54 Overview Introduction Methods Theory Questions & Discussion
Step 1: Mapping the outcomes Study A Audit context (C) Mechanisms of the audit (M) Audit outcomes (O) Supportive culture for quality improvements Collaboration between stakeholders Improved quality of healthcare Study B Audit context (C) Mechanisms of the audit (M) Audit outcomes (O) CEO s Commitment for quality improvement Weak participation of professionals Improved quality of healthcare 55 Overview Introduction Methods Theory Questions & Discussion
Step 2: Explaining outcomes Study B Same mechanisms within different contexts Audit context (C) Mechanisms of the audit (M) Audit outcomes (O) Supportive culture Active participation Improved quality Study A Study C Audit context (C) Mechanisms of the audit Audit outcomes (O) Audit context (C) Mechanisms of the audit Audit outcomes (O) (M) (M) External pressure Active participation No improved quality CEO s Commitment Active participation Improved quality 56 Overview Introduction Methods Theory Questions & Discussion
Step 3: CMO-configurations Within a context with a supportive culture for quality improvement (context) the active participation of healthcare professionals in the audit process (mechanism) leads to improved quality of healthcare (outcome). 57 Overview Introduction Methods Theory Questions & Discussion
Go to www.govote.at and enter the code 22 41 55 to vote Vote with your mobile or any other device 58 Overview Introduction Methods Theory Questions & Discussion
Discussion I Quality improvement stands or falls with the active involvement of healthcare professionals Go to www.govote.at and enter the code 22 41 55 to vote https://www.mentimeter.com/app/edit/a907bbcce555b42ebd44b49c33500dc0/5e25f93f4968 59 Overview Introduction Methods Theory Questions & Discussion
60 Overview Introduction Methods Theory Questions & Discussion
Take home messages Audits are complex interventions Theory and context are important to understand how complex interventions work Realist review is a promising approach to understand contextual influences on whether, why and how audits work No one-size-fits-all set to predict success or failure of audits 61 Overview Introduction Methods Theory Questions & Discussion
Discussion II Which other mechanisms are of influence on the effectiveness of audits? 62 Overview Introduction Methods Theory Questions & Discussion
Thank you for your attention Lisanne Hut-Mossel RN MSc http://www.rug.nl/staff/p.a.hut-mossel/research Email: p.a.mossel@umcg.nl 63 Overview Introduction Methods Theory Questions & Discussion
Literature I Bohigas, L., & Heaton, C. (2000). Methods for external evaluation of health care institutions. International Journal for Quality in Health Care: Journal of the International Society for Quality in Health Care/Isqua, 12(3), 231-238. Brubakk, K., Vist, G. E., Bukholm, G., Barach, P., & Tjomsland, O. (2015). A systematic review of hospital accreditation: The challenges of measuring complex intervention effects. BMC Health Services Research, 15, 280-015-0933-x. Dixon, N. (2014). Getting clinical audit right to benefit patients. Romsey, England: Healthcare Quality Quest. Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature (FMHI 231). Tampa: University of South Florida, Louis De La Parte Florida Mental Health Institute, National Implementation Research Network., Flodgren, G., Pomey, M. P., Taber, S. A., & Eccles, M. P. (2011). Effectiveness of external inspection of compliance with standards in improving healthcare organisation behaviour, healthcare professional behaviour or patient outcomes. The Cochrane Database of Systematic Reviews, (11):CD008992. doi(11), CD008992. Greenfield, D., & Braithwaite, J. (2008). Health sector accreditation research: A systematic review. International Journal for Quality in Health Care: Journal of the International Society for Quality in Health Care/ISQua, 20(3), 172-183. Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly, 82(4), 581-629. Ivers, N., Jamtvedt, G., Flottorp, S., Young, J. M., Odgaard-Jensen, J., French, S. D., et al. (2012). Audit and feedback: Effects on professional practice and healthcare outcomes. The Cochrane Database of Systematic Reviews, 6, CD000259. Johnston, G., Crombie, I. K., Davies, H. T., Alder, E. M., & Millard, A. (2000). Reviewing audit: Barriers and facilitating factors for effective clinical audit. Quality in Health Care : QHC, 9(1), 23-36. 64
Literature II Ovretveit, J. (2011). Understanding the conditions for improvement: Research to discover which context influences affect improvement success. BMJ Quality & Safety, 20 Suppl 1, i18-23. Pawson (2003) Nothing as Practical as a Good Theory. Evaluation 9 (3) pp 309-321 Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005). Realist review--a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy, 10 Suppl 1, 21-34. Scrivener, R., Morrell, C., Baker, R., Redsell, S., Shaw, E., Stevenson, S., et al. (2002). Principles for best practice in clinical audit. Abingdon, UK: Radcliffe Medical Press Ltd. Shaw, C. D., Groene, O., Botje, D., Sunol, R., Kutryba, B., Klazinga, N., et al. (2014). The effect of certification and accreditation on quality management in 4 clinical services in 73 european hospitals. International Journal for Quality in Health Care : Journal of the International Society for Quality in Health Care / ISQua, 26 Suppl 1, 100-107. Shojania, K. G., McDonald, K. M., Wachter, R. M., & Owens, D. K. (2004). Closing the quality gap: A critical analysis of quality improvement strategies (vol. 1: Series overview and methodology). Rockville (MD): UCSF Evidence-based Practice Center. Walshe, K. (2007). Understanding what works--and why--in quality improvement: The need for theorydriven evaluation. International Journal for Quality in Health Care: Journal of the International Society for Quality in Health Care/Isqua, 19(2), 57-59. Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013). RAMESES publication standards: Realist syntheses. BMC Medicine, 11, 21-7015-11-21. World Health Organization. (2006). Quality of care - A process for making strategic choices in health systems. France: World Health Organization. 65
Search strategy MEDLINE (Pubmed) ("Clinical Audit"[Majr:noexp] OR "Medical Audit"[Majr] OR "Nursing Audit"[Majr] OR "Accreditation"[Majr] OR "Certification"[Majr:noexp] OR "Peer Review, Health Care"[Majr] OR ((extern*[tiab] OR internal*[tiab]) AND audit[tiab]) OR medical audit*[tiab] OR clinical audit*[tiab] OR nursing audit*[tiab] OR audit[ti] OR audits[ti] OR accreditat*[ti] OR visitation*[ti]) AND ("Academic Medical Centers"[Mesh:noexp] OR "Hospitals, Teaching"[Mesh] OR "Outpatient Clinics, Hospital"[Mesh:noexp] OR "Hospitals"[Mesh:noexp] OR hospital*[tiab] OR ((health*[tiab] OR clinical[tiab]) AND (organisation*[tiab] OR organization*[tiab] OR center*[tiab] OR centre*[tiab])) OR health sector*[tiab] OR healthcare sector*[tiab] OR health care sector*[tiab]) AND ("Efficiency, Organizational"[Mesh] OR efficien*[tiab] OR effectiveness*[tiab] OR performan*[tiab] OR improvement*[tiab] OR "Quality Improvement"[Mesh:noexp] OR "Quality Assurance, Health Care"[Majr:noexp] OR quality improv*[ti] OR quality assur*[ti]) NOT ("Animals"[Mesh] NOT "Humans"[Mesh]) AND english[la] AND ("last 10 years"[pdat]) 66