The measurement and monitoring of safety Health Foundation summary April 2013 For more information and to download or order the full report, please visit: www.health.org.uk/measuresafety
Introduction Over the past 10 years there has been a huge volume of data collected on medical error and harm to patients, many tragic cases of healthcare failure and a growing number of major government and professional reports on the need to make healthcare safer. There is now widespread acceptance and awareness of the problem of medical harm, and considerable efforts have been made to improve the safety of healthcare. But if we ask whether patients are any safer than they were 10 years ago, the answer is curiously elusive. What we currently measure is not how safe healthcare systems are now but how harmful they have been in the past. The Health Foundation believes that we cannot improve patient safety until we have a clear understanding of how to know if care is safe in the first place. About this document This document provides a short overview of the full report. It introduces the framework for safety measurement and monitoring and gives a short summary of the full report s discussion of each of its dimensions. In the full report, the authors conclude by setting out ten guiding principles for safety measurement and monitoring. These are also summarised here. In The measurement and monitoring of safety, Professor Charles Vincent and colleagues from Imperial College London draw together evidence from a range of sources (published research, public data, case studies and interviews), both from within healthcare settings and from other safety critical industries. The authors have synthesised this evidence and propose a single framework for safety measurement and monitoring that brings together a number of conceptual and technical facets of safety. The framework highlights the key dimensions that any healthcare organisation should consider in its safety measurement plans. It also provides a starting point for discussions about what safety means and how it can be actively managed. This document summarises The measurement and monitoring of safety, a Health Foundation Spotlight report by Charles Vincent, Susan Burnett and Jane Carthey. The full report is available to order or download from: www.health.org.uk/measuresafety It is also available as a free ebook from itunes or Google Play. 2 THE HEALTH FOUNDATION A printed version of this In brief is available as an A5 booklet.
A framework for safety measurement and monitoring When we ask whether a healthcare organisation is safe, what exactly do we want to know? One reason this question is so difficult to answer is that it has a number of different facets, which are not always clearly distinguished. We can approach the different dimensions of safety in an organisation by asking five critical questions: Has patient care been safe in the past? Are our clinical systems and processes reliable? Is care safe today? Will care be safe in the future? Are we responding and improving? In The measurement and monitoring of safety, Professor Vincent and colleagues develop these questions into a framework, underpinned by a rigorous review of the relevant literature and survey of current practice. This framework highlights five dimensions, which the authors believe should be included in any safety and monitoring approach in order to give a comprehensive and rounded picture of a healthcare organisation s safety. The five dimensions are: Past harm: this encompasses both psychological and physical measures. Reliability: this is defined as failure free operation over time and applies to measures of behaviour, processes and systems. Sensitivity to operations: the information and capacity to monitor safety on an hourly or daily basis. Anticipation and preparedness: the ability to anticipate, and be prepared for, problems. Integration and learning: the ability to respond to, and improve from, safety information. Professor Vincent and colleagues believe that this framework encompasses the principal facets of safety while also providing a simplicity and clarity with which to guide and inform safety measurement and monitoring. Each of the framework s dimensions are briefly explained and discussed in the following pages. Figure: A framework for safety measurement and monitoring IN BRIEF: THE MEASUREMENT AND MONITORING OF SAFETY 3
Past harm Traditionally, patient safety has focused on comparatively rare, often tragic, events. But over time it has become clear that the frequency of error and harm is much greater than previously realised. Patient safety now involves much more than preventing tragic but rare events. We also need to address healthcare-acquired infections, adverse drug events, complications and harm from falls and pressure ulcers, together with a host of other rare and less predictable incidents. To assess harm from healthcare, we have to consider all these kinds of events. Most patients are vulnerable, to some degree, to infections, adverse drug events, falls, and the complications of surgery and other treatments. Patients who are older, frailer or have several conditions may be affected by over-treatment, polypharmacy and other problems such as delirium, dehydration or malnutrition. Patients may also suffer harm from rare and perhaps unforeseeable events stemming from new treatments, new equipment or rare combinations of problems that could not easily have been foreseen. The report divides these various forms of harm into the following six categories. Treatment-specific harms, such as adverse drug reactions or complications of treatment. Harm due to overtreatment, such as falls resulting from excessive use of sedatives. General harm from healthcare, such as hospitalacquired infection. Harm due to failure to provide appropriate treatment, such as failure to provide prophylactic antibiotics before surgery. Harm resulting from delayed or inadequate diagnosis, such as a slow diagnosis or misdiagnosis of cancer symptoms. Psychological harm and feeling unsafe, such as clinical depression following mastectomy. In recent years, healthcare organisations and researchers have taken several different approaches to measuring harm, using a range of methods and data sources. Some, such as mortality, focus on a very specific issue. Others, such as record review, attempt to cover a very broad range of possible types of harm. The available measures can be grouped into four broad types: mortality statistics methods that rely on record review methods that rely on staff reporting routine databases. Each of these groups of measures has strengths and limitations, and none can claim to reflect all the kinds of harm discussed above as they all focus on different issues. An organisation may have low levels of mortality but a high rate of adverse events overall, or vice versa. Each measure is a useful but partial assessment of the underlying broad issue of harm, and the usefulness and validity of each measure is still being explored. Reflections Measuring harm is not equivalent to measuring safety but is an essential foundation. Whatever approach is taken to measure harm, it must be valid and reliable. It is important to pay attention to the reliability of the data source and to clearly define what kind of harm is being measured. We need to devise more specific and more nuanced measures of harm that can be tracked over time and clearly demonstrate whether healthcare is becoming safer. However, if we want to provide a more rounded approach to safety measurement and monitoring, we need to look beyond the measurement of harm. This is what the other four framework dimensions seek to do. 4 THE HEALTH FOUNDATION
Reliability Reliability is an essential foundation of safety, but reliability alone is not enough to ensure safety. It is concerned only with the probability of occurrence of a failure, rather than with the severity or otherwise of its consequences. Reliability is defined as failure-free operation over time and has been a focus of safety conscious industries such as aviation and nuclear power for many years, with impressive results. In healthcare, by contrast, it is well established that many systems have poor reliability. Some studies have found reliability as low as 50% in delivering recommended evidence-based care for clinical conditions. Some variation might be explained by different patient characteristics. However, one might reasonably expect the routine processes that support clinical care such as ensuring that relevant information is available to doctors in clinics will have high reliability. The concept of reliability can be applied most meaningfully to those aspects of healthcare systems that have a higher degree of agreement and standardisation for example: Reliability of clinical systems. Where staff accept poor reliability, they do not report or challenge problems. The report findings suggest that improving common system factors in organisations could have a bigger impact on patient safety than current approaches focusing on individual areas of risk. Perhaps more important is the need to develop a culture of challenge, so that staff no longer accept poor reliability and the associated potential for patient harm as a normal part of everyday work. Reliability of human behaviour. For essential standardised procedures, safety is maintained by the conscientious, disciplined adherence to rules. Three such areas that require a protocol approach are hand washing, medication errors and intravenous drug administration. In the English NHS, reliability is typically assessed through a rolling programme of clinical audits some locally determined and some in response to national imperatives. These are reported to clinical and management groups and summarised in the organisation s quality accounts. Organisations carry out a wide variety of assessments of reliability of processes, of staff compliance with procedures and of the maintenance and use of equipment. In most cases these are not seen as contributing to an assessment of the overall reliability of a system. There are three reasons for this. Staff are unaccustomed to thinking in terms of standardisation and reliability of processes that, for example, would come naturally to engineers. Many of these assessments are made in response to external demands from different organisations and therefore tend to be viewed in isolation. There is seldom any attempt to make an initial assessment of what processes in a clinical unit or organisation are essential to safety or to set targets for reliability. Reflections As outlined above, organisations already carry out a variety of reliability assessments. For many organisations, the next step is to identify all safety critical processes and specify the levels of reliability expected. This seemingly simple step would be a massive transformation in healthcare, representing a move from gradual improvement towards an engineering perspective in which systems are designed to operate to certain specifications under a range of conditions. Monitoring reliability across a system would be a major challenge, although all processes could be assessed periodically, but this is undoubtedly the direction of travel if healthcare is to achieve true reliability. IN BRIEF: THE MEASUREMENT AND MONITORING OF SAFETY 5
Sensitivity to operations If we want to be safe when we drive a car, operate machinery or cross the road, we have to continuously monitor our own actions, attend to the environment, and adapt and respond to changing circumstances and hazards. Those working in risky environments whether in a cockpit, an operating theatre or a primary care clinic have to maintain alertness and safety awareness in a similar way. This vision can be expanded to consider the safe running of an organisation. Certainly one must monitor harm and consider the reliability of systems over time. But safety also requires monitoring the workings of the organisation on a day-to-day basis. High reliability organisations use the phrase sensitivity to operations (with operations referring to the workings of an organisation, rather than surgical procedures) to describe their staff s acute awareness of the workings of the organisation and sensitivity to subtle changes and disturbances. Sensitivity to operations enables people to identify problems early so that actions can be taken before they threaten safety. Specific mechanisms that support sensitivity to operations in healthcare include the following. Safety walk-rounds. An important source of safety intelligence, where senior managers discuss safety concerns with the workforce. Using designated patient safety officers. Clinicians and others with a specific role to actively seek out, identify and resolve patient safety issues in their clinical units. Meetings, handovers and ward rounds. Opportunities for cascading patient safety information within and across staff teams and between staff and patients or carers. Day-to-day conversations. Informal dialogue between healthcare teams and managers, used to identify attitudes and behaviours that could indicate poor team safety culture. Patient interviews to identify threats to safety. Highlighting practical difficulties and harms experienced by patients that might not be immediately obvious to staff, such as assumptions by staff that a patient has understood the information provided at discharge. The other key component of sensitivity to operations is timely action and intervention to thwart potential safety risks. Real-time information from safety measurement performance systems supports sensitivity to operations by improving the timeliness with which healthcare teams receive safety intelligence. For example, a clinical director receives weekly hand-hygiene data that indicates a decrease in compliance with hand washing on a ward. However, in practice, different timescales are appropriate to different contexts. Sometimes, in clinical settings, safety needs to be monitored on a minute-byminute basis. Managers typically may have to resolve the bulk of minor problems either on a daily basis or within a week or so. Reflections Different healthcare organisations have different approaches to monitoring safety on a day-to-day basis. Some have more evolved approaches than others perhaps because they have a greater appreciation of the importance of this kind of information. However, it may also be that some approaches work more effectively in one organisation than in another. This would indicate the importance of the fit between the organisation and the methods it uses to achieve sensitivity to operations. The timeliness aspect of sensitivity to operations does not fit well in an NHS culture, where safety information might not be acted on until action is sanctioned by a monthly or even quarterly committee. If we want to enable healthcare organisations to act on information more quickly, we may need to rethink some of the structures and processes that are accepted as the architecture of a safe organisation. 6 THE HEALTH FOUNDATION
Anticipation and preparedness Anticipation is a key component of expertise in many areas and forms a critical element of safety. Essentially, it involves thinking ahead and envisioning possible problems and hazards, enabling those involved to make plans and be prepared. The ability to anticipate and respond is an essential part of delivering safe clinical care. In clinical work, treating complex, fluctuating conditions requires thinking ahead and being prepared to adjust treatment as the patient s condition changes. Considering the safety of an organisation requires a broader vision. Clinicians and managers need to use information to anticipate the safe functioning of the organisation in which they work, assessing the hazards and taking action to reduce the risks over time. Safety, from this broader perspective, requires anticipation, preparedness, and the ability to intervene to reduce risks at the ward, department or systems level. Anticipation and preparedness require that questioning is encouraged, even when things are going well, and creating opportunities for staff to think about potential problems and hazards. Reviewing trends in harm to patients, reliability of procedures or reflections on the organisational culture could all provoke questions about how resilient the organisation might be in more hostile circumstances. The following approaches have been used to anticipate and prepare for risk. Risk registers. These are commonly used across healthcare settings to capture and grade levels of risk and put in place action plans to mitigate the risks identified. Disadvantages include an unresponsive quarterly timeframe, the retrospective nature of identified risks gleaned from lessons learned, and the risk of being seen as a tick-box exercise. Human reliability analysis. These techniques take a process of care and systematically examine it to identify and anticipate possible failure points. They provide a structured way to anticipate factors such as workload, patient familiarity, communication across interfaces and levels of decision-making expertise in the system design phase. Safety cases. These comprise processes to build an argument and present the evidence base to demonstrate that a system is designed safely. They are typically used in safety-critical industries but have recently been proposed for use in healthcare to overcome the assumptions and dependencies that can result from a health regulation system that focuses on certification and audit. Safety culture analysis. Research has found that safety culture is associated with accident rates and a variety of indices of safety, but few studies have attempted to forecast future accidents from current measures of culture. Similarly, safety climate among nurses has been strongly associated with patient outcomes and staff injury. Staff indicators. Safety indicators relating to staff can be used to anticipate whether care will be safe in the future. These include sickness absence rates, the number of staff who have attended training on medication safety, and the frequency of sharps injuries per month. Reflections At an organisational level, anticipation and preparedness is relatively undeveloped in healthcare and within the NHS. The different dimensions of safety and the associated analysis for anticipation need to be further explored, both in research and practice. There is a plethora of safety-related information in trusts, but the extent to which this is systematically used to anticipate whether care will be safe in the future varies across healthcare organisations and between care settings. Meanwhile, some potentially useful methods, such as human reliability analysis or safety cases, are not widely known about or used in service design. This is perhaps the next challenge if we are to really begin to improve patient safety. IN BRIEF: THE MEASUREMENT AND MONITORING OF SAFETY 7
Integration and learning All healthcare organisations will, if they look, discover numerous incidents and deviations from best practice. Safe organisations actively seek out such incidents, and respond by attempting to harness the learning to influence their future functioning. However, in healthcare, with the wide variety of safetyrelated data available, it is often hard to know how to integrate the wealth of information, analyse it in a meaningful way, and use it to support organisational learning and implement sustainable improvements. One of the challenges facing risk management or patient safety departments is to understand how best to integrate and weight the multiple sources of data that potentially shed light on safety issues, in order to effectively prioritise them. Data sources could include: incidents reported, incidents detected from administrative data, complaints, health and safety incidents, inquests, claims, clinical audits, routine data, observations and informal conversations with patients, families and staff. Ways of integrating the different data sources include the following. Integration at clinical unit level. For example, through an automated information management system highlighting details such as medication errors and hand hygiene compliance rates. Integration and learning at board level. For example, using dashboards and reports with indicators, set alongside financial and access targets, with priorities colour-coded red, amber or green. Integration across a whole system of care. For example, developing an online reporting portal for quality and patient safety with web-enabled reporting and statistical process control (SPC) charts on demand. Using multiple information systems at population level. For example, bringing together one dashboard relating to safety and quality from a wide variety of data sources across an entire population. A safety information system should really be seen as an information, analysis, learning, feedback and action system. Few healthcare organisations have achieved this, but one can imagine the kind of data collection, analysis and response that this would entail. Many organisations currently expend most of their efforts on data collection, to the detriment of other aspects. Too narrow a focus on reporting inevitably leads to less resource invested in the more critical areas of feedback and learning. Incident analysis is an important element not to retrospectively search for root causes, but to highlight continuing system weaknesses that could lead to another incident. After this analysis, a range of feedback methods can be used to disseminate lessons learned to healthcare teams, including intranet lessons weblinked back to clinical guidelines, safety newsletters and information sharing by team leaders. The final process is to integrate the information, analyse it meaningfully, draw lessons and, where necessary, initiate improvement programmes. One way of doing this is developing annual organisational learning reports which integrate and analyse all safety indicators to bring out learning themes for the organisation. Such reports enable different services to benchmark themselves against each other and provide an organisation-wide picture, highlighting where action is required. Reflections Healthcare organisations take many different approaches to integrating and learning from the various sources of safety information. This is understandable, given the diversity of the clinical services provided across different care settings and the different patient populations served. However, whatever the approach, feedback, action and improvement are key elements in integration and learning. It is essential that healthcare organisations balance the focus of collecting and integrating safety information with appraising how it is used to deliver meaningful feedback, action and improvement. 8 THE HEALTH FOUNDATION
Ten guiding principles Professor Vincent and colleagues conclude their report by suggesting ten guiding principles for safety measurement and monitoring. These are not set in stone, but rather potential directions of travel, derived from the authors synthesis of the experiences of many people and organisations and the wider safety literature. 1. A single measure of safety is a fantasy The search for simple metrics has sometimes led organisations to use a single measure, such as standardised mortality, as a generic indicator of safety performance. However, safety cannot be encapsulated in a single measure. This reductionist approach could even make healthcare organisations less safe, by providing false reassurance. 2. Safety monitoring is critical and does not receive sufficient recognition Healthcare organisations use a variety of approaches, both formal and informal, to elicit safety information that enables them to understand how frontline healthcare services are delivered. But staff need the time, freedom and authority to monitor and intervene when necessary. Patients and carers play an essential role in safety monitoring, too for example, by listening, perceiving and anticipating risk. However, all too often they are an underused defence in preventing patient harm. 3. Anticipation and proactive approaches to safety As safety measurement evolves within an industry, it tends to move away from an over-reliance on lagging (after the event) indicators to a mixed model, combining both lagging and leading (before an event) indicators. However, the healthcare case studies in the full report generated comparatively few examples of anticipation and preparedness metrics. 4. Integration and learning: invest in technology and expertise in data analysis Currently, safety information is fragmented within NHS organisations and across the wider system. Probably the greatest challenge is to integrate it into a useable and comprehensible format. Some organisations are already achieving this, having invested in data analysis teams and automated data capture. 5. Mapping safety measurement and monitoring across the organisation Safety measurement and monitoring must be customised to local settings and circumstances to some extent. In each clinical context, we need to consider what kinds of harm are prevalent, what features of care must be reliable, and how we monitor, anticipate and integrate safety information. 6. A blend of externally required metrics and local development Safety measurement and monitoring needs to be customised to local settings and local circumstances. Many measures should be agreed nationally or even internationally, although they can be complemented by locally developed measures. But day-to-day monitoring, anticipation and preparedness are necessarily local activities, whether at ward or board level. 7. Clarity of purpose is needed when developing safety measures Healthcare regulators, national agencies and commissioners need to consider the criteria for safety measures and be clear about the purpose of each measure. They must also be wary of excessively complex data collection and need to test safety measures before implementation. 8. Empowering and devolving responsibility for the development and monitoring of safety metrics is essential Clinical units need the flexibility to develop measures that are relevant and adapted to their clinical context, so that clinicians do not become disenfranchised. Healthcare regulators need to adopt a goal-setting approach that allows organisations some flexibility in how they demonstrate that their care is safe. 9. Collaboration between regulators and the regulated is critical The fragmentation of key safety information across multiple national and local stakeholders, combined with the fragmented approach to regulation, are potential threats to safety. The multiplicity of regulators means that huge resources are consumed in meeting external demands, to the detriment of critical activities such as monitoring, anticipation and improvement. 10. Beware of perverse incentives Some types of measurement introduce perverse incentives that can lead to unwanted behaviour. For example, imposing financial penalties may promote under-reporting or excessive focus on one type of harm. Instead, we need a more holistic approach to measuring, monitoring and implementing interventions for all potential types of harm. IN BRIEF: THE MEASUREMENT AND MONITORING OF SAFETY 9
The Health Foundation 90 Long Acre London WC2E 9RA T 020 7257 8000 F 020 7257 8001 E info@health.org.uk Registered charity number: 286967 Registered company number: 1714937 For more information, visit: www.health.org.uk Follow us on Twitter: www.twitter.com/healthfdn Sign up for our email newsletter: www.health.org.uk/enewsletter 2013 The Health Foundation