Reviewing Methods Used in Patient Safety Research: Advantages and Disadvantages Dr Jeanette Jackson (j.jackson@abdn.ac.uk) This SPSRN work is funded by
Introduction Effective management of patient safety in healthcare requires: 1. an understanding of the causes of adverse events and related outcomes 2. a capacity to measure adverse events and their causes as well as related outcomes at different levels (individual, unit, organization, industry, national, international) Measurement of industry safety status is achieved by a range of methods based on key performance indicators for risk factors and safety events as well as leading indicators for safety (including causes like cultural factors)
Objectives 1. To review possible methods for measuring patient safety 2. To evaluate available methods for measuring patient safety
Methods Measuring Patient Safety Methods can be classified based on different criteria: 1. Related to the source of data (Runciman et al., 2008): e.g., research data from previous studies, routine surveillance data, quality assurance and risk management data 2. Human factor and ergonomic methods used in system design and evaluation (Carayon, 2007): e.g., cognitive work analysis, incident analysis, work systems and process analysis, usability evaluation, assessing safety culture and climate 3. WHO report on methods measuring patient safety (Jeffs, Law & Baker, 2008): retrospective, prospective and concurrent states of patient safety
Retrospective Methods: Methods Measuring Patient Safety Medical Records, Incident Reporting Systems, Claims and Complaints, Staff Questionnaires, Interviews Exploring adverse event, errors and near misses Awareness about patient safety issues Inform improvement programmes
Prospective Methods: Methods Measuring Patient Safety Prospective Analysis Tools, Simulations, Safety Culture Assessment Identifying sources of potential error based on knowledge about error rates
Concurrent Methods: Methods Measuring Patient Safety Direct Observations, Video Techniques, Checklists and Audits Systems that update information once they receive data Monitoring and controlling patient care processes Estimates of error rates Reaction to error as it occurs
Retrospective Methods Medical Records: Indicators of the cause, nature and incidence of harm Most frequently used for researching medical error Using triggers, e.g., high risk medication or laboratory values Adverse events studies: 3-17%
Medical Records Advantages: Highlighting particular incidents / areas that require further understanding / training Disadvantages: Missing incidents that are not reported Not everything documented, e.g., confounding factors or human factors Time consuming, staff costs (training for screening and reviewing)
Retrospective Methods Incident Reporting Systems: Different systems implemented in different health care systems and settings Reporting adverse outcomes for patients, errors and near misses (e.g., adverse drug events, problems with medical devices, safety of blood products) Information on causes of adverse events, provide warnings, point to important problems, raise awareness and enhance safety
Incident Reporting Systems Advantages: Systems can be implemented across multiple sites Minimal costs for volume of data (?) Can highlight effectiveness of existing policies / procedures Disadvantages: Depending on coding systems used as well as the ability of using these systems reliably What and how to report (high hazard events, willingness to report) Number of reported incidents (lack of resources, varying quality) Ensuring that feedback (e.g., newsletter, meetings) is given and actions are taken - effectiveness of learning from incident data unclear (lack of documentation) Lack of clarity in roles and responsibilities A large amount of time and resources involved
Retrospective Methods Claims and Complaints: Claims reflect a patients demand for compensation (general safety issues) whereas complaints reflect a patients subjective impressions regarding their healthcare (specific incidents) Incidence data, experiences with intervention programmes, starting point for reviews and patient safety activities Identifying potential problem areas or clinical issues such as organisational and management issues, teamwork skills, individual clinicians behaviours
Claims and Complaints Advantages: When I first started out in practice, I lied awake at night worrying about my patients. Now I lie awake worrying about their lawyers. (Howard Fischer, MD) Themes can be picked up from a large number of claims / complaints Looking into 1 case in depth can provide important information for learning Disadvantages: Subjective perspective Underreporting by elderly people Expensive One major event might influence the whole picture
Detecting Adverse Events (adapted from P. Hebert) Method AE / 1000 admissions 1. Incident reports 5 2. Retrospective chart reviews 30 3. Stimulated voluntary reports 30 4. Automated flags 55 5. Daily chart reviews 85 6. Automated flag and daily review 130 7. Trigger tool 400 Jha et al. (1998). Identifying Adverse Drug Events: Development of a Computer-based Monitor and Comparison with Chart Review and Stimulated Voluntary Report. Journal of the American Medical Informatics Association, 5, 305-315. O Neill et al. (1993). Physician Reporting Compared with Medical-Record Review to Identify Adverse Medical Events. Annals of Internal Medicine, 119(5), 370-376.
Prospective Methods Prospective Analysis Tools: Proactively identifying, prioritising and mitigating patient safety risk Wide range of methods such as Failure Modes Effects and Critical Analysis, Health Care Failure Mode and Effect Analysis, Hazard Analysis Use at local level, discovered information is not shared throughout organisations, not identifying combinations of events leading to incidents Probabilistic risk assessment (mixture of process analysis techniques and decision making processes) to balance and prioritise between competing goals as well as to identify combinations of events leading to potential harm
Prospective Analysis Tools Advantages: Uncovering unrecognised system level problems Boosting staff communication and morale Process of scoring failure modes regarding their probability of occurrence and severity of consequences (group census) Disadvantages: Logistic challenge Complex methodology involved Process of scoring failure modes probability of occurrence and severity of consequences (depending on setting, people)
Prospective Methods Simulations: Training of performance skills in a scenario with the same realistic problems and demands as in real life settings Learning opportunity for healthcare workers and teams about consequences of their actions Identify problem conditions / actions to improve healthcare delivery Teach and test specific technical (e.g., performing a new procedure) and non-technical (e.g., communication) skills
Simulations Advantages: Can identify additional risks that may not have been considered Control of the situation and problem area Disadvantages: Challenge to create realistic teams with real life skills mix (by experience and profession) Expensive (staff involvement, equipment, ) Transfer from simulation to real world
Concurrent Methods Direct Observations: Data collection on errors, adverse events, near misses, team performance, decision making and organisational culture Information related to the analysis of specific types of procedures (e.g., surgery) Structured observations using rating systems, e.g., surgeons nontechnical skills, to rate skills and provide feedback during a postoperative debrief (Yule et al., 2006) Range of behaviour rating tools available for individual and team assessment in acute medicine (see Flin & Mitchell, 2009)
Direct Observations Advantages: Can highlight particular factors of concern Can influence local behaviours / policies Disadvantages: Observer as a distraction / interruption Observer training Time consuming (data collection and analysis)
Concurrent Methods Checklists and Audits: Identifying active errors in the environment such as delay in care, equipment failure, information transfer, non-compliance with hospital policy, Checklists are context specific (e.g., ICU versus general ward) Safety audits can be performed using checklists, e.g., during and after morning rounds to identify unlabeled medication at the bedside, missing ID bands or inappropriate pulse oximeter alarm setting
Checklists and Audits Advantages: Public charts to display ongoing results Influence of culture (motivation to get it right) Disadvantages: Outcome measure identification (who decides what to display?) Lots of pressure to ward staff Lack of compliance
Selection of Methods Appropriate selection of methods depends on the question being addressed, the resources available, and on contextual factors
Framing Patient Safety Research Multilevel Framework of Patient Safety Research (Jackson & Flin, in prep): Organizational Structure Unit Management Worker Behaviours Outcomes Individual Differences Based the causal chain and different levels of analysis (i.e., individual, team, unit, and organisational) proposed by industrial and patient safety models Applies within an organisation even though external factors such as government and regulators responsibilities exist outside an organisation
Any Questions? Dr Jeanette Jackson (j.jackson@abdn.ac.uk) This SPSRN work is funded by