The Safety Attitudes Questionnaire (SAQ) 1 Guidelines for Administration. Sexton, J.B., Thomas, E.J. and Grillo, S.P.

Transcription

1 The University of Texas Safety Attitudes Questionnaire 2/03 Page 1 The Safety Attitudes Questionnaire (SAQ) 1 Guidelines for Administration Sexton, J.B., Thomas, E.J. and Grillo, S.P. This technical paper is to serve as a users manual for the Safety Attitudes Questionnaire. The University of Texas Center of Excellence for Patient Safety Research and Practice January 11, 2003 Technical Report The University of Texas Center of Excellence for Patient Safety Research and Practice (AHRQ grant # 1PO1HS and U18HS ) 1 The development of this instrument was supported grants from the Agency for Healthcare Research and Quality, Eric J. Thomas, Principal Investigator, and the Gottlieb Daimler and Karl Benz Foundation, Robert L. Helmreich, Principal Investigator.

2 The University of Texas Safety Attitudes Questionnaire 2/03 Page 2 Administration Process Summary o o o o o This users manual is a guide to hospital administrations of the SAQ family of surveys. Each hospital conducting the survey needs a primary point of contact responsible for survey logistics, and one back-up person. High and accurate response rates are key to conducting the survey and interpreting results: o response rate numerator is the number of surveys returned o response rate denominator is the number of surveys that made it into the hands of respondents Surveys are preprinted and numbered sequentially. These serial numbers are critical for response rates and for administration logistics. o Ranges of serial numbers are assigned to clinical areas, and to job categories within a clinical area o Do not use names, only track data by job category of provider Hospital to be surveyed must calculate the number of questionnaires needed for each clinical area by estimating the number of questionnaires for each job category within a clinical area. o The number of questionnaires estimated for each job category is the preliminary denominator in the response rate for that job category, and will be updated at least once after the data are collected. o Hospital submits preliminary response rate denominators for all clinical areas and in return receives the appropriate number of surveys (along with some cushion. ) o Before the completed questionnaires are returned to the Center of Excellence for processing, the response rate denominators must be verified within each job category (within each clinical area) to reflect the number of individuals who, for whatever reason, did not receive a questionnaire e.g., vacation, maternity leave, sick day, etc., in an effort to make the response rates as accurate as possible. Note that these adjustments to the denominator often make a significant difference in overall response rates, as well as making them more accurate.

3 The University of Texas Safety Attitudes Questionnaire 2/03 Page 3 Background In the wake of recent reports from the Institute of Medicine (1) and the National Health Service (2) there has been a tsunami of interest in patient safety research and improvement. Experience in other safety critical industries suggests that measuring attitudes about teamwork and the overall context of work is an important step in improving safety. (3,4) In healthcare, quality of care must also be investigated within the framework of systems and contextual factors that provide the environments in which errors and adverse events occur. (5,6,7,8) For example, Charles Vincent and his colleagues describe several factors that influence clinical practice: organizational factors such as safety climate and morale, work environment factors such as staffing levels and managerial support, team factors such as teamwork and supervision, and staff factors such as overconfidence and being overly self assured. (8) These factors are believed to influence the safe delivery of care, and to date, caregiver attitudes about these key factors remain largely unexplored. (9,10) Influential organizations in healthcare agree with the need to examine caregiver attitudes about these issues. Research agencies (Agency for Healthcare Research and Quality, National Patient Safety Foundation, National Health Service), regulators (Joint Commission on Accreditation of Healthcare Organizations [JCAHO]), managed care organizations (Kaiser Permanente), professional organizations (e.g., American Hospital Association) and quality improvement experts (e.g., Institute for Healthcare Improvement) are encouraging the measurement of caregiver attitudes about the context of work. These attitudes, taken together, are generally referred to as the climate or culture in a given setting. Targeting, measuring and understanding caregiver attitudes about safety is critical if we are to translate the contemporary national will to improve safety into actionable local improvement. Despite considerable interest, there is not a commonly used metric to measure these attitudes, nor is there benchmarking data for evaluating results. We set out to fill this need for a common metric by creating a psychometrically sound survey instrument that assesses caregiver attitudes relevant to patient safety. When individual attitudes are aggregated by clinical area, this tool provides a snapshot of the climate in a given clinical area (i.e., one attitude is an opinion, but the aggregate attitudes of everyone in a work unit is climate). Survey Etiology The Safety Attitudes Questionnaire (SAQ) is a refinement of the Intensive Care Unit Management Attitudes Questionnaire (ICUMAQ), (11,12) which was derived from a questionnaire widely used in commercial aviation, the Flight Management Attitudes Questionnaire (FMAQ). (13,14) The SAQ differs from other medical attitudinal surveys (15) in that it maintains continuity with its predecessor (the FMAQ) - a traditional human factors survey with a 20 year history. (16,17) Preserving continuity allows for comparisons between professions, and assists with the search for universal human factors issues. There is a 25% overlap in item content between the SAQ and the FMAQ because the same items demonstrated utility in medical settings in terms of the subject covered and factor loadings. The new (non-overlapping) SAQ items were generated by focus groups of health-care providers, review of the literature, and round-table discussions with subject-matter experts. This generated a pool of over 100 new items covering four themes: safety climate, teamwork climate, stress recognition, and organizational climate. These targeted themes were based on prior research in the aviation industry and in medicine. (11,12, 15, 18,19, 20, 21) Items were evaluated through pilot testing and exploratory factor analyses. This phase of survey development yielded 6 factoranalytically derived attitudinal domains containing 40 items from the survey. Three of the targeted themes, safety climate, teamwork climate, and stress recognition, emerged as factors. The fourth

4 The University of Texas Safety Attitudes Questionnaire 2/03 Page 4 targeted theme, organizational climate, consistently emerged as three distinct but related factors, perceptions of management, working conditions, and job satisfaction. An additional 20 items were retained as individually diagnostic items because they were deemed interesting and valuable to the unit managers and senior hospital leadership to whom we reported the results of our pilot studies. The SAQ has been adapted for use in intensive care units (ICU), operating rooms (OR), general inpatient settings (Medical Ward, Surgical Ward, etc.), and Ambulatory Clinics. Pilot research is being conducted on additional SAQ versions (Pharmacies, Emergency Departments, and Labor & Delivery units), and was not available at the time of this writing. For each version of the SAQ, item content is the same, with minor modifications to reflect the clinical area. For example, In this ICU, it is difficult to discuss mistakes, vs. In the ORs here, it is difficult to discuss mistakes. The SAQ elicits caregiver attitudes through the 6 factor analytically derived scales: teamwork climate; job satisfaction; perceptions of management; safety climate; working conditions; and stress recognition (Figure 1). Factor: Definition Teamwork climate: perceived quality of collaboration between personnel Job satisfaction: positivity about the work experience Perceptions of management: approval of managerial action Safety climate: perceptions of a strong and proactive organizational commitment to safety Working conditions: perceived quality of the ICU work environment and logistical support (staffing, equipment etc.) Example items Disagreements in the ICU are appropriately resolved (i.e., what is best for the patient) Our doctors and nurses work together as a well coordinated team I like my job This hospital is a good place to work Hospital management supports my daily efforts in the ICU Hospital management is doing a good job I would feel perfectly safe being treated in this ICU ICU personnel frequently disregard rules or guidelines developed for our ICU Our levels of staffing are sufficient to handle the number of patients The ICU equipment in our hospital is adequate Stress recognition: acknowledgement of how performance is influenced by stressors Figure 1. SAQ factor definitions and example items I am less effective at work when fatigued When my workload becomes excessive, my performance is impaired The SAQ is a single page (double sided) questionnaire with 60 items and demographics information (age, sex, experience, and nationality). Each of the 60 items is answered using a fivepoint Likert scale (Disagree Strongly, Disagree Slightly, Neutral, Agree Slightly, Agree Strongly). Some items are negatively worded. There is also an open-ended section for comments: What are your top three recommendations for improving patient safety in this clinical area. Each version of the SAQ has a Collaboration and Communication section, where respondents are asked to indicate the quality of collaboration and communication they have experienced with each of the types of providers in their clinical area (e.g., Staff Surgeons, Surgical Residents, Staff

5 The University of Texas Safety Attitudes Questionnaire 2/03 Page 5 Anesthesiologists, OR Nurses, etc.) using a five-point Likert scale (Very Low, Low, Adequate, High, Very High). Administration Guidelines Administration & Logistics of the SAQ Once you have established the specific areas you wish to survey and which comparisons you would like to generate (e.g., by clinical area, by institution, by department, by job category, etc.), the critical task is to get a high response rate from those areas. Survey results are meaningless without representative response rates. In our experience, high response result in findings that are easy to interpret, enthusiastically received during feedback sessions, and hard to ignore (whereas the opposite is true for low-response rates). The 10 to 15 minutes required to complete the survey (nurses typically take longer than physicians because they provide more open-ended comments) should be taken into account when deciding an appropriate administration methodology. Typical administration methods include meeting administrations, hand-deliveries, and mailing administrations. By far, the best response rates are garnered through meeting administrations (e.g., during a department meeting, staff meeting, in-service, etc.), whereby 15 minutes of the meeting are allotted for respondents to complete the survey. We administer the SAQ with a cover letter, a sharpened #2 pencil (with an eraser), and a return envelope. We strongly recommend using full size envelope to prevent the need for folding the survey. In our experience, the pencil, envelope, and full-color scannable version of the survey each contribute to higher overall response rates. Meeting administrations will typically generate a 90% (or higher) response rate. After meeting administrations, the second most productive method of administration is hand-delivery of the survey (typically a 60-70% response rate). Hand deliveries put a face to the survey and allow respondents to ask a question immediately if they would like additional information. The most important factor in hand deliveries is who hands over the surveys, i.e., physicians should be given a survey by another physician not, e.g., a departmental secretary. Hand delivery requires the use of a self-addressed, stamped envelope or a secure lock-box with a slit for the return envelope. Lastly, mailing administrations (e.g., through departmental mail, postal mailings to home addresses, etc.) typically generate the lowest response rates (35-45%). Moreover, these administrations usually require multiple mailings, 8 weeks of follow-up, and genuine cajoling on the part of project managers to get a response rate high enough to be interpretable. In an effort to maintain methodological rigor in the survey process, we highly recommend response rates of 60% or higher. In sum, meeting administrations and hand deliveries require little follow-up and maximize return. If at all possible, use a return envelope large enough to contain the survey without folding it. If this is not possible use a 6 and a half by 9 and a halfinch envelope, and fold the survey only once from top to bottom, creating a single horizontal crease. In other words, if the top of the survey were North, think of this crease as being the West-East axis. Please ensure that the front side of the survey is showing when folded. This process minimizes problems with the survey scanner, and allows us to return your data more quickly and completely. Note to Survey Administrators Maintaining respondent confidentiality is an absolute priority with this survey research, as it impacts upon response rates and upon the quality (honesty) of the responses that respondents provide. To ensure confidentiality, administer the survey with a return envelope so that the person

6 The University of Texas Safety Attitudes Questionnaire 2/03 Page 6 collecting completed surveys can not see the completed survey. Store all completed surveys in a locked box or locked file cabinet until sent to the University of Texas Survey Processing Facility for scanning, data entry, and analysis. Another method that has been applied with great success is to have someone from your mailroom present during a meeting administration. In this scenario, the mailroom representative collects the return envelopes in an addressed box, which is sealed and walked to the mailroom immediately after the meeting. Response Rates Although it may seem counter-intuitive, a good rule of thumb for response rates within personnel categories is the following: the smaller the number of personnel in that job category, the higher the response rate you will need to get a representative sample; the larger the number of personnel in a category, the lower the response rate will need to be for representativeness. Good response rates for climate research are generally 65% to 85%, and these are typical response rates using the SAQ. Criteria for Inclusion as a respondent The goal is to survey individuals who influence or are influenced by the culture in their clinical area. We typically require that respondents have worked in the clinical area for at least 4 weeks before they are sufficiently exposed to the culture of their unit to allow them to adequately respond to the survey. To be considered for inclusion in this survey, respondents must work approximately 20 hours per week in/for the clinical area. Physicians who admit 2 or more patients per month are also eligible for inclusion. Tracking Data Each individual survey has a unique number printed on the front. This unique number simplifies the logistics of administration and database management. For example, if you ask for 400 Operating Room surveys, you may get unique numbers through In this way, we can keep track of which surveys went to which facilities. Response rates are important to keep track of by facility and by job category within facility (and if applicable by unit within a facility). You can calculate accurate response rates without identifying respondents by using the unique number on each survey. For example, if you are administering the survey to OR personnel, you could get an up-to-date roster of Surgery, Anesthesia and Nursing personnel delete the names, and use number of personnel within each job category to establish your response rate denominator. Rosters provided from a central source such as the human resources department or payroll may not be accurate. We recommend the final list of eligible respondents be determined by a unit manager who refers to a list from human resources. Assign a range of unique numbers to go with each job category (see Appendix B as an example). To account for the possible need for additional surveys after the survey administration process has begun, we highly recommend adding extra surveys in each job category (i.e., to provide a cushion ) in order to maintain sequential numbering. The number of surveys returned from that range is your response rate numerator. There is no need to identify the respondents by name. Please do not do this, and under no circumstances are you to share that information or other respondent anonymity compromising information with us. The reason we focus on job categories within a clinical area is because some people fill out the entire survey, but we don t know if they are a physician or a nurse which renders their data useless in a number of analyses. By tracking this information initially (i.e., what ranges of numbers went to a job category), you can maximize the utility of the data and much more accurately assess your response rates. To

7 The University of Texas Safety Attitudes Questionnaire 2/03 Page 7 assist you with accurate response rate tracking by clinical area and by job categories within a clinical area, we have created a spreadsheet in excel which you can download from our website ( Also, once you have determined the total number of surveys you will need, we suggest that you add an additional 20% to cover the need for additional surveys (e.g., if surveys are lost, left at home, damaged, etc.). The process of identifying the number of eligible respondents in a facility or unit and assigning serial number ranges is very important. We suggest that this job be coordinated by one project manager who is steadfast in paying attention to details. Numbers of respondents should be double checked, as should the serial numbers assigned to each unit. Typographical errors often occur when recording these long serial numbers. If you are administering surveys to multiple facilities, or multiple clinical areas, it is very important to assign a range of unique numbers to each facility/clinical area, as this information is critical during analysis. For example, if you have 300 Ambulatory Surveys (unique numbers ) and 3 facilities, you could assign: Facility A = Facility B = Facility C = Before you administer the survey, please make note of the date and administration method, as well as the following information, so that we can take it into consideration when writing up the results for you. 1) Hospital Type o Tertiary Care (may be teaching or non) o Academic / Teaching o Community / Non-teaching o Military / Veterans 2) Setting o Urban o Sub-urban o Rural 3) Size o <100 beds o beds o beds o beds o > 1000 beds 4) Structure o Independent o Part of multi-hospital system o Government owned o For-profit o Non-for-profit 5) Inpatient, Non-Hospital o Nursing Home / Long Term Care o Assisted Living o Rehabilitation o Sub Acute 6) Outpatient o Clinic o Office private practitioners o Diagnostic testing center o Therapy center PT, OT, etc o Psychiatric / Mental Health o Geriatric / Adult Day Care Completed Surveys & Results Once all of the surveys have been collected, the tracking spreadsheet needs to be updated to account for the actual number of surveys that were administered (i.e., made it into the hands of a respondent) and the actual number of surveys that were returned by job category, by department, and for the overall site. This spreadsheet will be returned electronically to the Center of Excellence project manager.

8 The University of Texas Safety Attitudes Questionnaire 2/03 Page 8 Please send the Center of Excellence Survey Processing Facility your SAQs only after you have fully completed data collection, and then send them in one single batch. No staples, paperclips, or post-it notes can be affixed to the survey itself. If someone provides an additional sheet with comments (less than 1% do this), please clearly write the relevant unique number from the survey onto the sheet of comments and send the comments with the batch of surveys. Important: NO STAPLES, NO PAPERCLIPS. We will provide a summary report of results for your survey administration. This report can take 6 weeks or more to prepare, and is provided in Microsoft PowerPoint. We have data from over 400 medical organizations for comparison purposes. No individual-level results will be provided, only aggregate-level results from groups of 5 or more respondents will be reported (i.e., grouped by job category or clinical area). The report of results can include: response rate information, the overall results for each of the six factors assessed by the survey, results by provider type, results by clinical area, results of each survey item broken down by content area, and comparisons between your institution and other institutions in our archives. Frequently Asked Questions Q: Can we do this online? A: Paper and pencil, for the time being, is better suited to collecting data on climate. We have tried online administrations using the SAQ without success. There are often issues of access that organizations don't realize until it is too late. For instance, we tried a web-based administration at a site that purported to do everything online, and they received a 17% response rate. With the paper version we get 65-85%. The convenience of web-based administrations is not lost on us, and we are keen to migrate to the web within the next few years. For the time being, access to computers and trust of online confidentiality are the two main roadblocks to this process. In particular, junior personnel (e.g., residents) do not feel comfortable saying something negative on the survey and then hitting the send, button. Paper and pencil just seems less threatening. One advantages of paper & pencil versions is that they can be administered by anyone in any venue, and all you need is the survey, a pencil, a return envelope, and 15 minutes (no computer and no internet access is required). Q: Do I need Institutional Review Board (Human Subjects Committee) approval to collect these data? A: This depends. If you are collecting survey data as part of internal audit for quality improvement purposes, and do not intend to publish the data, then you do not need human subjects approval. However, if you intend to publish the results or share them outside of your organization, then you do need human subjects approval. In which case we recommend the following language be used in seeking approval: The SAQ is a voluntary, confidential survey. As a voluntary survey, respondents can choose to leave blank any part of the survey or they may elect to not respond at all. Results are only reported for groups of 5 or fewer respondents to protect the confidentiality of individual respondents. Administration technique: respondents will receive a survey, a pencil, and a return envelope. Respondents deliver completed surveys in the return envelope either to lockboxes, or to a shipping box for immediate batched mailing. No patient information is collected.

9 The University of Texas Safety Attitudes Questionnaire 2/03 Page 9 References 1. Kohn LT, Corrigan JM, Donaldson MS (Eds) : To Err is Human. Building a Safer Health System. Washington DC: National Academy Press, Department of Health. Organisation with a memory. The Stationary Office, London; Reason JT. Managing the risks of organizational accidents. Ashgate Publishing, Aldershot; Maurino DE, Reason J, Johnston N, Lee RB. Beyond Aviation Human Factors. Ashgate Publishing, Aldershot; Leape LL. Error in medicine. Journal of the American Medical Association 1994;272: Cook RI, Woods DD. Operating at the sharp end: the complexity of human error. In M.S. Bogner (Ed.), Human error in medicine (pp ). Hillside, NJ: Lawrence Erlbaum and Associates; Reason JT. Understanding adverse events: Human factors. In C.A. Vincent (Ed.), Clinical Risk Management (pp ). British Medical Journal Publications, London; Vincent CA, Taylor-Adams S, Stanhope N. Framework for analyzing risk and safety in clinical medicine. British Medical Journal; 316: ; Pronovost PJ, Morlock L, Dorman T. Creating Safe Systems of ICU Care. In J. L. Vincent (Ed) Year Book of Intensive Care and Emergency Medicine. (pp ). Springer Verlag; Vella K, Goldfrad C, Rowan K, Bion J, Black N. Use of consensus development to establish national research priorities in critical care. British Medical Journal, 320; ; Sexton BJ, Thomas EJ, Helmreich RL. Error, stress, and teamwork in medicine and aviation: cross sectional surveys. BMJ 2000; 320: Thomas EJ, Sexton JB, Helmreich RL. Discrepant attitudes about teamwork among critical care nurses and physicians. Crit Care Med 2003;31(3) 13. Merritt AC. National culture and work attitudes in commercial aviation: A cross-cultural investigation. Unpublished doctoral dissertation. The University of Texas at Austin; Helmreich RL, Merritt AC, Sherman PJ, Gregorich SE, Wiener EL. The Flight Management Attitudes Questionnaire (FMAQ). NASA/UT/FAA Technical Report Austin, TX: The University of Texas; Shortell, S.M., Denise, M., Rouseau, D.M., Gillies, R.R., Devers, K.J. & Simons, T.L. (1991). Organizational assessment in intensive care units (ICUs): Construct Development, Reliability, and Validity of the ICU Nurse-Physician Questionnaire. Medical Care. 29(8): Helmreich RL. Cockpit management attitudes. Human Factors. 1984; 26, Gregorich SE, Helmreich RL, Wilhelm JA. The structure of Cockpit Management Attitudes. Journal of Applied Psychology, 1990;75(6), Helmreich RL, Merritt AC. Culture at work in aviation and medicine: National, organizational, and professional influences. Aldershot, U.K.: Ashgate; Sexton JB, Klinect JR. The link between safety attitudes and observed performance in flight operations. In Proceedings of the Eleventh International Symposium on Aviation Psychology. Columbus, OH: The Ohio State University; Sexton, JB. A Matter of life or death: Social psychological and organizational factors related to patient outcomes in the intensive care unit. Unpublished doctoral dissertation. The University of Texas at Austin; Sexton, J B, Helmreich, R L, Wilhelm, JA, Merritt, A C, & Klinect, J R (2001). The Flight Management Attitudes Safety Survey (FMASS). The University of Texas Human Factors Research Project Technical Report Austin, TX: The University of Texas.

10 The University of Texas Safety Attitudes Questionnaire 2/03 Page 10

11 The University of Texas Safety Attitudes Questionnaire 2/03 Page 11 Appendix A: The Safety Attitudes Questionnaire (ICUVersion)

12 The University of Texas Safety Attitudes Questionnaire 2/03 Page 12

13 The University of Texas Safety Attitudes Questionnaire 2/03 Page 13 Appendix B: Example ICU Response Rate Tracking Form Available on the web: Serial # Tracking Sheet for: Instructions: 1) assign serial number ranges to each job category (blue cells) 2) insert numbers for response rate calculaions (number administered, any additional, and number returned) into yellow cells Serial # Range assigned to your unit From To Job Category # administer ed additional TOTAL Serial From Serial To # Returned Respons Rate for each Job Category Charge Nurse Nurse Manager/Head Nurse Critical Care RN Critical Care LVN/LPN Critical Care Attending/Intensivist Critical Care Fellow/Resident 0 0 Attending/Staff Physician (Non-Critical Care) Fellow/Resident (Non-Critical Care) 0 0 Pharmacist Respiratory Therapist Physician Assistant/Nurse Practitioner 0 0 Nursing Aide/Assistant Other (specify): Totals Administered 141 Returned 107 Grand Total Returned 107 Grand Total Administered Overall Response Rate 82