ECONOMICS AND HEALTH SYSTEMS RESEARCH SECTION EDITOR PETER G. DUNCAN Which Clinical Anesthesia Outcomes Are Both Common and Important to Avoid? The Perspective of a Panel of Expert Anesthesiologists Alex Macario, MD, MBA*, Matthew Weinger, MD, P. Truong, and M. Lee *Departments of Anesthesia and Health Research and Policy Stanford University Medical Center, Stanford; and Department of Anesthesiology, University of California San Diego and the San Diego Veterans Affairs Healthcare System, San Diego, California Anesthesia groups may need to determine which clinical anesthesia outcomes to track as part of quality improvement efforts. The goal of this study was to poll a panel of expert anesthesiologists to determine which clinical anesthesia outcomes associated with routine outpatient surgery were judged to occur frequently and to be important to avoid. Outcomes scoring highly in both scales could then be prioritized for measurement and improvement in ambulatory clinical practice. A mailed survey instrument instructed panel members to rate 33 clinical anesthesia outcomes in two scales: how frequently they believe the outcomes occur and which outcomes they expect patients find important to avoid. A feedback process (Delphi process) was used to gain consensus rankings of the outcomes for each scale. Importance and frequency scores were then weighted equally to qualitatively rank order the outcomes. Of the 72 anesthesiologists, 56 (78%) completed the questionnaire. The five items with the highest combined score were (in order): incisional pain, nausea, vomiting, preoperative anxiety, and discomfort from IV insertion. To increase quality of care, reducing the incidence and severity of these outcomes should be prioritized. Implications: Expert anesthesiologists reached a consensus on which low-morbidity clinical outcomes are common and important to the patient. The outcomes identified may be reasonable choices to be monitored as part of ambulatory anesthesia clinical quality improvement efforts. (Anesth Analg 1999;88:1085 91) Anesthesia groups need to determine which clinical anesthesia outcomes to track as part of their quality improvement efforts. Serious adverse outcomes (i.e., death) from anesthesia are now rare due to improvements in clinical care (1,2). A next step in quality improvement in anesthesia may come from addressing less morbid, yet more common, anesthesia outcomes. The term outcome may be used by clinicians to mean the results of patient care, such as an intermediate end point or adverse event. Donabedian (3) defined outcome more broadly as a change in a patient s... health status that can be attributed to antecedent healthcare. For purposes of this study, we use the term clinical anesthesia outcome to refer to adverse clinical events associated with anesthesia (e.g., postoperative nausea). This study was funded in part by a FAER/Hoechst Marion Roussel, Inc/Society for Ambulatory Anesthesia Clinical Research Starter Grant from the Foundation for Anesthesia Education and Research. Accepted for publication January 28, 1999. Address correspondence to Alex Macario MD, MBA, Department of Anesthesia (H3580), Stanford University Medical Center, Stanford, CA 94305-5640. Address e-mail to amaca@leland.stanford.edu. The clinical anesthesia outcomes deserving highest priority for monitoring and improvement are unknown. Physicians make most medical care decisions; thus, their opinions on clinical anesthesia outcomes are integral to the quality improvement process. Therefore, we were interested in identifying which anesthesia outcomes associated with routine outpatient surgery anesthesiologists believe occur frequently and which outcomes they believe that patients would most want to avoid. The importance of these outcomes could then be studied further in patients. In fact, the occurrence of minor adverse events leads to patient dissatisfaction with anesthesia (4). For example, avoiding postoperative nausea is reported to be high among patient concerns (6). 1 However, patients may not fully understand or be able to judge the care (and outcomes) that they receive. This is, in part, due to the finding that the interpersonal characteristics of care (i.e., how nice the staff is to the patient) are often indistinguishable to the patient from the clinical outcomes of care (7,8). 1 Orkin F. What do patients want? Preferences for immediate postoperative recovery [abstract]. Anesth Analg 1992;74:S225. 1999 by the International Anesthesia Research Society 0003-2999/99 Anesth Analg 1999;88:1085 91 1085
1086 ECONOMICS AND HEALTH SYSTEMS RESEARCH MACARIO ET AL. ANESTH ANALG CONSENSUS ON ANESTHESIA OUTCOMES 1999;88:1085 91 Because patients may have difficulty identifying which common, less morbid clinical anesthesia outcomes are most important to them, the goal of this study was to poll expert anesthesiologists (using a modified Delphi technique) (9) to quantify which clinical anesthesia outcomes they judged to occur frequently and to be important to avoid (from a patient s perspective). Given the large number of possible outcomes, the challenge was to find a meaningful rationale that can be used to develop a selection process. Expert panels convened around medical practice are used to attain consensus about specific clinical issues (10). By allowing anesthesiologists to identify key patient end points associated with outpatient surgery, anesthesiologists (instead of health maintenance organization administrators, for example) can define which outcomes should be measured during quality improvement processes. Once these highly rated clinical outcomes are identified, physicians can work together to reduce their incidence and severity. Methods This study was approved by Stanford University s Human Subjects Committee. A comprehensive list of clinical anesthesia outcomes was developed from a computerized literature search (MEDLINE) for 1986 1997 using the term anesthetic outcome, complications. This yielded 100 published studies (a sample of these studies includes References (11 18), which were read by AM to generate a list of clinical anesthesia outcomes associated with routine outpatient surgery. Death was included, in part, to test whether panel members correctly understood and completed the questionnaire (we expected death to be rated very important and very infrequent). The list of outcomes was reviewed to focus on outcomes that were associated with low morbidity, were applicable in the ambulatory surgery setting, and were likely to be noticed by patients. We edited (i.e., duplicative outcomes were eliminated) the list to a total of 33 items with the assistance of four senior, board-certified anesthesiologists in the Stanford anesthesia department to ensure that significant elements of care were not excluded. A pilot study of 97 surgical patients (19) was completed to ascertain which outcomes patients actually value. In the patient survey, patients were asked whether there were any other outcomes not mentioned in the survey that were important to them. All of the outcomes that were spontaneously identified by patients in this study were included in the final list of outcomes. Thus, the survey items had content validity to patients. Table 1. Demographics of Survey Respondents First mailing cycle Second mailing cycle Questionnaires mailed 72 57 Questionnaires returned 57 35 a Respondents Clinical experience (yr) 16 8.15 (3 33) Age (yr) 46 7.62 (32 64) Community practice 17 (30) Academic practice 40 (70) Values are mean sd or n (%). a Twenty-five had changes. The eligible population of anesthesiologists included members of three different 1998 ASA committees (Quality Improvement, Value-Based Anesthesia Care, and Patient Safety) with expertise in anesthesia outcomes and a list of investigators in the field of anesthesia outcomes generated from a computerized literature search (MEDLINE) for 1986 1997 with the keyword anesthesia outcome. Using a random number generator and an 80% sampling fraction of all members of the committees and all authors of the articles, 72 possible respondents were included in the panel for this study. Rather than convening the panel members in one location, we mailed a survey instrument (available from the authors) to each physician on this panel. If a panel member failed to respond after 1 mo, a second request letter and questionnaire were mailed to the panel member s home address. The instrument was organized into three parts: preoperative, intraoperative, and postoperative outcomes. The written questionnaire instructed anesthesiologists to judge (on a 5-point Likert scale) the frequency and the importance to the patient (as perceived by the anesthesiologist) of 33 clinical anesthesia outcomes associated with routine outpatient surgery. Panel physicians were given the following instructions. The following list includes many possible outcomes of anesthesia for patients having routine surgery. For patients undergoing routine anesthetics, please rate each of the items based on your perception of the incidence of these events on a scale of 1 5, with 1 corresponding to very infrequent and 5 corresponding to very common. Then, rate the outcomes you think are most important to avoid from an educated patient s perspective using a scale of 1 5, with 1 corresponding to no importance to avoiding and 5 corresponding to very important to avoid. The Delphi method was developed in the 1950s to obtain consensus among a group of experts by using a series of questionnaires interspersed with feedback (20). The theory behind Delphi is that the aggregate of a group will provide judgment that is superior to that
ANESTH ANALG ECONOMICS AND HEALTH SYSTEMS RESEARCH MACARIO ET AL. 1087 1999;88:1085 91 CONSENSUS ON ANESTHESIA OUTCOMES Table 2. Scores for Frequency of Clinical Anesthesia Outcomes Clinical anesthesia outcome Frequency score assigned by experts a 1 2 3 4 5 Incisional pain 0 2 5 25 24 Nausea 0 3 18 24 11 Propofol injection pain 1 3 15 28 9 Vomiting 1 6 22 18 9 Sore throat 2 6 20 22 6 Somnolence 3 9 16 19 9 Preoperative anxiety-inadequate anxiolysis 2 12 13 19 8 Shivering 0 8 24 22 2 Fatigue after anesthesia 3 9 17 23 4 Discomfort from IV catheter insertion 4 13 11 21 7 Hangover after anesthesia 2 10 22 21 1 Hypothermia 3 13 19 17 4 More than one stick to place IV catheter 3 22 20 9 2 Postoperative confusion 5 23 18 6 4 Waking up and gagging on endotracheal tube 7 22 17 8 2 Bruising at IV catheter site 10 24 13 6 3 Prolonged stay in recovery room 8 27 17 3 1 Pruritus 9 25 14 7 1 Urinary retention 9 24 18 4 1 Back pain after regional anesthesia 8 28 16 3 1 Anxiety during monitored anesthesia care 10 27 13 6 0 Succynlcholine myalgias 11 22 19 4 0 Discomfort from face mask for preoxygenation 17 20 15 3 1 IV catheter site infiltration 17 25 12 2 0 Residual neuromuscular weakness 22 27 7 0 0 Postdural headache 28 24 3 1 0 Revealing something personal while sedated 43 8 5 0 0 Dental injury 38 17 1 0 0 Corneal abrasion 46 9 1 0 0 Peripheral nerve injury 47 9 0 0 0 Recall without pain 49 7 0 0 0 Recall with pain 54 1 1 0 0 Death 55 1 0 0 0 Values are number of experts who assigned each score. a 1 very infrequent to 5 very common. of most individuals of the group. Features of the Delphi method include anonymity of the participants, iterative adjustment of members responses to permit them to change their opinions, and a final group response expressed, when appropriate (as in this study), as a ranking of outcomes (21). Each panel member s survey was returned, together with the tabulated group responses. Physicians were asked to review their individual responses in relation to the collective (group) data and to indicate any changes that they wanted to make. This feedback was used to gain consensus among the experts. The data used in this study are from responses that were reviewed and confirmed by the physicians who provided them. Thus, having passed the scrutiny of expert panel members, the model has face validity. Outcomes were ranked according to their mean score. The Spearman correlation coefficient was computed for pairs of outcomes that seemed to be clinically associated. To obtain a final qualitative ranking of the outcomes, and understanding the limitations of noncontinuous Likert scales, importance and frequency scores were combined. A combined score was computed for each outcome by combining (multiplying) importance and frequency mean scores (weighted equally) to rank order the outcomes. An additive (importance and frequency scores were summed) model was also used to assure that the results were robust to choices of the qualitative model. Results Of the 72 anesthesiologists, 56 (78%) completed the questionnaire (Table 1). One questionnaire was returned for having an incorrect mailing address (we were unable to contact the panel member). One member was a nonpracticing anesthesiologist who returned the questionnaire without completing it. Fourteen panel members not return the questionnaire.
1088 ECONOMICS AND HEALTH SYSTEMS RESEARCH MACARIO ET AL. ANESTH ANALG CONSENSUS ON ANESTHESIA OUTCOMES 1999;88:1085 91 Table 3. Score for Importance to Avoid Clinical Anesthesia Outcomes Clinical anesthesia outcome Importance to avoid score assigned by experts a 1 2 3 4 5 Death 0 0 0 1 54 Recall with pain 0 0 2 6 47 Peripheral nerve injury 0 1 2 9 43 Recall without pain 0 1 1 13 40 Dental injury 0 2 5 15 33 Corneal abrasion 0 2 5 19 29 Vomiting 1 1 2 27 24 Incisional pain 2 0 3 29 21 Postdural headache 0 1 5 27 22 Nausea 2 0 8 27 18 Discomfort from IV catheter insertion 2 3 13 22 15 Discomfort from face mask for preoxygenation 0 2 24 11 18 Residual neuromuscular weakness 0 5 15 22 13 Preop anxiety-inadequate anxiolysis 2 3 12 27 11 Anxiety during monitored anesthesia care 0 0 17 26 12 More than one stick to place IV catheter 4 3 11 23 13 Back pain after regional anesthesia 0 4 17 24 9 Urinary retention 0 5 18 23 9 Revealing something personal while sedated 3 5 18 19 10 Postoperative confusion 1 8 17 20 9 Succynlcholine myalgias 1 10 18 25 1 IV catheter site infiltration 2 4 30 12 7 Shivering 0 9 21 22 3 Bruising at IV catheter site 2 12 21 17 3 Hangover after anesthesia 1 13 18 21 2 Waking up and gagging on endotracheal tube 7 9 17 15 7 Hypothermia 2 10 22 19 2 Sore throat 0 11 25 17 2 Fatigue after anesthesia 0 16 21 15 3 Propofol injection pain 0 16 27 11 1 Pruritus 1 17 25 10 2 Prolonged stay in recovery room 5 17 15 16 2 Somnolence 6 20 11 17 1 Values are the number of experts who assigned each score. One panel member declined to answer the importance section of questionnaire; therefore, there were 55 respondents. a 1 no importance to avoid to 5 very important to avoid. Panelists who did not respond were roughly equivalent in available variables to those who responded (e.g., gender, type of practice, geographical location). Of the 56 respondents, 50 reported that 46% (range 25% 100%) of their practice was in ambulatory anesthesia. The 56 respondents were given the mean results of their colleagues responses via a secondary mailing. In response to the second mailing of the survey, 25 of the 56 respondents made an average of 9.1 7 changes (range 0 30) to their original responses. There was no obvious pattern to the changes. The 10 highest ranked items did not change between the first and second mailing. Therefore, after analysis of the second mailing data, we did not send out the survey a third time. Incisional pain received the highest frequency score. Death received the highest score for importance to avoid (Tables 2 4). A combined outcome score was obtained by multiplying the mean importance and frequency scores to rank order the outcomes (Fig. 1). The larger the product of the two mean scores, the further the distance from the intercept of the two axes. The mean scores were also added to determine whether the resultant scores and ranks differed from the multiplicative model. The top 15 items, weighting equally for importance and frequency, did not change under the additive model relative to the multiplicative model. The top five items were (in order): incisional pain, nausea, vomiting, preoperative anxiety, and discomfort from IV catheter insertion (Table 5). The results showed internal consistency. Of 56 respondents, 55 ranked death as 1 (very infrequent) on the frequency scale. Of 55 respondents who completed the importance questions, 54 ranked death as 5 (very important) on importance to avoid from the patient s
ANESTH ANALG ECONOMICS AND HEALTH SYSTEMS RESEARCH MACARIO ET AL. 1089 1999;88:1085 91 CONSENSUS ON ANESTHESIA OUTCOMES Table 4. Experts Final Ranking of Clinical Anesthesia Outcomes Clinical anesthesia outcome Frequency Importance to avoid Incisional pain 1 9 Nausea 2 10 Propofol injection pain 3 30 Vomiting 4 7 Sore throat 5 25 Somnolence 6 33 Preop anxiety-inadequate anxiolysis 7 15 Shivering 8 21 Fatigue after anesthesia 9 29 Discomfort from IV catheter 10 12 insertion Hangover after anesthesia 11 24 Hypothermia 12 26 More than one stick to place IV 13 16 catheter Postoperative confusion 14 20 Waking up and gagging on 15 28 endotracheal tube Bruising at IV catheter site 16 27 Prolonged stay in recoveryroom 17 32 Pruritus 18 31 Urinary retention 19 18 Back pain after regional anesthesia 20 17 Anxiety during monitored 21 11 anesthesia care Succynlcholine myalgias 22 23 Discomfort from face mask for 23 13 preoxygenation IV catheter site infiltration 24 22 Residual neuromuscular weakness 25 14 Postdural headache 26 8 Revealing something personal 27 19 while sedated Dental injury 28 5 Corneal abrasion 29 6 Peripheral nerve injury 30 3 Recall without pain 31 4 Recall with pain 32 2 Death 33 1 perspective. There was positive correlation among pairs of clinically related outcomes expected to be associated (Table 6). Discussion Healthcare organizations may be ready to define and monitor some version of clinical quality. However, health plans, employers, accreditation agencies, and federal regulators are poorly positioned to define clinical quality. Many of these groups believe that they have been forced to ensure accountability for clinical quality because providers have failed to provide data that demonstrate the quality of their practice. Anesthesia groups want to know which clinical anesthesia outcomes they should track. In this study, we used a process (expert panel consensus) to identify (frequent and important to avoid) clinical anesthesia outcomes that deserve priority for monitoring and quality improvement. Although there are numerous outcomes to monitor, the results of this anesthesiology expert panel, which used a modified Delphi process to gain consensus, suggest that clinicians judge the top five clinical anesthesia outcomes associated with ambulatory anesthesia to be (in order) incisional pain, nausea, vomiting, preoperative anxiety, and discomfort from insertion of the IV catheter (when weighted equally for importance and frequency). Once these outcomes are tracked over time, unnecessary variation in their occurrence can be detected, and efforts to systematically improve these clinical end points can be made (22). However, before assessing whether there are differences in these outcomes among clinicians and institutions, issues of differing case mixes and imprecise definitions must be addressed. Patient satisfaction as measured in most anesthesia surveys may not be a fine enough measure of quality of care because patients have different expectations about the anesthesia experience, the validity of surveys has not been established, and nonmedical factors may affect a patient s satisfaction with care (23). The unstandardized, simple ratings of patient satisfaction used in most anesthesia surveys are inadequate to address the complexity of measuring satisfaction (24). For example, in the setting of perceived risk (anesthesia), satisfaction ratings may be dominated by a sense of relief. Monitoring clinical anesthesia outcomes, instead of measuring patient satisfaction, may be a more useful indicator of quality. In this study, we focused on clinical anesthesia outcomes associated with routine outpatient surgery, rather than other aspects of care, such as customer service (e.g., the effect of care or how nice providers are to patients) or the timeliness of care (does surgery start on time?). In fact, these other aspects of care may be more noticeable and important to patient satisfaction than are the clinical outcomes about which physicians may be concerned. For example, one study suggested that friendliness of the operating room staff is the primary determinant of patient satisfaction with outpatient surgery (25). In a study of how surgeons prioritize different aspects of anesthesia service, we found that surgeons (another important customer of the anesthesia service) rank timeliness of care (e.g., time that the first case of the day starts) as a key item in improvement in the quality and function of the anesthesia service (26). Like surgeons, patients may perceive nonclinical issues to be important in evaluating anesthesia care. The outcomes deemed important and frequent in this study are consistent with the findings of other investigators (4 6,11 19). Although major morbidity is uncommon after ambulatory surgery, symptom distress and reduced function are common 7 days postoperatively (27). Investigators are continuing to determine how the results of this study compare with
1090 ECONOMICS AND HEALTH SYSTEMS RESEARCH MACARIO ET AL. ANESTH ANALG CONSENSUS ON ANESTHESIA OUTCOMES 1999;88:1085 91 Figure 1. The 33 clinical anesthesia outcomes plotted according to their mean scores for frequency and importance to avoid. The items in the upper right quadrant may deserve highest priority for clinical improvement. The x and y axes have different scales to aid in the visual representation of the results by spreading the outcomes apart on the graph. patients actual perceptions regarding which clinical anesthesia outcomes are important to avoid (19). In a pilot study of 97 patients, we found that although there was a substantial variability in patient preferences for postoperative outcomes, patients ranked vomiting, gagging on the endotracheal tube, and pain as their three least desirable outcomes. There are several potential limitations to the study. Of the respondents, 70% were in academic practice, which may bias the results. However, subgroup analysis (academic versus community practice) showed no difference in rankings of outcomes. With any surveybased study, the results may be affected by a variety of cognitive biases or response bias (28). The possibility of response bias is somewhat diminished by the response rate in this study of 78%. In addition, there may be an interaction (i.e., scales are not independent) between importance and frequency scales, in that clinical outcomes judged to be frequent may be judged to be important to avoid because they occur frequently. Furthermore, the consensus of a panel does not mean that the correct answer has been found, and it is not a substitute for rigorous scientific review (29). Panel members were asked to list other outcomes that should be included that were not on the questionnaire. No one clinical outcome was suggested by more than one respondent. Thus, it seems that our study did not miss any important outcomes. The generalizability of these consensus data to all North American anesthesia providers is supported because the panel population was comprised of practitioners from 26 American states and Canadian provinces. However, it is possible that opinion in other parts of the world could differ considerably. Conclusion In this study, we identified clinical anesthesia outcomes associated with routine surgery that are common and important to avoid, at least from the physician s perspective. It is important to validate these
ANESTH ANALG ECONOMICS AND HEALTH SYSTEMS RESEARCH MACARIO ET AL. 1091 1999;88:1085 91 CONSENSUS ON ANESTHESIA OUTCOMES Table 5. Final Rankings Using Combined Scores Combined Clinical anesthesia outcome score Incisional pain 18.00 Nausea 15.35 Vomiting 15.08 Preop anxiety-inadequate anxiolysis 12.62 Discomfort from IV catheter insertion 12.41 Shivering 11.11 Propofol injection pain 10.99 Sore throat 10.91 More than one stick to place IV catheter 10.28 Fatigue after anesthesia 10.16 Hangover after anesthesia 10.06 Hypothermia 9.83 Somnolence 9.38 Postoperative confusion 9.34 Anxiety during monitored anesthesia care 8.87 Urinary retention 8.61 Back pain after regional anesthesia 8.53 Discomfort from face mask for preoxygenation 8.11 Waking up and gagging on endotracheal tube 7.99 Bruising at IV catheter site 7.59 Succynlcholine myalgias 7.48 Pruritus 6.96 Postdural headache 6.79 Prolonged stay in recovery room 6.67 IV catheter site infiltration 6.60 Residual neuromuscular weakness 6.55 Dental injury 5.94 Peripheral nerve injury 5.47 Recall without pain 5.26 Corneal abrasion 5.22 Recall with pain 5.08 Death 5.05 Revealing something personal while sedated 4.64 Table 6. Spearmann Correlation Coefficients Among Pairs of Outcomes Frequency Importance P value Nausea/vomiting 0.72 0.86 0.001 Shivering/hypothermia 0.40 0.49 0.001 IV bruising/iv infiltration 0.38 0.28 0.01 findings with comparable data obtained from actual patients. Preliminary data from our studies of how these key clinical outcomes are perceived by patients suggest significant variability among patients as to what they believe is most important to avoid. Targeting the improvement of these outcomes prospectively, via physician-led scientific data collection, analysis, and feedback, is likely to improve patient care. References 1. Forrest J, Rehder K, Cahalan M, et al. Multicenter study of general anesthesia. III. Predictors of severe perioperative adverse outcomes. Anesthesiology 1992;76:3 15. 2. Cohen M, Duncan D, Pope W, Vourch G. A survey of 112,000 anesthetics at one teaching hospital (1975 1983). Can J Anesth 1987;33:22 31. 3. Donabedian A. Evaluating the quality of medical care. Millbank Q 1966;44:166 206. 4. Tong D, Chung F, Wong D. Predictive factors in global and anesthesia satisfaction in ambulatory surgical patients. Anesthesiology 1997;87:856 64. 5. Deleted in proof. 6. Fisher D. The big little problem of postoperative nausea and vomiting: do we know the answer yet? Anesthesiology 1997;87: 1271 3. 7. Donabedian A. The quality of care: how can it be assessed? In: Graham N, ed. Quality assurance in hospitals. Rockville, MD: Aspen Publishers, 1990:14 29. 8. Sira Z. Effective and instrumental components in the physicianclient relationship. J Health Soc Behav 1980;21:170 80. 9. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ 1995;311:376 80. 10. Pill J. The Delphi method: substance, context, a critique and an annotated bibliography. Socioecon Planning Sci 1971;5:57 71. 11. Chye EPY, Young IG, Osborne GA, et al. Outcomes after sameday oral surgery. J Oral Maxillofac Surg 1993;51:846 9. 12. King B. Patient satisfaction survey: day surgery unit. Aust Clin Rev 1989;9:127 9. 13. Burrow B. The patient s view of anaesthesia in an Australian teaching hospital. Anaesth Intensive Care 1982;10:20 4. 14. Philip B. Patients assessment of ambulatory anesthesia and surgery. J Clin Anesth 1992;4:355 8. 15. Keep P, Jenkins J. From the other end of the needle: the patient s experience of routine anaesthesia. Anaesthesia 1978;33:830 2. 16. Osborne GA, Rudkin GE. Outcome after day-care surgery in a major teaching hospital. Anaesth Intensive Care 1993;21:822 7. 17. Moerman N, van Dam F, Oostino J. Recollections of general anaesthesia: a survey of anesthesiological practice. Acta Anaesthesiol Scand 1992;36:767 71. 18. Dodds CP, Harding MI, More D. Anaesthesia in an Australian private hospital: the consumer s view. Anaesth Intensive Care 1985;13:325 9. 19. Macario A, Weinger M. Which clinical anesthesia outcomes do patients find most undesirable? [abstract]. Anesthesiology 1998; 89:A1330. 20. Dalkey N, Helmer Q. An experimental application of the Delphi method to the use of experts. Manage Sci 1963;9:458 67. 21. Rowe G, Wright G, Bolger F. Delphi: a reevaluation of research and theory. Technol Forecasting Social Change 1991;39:235 51. 22. Wennberg J, Gittelshon A. Small area variations in health care delivery: a population based health information system can guide planning and regulatory decision making. Science 1973; 182:1102 8. 23. Donabedian A. The definition of quality and approaches to its measurement. Ann Arbor, MI: Health Administration, 1980. 24. Fung D, Cohen M. Measuring patient satisfaction with anesthesia care: a review of current methodology. Anesth Analg 1998; 87:1089 98. 25. Tarazi E, Philip B. Friendliness of OR staff is top determinant of patient satisfaction with outpatient surgery. Am J Anesth 1998; 4:154 7. 26. Vitez T, Macario A. Setting performance standards for an anesthesia department. J Clin Anesth 1998;10:166 75. 27. Swann B, Maislin G, Traber K. Symptom distress and functional status changes during the first seven days after ambulatory surgery. Anesth Analg 1998;86:739 45. 28. Tversky A, Kahneman D. Judgment under uncertainty: heuristics and biases. Science 1974;185:1124 31. 29. Scott E, Black N. Appropriateness of cholycystectomy: a consensus panel approach. Gut 1991;9:1066 70.