ORIGINAL ARTICLE Evaluating Popular Media and Internet-Based Hospital Quality Ratings for Cancer Surgery Nicholas H. Osborne, MD; Amir A. Ghaferi, MD; Lauren H. Nicholas, PhD; Justin B. Dimick; MD MPH Objective: We sought to determine whether best hospitals, as defined by the US News & World Report or, have lower mortality rates than all other American hospitals for cancer surgery. Design: Retrospective cross-sectional study. Setting: Medicare database (2005-2006). Patients: All patients with a diagnosis of malignancy who underwent pancreatectomy, esophagectomy, and colectomy (n=82 724). Main Outcomes Measures: Risk-adjusted mortality rates at best hospitals according to both the US News & World Report and, was compared with all other US hospitals, adjusting for differences in patient factors and surgical acuity. Risk-adjusted mortality rates between best hospitals and all other hospitals was compared after controlling for differences in hospital volume. Results: Risk-adjusted mortality was significantly lower in US News & World Report best hospitals for all 3 procedures: pancreatectomy (odds ratio [OR], 0.42; 95% confidence interval [CI], 0.30-0.58), esophagectomy (OR, 0.48; 95% CI, 0.37-0.62), and colectomy (OR, 0.69; 95% CI, 0.55-0.86). Risk-adjusted mortality was significantly lower in best hospitals for colectomy (OR, 0.79; 95% CI, 0.65-0.95). However, after accounting for hospital volume, risk-adjusted mortality was only significantly lower at the US News & World Report best hospitals for colectomy (OR, 0.79; 95% CI, 0.62-0.99) and was not significantly lower at best hospitals for any of the 3 oncologic procedures. Conclusions: Publicly marketed hospital rating systems of surgical quality such as the US News & World Report America s Best Hospitals and Best Hospitals may identify high-quality hospitals for some oncologic surgeries. However, these ratings fail to identify other high-volume hospitals of equal quality. Arch Surg. 2011;146(5):600-604 Author Affiliations: Michigan Surgical Collaborative for Outcomes Research and Evaluation, Department of Surgery, University of Michigan, Ann Arbor. THE US NEWS &WORLD REport America s Best Hospitals and the Best Hospitals are two of the most recognized publicly available hospital ratings. The US News & World Report America s Best Hospitals is an annual rating of the fifty best hospitals across 16 medical and surgical specialties including the 50 best cancer hospitals. markets hospital report cards to health care consumers on the internet, reporting the quality of health care of 5000 American hospitals. 1 See Invited Critique at end of article Both the US News & World Report and hospital quality ratings receive significant publicity. Hospitals frequently use these ratings in advertising, increasing the influence of the ratings in public perception of hospital quality. 2 The US News & World Report is widely available, boasting a paper circulation of more than 2 million readers and an online readership with free access to all reports free of charge. Similarly, reports more than 5 million visits monthly to it s consumer Web site and markets itself as the nation s leading healthcare ratings organization. 1 Although these quality ratings are marketed as a consumer aid for choosing hospitals, it is unclear whether these ratings appropriately identify high-quality hospitals. Previous studies have shown that competing hospital quality ratings frequently provide inconsistent results. 3 No previous studies have addressed the question of whether the best hospitals have better cancer surgery outcomes than other hospitals in the United States. We sought to determine independently whether these two quality ratings truly identify the best hospitals for cancer surgery. We used the 2005-2006 MedPar data set to compare the 30-day mortality in the best hospitals and all other hospitals for 3 cancer surgery procedures. 600
METHODS US NEWS & WORLD REPORT AMERICA S BEST HOSPITALS The 50 top-ranked cancer hospitals were determined from the 2008 publication of the US News & World Report America s Best Hospitals. 4 These hospitals are determined by a combination of 3 equally weighted measures: hospital infrastructure, hospital reputation among subspecialists, and 30-day mortality rate. Only hospitals that are either a teaching hospital or have a minimum number of advanced technologies are considered eligible for the ratings. In addition, hospitals must meet a minimum discharge and surgical volume threshold for 12 subspecialties (cancer; ear, nose, and throat; endocrinology; gastrointestinal disorders; geriatric care; gynecology; heart and heart surgery; kidney disease; neurology and neurosurgery; orthopedics; respiratory disorders; and urology). The US News & World Report evaluates hospital infrastructure using measures from the American Hospital Association annual survey, including hospital teaching status, Nurse Magnet status, staffto-bed ratios, hospital volume, and the presence of advanced technologies. The most subjective portion of the rating system, reputation of the hospitals among subspecialists, is based on a survey sent out to fewer than 100 members of each subspecialty to rank the 5 best hospitals. Risk-adjusted mortality rates for each hospital are calculated using Medicare claims data from the previous year. Using a composite score of these 3 measures, hospitals are ranked and the top 50 hospitals for each specialty are published. HEALTHGRADES BEST HOSPITALS The America s 50 Best Hospitals were collected directly from the Web site in October 2008. 1 In contrast to the US News & World Report ratings, the ratings are based solely on Medicare data. generates hospital ratings using a proprietary method, comparing predicted with observed mortality rates for all hospitals. The predicted 30-day mortality rate is derived from Medicare Part A billing data (MedPar 2005-2006). These predicted mortality rates were then compared with the observed mortality rates for each hospital, generating an observed to expected ratio. The hospitals are then rated based on a ratio of observed to expected mortality rate for 27 procedures and diagnoses. The 50 hospitals with the best observed to expected ratios are then included in the final list available on the Web site. STUDY SAMPLE All patients who underwent the 3 cancer operations, pancreatectomy, esophagectomy, and colectomy, were identified from the MedPar database from 2005 through 2006 using International Statistical Classification of Diseases, 10th Revision, Clinical Modification (ICD-9-CM) diagnosis and procedure codes (n=82 724). A total of 6455 patients underwent surgery in the 50 US News & World Report America s Best Cancer Hospitals, compared with 76 369 patients in the remaining 4445 hospitals nationwide. A total of 3452 patients from the 50 Best Hospitals were compared with 79 112 patients from the remaining 4445 hospitals nationwide. A total of 639 hospitals that do not participate in the American Hospital Association Annual Survey were not included in the analysis. All of the best hospitals from the US News & World Report America s Best Cancer Hospitals and the Best Hospitals participate in the American Hospital Association Annual Survey. STATISTICAL ANALYSIS Given the lack of agreement between the two quality ratings, the 50 best hospitals in each rating were compared with all other hospitals separately for each individual procedure. Membership in the Council of Teaching Hospitals was obtained from the 2003 American Hospital Association Annual Survey. Patient age, race, comorbidities, surgical acuity, and hospital characteristics were compared between best hospitals and all other hospitals using 2 statistics, t tests, and analyses of variance. Using all patients who underwent each of these 3 operations, we generated expected mortality rates using logistic regression models, adjusted for comorbidity burden, patient demographics, and emergent operation. Our independent variable was whether patients received care in a best hospital or all other hospitals. Random-effects logistic regression was used to provide robust adjustment for the effect of hospital clustering. Next, to assess the relative effect of procedural volume on mortality in best hospitals, we created quintiles of procedurespecific hospital volume. Using random-effects logistic regression, we compared mortality following each of the 3 operations between best hospitals and all other hospitals, adjusting for patient covariates and hospital volume quintile. Using the relative attenuation of the odds ratio (OR), we estimated the relative effect of hospital volume on mortality in the best hospitals compared with all other hospitals. 5 RESULTS Comparing the US News & World Report America s Best Cancer Hospitals and the Best Hospitals, only 2 hospitals are considered best hospitals in both ratings. Patient characteristics were compared between the 50 best hospitals in each rating and all other hospitals for each individual operation and each rating system (Table 1). The patients treated in the best hospitals were significantly different from the patients in all other hospitals. Across all procedures, US News & World Report America s Best Hospitals were less likely to operate emergently or urgently. Best Hospitals treated fewer black patients and were less likely to operate emergently or urgently. Patient comorbidity burden did not significantly differ between best hospitals from either rating system and all other hospitals. The characteristics of hospitals in each rating differed significantly from all other hospitals (Table2). US News & World Report America s Best Cancer Hospitals were significantly less likely to be very-low-volume hospitals for all 3 operations. America s Best Cancer Hospitals were also more likely to be teaching hospitals as compared with all other hospitals. Similarly, Best Hospitals were less likely to be very-low-volume for any of the cancer operations. Best Hospitals were also more likely to be teaching hospitals as compared with all other hospitals. Adjusting for patient characteristics and surgical acuity, 30-day mortality rates were lower in US News & World Report America s Best Cancer Hospitals for all 3 operations: pancreatectomy (OR, 0.42; 95% confidence interval [CI], 0.30-0.58), esophagectomy (OR, 0.48; 95% CI, 0.37-0.62), and colectomy (OR, 0.69; 95% CI, 601
Table 1. Patient Characteristics of US News & World Report America s Best Cancer Hospitals, Best Hospitals, and All Other Hospitals (2005-2006) Characteristic National Hospital P Value US News & World Report National Hospital P Value Esophagectomy, No. 904 5035 1516 4523 Male, % 72.14 71.44.74 72.82 71.04.18 Mean age, y 73.97 73.77.48 73.27 73.96.001 Black, % 2.29 6.16.001 4.38 6.35.05 Emergent/urgent surgery 13.51 19.15.002 9.56 21.82.001 Comorbidities 2, % 87.11 90.51.06 90.17 90.26.99 Pancreatectomy, No. 1135 5166 2081 4220 Male, % 50.75 48.73.40 50.41 48.13.09 Mean age, y 74.02 74.1.78 74.02 74.13.49 Black, % 4.1 6.69.03 5.22 7.13.004 Emergent/urgent surgery 14.13 20.4.001 12.35 23.67.001 Comorbidities 2, % 83.94 84.35.17 84.29 84.34.91 Colectomy, No. 1413 68 911 2858 67 626 Male, % 44.78 42.54.01 43.32 42.61.45 Mean age, y 78.45 78.48.80 77.81 78.5.001 Black, % 7.39 8.85.005 11.41 8.67.001 Emergent/urgent surgery 33.65 37.66.001 20.85 38.18.001 Comorbidities 2, % 100 100.99 100 100.99 Table 2. Hospital Characteristics of US News & World Report America s Best Cancer Hospitals, Best Hospitals, and All Other Hospitals (2005-2006) Percentage Percentage Characteristic National Hospital P Value US News & World Report National Hospital P Value Esophagectomy Member of council of teaching hospitals 28.26 18.72.06 95.83 16.07.001 Very-low-volume hospital 34.78 64.35.001 0 65.72.001 Pancreatectomy Member of council of teaching hospitals 24.39 24.42.89 95.83 20.58.001 Very-low-volume hospital 53.66 71.68.02 0 74.64.001 Colectomy Member of council of teaching hospitals 26.0 7.94.001 93.88 6.98.001 Very-low-volume hospital 4.0 62.12.001 2.04 62.13.001 0.55-0.86). In contrast, risk-adjusted mortality was significantly lower in Best Hospitals following 1 of 3 procedures: colectomy (OR, 0.79; 95% CI, 0.65-0.95). The risk-adjusted mortality rates of the US News & World Report and rated hospitals and all other hospitals are shown in Figure 1 and Figure 2, respectively. After accounting for hospital volume, US News & World Report America s Best Cancer Hospitals 30-day mortality rates were significantly lower for only 1 of 3 procedures: colectomy (OR, 0.69; 95% CI, 0.62-0.99). In comparison, accounting for hospital volume, mortality rates did not differ significantly between Best Hospitals and all other hospitals for any of the 3 procedures (Table2). Using the relative attenuation of the OR to determine how much of the differences in mortality can be explained by volume, a significant proportion of the differences in mortality between Best Hospitals and all other hospitals in both rating systems can be attributed to hospital volume (Table 3). As much as 71% of the observed differences in mortality following pancreatectomy between US News & World Report Best Hospitals and other hospitals is owing to differences in hospital volume. COMMENT Current consumer-marketed ratings of hospital quality may identify hospitals with lower surgical mortality rates for 3 cancer surgery procedures. America s Best Cancer Hospitals and Best Hospitals had significantly lower risk-adjusted mortality rates for all 3 and 1 of 3 operations, respectively. Previous studies of hospital ratings have shown that both and US News & World Report best hospitals have lower mortality rates for patients admitted with acute myocardial infarction and some cardiovascular procedures. 6-8 Interestingly, these differences in mortality rates did not exist for all operations in both rating systems. Of note, rating system did not identify highquality hospitals for patients who underwent high-risk cancer surgery, pancreatectomy, and esophagectomy. Although these ratings systems may identify hospitals with lower-than-average mortality, a significant 602
12.0 10.0 Other hospitals 14.0 12.0 US News & World Report Other hospitals Risk-Adjusted Mortality Rate 8.0 6.0 4.0 Risk-Adjusted Mortality Rate 10.0 8.0 6.0 4.0 2.0 2.0 0.0 Esophagectomy Pancreatectomy Colectomy 0.0 Esophagectomy Pancreatectomy Colectomy Figure 1. Risk-adjusted mortality rates following cancer surgery in Best Hospitals and all other hospitals (2005-2006). amount of the differences in surgical mortality are attributable to differences in procedural volume. In this analysis, procedural volume completely explained the differences in mortality for pancreatectomy and esophagectomy. These differences in procedural volume have been previously reported as important predictors of riskadjusted mortality, especially for less-common, highmorbidity surgical procedures such as esophagectomy and pancreatectomy. 9 Similarly, when comparing hospitals performing cardiovascular procedures, volume has a significant effect on quality ratings. 8 Although both the US News & World Report and the quality ratings rely on risk-adjusted mortality rates, neither rating controls for differences in procedural volume. Our findings show that both ratings fail to identify equally wellperforming hospitals of similar volume. Mortality rates between best hospitals and hospitals of similar volume were not significantly different. This study has important implications for patients trying to choose safe hospitals for oncologic surgery. While we found a mortality rate benefit to choosing these highly rated hospitals, patients may equally benefit from going to a similarly high-volume hospital closer to their home. Pervious study by Birkmeyer and colleagues has shown that most patients live close to a high-volume hospital. In an analysis of patients who underwent pancreatectomy or esophagectomy, more than one-quarter of patients lived closest to a high-volume hospital and nearly 75% of patients would travel fewer than 30 minutes to reach a high-volume hospital. 10 Because these ratings list only a selected number of best hospitals, these consumer aids may dissuade patients from seeking care at closer high-volume, equal-quality hospitals. This study has several limitations. First, the study assessed the validity of these hospital ratings using the outcome of 30-day mortality. Mortality represents just one domain of surgical quality; other outcomes are equally important such as readmission rates, complications, and cost. In addition, in the Donabedian structure-processoutcomes model of quality, 11 structural and process measures may play important roles in measurement of surgical quality. Unfortunately, current databases do not Figure 2. Risk-adjusted mortality rates following cancer surgery in US News & World Report America s Best Cancer Hospitals and all other hospitals (2005-2006). Table 3. Odds Ratios of 30-Day Mortality Rate Following Cancer Surgery for Best Hospitals and US News & World Report America s Best Hospitals Compared With All Other Hospitals (2005-2006) Surgery Odds Ratio (95% Confidence Interval) Multivariate Multivariate With Hospital Volume Proportion of Variation Explained by Volume, % Esophagectomy 0.89 (0.59-1.33) 1.02 (0.73-1.44) Pancreatectomy 0.62 (0.34-1.11) 0.73 (0.45-1.19) Colectomy 0.79 (0.65-0.95) 0.89 (0.73-1.08) 48 US News & World Report Esophagectomy 0.48 (0.37-0.62) 0.76 (0.55-1.05) 54 Pancreatectomy 0.42 (0.30-0.58) 0.83 (0.57-1.20) 71 Colectomy 0.69 (0.55-0.86) 0.79 (0.62-0.99) 32 include robust measures of these domains. Although this analysis was limited to mortality rate, this is a key measure used in the methodologies of both hospital ratings. Second, our study relies on Medicare claims data, limiting our study population to patients older than 65 years. However, of elderly persons represent most patients undergoing these procedures, and elderly patients have the highest risk associated with these surgeries. In addition, administrative data lack robust clinical detail to entirely account for differences in case-mix and patient illness severity that may confound the differences in mortality rate between hospitals. We have controlled for available patient risk factors including demographics, comorbidities, and acuity of admission/surgery. Although differences in patient factors may exist between hospitals, there is a lack of clinical data to adequately account for such differences. More robust patient and hospital data will likely improve our ability to accurately rate the quality of hospitals. This study has significant policy implications. As politicians, patients, and payers continue to demand acces- 603
sible quality measurements of medical care, accurate hospital quality ratings must be developed. Reliable and accessible methods of rating hospital quality will require a collaboration of not only independent ratings organizations, but hospitals and payers as well. Unfortunately, current quality ratings may confuse patients and dissuade them from receiving care in high-quality hospitals close to home. Future methods of rating hospital quality should consider not only quality but access to those quality hospitals. Both US News & World Report and may identify high-quality hospitals for receiving cancer surgery. However, these ratings fail to identify other highvolume hospitals with equivalent outcomes. Further research into the measurement of surgical quality will be necessary to help aid patient decision making. Accepted for Publication: April 9, 2010. Correspondence: Nicholas H. Osborne, MD, 6312 Medical Science Building I, 1150 W Medical Center Dr, SPC5604, Ann Arbor, MI 48109-5604 (nichosbo@umich.edu). Author Contributions: Study concept and design: Osborne and Dimick. Acquisition of data: Osborne, Ghaferi, Nicholas, and Dimick. Analysis and interpretation of data: Osborne, Ghaferi, Nicholas, and Dimick. Drafting of the manuscript: Osborne, Ghaferi, Nicholas, and Dimick. Critical revision of the manuscript for important intellectual content: Osborne, Ghaferi, Nicholas, and Dimick. Statistical analysis: Osborne, Ghaferi, Nicholas, and Dimick. Obtained funding: Osborne. Study supervision: Dimick. Financial Disclosure: None reported. REFERENCES 1. Web site. http://www.healthgrades.com/. Accessed November 24, 2008. 2. Larson RJ, Schwartz LM, Woloshin S, Welch HG. Advertising by academic medical centers. Arch Intern Med. 2005;165(6):645-651. 3. Leonardi MJ, McGory ML, Ko CY. Publicly available hospital comparison web sites: determination of useful, valid, and appropriate information for comparing surgical quality. Arch Surg. 2007;142(9):863-869. 4. US News & World Report America s Best Hospitals 2008. US News & World Report Web site. http://health.usnews.com/best-hospitals. Accessed March 30, 2009. 5. Finlayson EV, Goodney PP, Birkmeyer JD. Hospital volume and operative mortality in cancer surgery: a national study. Arch Surg. 2003;138(7):721-726. 6. Chen J, Radford MJ, Wang Y, Marciniak TA, Krumholz HM. Do America s Best Hospitals perform better for acute myocardial infarction? N Engl J Med. 1999;340 (4):286-292. 7. Krumholz HM, Rathore SS, Chen J, Wang Y, Radford MJ. Evaluation of a consumeroriented internet health care report card: the risk of quality ratings based on mortality data. JAMA. 2002;287(10):1277-1287. 8. Osborne NH, Nicholas LH, Ghaferi AA, Upchurch GR Jr, Dimick JB. Do popular media and internet-based hospital quality ratings identify hospitals with better cardiovascular surgery outcomes? J Am Coll Surg. 2010;210(1):87-92. 9. Birkmeyer JD, Dimick JB, Staiger DO. Operative mortality and procedure volume as predictors of subsequent hospital performance. Ann Surg. 2006;243 (3):411-417. 10. Birkmeyer JD, Siewers AE, Marth NJ, Goodman DC. Regionalization of high-risk surgery and implications for patient travel times. JAMA. 2003;290(20):2703-2708. 11. Donabedian A. The quality of medical care. Science. 1978;200(4344):856-864. INVITED CRITIQUE Choosing The Best H ow frightening and perplexing it must be to learn of a cancer diagnosis and not know where to go for care. How do our patients, friends, and family members assure themselves that they are committing their lives to a high-quality surgeon and hospital? A physician colleague just emailed one of us for a recommendation for care of a family member with a complex cancer problem. Some things we did not do: (1) go to www.healthgrades.com or (2) drop down to the magazine section in our hospital gift shop. But we did know where the patient lived and had first-hand knowledge of the quality of the hospital and the surgeon that we recommended. That insider information is both priceless and not easily available to the general public. Had our patient s family member checked on the US News and World Report site, he would have found the name of the hospital we recommended and 4 others in the tri-state area where he lives and may have been reassured. A sixth hospital was listed in and none of the other 5 was mentioned in that grading system. In this issue, Osborne and colleagues examine the riskadjusted 30-day mortality rate of patients undergoing 1 of 3 surgical procedures for cancer: pancreatectomy, esophagectomy, and colectomy. In doing this, the authors compare the 30-day mortality rates of these procedures between institutions designated best hospitals and all other hospitals to determine if outcomes are really better in hospitals fortunate enough to carry the coveted designation. The authors used a media-based (US News and World Report) and an internet-based () hospital rating system to compare the 50 best hospitals in each ranking with outcomes from all other hospitals. As we found when we looked, there was very little concordance between the two rating systems as to which were the best hospitals in the nation, since only a few hospitals appeared on both lists. The article by Osborne and colleagues supports the recommendation that we made to our friend, as the 6455 patients treated at the US News and World Report top 50 cancer hospitals have a significantly lower mortality rates than the 76 369 patients treated at the 4445 other hospitals. The authors show that indeed there is a significantly lower risk-adjusted mortality rate in the US News and World Report best hospitals for all 3 procedures exam- 604