Nursing Home Administrators Opinions of the Nursing Home Compare Web Site

Transcription

1 The Gerontologist Vol. 45, No. 3, Copyright 2005 by The Gerontological Society of America Nursing Home Administrators Opinions of the Nursing Home Compare Web Site Nicholas G. Castle, PhD 1 Purpose: In November of 2002 the Centers for Medicare and Medicaid Services publicly reported on a national basis the quality of nursing homes on the Nursing Home Compare (NHC) Web site. This study examines administrators opinions of this initiative and whether it has fostered quality improvement. Design and Methods: Data used in this investigation come from a questionnaire mailed to the administrators of 477 nursing homes. Three hundred and twenty-four responses were received, giving a response rate of 68%. Facilities were located in four states: Connecticut, Maryland, Pennsylvania, and Tennessee. Results: Ninety percent of the administrators examined the NHC Web site, 51% stated that they would be using the information for quality improvement in the future, and 33% stated that they were using NHC information in quality-improvement initiatives. Implications: For NHC to influence quality as it is intended, nursing home quality-of-care initiatives are essential. Study results show that administrators are generally accepting of the NHC Web site, and some are using the information in quality initiatives. Key Words: Report cards, Nursing Home Compare, Quality Public report cards have been a feature of the U.S. health care landscape for several decades. These report cards attempt to pull together concise quality assessments to assist consumers choose among providers (Kroll Letwat, 1999, p. 325). Notable initiatives in this area include the Health Plan Employer Data and Information Set (HEDIS), the Consumer Assessment of Health Plans Survey (CAHPS), and the mortality reports of the Health Care Financing Administration (now known as the Centers for Medicare and Medicaid Services, or CMS; see Scanlon, Chernew, Sheffler, & Fendrick, 1998). Address correspondence to Nicholas G. Castle, PhD, Health Policy and Management, University of Pittsburgh, A649 Crabtree Hall, 130 DeSoto Street, Pittsburgh, PA castlen@pitt.edu 1 Health Policy and Management, University of Pittsburgh, PA. Report cards derive their popularity from their utility in several areas. First, they can assist health care purchasers in evaluating the value they are receiving from their expenditures (Scanlon et al., 1998). As Mukamel and Mushlin described, public dissemination of accurate information about quality restores the imbalance between quality and costs in the value equation (2001, p. 21). Second, the collecting and reporting of information regarding quality may encourage providers to improve their performance. As Davies, Washington, and Bindman reported, very public judgments about the quality of services affect the reputation of organizations, and organizations so affected might be expected to respond (2002, p. 382). Third, report cards may educate health care purchasers about the care provided (Kroll Letwat, 1999), by providing information on quality indicators, for example. Despite these potential benefits of report cards, they also have their critics. The public may not actually use the information they contain (Marshall, 2001). Robinson and Brodie (1997), for example, found that only 34% of consumers with report-card information actually used it. The data used by report cards may not be timely, and may not report quality measures that providers can improve in any significant way (Thomas, 1998). In addition, if report cards are successful, Mukamel and Mushlin (2001) noted that high-quality providers may not be able to respond to any resulting increase in demand for their services. With these advocates and detractors as a backdrop, in November of 2002 the CMS publicly reported on the quality of nursing homes on a national basis on the Nursing Home Compare (NHC) Web site ( This was significant. Although report cards have been used for a considerable period of time (as cited in Marshall, 2001, Florence Nightingale is credited as producing one of the earliest report cards in 1863, describing mortality rates in London hospitals), largescale report-card initiatives are relatively recent developments and generally have not diffused to long-term-care settings. Large-scale report-card initiatives had generally been used for hospitals and managed care providers. The NHC report card now Vol. 45, No. 3,

2 expanded the availability of public information on nursing homes and the quality of care provided (General Accounting Office [GAO], 2002, p. 1). However, as with most report cards preceding NHC, questions remain regarding the use and usefulness of the information collected and reported. In this investigation, I used a questionnaire to examine administrators opinions of the NHC initiative and to determine whether the NHC Web site has fostered or helped with quality-improvement initiatives. Administrators are in contact almost daily with consumers; their opinions are important, as they can give some indication of the use and usefulness of the NHC website from the perspectives of both the facility and the consumer. Whether NHC information has fostered or helped with quality improvement is a particularly important question. For the NHC Web site to influence quality, nursing home quality-of-care initiatives are essential. A prior literature review of report cards identified quality improvement by providers to be a likely outcome of disclosure of report-card information (Marshall, Shekelle, Leatherman, & Brook, 2000). Indeed, the New York Cardiac Surgery Reporting System influenced provider behavior (Hannan et al., 1997), as did the obstetrics consumer report developed by the Missouri Department of Health (Longo et al., 1997). The NHC Web site is intended to provide assistance to consumers in choosing nursing homes and to improve the quality of care of nursing homes (GAO, 2002). Because this information could affect each nursing home s market share of incoming residents, one would expect most administrators to examine this report card. Administrators also may be sensitive to the public image of their facility (Marshall et al., 2000) or to peers opinions (Longo et al., 1997). Professional norms also can create pressure for changes in quality. A recent study by Frayne and Geringer (2000) supports the notion that professional norms can influence outcomes. In turn, administrators are likely to have an opinion on the validity of the information presented, and some may have developed quality-of-care initiatives as a result of the information contained in the NHC Web site. Thus, on the basis of (a) prior literature showing quality improvement by providers to be a likely outcome of disclosure of report-card information; (b) the influence of market forces; (c) public image; and (d) professional norms, I hypothesize that public dissemination of nursing home quality through the NHC report card has motivated administrators to improve quality of care. Nursing Home Compare The NHC report card began in a small way in The quality data reported on the website first consisted of deficiency citations. These citations are given to Medicare- or Medicaid-certified nursing homes when they do not meet preset quality standards. Very soon after the initial NHC beginnings in 1998, the CMS expanded the quality data to include the percentage of residents with pressure ulcers; information regarding the use of physical restraints; and caregiver staffing levels. In April 2002, the CMS included further revisions in the NHC report card. These revisions included a set of quality measures, which are based on individual resident data found in the Minimum Data Set (MDS). It is worth noting that the quality measures used in the NHC report card are not all the more familiar quality indicators used by surveyors during the survey and certification process. These quality indicators were developed by the Center for Health Systems Research and Analysis (CHSRA) at the University of Wisconsin (Zimmerman, 2003); they include, for example, use of nine or more different medications, prevalence of bedfast residents, and incidence of new fractures. They were developed for use by surveyors as indicators of quality problems, and not for public reporting. The CMS wanted to use quality measures in the NHC report card that were reliably associated with quality, and of course could be publicly reported. In short, some quality measures may be quality indicators, but not all quality indicators are quality measures (Zimmerman). Therefore, under a contract from the CMS, Abt examined the quality indicators and other quality measures specifically for use in the NHC report card. This development is described in a GAO report (GAO, 2002). The Abt research was intended to identify quality measures most reflective of nursing home quality. This included risk adjustment, to take into account resident and facility characteristics. The CMS selected 10 measures for use in the NHC report card these are called the core quality measures. Loss of ability in basic daily tasks, pressure ulcers, physical restraint use, and infections were used for long-stay residents with no risk adjustment, along with pain with resident-level risk adjustment, and pressure ulcers with facility-level risk adjustment. Delirium and pain were used for short-stay residents with no risk adjustment, along with delirium with resident-level risk adjustment, and walking with facility-level risk adjustment. The walking quality measure indicates the percentage of residents whose independence in walking has been maintained or improved. The assessments used to calculate this measure are those conducted between Days 5 and 14 for short-stay residents. Resident-level risk adjustment is not unusual in the creation of report-card information. This controls for confounding effects that might be attributed to differences in resident populations (Mor et al., 2003). Facility-level risk adjustment is more uncommon than resident-level risk adjustment. Rather than account for individual resident characteristics alone, this approach accounts for the fact that, in the 300 The Gerontologist

3 aggregate, some facilities may admit more impaired residents than other facilities. This facility admission profile (FAP) was used to risk-adjust quality measures that are considered more difficult for a facility to change, or in which one could reasonably anticipate some reoccurrence of the condition (e.g., pressure ulcers). Mor and associates recently outlined the rationale behind this approach to risk adjustment. The CMS initiative in early 2002 was intended as a pilot program, and NHC information was included only for six states: Colorado, Florida, Maryland, Ohio, Rhode Island, and Washington. These pilot activities were expanded into a national initiative in November 2002, including all 50 states. In addition, as part of this initiative, Medicare Quality Improvement Organizations (QIOs) also were supposed to work with nursing homes to improve their quality and help consumers understand and use the NHC report card. Kissam and colleagues (2003) provide some information on how QIOs are responding to this initiative. Finally, in January 2004, the CMS further changed the core quality measures to include a total of 14 indicators. For long-stay residents this included the percentage with an increased need for help with daily activities; the number of high-risk and low-risk residents with pressure ulcers; physical restraint use; the number of low-risk residents with loss of bladder or bowel control; the number of residents that had a catheter inserted and left in the bladder, spent most time in bed or in a chair, and had a urinary tract infection, with no risk adjustment; and percentage with moderate to severe pain, who were more depressed or anxious, and whose ability to move in or around the room got worse, with resident-level risk adjustment. For short-stay residents this included the percentage with delirium, with moderate to severe pain, and with pressure ulcers, all with resident level risk-adjustment. Thus, the quality measures were expanded, and the FAP, which was a little controversial (GAO, 2002), was no longer used. Methods Sources of Data The data used in this investigation primarily came from a survey that I conducted of nursing home administrators. I mailed a questionnaire to administrators of 477 nursing homes during January of Facilities were located in four states: Connecticut (CT), Maryland (MD), Pennsylvania (PA), and Tennessee (TN). I chose these states because two had their own Web site report cards (MD and PA) prior to the full implementation of the NHC report card and the remaining two states did not have their own Web site report cards (CT and TN). The sample was divided in this way because I was interested in examining whether prior experience with Web site report cards influenced administrators responses to the NHC report card. Previous research has shown that prior exposure to report cards may influence how they are used and their usefulness; however, the literature has mixed results as to whether prior exposure increases users receptivity toward report cards, or decreases receptivity. Results from the study by Mukamel and Mushlin (1998) show that facilities are likely to react to the first report card, whereas results from other analyses show that one report card may be a catalyst to the use of the next (Longo et al., 1997). For the questionnaire mailing to administrators, I stratified facilities by state and then chose a random sample of approximately 30% of facilities from each state s pool of eligible facilities. I defined eligible facilities simply as those facilities participating in Medicare or Medicaid certification. I used this eligibility definition because these are the facilities included in the NHC report card. Eligible facilities included 252 nursing homes from CT, 749 from PA, 246 from MD, and 339 from TN. I sent the questionnaires to administrators in January of I used some facility characteristics in the analyses (e.g., bed size, ownership, chain membership, Medicaid occupancy, and overall resident census), which I obtained from the 2003 Online Survey, Certification, and Reporting (OSCAR) data. These data come from the annual survey inspection in which all Medicare- or Medicaid-certified nursing homes must partake. The facility characteristics are considered reliable, and the data have been extensively described by others (e.g., Hughes, Lapane, & Mor, 2000). Questionnaire To my knowledge, no previous questionnaire has examined administrators experiences with the NHC or any other nursing home report card. However, numerous publications have examined health-plan report cards. I examined this literature for guidance in developing my questionnaire. From this literature search, I found that Harris-Kojetin, McCormack, Jael, Sangl, and Garfinkel (2001) determined from interviews of consumers that report-card issues often fall into four main areas: content, comprehension, navigation, and decision processes. I used these issues in developing the questionnaire. Content refers to whether the report-card materials are relevant and complete. Comprehension questions examine whether the information contained in report cards is understood. Navigation examines how easily the report-card information is found. Decision process questions examine whether the report-card materials are helpful in making health care choices. In general, my questions first sought the administrators opinions, and then asked their opinions from the perspective of consumers. Most questions used a 1-to-10 visual-analog rating scale, with 1 anchored by the Vol. 45, No. 3,

4 Table 1. Descriptive Statistics of Administrators and Their Use of NHC Factor MD and PA (Used a Report Card Prior to NHC) a %orm CT and TN (Did not Use a Report Card Prior to NHC) b Total Sample Respondent characteristic Female 59% 42%* 54% Age (years) Highest level of education c Associate s degree 3% 2% 3% Bachelor s degree 51% 44%* 48% Master s degree or higher 45% 53%* 50% Tenure as administrator in current facility (years) Tenure as a nursing home administrator (years) Member of professional society of organization 74% 71% 73% Use of NHC Have examined the NHC Web site 91% 88% 90% Number of times visited NHC Web site * 4.1 Total time spent examining NHC (hr) * 3.2 First source of information about NHC c Newspaper 8% 10% 9% Journal or Society 72% 68% 71% Coworker 21% 23% 22% Facility characteristic Organizational size For-profit ownership 67% 69% 69% Chain membership 53% 54% 53% Medicaid occupancy 56% 58% 57% Census 88% 87% 88% Notes: NHC = nursing home compare. For the table, data were collected from 324 administrators in Connecticut, Maryland, Pennsylvania, and Tennessee in January For this investigation, a report card is defined as materials listing quality-of-care information and the information is standardized and released to the public. a N = 198 administrators. b N = 126 administrators. c Values may not total 100% because of rounding error. least positive rating (e.g., not at all useful) and 10 anchored by the most positive rating (e.g., extremely useful; see Castle & Engberg, 2004). I pilot tested the 45 items in the questionnaire with 10 administrators. This included mailing the questionnaire to the administrators and following up with phone calls. These calls lasted between 15 and 35 minutes. This resulted in minor changes to the questions and minor wording revisions to the instructions given at the beginning of the questionnaire. Analyses Descriptive analyses are presented consisting of the percent or mean for each of the questionnaire items. I calculated the average values for each of the two groups of states, and I used t tests to compare the significance of the difference in values between the groups. The total sample percent or mean also is presented, along with bar charts showing the distribution of scores. The score distributions show the percentage of responses for each response category, which is useful in determining whether the means are centered on a normal distribution or whether the data were skewed in some way. In addition, I used bivariate comparisons for respondent and nonrespondent facilities by using the OSCAR data. For those nursing homes using the NHC information, I included on the questionnaire two openended questions that asked (a) how the information was used by the facility and (b) for any additional comments. I also examined these qualitative data and areas where NHC information was used are presented. In addition, areas where other comments were made are presented. Results Three hundred and twenty-four responses were received from the nursing home sample (response rate = 68%). The response rate varied a little across the states, with a response rate of 62% for MD (n = 46), 68% for PA (n = 152), 79% for CT (n = 59), and 66% for TN (n = 67). In general, most items on the questionnaire were answered. Missing data occurred in less than 1% of cases and were evenly distributed across questions and states. Most (81%) of the questionnaires were returned by mail within 1 month. In addition, because I was able to link facilities with OSCAR data, I determined that no significant differences on facility characteristics (i.e., bed size, ownership, chain membership, Medicaid census, staffing levels, and deficiencies) existed for respondents compared with nonrespondents. Table 1 presents descriptive statistics of administrators and their use of the NHC report card, along with facility characteristics. Of most significance, I found a majority (90%) of administrators to have examined the NHC Web site. In general, few differences among the state samples were observed, but the gender and education level of administrators were different (p,.05). This may reflect employment conditions in these states. In examining administrators opinion of the content of the NHC website (see Table 2), a mean score of 6.9 resulted, whereas administrators had a mean score of 7.1 for the relevance of NHC. However, from the perspective of potential residents (or family members), administrators found the information to be less relevant (mean score of 6.0) and less complete (mean score of 4.8). Table 2 also 302 The Gerontologist

5 Table 2. Descriptive Statistics of Administrators Opinion of the Content and Comprehension of NHC Description MD and PA a CT and TN b Total Sample Score Distribution c Content Administrators opinion d How relevant you found the NHC information 7.4 (2.8) 6.2 (3.1)* 6.9 (2.9) How complete you found the NHC information 6.3 (2.9) 7.9 (2.7)* 7.1 (2.8) How much unnecessary information you found in NHC 4.1 (3.4) 4.7 (3.3) 4.3 (3.3) Administrators opinion from consumers perspective d How relevant you think the NHC information is in helping potential residents (or their families) choose a nursing facility 6.2 (3.0) 5.7 (2.8) 6.0 (2.8) How complete you think the NHC information is in helping potential residents (or their families) choose a nursing facility 4.9 (3.3) 4.7 (2.9) 4.8 (3.2) How much unnecessary information for potential residents (or their families) is in NHC 3.6 (2.1) 4.2 (3.2)* 3.8 (2.5) Comprehension Administrators opinion d How easy you found the NHC information to understand 8.7 (1.9) 8.5 (2.0) 8.6 (2.0) How much of the NHC information did you understand 8.5 (2.2) 8.2 (2.0) 8.4 (2.1) Administrators opinion from consumers perspective d How easy you think the NHC information is to understand in helping potential residents (or their families) choose a nursing facility 6.5 (3.2) 6.2 (3.4) 6.4 (3.2) How much of the NHC information you think potential residents (or their families) will understand 5.2 (2.8) 5.8 (3.0)* 5.4 (2.9) Notes: NHC = nursing home compare. For the table, data were collected from 324 administrators in Connecticut, Maryland, Pennsylvania, and Tennessee in January For this investigation, a report card is defined as materials listing quality-of-care information and the information is standardized and released to the public. a These states used a report card prior to NHC (N = 198 administrators). b These states did not use a report card prior to NHC (N = 126 administrators). c The score distributions used data from the total sample. The X-axis categories are the response scale scores of 1 to 10, and the Y-axis is a percent value from 0 to 100. d Questions used a 1 to 10 scale, with 10 representing the most positive rating and 1 the least positive rating. *Difference between pairs of states is significant at p,.05. presents descriptive statistics of administrators opinion of the comprehension of the NHC website. Administrators had a mean score of 8.6 for how easy the information was to understand and a mean score of 8.4 that they actually understood most of it. However, again from the perspective of potential residents (or family members), they thought the information would be less easy to understand in general (mean score of 6.4) and information would be less well understood (mean score of 5.4). Some results from the two groups of states are significantly different. This includes relevance, completeness, unnecessary information, and understanding. The score distributions for each of the questions appear Vol. 45, No. 3,

6 Table 3. Descriptive Statistics of Administrators Opinion of the Navigation and the Usefulness of NHC With the Decision Processes of Choosing a Nursing Home Description M (SD) MD and PA a CT and TN b Total Sample Score Distribution c Navigation Administrators opinion d How easy you found the NHC information was to explore 8.2 (2.1) 8.0 (2.4) 8.1 (2.3) How easy you found the NHC information was to find information you needed 8.4 (2.0) 7.5 (2.6)* 8.2 (2.1) If you found the NHC information helpful in interpreting the information it presented 7.9 (2.1) 8.2 (2.5) 8.0 (2.2) Administrators opinion from consumers perspective d How easy you think the NHC information is to explore for potential residents (or their families) choosing a nursing facility 6.0 (3.1) 6.4 (2.6) 6.1 (2.9) How easy you think the NHC information is for potential residents (or their families) choosing a nursing facility to find the information they need 7.7 (2.1) 7.2 (2.4)* 7.5 (2.2) Rate the NHC information in how helpful it is in interpreting the information presented for potential residents (or their families) choosing a nursing facility 7.4 (2.2) 7.0 (2.4) 7.2 (2.2) Decision Process Administrators opinion d Rate if you think the NHC information would be helpful in choosing a nursing facility 7.1 (2.2) 6.7 (2.5) 6.9 (2.3) Rate if you think the NHC information would be helpful to you in choosing your facility 5.6 (2.7) 6.3 (3.2)* 5.8 (2.8) To what extent you think the NHC information would help you be more confident in choosing a nursing facility 6.4 (2.3) 6.2 (3.0) 6.3 (2.5) Administrators opinion from consumers perspective d Rate how helpful you think the NHC information is for potential residents (or their families) choosing a nursing facility 6.7 (2.4) 6.3 (2.3) 6.5 (2.3) Rate if you think the NHC information would be helpful to potential residents (or families) in choosing your facility 5.6 (2.7) 6.3 (3.2)* 5.8 (2.8) To what extent you think the NHC information helps potential residents (or their families) be more confident in choosing a nursing facility 5.7 (2.4) 6.4 (3.1)* 5.9 (2.8) Notes: NHC = nursing home compare. For the table, data were collected from 324 administrators in Connecticut, Maryland, Pennsylvania, and Tennessee in January For this investigation, a report card is defined as materials listing quality-of-care information and the information is standardized and released to the public. a These states used a report card prior to NHC (N = 198 administrators). b These states did not use a report card prior to NHC (N = 126 administrators). c The score distributions use data from the total sample. The X-axis categories are the response scale scores of 1 to 10, and the Y-axis is a percent value from 0 to 100. d Questions used a 1 to 10 scale, with 10 representing the most positive rating and 1 representing the least positive rating. *Difference between pairs of states is significant at p, The Gerontologist

7 to be evenly distributed in most cases; however, the two questions addressing understanding of the NHC Web site are notable with a heavy skew toward higher scores. Some of the highest scores from the questionnaire were given in the section addressing administrators opinion of navigation of the NHC Web site (shown in Table 3). Administrators had a mean score of 8.1 for ease of exploring the NHC information and a mean score of 8.2 for how easy it was to find. From the perspective of potential residents (or family members), they thought the information provided would be less easy to explore (mean score of 6.1) and less helpful in interpreting the quality measures (mean score of 7.5). This table also presents descriptive statistics of administrators opinion of the usefulness of the NHC information in decision processes. Administrators had a mean score of 6.9 in rating the helpfulness of the information in choosing a facility, but the scores for whether the information would make them more confident in choosing a facility were lower (mean score of 5.8). From the perspective of potential residents (or family members), they thought the information would be less helpful in choosing a facility (mean score of 6.5) and have less utility in making them more confident in choosing a facility (mean of 5.8). Few results from the two groups of states are significantly different. Ease in finding information from the administrators perspective and from the perspective of potential residents (or family members) varied by state sample, as did helpfulness in choosing a facility. The score distributions show an even distribution of scores across response categories for most questions. Descriptive statistics of the uses of the NHC Web site are presented in Table 4. Administrators had a mean score of 7.2 for whether the information fairly represented the quality of their facility, a mean score of 3.8 for the extent to which NHC information was used by consumers, and a mean score of 2.8 for how much the information discouraged potential residents from choosing their facility. The score distributions for these latter two questions show a heavy skew toward lower scores. In addition, 33% of administrators indicated that they had used NHC information in their facility. For these questions, none of the results from the two groups of states are significantly different. Comments were provided by 122 administrators (38%) to the open-ended question concerning any additional comments. Most commonly, administrators objected to the use of deficiency-citation information (29% of all comments). Administrators also reported concerns with the risk adjustment (22% of all comments). These concerns included understanding the risk adjustment and whether it was needed. Administrators also expressed the opinion that the information presented on the NHC website was not accurate (19% of all comments). Finally, administrators also reported concerns that the measures did not accurately represent their facilities (17% of all comments). They indicated that the inclusion of other services, such as special care units, would more fairly represent the care they give. Information was provided by 73 administrators (22%) to the open-ended question asking whether quality improvements were made as a result of the NHC information. Most commonly, administrators gave general or nonspecific comments (42% of all comments) indicating that the information was available, used, or passed on to the correct person for quality improvement. A number of administrators (17% of all comments) indicated that the information was used in specific ongoing quality initiatives (e.g., restraint reduction). Other responses included using the information to initiate restraint reduction, pressure ulcer prevention, and pain reduction (7%, 9%, and 12% of all comments, respectively). Discussion Report-card advocates believe that informed choices will lead to improvements in quality of care. For these improvements to occur, consumers must use report cards, but providers also must access report-card information and use the information in quality initiatives. However, providing information in report cards does not necessarily guarantee their use by consumers, nor does it necessarily initiate quality improvement by providers. With this in mind, in this study I focused on the impact the NHC website has had on nursing home administrators. My hypothesis was that public dissemination of nursing home quality in NHC information would motivate administrators to improve quality of care. This research would appear to be especially warranted given that the NHC Web site has received some criticism. A recent GAO report (2002) cites concerns over the appropriateness of the quality indicators used in NHC and whether the CMS could help consumers use the reports. In general, the results would seem to support my hypothesis. I found 33% of facilities to be using NHC information, and a further 51% state that they would be using the information for quality improvement in the future. Given that the NHC information was released only 3 months prior to data collection, this would seem to be a large number of facilities. However, clearly interpreting my findings as positive and supporting my hypothesis depends somewhat on whether one considers 33% of facilities to be using NHC information a high percentage or not. I found this number to be surprisingly large given the short time frame since the release of the NHC information, and the fact that administrators also reported that they believed few consumers were using the NHC Web site. However, that is not to say that administrators were not critical of the NHC information. Vol. 45, No. 3,

8 Table 4. Descriptive Statistics of Administrators Opinion of the Overall Usefulness of NHC and Use in Their Facility M(SD) Description MD and PA a CT and TN b Total Sample Score Distribution c Administrators opinion d Rate how fairly you think the NHC information represents the quality of your facility 7.1 (2.4) 7.4 (2.7) 7.2 (2.5) Which measure (for long-stay residents) is most credible at measuring the quality of your facility e Loss of ability in basic daily tasks (%) 17 25* 20 Pressure sores (%) 6 11* 8 Pain (%) Physical restraint use (%) 48 34* 42 Infections (%) Which measure (for short-stay residents) is most credible at measuring the quality of your facility e Delirium (%) Pain (%) Walk as well or better (%) Administrators opinion from consumers perspective d Rate to what extent you think the NHC information has been used by potential residents (or their families) choosing your facility 3.7 (3.4) 3.9 (3.2) 3.8 (3.3) Rate to what extent you think the NHC information has discouraged potential residents (or their families) from choosing your facility 2.9 (2.4) 2.5 (2.2) 2.8 (2.3) Facility use of NHC and other reports cards d Have you used NHC information in your facility in any way (% yes) Do you plan to use NHC information in your facility in any way in the future (% yes) Have you ever used other nursing home report cards in your facility (% yes) Notes: NHC = nursing home compare. For the table, data were collected from 324 administrators in Connecticut, Maryland, Pennsylvania, and Tennessee in January For this investigation, a report card is defined as materials listing quality-of-care information and the information is standardized and released to the public. a These states used a report card prior to NHC (N = 198 administrators). b These states did not use a report card prior to NHC (N = 126 administrators). c The score distributions use data from the total sample. The X-axis categories are the response scale scores of 1 to 10, and the Y-axis is a percent value from 0 to 100. d Questions used a 1 to 10 scale, with 10 representing the most positive rating and 1 representing the least positive rating. e Values may not total 100% because of rounding error. *Difference between pairs of states is significant at p,.05. Administrators were critical of, if not a little perplexed by, the risk adjustment used in the NHC information. Most issues that administrators had with risk adjustment were simply related to understanding the techniques used. This understanding and potential misunderstanding of risk adjustment may be important. Previous research has shown that clarity is important in report cards. As Tumilson and associates (1997) found, clarity may facilitate getting consumers to use report cards. The same also may be true for nursing home administrators. It should be noted, however, that a gold standard to risk-adjust nursing home outcomes does not exist (Mor et al., 2003). Administrators also were critical about using deficiency-citation information. They did not believe that deficiency citations well represented the quality of their own facility, and in general they believed that other facilities may be able to game the inspection process. This information came from an open-ended question and may not be representative of all administrators. Still, this belief is in line with 306 The Gerontologist

9 recent GAO (1999) conclusions regarding the survey and certification system. Regarding the specific MDS quality measures, administrators were asked which measures they believed were most and least credible. The pressure ulcer measure was identified as the least credible measure. This is congruent with the belief that pressure ulcers can develop during relatively short stays in other facilities. Nursing homes that hospitalize residents, for example, may experience higher rates of pressure ulcers and be unfairly classified as providing poor-quality care. The high degree of concern with pressure ulcers as a quality measure is important. As Mennemeyer, Morrisey, and Howard described, either the providers will lose credibility, or the quality measures will (1997, p. 126). It certainly is the case that, from administrators perspective, the number of residents with pressure ulcers is not a credible measure. It is interesting that administrators cite hot-button issues such as certification and pressure ulcers as important, yet, overall, they rate the NHC information as useful. It also is interesting that administrators appeared to rate the NHC information as more useful for themselves than for elders. In one sense this is rational, as some elders and family members tend to have low levels of education, limited reading skills, limited access to the Internet, and difficulty seeing (Goldstein & Fyock, 2001). In another sense this in not rational, because this is certainly not the case for all elders or family members and it is a little surprising that administrators would have such a dim view of the capabilities of consumers. This view that the information is more useful for themselves than elders also is contrary to the belief that the NHC information will help consumers. Therefore, I speculate that administrators appear to be responding to the NHC information more because of a professional responsibility than because they fear potential consumers will use the information for admission decisions. Facility characteristics (listed in Table 1) and administrator characteristics, and items from the content, comprehension, navigation, and decision processes sections of the questionnaire, were also used in multivariate analyses (not shown) examining the association of administrators opinions of the usefulness of NHC information. The results were highly similar to those presented in the descriptive analyses. That is, relevance, completeness, ease of understanding, amount understood, ease of exploration, information interpretation, and confidence in choosing facility all had a significant association with whether administrators thought the NHC Web site was useful. Some of the results for the facility characteristics also warrant mentioning. Administrators in facilities with larger bed size, for-profit ownership, chain membership, and high Medicaid occupancy were associated with a lower likelihood of finding the NHC Web site extremely useful. Limitations I used MD and PA as states using report cards prior to the NHC report card. One limitation of this approach is that the MD and PA report cards are quite different. For example, the MD report card provides more information than the PA report card. This could influence administrators opinions of the usefulness of report cards. We found that administrators in MD rated the usefulness of the NHC report card lower than did administrators from other states (analyses not shown). Alternatively, the information found on the state report cards may be seen as better (or worse) by administrators than that found in the NHC report card. The length of time the state report cards have been available also could influence administrators opinions. For example, in states where report cards have been available for several years and consumers have (or have not) been influenced by the information, administrators opinions of report cards in general could be influenced. In retrospect, a series of questions asking about administrators prior experience and use of report cards may be a better approach to examining whether prior experience with state website report cards influenced administrators response to the NHC report card in any way. My results represent the opinions of administrators. These administrators may not be good proxies for opinions of residents and family members. In addition, the NHC web site is intended to be used primarily by consumers. As Harrington, O Meara, Kitchener, Simon, and Schnelle described, ultimately, report cards cannot serve the interests of all users equally (2003, p. 48). A further limitation of using administrators as respondents is the disparity between what administrators report on our questionnaire regarding report cards and whether they actually use the information. An observational examination of whether and how nursing homes use NHC information is still needed as a follow-up to this study. A related issue is that I did not define what I meant by quality initiatives in the questionnaire. This was left to the discretion of the administrator. Thus, the results I present for use of NHC information in quality initiatives are likely to vary from widespread to minimal quality-improvement activities. The visual-analog rating scale used on the questionnaire was well received by administrators during pilot testing. Nevertheless, only two points on these scales were labeled at the highest and lowest values. Thus, some caution is required in interpreting scores between the poles. A score of 8 is more positive than a score of 7, for example, but further interpretation of either of these scores as representing a moderate rating cannot truly be made. The questionnaire was administered during the first 3 months of the release of the NHC information. Administrators may have visited the website more Vol. 45, No. 3,

10 frequently because of its novelty or newness, so the information reported for the number of visits to the NHC Web site may be biased. Of course, the alternative also is possible. That is, over time, the NHC report card will influence a greater number of administrators, and we will see more use of this Web site. The questionnaire also may have prompted administrators to visit the NHC Web site. I have no way of knowing whether this was the case. An examination of administrators opinions of the NHC Web site now that it has been in operation for more than 1 year would probably be a useful extension to this work, and it should include an evaluation of whether and how NHC information has become institutionalized in the daily operations of the facility. Conclusions We know very little about nursing home report cards and the impact they are having on consumers and providers. With the exception of a recent GAO report (2002) and four recent articles (Castle & Lowe, 2005; Harrington, Collier, & O Meara, 2003; Harrington, O Meara, et al., 2003; Mukamel & Spector, 2003), I could not identify any research on this topic. Furthermore, these prior works are descriptive; I could not find any empirical research on this topic. So, despite the unquestionable need for report cards on nursing home care, some research in this area is clearly warranted. In this research, I focus on the impact that the NHC report card has had on administrators. My results show that administrators believe the NHC information is more useful for themselves than for potential consumers, and it is equivocal as to whether the information has stimulated much in the way of quality-improvement initiatives. Many other research topics readily come to mind. These include whether report-card information influences consumer cost quality tradeoffs, if the information is used by family members or elders, if market conditions influence consumer and provider reactions to NHC information, and if some groups of consumers find the NHC information more (or less) useful. In my research I have identified nursing homes to be using NHC information; the most significant future area of research I can think of is whether this improves quality of care a badly needed outcome for our nation s nursing homes. References Castle, N. G., & Engberg, J. (2004). Response scales and satisfaction of elders. The Gerontologist, 44, Castle, N. G, & Lowe, T. (2005). Report cards and nursing homes. The Gerontologist, 45, Davies, H. T., Washington, A. E., & Bindman, A. B. (2002). Health care report cards: Implications for vulnerable patient groups and the organizations providing them care. Journal of Health Politics, Policy and Law, 27, Frayne, C. A., & Geringer, J. M. (2000). Self-management training for improving job performance: A field experiment involving salespeople. Journal of Applied Psychology, 85, General Accounting Office. (1999). Nursing homes: HCFA initiatives to improve care are under way but will require continued commitment. Washington, DC: Author. General Accounting Office. (2002). Nursing homes: Public reporting of quality indicators has merit, but national implementation is premature. Washington, DC: Author. Goldstein, E., & Fyock, J. (2001). Reporting of CAHPS quality information to medicare beneficiaries. Health Services Research, 36, Hannan, E. L., Siu, A. L., Kumar, D., Racz, M., Pryor, D. B., & Chassin, M. R. (1997). Asessment of coronary artery bypass graft surgery performance in New York. Is there a bias against taking high-risk patients? Medical Care, 35, Harrington, C., Collier, E., & O Meara, J. (2003). Federal and state nursing facility web sites: Just what the consumer needs? American Journal of Medical Quality, 18, Harrington, C., O Meara, J., Kitchener, M., Simon, L., & Schnelle, J. F. (2003). Designing a report card for nursing facilities: What information is needed and why. The Gerontologist, 43(Special Issue II), Harris-Kojetin, L. D., McCormack, L. A., Jael, E. F., Sangl, J. A., & Garfinkel, S. A. (2001). Creating more effective health plan quality reports for consumers: Lessons from a synthesis of qualitative testing. Health Services Research, 36, Hughes, C. M., Lapane, K. L., & Mor, V. (2000). Influence of facility characteristics on use of antipsychotic medications in nursing homes. Medical Care, 38, Kissam, S., Gifford, D. R., Parks, P., Patry, G., Palmer, L., Wilkes, L., et al. (2003). Approaches to quality improvement in nursing homes: Lessons learned from the six-state pilot of CMS Nursing Home Quality Initiative. BMC Geriatrics, 3, 1 8. Kroll Letwat, J. (1999). The current quality of health plan report cards. Journal of Medical Systems, 23, Longo, D. R., Land, G., Schramm, W., Fraas, J., Hoskins, B., & Howell, V. (1997). Consumer reports in health care: Do they make a difference in patient care? Journal of the American Medical Association, 278, Marshall, M. N., Shekelle, P. G., Leatherman, S., & Brook, R. (2000). Public release of performance data. What do we expect to gain? A review of the evidence. Journal of the American Medical Association, 283, Marshall, M. N. (2001). Accountability and quality improvement: The role of report cards. Quality in Health Care, 10, Mennemeyer, S. T., Morrisey, M. A., & Howard, L. Z. (1997). Death and reputation: How consumers acted upon HCFA mortality information. Inquiry, 34, Mor, V., Berg, K., Angelelli, J., Gifford, D., Morris, J., & Moore, T. (2003). The quality of quality measurement in U.S. nursing homes. The Gerontologist, 43(Special Issue II), Mukamel, D. B., & Mushlin, A. I. (1998). Quality of care information makes a difference: An analysis of market share and price changes following publication of the New York State Cardiac Surgery Report Card. Medical Care, 36, Mukamel, D. B., & Mushlin, A. I. (2001). The impact of quality report cards on choice of physicians, hospitals, and HMOs: A midcourse evaluation. Journal on Quality Improvement, 27, Mukamel, D. B., & Spector, W. D. (2003). Quality report cards and nursing home quality. The Gerontologist, 43(Special Issue II), Robinson, S., & Brodie, M. (1997). Understanding the quality challenge for health consumers: The Kaiser/AHCPR survey. Journal of Quality Improvement, 23, Scanlon, D. P., Chernew, M., Sheffler, S., & Fendrick, A. M. (1998). Health plan report cards: Exploring differences in plan ratings. Journal on Quality Improvement, 24, Thomas, J. W. (1998). Report cards Useful to whom and for what? Journal on Quality Improvement, 24, Tumlinson, A., Bottigheimer, H., Mahoney, P., Stone, E. M., & Hendricks, A. (1997). Choosing a health plan: What information will consumers use? Health Affairs, 16, Zimmerman, D. R. (2003). Improving nursing home quality of care through outcomes data: The MDS quality indicators. International Journal of Geriatric Psychiatry, 18, Received January 29, 2004 Accepted November 2, 2004 Decision Editor: Linda S. Noelker, PhD 308 The Gerontologist