Risk Adjustment of Nursing Home Quality Indicators 1

Transcription

1 Copyright 1997 by The Cerontological Society of America The Cerontologist Vol. 37, No. 6, The purpose of this study was to develop a method for risk adjusting nursing home quality indicators (Ql's). The Ql's measure incidence and prevalence of resident-level care processes and outcomes which are indicative of care quality. Risk adjustment was carried out by stratifying residents into risk groups (high and low), calculating Ql rates within groups, and then drawing comparisons across facilities. The method was examined through analysis of data from over 800 nursing homes in four states. Results showed that facilities differed substantially in Ql rates even after risk had been taken into account. Also, results suggested differences in care quality which may not have been apparent without controlling for risk. Key Words: Long-term care, Outcomes, Assessment, Measurement, Risk adjustment, Quality indicators Risk Adjustment of Nursing Home Quality Indicators 1 Greg Arling, PhD, 2 Sarita L Karon, PhD, 3 Francois Sainfort, PhD, 4 David R. Zimmerman, PhD, 4 and Richard Ross, BS 3 Quality of care for nursing home residents has received growing attention from health care professionals, consumers, policy makers, and researchers in the last decade (Institute of Medicine, 1986; Kane & Kane, 1988). This growing concern led to the passage of the Omnibus Reconciliation Act of 1987 (OBRA 87), which, among other important provisions, requires that a comprehensive assessment of all nursing home residents be conducted periodically using resident level data from the Resident Assessment Instrument (RAI) (Morris et al., 1990,1991). The RAI consists of the Minimum Data Set (MDS) assessment form and the Resident Assessment Protocols (RAP's). The MDS includes information about a resident's physical functioning, cognitive, medical, emotional and social status. The RAP's are corresponding care-planning tools used to help identify potential care issues (Morris et al., 1991). Not long after the passage of OBRA 87, the Health Care Financing Administration (HCFA) began its Multistate Nursing Home Case Mix and Quality Demonstration. The Demonstration uses resident assess- The work reported here was supported by the Health Care Financing Administration (HCFA) under Cooperative Agreement 18-C-99256/5-04 and Contract WI-1. We wish to thank Elizabeth Cornelius, HCFA Project Officer, for her advice and support. We also would like to acknowledge contributions of other Ql project staff, particularly Ted Collins and Brenda Ryther, as well as the many individuals in the Multistate Demonstration, nursing home survey teams, and nursing facilities who assisted in design and testing of the Ql's. Finally, we wish to thank two anonymous reviewers for their helpful comments. The opinions expressed in the article are those of the authors and do not necessarily reflect those of HCFA or the authors' institutions. 'Address correspondence to Greg Arling, PhD, Bloch School of Business and Public Administration, University of Missouri-Kansas City, 5100 Rockhill Road, Kansas City, MO garling@cctr.umkc.edu 'Center for Health Systems Research and Analysis, University of Wisconsin, Madison. 'Center for Health Systems Research and Analysis and Department of Industrial Engineering, University of Wisconsin, Madison. ment information from the MDS in order to: (1) develop and implement a case mix classification system for Medicaid and Medicare reimbursement, and (2) create a quality monitoring system to increase the effectiveness and efficiency of the nursing home survey process. The Demonstration, which began in 1989, is now in its implementation stage. Four states (Kansas, Maine, Mississippi, and South Dakota) are participating in the reimbursement and quality components of the Demonstration; two additional states (New York and Texas) are included only in the Medicare reimbursement component. As part of the Demonstration, we developed the nursing home quality indicator (Ql) system (Zimmerman et al., 1995). We designed the Ql's to take advantage of the routinely collected MDS information, and to be incorporated efficiently into the nursing home survey process. In addition, we made Ql's accessible to nursing homes for their own quality assurance and management activities. In these respects, the Ql's represent an advance over earlier quality assessment systems, many of which relied on specialized data and complex decision rules (Davis, 1991; Sainfort, Ramsay, & Monato, 1995). The Ql's cover health and functional conditions, emotional and cognitive status, and use of specific services or procedures. At the resident level, each Ql is measured by the presence or onset of a condition or service. At the facility level, the Ql represents the proportion of residents who have the condition at a point in time ("prevalence Ql") or who develop the condition over a specific period of time ("incidence Ql"). Application of the Ql System involves a series of reports that are generated from routinely collected MDS data and that describe prevalence or incidence of each Ql for the nursing home population as a whole and within each facility. In addition, re- Vol. 37, No. 6,

2 ports list Ql's associated with each resident, i.e., presence or absence, or development of a condition or use of a specific service. A central issue in designing the Ql system was the need to adjust for variation in the risk of deleterious outcomes. We have conceptualized risk as the probability that, given a particular combination of health or functional conditions, a resident will require certain care processes or will experience specific adverse health outcomes. Ideally, quality assessment systems should distinguish adverse outcomes that are "avoidable" or that result from the quality of resident care, from those that are "unavoidable" or that follow from the natural course of disease or disability. Moreover, systems should be able to control for inter-facility differences in clinical and functional pre-conditions of residents. The purpose of riskadjustment is to remove effects of resident risk from those associated with quality of care provided by the facility. Thus, facility comparisons based on riskadjusted Ql's should be a more accurate reflection of care quality than unadjusted indicators. They should result in more effective targeting of quality of care problems, at both the facility and resident levels. Yet, as we will discuss, the assumptions underlying risk adjustment are quite complex and have important implications for the quality assessment process. The main purpose of this article is to explain the methods we used in risk-adjustment of the Ql's. First, we discuss the conceptual basis for risk adjustment and our methods for defining risk and making this concept operational. Next, we describe the risk adjustment method employed in the project and we illustrate this method with selected findings based on resident and facility-level data from the Multistate Demonstration. Finally, we outline remaining issues in the application of risk to assessment of quality of long-term care. Conceptual and Methodological Issues The application of risk in the assessment of health care quality has been the subject of considerable debate. Much of this has centered around the availability of relevant data and the validity of risk adjustment models. The Health Care Financing Administration's use of Medicare data to assess hospital performance based on risk-adjusted mortality rates has been at the center of much of this controversy. Critics have pointed out that administrative data systems, such as Medicare claim files, contain a limited set of information on risk factors. Demographic characteristics, such as age or sex, or even primary diagnoses may not be sensitive enough indicators of mortality risks to provide meaningful adjustment. Also, only a few attempts have been made to establish the validity of risk adjustment models (Blumberg, 1986; Landon et al., 1996; Lohr, 1988; Rubin & Wu, 1992; Thomas, Holloway, & Cuire, 1993). Quality assessment systems in long term care have only recently employed risk adjustment methods. For example, in their review of 24 nursing home quality studies carried out before 1990, Sainfort et al. (1995) found no study which attempted to adjust for risk at the resident level. Only a few studies applied global measures of risk at the facility level. Some more recent studies, however, have attempted to incorporate risk when examining care practices and outcomes. Zinn, Aaronson, and Rosko (1993) applied a multivariate regression model to adjust for facility variation in outcomes, e.g., mortality, pressure sores, and catheter and restraint use, among residents in Pennsylvania nursing homes. Berlowitz et al. (1996) found that risk adjustment made a significant difference in overall rates of pressure sore incidence (observed versus adjusted) in long term care facilities operated by the Veterans Administration. Adjustment and standardization of rates led to a substantial shift in facilities between outlier and nonoutlier status. Shaughnessy, Kramer, Hittle, and Steiner (1995) used a logistic regression model to control for interfacility differences in risk among teaching nursing homes and a sample of comparison facilities. In work related to home health care, Shaughnessy et al. (1994) attempted to control for risk by stratifying the patient population. They formed quality indicator groups (QUIG's), defined according to clinically significant risk factors, and analyzed particular care outcomes within each group. They argued that stratification of patients according to risk may obviate the need for more complex statistical adjustment procedures. We addressed a number of issues related to risk in our development of the nursing home quality indicators. Many of these issues were raised in the studies referenced above. They can be summarized as follows. Outcome and Process Indicators. Although risk adjustment has typically been applied to outcomes of care, it also may be appropriate when assessing process-oriented indicators (Zinn et al., 1993). For example, use of indwelling catheters is a controversial procedure that may be necessary to compensate for loss of bladder function, but also may have deleterious consequences, such as infection or lowering of resident self-esteem. Catheter use among nursing home residents may result from careful review of clinical conditions indicating need for this procedure. On the other hand, facilities may apply catheters for nonclinical reasons, such as perceived labor cost savings (compared to bladder training or toileting) or because of financial incentives, such as favorable reimbursement for catheter supplies. To adequately assess the appropriateness of catheter use when drawing comparison among facilities, we believed it was appropriate to adjust for inter-facility differences in risk factors. Available Resident Level Data. The adequacy of available data on risk factors has been a persistent criticism of health care quality assessment models. Until recently, long term care quality assessment also has been hampered by a lack of routinely collected resident-level data. Introduction of the MDS, however, has provided a comprehensive set of information that can be employed in measurement of both 758 The Gerontologist

3 indicators and risk factors. These data are collected at the resident level, so resident-specific measures of risk can be incorporated into the quality assessment process. Indicator-Specific Risk Definitions. Risk adjustment in previous studies has sometimes involved a global measure (e.g., illness severity or functional status index) to risk adjust a number of specific care outcomes. Rubin and Wu (1992) have noted that many of these global indices were not designed specifically for quality assessment. By incorporating a wide range of variables, some of which may be unrelated to the outcome of interest, these global risk adjusters may dilute the effect of important risk factors. For this reason, we have chosen to define a separate set of risk factors for each quality indicator. We selected risk variables based on review of relevant research, clinical input, and empirical analysis. Specification of Risk Models. Variation in an indicator, such as a Ql, is a function of quality, risk, and an "error" component that represents unmeasured variables, potential systematic bias, and random error. In theory, we can infer that inter-facility difference in quality will be the portion of variation remaining after risk factors and error components have been removed. In practice, as Thomas et al. (1993) have pointed out, it is impossible in advance to know what portion of variance can be attributed to each component. Inference about quality can be problematic because a facility's outlier status may be due to chance rather than underlying differences in quality of care. An even more serious problem arises if risk factors used for adjustment are, themselves, a function of poor care quality or if they represent problematic care practices. For example, use of physical restraints relates to falls and other injuries. Yet, excessive use of restraints is a reflection of poor care practices. Including restraint use as a risk factor for the prevalence of falls would seriously bias facility comparisons because potentially important quality differences would be adjusted out of the model. Although it is difficult to totally avoid random error or to fully specify risk for each Ql, we have taken steps to minimize these problems. First, we have selected only a sub-set of Ql's for risk adjustment. We regard several Ql's as "sentinel events," which are so severe or so closely related to quality of care problems, that they are left unadjusted. The possibility of overlooking serious quality of care problems outweighs the potential benefits of risk adjustment. Fecal impaction and dehydration fall into this category of indicators. In other cases, we felt that we did not have sufficiently reliable measures of risk factors, or we felt that available measures of risk were too closely related to the quality of care practices. Examples include the prevalence of falls, and the use of nine or more scheduled medications. Among Ql's that were subject to adjustment, we limited the number of variables that we placed in the risk model. We included in the risk definition those variables significantly related to the Ql, based on multivariate logistic modeling. In a few cases, we also included variables that were of major clinical importance, even when they were not statistically significant. When we believed that a statistically significant predictor of the Ql might itself be a result of poor care practices, we omitted it from the risk adjustment definition. Adjustment Method. Most studies of long-term care quality have used statistical methods of risk adjustment, such as multivariate models or standardization (Berlowitz et al., 1996; Shaughnessy et al., 1995; Zinn et al., 1993). A statistical approach offers the advantages of producing a single number (e.g., standardized rate, or difference or ratio observed and expected rates) to represent the care of each facility relative to others. Also, with a multivariate model the independent effects of a potentially wide range of risk variables can be taken into account in the adjustment process. The multivariate statistical model, on the other hand, may not deal well with interactions among risk factors or between risk and the care outcome (Thomas et al., 1993). In addition, clinical staff and other users (e.g., surveyors) may not fully understand methods of statistical adjustment. Lack of understanding may inhibit use of risk-adjusted measures and impede the quality assessment process. After considering various statistical risk adjustment techniques, we decided to use a stratification approach. Using a stratification method, one could set several levels of risk based for example on the number of risk elements that one resident presents or, alternatively, on different combinations of such risk elements. For our study, however, we adopted a simpler approach. We stratified residents into high and low risk groups and calculated Ql rates within each stratum. We then were able to draw inter-facility comparisons based on Ql rates within their high or low risk resident populations. Stratification offers several advantages. First, it is a relatively simple and straightforward process that those who do not have extensive training in epidemiology or multivariate statistics can understand. Second, the focus on high or low risk may aid in targeting resident review for quality of care problems. One may inquire, for example, why a low risk resident developed a particular condition, or why a high risk resident did not receive the types of preventive care that may have avoided a deleterious outcome. Third, the comparison of rates between strata can yield information about potentially important interactions between risk and the quality of care provided in a facility. For example, some facilities may be quite competent at treating low risk residents but fail to have the specialized care required for the high risk resident. Methods As indicated above, our primary method of risk adjustment is based on a set of binary risk groups for each Ql. and low risk groups are defined according to the presence or absence of specific risk elements. These elements consist of clinical conditions Vol. 37, No. 6,

4 that are believed to increase the likelihood that an individual will experience a Ql. An individual who has any one or more of these risk elements is placed in the high risk group; whereas an individual with none of the risk elements falls into the low risk group. Development of Risk Groups We identified risk elements and defined risk groups through an iterative process involving expert opinion and empirical analysis. We began with a review of research and clinical literature on conditions or services measured by the Ql's, and consideration of the RAP's component of the RAI. A preliminary set of risk elements from this review was examined by specially convened clinical panels made up of professionals from the range of disciplines, such as nursing, medicine, physical, occupational and speech therapy, nutrition, social work, and nursing home administration. Members of these panels suggested additions, deletions, and other changes to the risk elements associated with each Ql. After receiving clinical input, we subjected risk elements to extensive empirical analysis using MDS data from the Multistate Demonstration. We tested a broad range of risk factors, including those recommended by the clinical panel, as well as others that might have relevance to the Ql's. We pursued two basic questions in the analysis: (1) When combined with other risk elements in a multivariate model (logistic regression), did the risk element have a significant, independent effected on the Ql? (2) When risk elements were combined to form high and low risk groups, did the risk groups discriminate well between residents who did and did not experience the Ql? In general, we selected risk elements that had significant independent relationships to Ql's, and we chose risk groups that had a reasonable balance between sensitivity and specificity. That is, we attempted to maximize the proportion of residents with a particular Ql who fell into the high risk group, while minimizing the proportion of "false positives", i.e., residents in the high risk group that did not have the Ql. In addition, we wanted to maintain a reasonable size for each risk group so that facility-level Ql rates estimated within risk groups would be reliable even for facilities with small number of residents. The set of risk elements selected for each Ql were moderate to good predictors of each Ql in the logistic regressions performed. When examining the association between predicted and observed values, the Goodman-Kruskal gamma ranged from.306 (for incidence of contractures) to.885 (for prevalence of bladder or bowel incontinence). Not all our decisions were made on purely empirical grounds. We recognized that some risk elements can be so rare that we would not find a strong statistical relationship between the risk element and the Ql (e.g., being comatose, a condition experienced by less than 0.3% of residents), yet may be important clinical measures of risk and hence they should be included as risk elements. In addition, we removed risk elements that might be directly related to quality of care. For instance, people who are bedfast or who are incontinent may be at increased risk of developing pressure ulcers. Yet, either of these issues may also reflect a problem with the quality of care. For example, we would expect facilities to intervene in the care process by avoiding bedfastness or treating incontinence and thereby reducing the risk for pressure sores. If we were to risk adjust for bedfastness or incontinence, we might obscure potentially important differences in facility care practices. We should note, however, that bedfastness and incontinence also are important quality-related outcomes, and that we have included these as separate Ql's. We also retained several risk factors that were potentially quality-related, such as impaired transfer, impaired bed mobility, and history of pressure sores. With pressure sores and other Ql's we found it very difficult to select risk factors that were predictive of the Ql yet totally independent of care quality. We had to strike a balance for each Ql between the "purity" of the risk model and its predictive capability. We should point out that inclusion of quality-related risk factors may affect the sensitivity of the Ql's by increasing false negatives. Facilities with true quality problems may have their Ql rates reduced through risk adjustment. Data Sources For purposes of the present analysis, we utilized data from the MDS+ (an enhanced version of the MDS used in the HCFA Multistate Demonstration) from Kansas, Maine, Mississippi, and South Dakota for the period October 1,1994 to March 31, The MDS+ contains all the information found in the MDS (Morris et al., 1990), plus information on various processes of care, including a detailed inventory of prescription medications administered within 7 days of the assessment. Data for the analysis consist of assessments for 61,136 residents: 27,509 from Kansas, 9,568 from Maine, 15,140 from Mississippi, and 8,919 from South Dakota. The data set excluded admission and readmission assessments (explained below). A total of 834 facilities is represented in the data; 406 from Kansas, 143 from Maine, 171 from Mississippi, and 114 from South Dakota. The data set is a more recent version of data that were employed in the initial development of the Ql's and the risk adjustment methodology. The Ql's and associated risk elements were based on a subset of resident assessments performed during the six month data period. Since multiple assessments can be performed on each resident during the period, we needed a decision rule for selection of assessments. We chose the most recent assessment (closest in time to the end of the period) for the prevalence Ql's. Incidence Ql's were based on the two most recent assessments for each resident. Therefore, prevalence and incidence rates were calculated for all residents in the facility during the data period, but they reflected only the most recent assessments for each resident. Risk elements for prevalence Ql's were taken from the same assessment as the Ql, i.e., the most recent 760 The Gerontologist

5 assessment. For the two incidence Ql's, decline in late loss ADL's (i.e., eating, toileting, transferring, and bed mobility) and incidence of contractures, risk elements were measured at the most recent assessment, the previous assessment, or a combination of the two. For example, conditions such as hemiplegia, quadriplegia, coma, and terminal prognosis were measured at the most recent assessment. Total dependence in mobility ADL's (i.e., bed mobility, transfer, locomotion), a risk element for incidence of contractures, was measured at the previous assessment. Decline in cognitive status, development of an acute condition, and weight loss are risk elements reflecting changes between the previous and most recent assessments. Timing of conditions placing a resident at risk was difficult to establish because MDS data were gathered at discrete points and we lacked information about the time order of changes occurring between assessments. We assumed that certain conditions, such as hemiplegia and quadriplegia, were relatively stable over time and, thus, could be captured on the same assessment used to measure the Ql. Other conditions, such as cognitive status or acute illness, were more transitory and, therefore, we had to take into account changes in these conditions between assessments. We should acknowledge, however, that the timing of the data did not allow us to firmly establish causal order between the risk element and the Ql. Moreover, these health status changes may have been related at least indirectly to care quality. As we indicated above, we had to make difficult choices between the purity of our risk models and their ability to differentiate between high and low risk residents. Besides selecting the most recent or previous assessment for each resident, we further restricted our analysis to the subset of assessments that are most likely to reflect care provided in the facility, as opposed to conditions that might arise prior to admission or readmission. The MDS+ schedule requires that each resident be given a full assessment when first admitted to a nursing home; every 90 days thereafter; after a "significant change" in health status; or upon readmission to the facility if a resident leaves for at least 72 hours due to a hospitalization or temporary stay at home. For purposes of our analysis, we have excluded initial admission and readmission assessments. Health or functional conditions at initial admission do not reflect quality of care provided in the facility. Similarly, readmission data may be only marginally related to care being provided in the facility. Health conditions measured at readmission from a hospital, for example, may reflect care provided in the nursing facility prior to hospitalization, or may reflect care received in the hospital. We have chosen to use a conservative approach in calculating Ql's and, thus, we have excluded readmission assessment data from our analysis. An average of 5% of residents had missing information on a particular Ql or associated risk groups. If a Ql or risk group could not be calculated for a resident, the case was dropped from that analysis but was retained for other Ql's where full information was available. There are 31 Ql's now being used in the demonstration states. Of these, only seven currently are risk adjusted. Among the remaining Ql's, some were sentinel events, however in several cases we were unable to identify appropriate risk elements or to develop sufficiently predictive risk models. The results presented below focus on the seven Ql's that have been risk adjusted. Results We present results through a series of tables which describe prevalence or incidence of risk elements and Ql's at both resident and facility levels. Resident level results shed light on characteristics of residents, relative contribution of each risk element in defining risk groups, and ability of groupings to distinguish between high and low risk populations. Facility level results highlight differences in facility Ql scores between high and low risk populations, and they demonstrate methods we have used to identify outlier facilities. They also show that outlier status differs according to the risk group. This latter finding suggests potentially important interaction effects between facility care practices and resident characteristics. Residents by Individual Risk Elements and Risk Groups. Table 1 displays the proportion of residents who have each risk element, as well as the proportion who have one or more elements. By definition, residents with one or more risk elements are considered to belong to the high risk group; whereas, the remainder of residents fall into the low risk group. Several conditions, e.g., cognitive impairment, ADL dependency, hemiplegia and quadriplegia, appear as risk elements for more than one Ql. Cognitive impairment, severe cognitive impairment, total dependency in ADL's, total dependency in mobilityrelated ADL's, dependency in specific mobilityrelated ADL's, and weight loss are the most prevalent risk elements. As such, they have the greatest impact in placing individuals within high risk groups for their respective Ql's. Other conditions such as quadriplegia, terminal prognosis, multiple sclerosis diagnosis, and coma are much less prevalent. Although they represent clinically important risk elements, these conditions are responsible for placing only small numbers of individuals within the high risk groups. The size of the high risk group varies by Ql. Approximately two-thirds of residents are defined as high risk for Stage 1-4 pressure ulcers and problem behavior. We intentionally adopted broad risk group definitions for these Ql's because we wanted to focus attention on the low risk population, i.e., those residents who would be very unlikely to experience the Ql outcome. Only about one-fourth ( ) of residents fall into the high risk groups for the remainder of the Ql's. Here, we wanted to focus attention on residents who were more likely to have a Ql and, thereby, distinguish them from the remainder of the population. Vol. 37, No. 6,

6 Quality Indicator Table 1. Prevalence of Risk Elements Associated With Each Quality Indicator Prevalence of problem behavior toward others. Prevalence of bladder or bowel incontinence. Prevalence of indwelling catheters. Incidence of decline in late loss ADL's. Incidence of contractures. Prevalence of antipsychotic use, without psychotic and related conditions. Prevalence of stage 1-4 pressure ulcers. " = one or more condition. Risk Elements Cognitive impairment Alzheimer's disease Other dementia Psychotic conditions group' Severe cognitive impairment Totally dependent mobility ADL group" Coma Hemiplegia Quadriplegia Totally dependent mobility ADL Mutliple Sclerosis group" Weight loss Terminal prognosis Decline in cognitive status Development of acute condition Risk Group" Hemiplegia Quadriplegia Coma Totally dependent in mobility ADL's (prior assessment) group 8 group (Cognitive impairment and behavior problems) Impaired transfer Impaired bed mobility Hemiplegia Quadriplegia Coma Malnutrition Peripheral vascular disease History of pressure ulcers Desensitized skin Terminal prognosis Diabetes Pitting edema group 3 Prevalence Ql Prevalence or Incidence Within Risk Groups. Table 2 presents Ql prevalence or incidence rates overall and by risk groups. Several of the Ql's, such as use of indwelling catheters, incidence of incontinence, incidence of contractures, and pressure ulcers, are relatively uncommon events and, therefore, the proportions are low for both high and low risk groups. All seven of the Ql's, however, display statistically significant differences between high and low risk groups. With such a large sample size, one would expect statistical significance even with minor differences in Ql rates. The ratios reported in Table 2 are a better reflection of the magnitude of these differences. Prevalence or incidence rates for the high risk groups are from 2.6 to 3.1 times greater than rates for low risk groups for most of the Ql's. For prevalence of pressure sores, however, the high risk group has a rate that is 6.5 times as great as the low risk group. These findings demonstrate the ability of the risk groupings to discriminate between residents who either did or did not experience a Ql. Facility Level Ql Distributions by Risk Croup. The primary purpose of the risk groups is to adjust for inter-facility differences in risk among their resident populations. Even though we found significant differences in the Ql's between risk groups for the nursing home population as a whole, we cannot conclude necessarily that similar differences will appear when comparing facility level rates. Table 3 presents mean and median facility Ql rates, standard deviations, and coefficients of variation by risk group. Table 3 also contains mean Ql rates for facilities grouped by percentile categories (quartiles and 90-99th percentile). Percentile categories were based on facility Ql rankings from low to high within each risk group. For example, among high risk residents the mean facility prevalence of behavioral problems was.33; whereas, facilities in the first quartile had a mean prevalence of.13, facilities in the second quartile had a mean prevalence of.26, and so on. The table shows that mean facility Ql rates for the high risk population are significantly higher than rates for the low risk population. Moreover, this pattern is consistent across quartiles. In each quartile, the mean facility Ql rate for the high risk population is substantially higher than for the low risk population. In addition, coefficients of variation point to substantial inter-facility variation in Ql rates within risk groups, as well as for the unstratified nursing facility populations. Even though coefficients of variation differ across risk groups, in almost all cases they are larger within risk groups than they are for the overall population. Two risk groups high risk of incontinence and low risk for pressure sores have extremely skewed distributions. Among residents with high risk of incontinence, the mean facility prevalence is.906, and nearly half (49%) of facilities have a prevalence of For residents at low risk of pressure sores, the mean facility prevalence is only.017, and 70% of facilities have no low risk residents with pressure sores. Risk definitions for these groups appear to be highly predictive of the clinical outcome presence of incontinence or absence of pressure sores. Although facilities presumably vary in their care practices for these conditions, residents who are at high risk of incontinence (based on our definition of risk) have a consistently high probability of being incontinent, and residents who are at low risk of pressure sores are very unlikely to develop them. 762 The Gerontologist

7 Table 2. Ql Prevalence or Incidence by Risk Group Quality Indicator lisk Group Prevalence or Incidence Ratio ( : Risk) Prevalence of problem behavior toward others. Prevalence of bladder or bowel incontinence. Prevalence of indwelling catheters. Incidence of decline in late loss ADL's. Incidence of contractures. Prevalence of antipsychotic use, in the absence of psychotic and related conditions. Prevalence of stage 1-4 pressure ulcers..33* * * *.10.11* *.11.13* :1 2.6:1 2.8:1 2.7:1 2.8:1 3.1:1 6.5:1 *p <.001 (chi-square) for difference between high and low risk groups. Table 3. Facility-Level Ql Prevalence or Incidence by Risk Group and Facility Rank Quality Indicator Risk Group Mean Median Std. Dev. CV Means by Facility Rank (Percentile) Prevalence of problem behavior toward others. Prevalence of bladder or bowel incontinence. Prevalence of indwelling catheters. Incidence of decline in late loss ADL's. Incidence of contractures. Prevalence of antipsychotic use, in the absence of psychotic conditions. Prevalence of stage 1-4 pressure ulcers. a b The distribution is severely skewed. Nearly half (49%) of all facilities had a prevalence of The distribution is severely skewed, such that 70% of all facilities had a prevalence of On the one hand, this finding implies that our risk definitions may be too restrictive. That is, once risk is taken into account, the Ql may not be sensitive to inter-facility variations in care practices and resulting outcomes. On the other hand, the more restrictive risk definition offers an advantage because it provides greater specificity or more precise targeting of care outcomes. For example, the risk definition for pressure sores focuses attention on the small number of facilities where low risk residents develop these conditions. These are essentially sentinel events which are very likely to be related to care Vol. 37, No. 6,

8 quality and thus merit special review. In contrast, the risk definition for incontinence focuses attention on the lowest quartile, where the rate of incontinence for these facilities averages.76, indicating that nearly one out of four residents is continent despite their high risk. This finding suggests that facilities in the lowest quartile might have exemplary care practices which could be emulated by other facilities. Taken as a whole, these results illustrate the importance of risk definitions and their ability to meet different objectives in the quality assessment process. Outlier Status Before and After Risk Adjustment Risk adjustment can have a substantial impact on facility outlier status, i.e., facilities with inordinately high Ql rates relative to their peers. For this analysis we have defined outliers as facilities with Ql rates at or above the 90th percentile, or approximately 10% of all facilities. To assess the impact of risk adjustment, we compared outliers before and after adjustment. Table 4 contains the percentage of outlier facilities in each risk group that were not outliers prior to adjustment. For example, among facilities that were outliers in the high risk group for behavioral problems, 41.8% became outliers only after risk adjustment. That is, they were below the 90th percentile on the behavior problem Ql when facilities were ranked without regard to risk. On average across all risk groups, 42% of outlier facilities shifted from non-outlier status as a result of risk adjustment. Percentages ranged from 19% to 85% depending on the Ql and risk group. The greatest shifts in outlier status occurred within high risk groups for incontinence, ADL decline, and antipsychotics, and within the low risk group for pressure sores. Relatively little shifting occurred within the low risk group for contractures and high risk group for behavioral problems. Even though some Ql's seem to be more sensitive to inter-facility variation in risk, the adjustment process had a significant impact on all of the Ql's. We should note, however, that a shift into or out of outlier status may result from only a small change in percentile rank, e.g., from just below to just above the 90th percentile, so absolute changes in rank may not be as dramatic as they appear in the table. Nonetheless, risk adjustment helps to target facilities that otherwise would not have been suspect of quality problems, and it also gives consideration to facilities caring for a more at-risk population. Differences in Outlier Status for and Risk Populations. We suggested above that facility care practices might have different implications for quality depending on characteristics of resident populations. That is, certain facilities might be most effective at caring for high risk residents, whereas other facilities might focus on low risk residents. One method of exploring this issue is to look at the consistency of facility Ql rankings by risk groups. Will facilities that are Ql outliers among high risk residents also be outliers for low risk residents, or vice versa? To answer this question, we ranked facilities according to their Ql Table 4. Percentage of Facility Outliers After Risk Adjustment That Were Not Outliers Prior to Adjustment Quality Indicator Prevalence of problem behavior Prevalence of incontinence Prevalence of indwelling catheters Incidence of late loss ADL decline Incidence of contractures Prevalence of antipsychotics Prevalence of pressure ulcers Risk Croup Became Outliers After Risk Adjustment 41.8% 21.2% 36.5% 85.4% 27.1% 41.0% 32.1% 61.2% 19.3% 40.7% 27.1% 66.7% 60.7% 27.1% Note: Outlier status is defined as a Ql rate at or above the 90th percentile. The numerator for each percentage is the number of facilities that had a Ql rate below the 90th percentile when ranked overall (without regard to risk) but were at 90th percentile or above when ranked within the risk group. The denominator is the total number of facilities with a Ql rate at or above the 90th percentile for the risk group. rates and divided them into quartiles. Although the designation of outlier status is inherently arbitrary, we selected as outliers the top quartile of facilities on each Ql. We then compared the outlier status of the facility between low and high risk groups. For this comparison we defined facility outliers as the top quartile (at or above the 75th percentile). Table 5 shows the percentage of facilities that were in the top Ql quartile for both high and low risk groups, for high risk only, for low risk only, or for neither high nor low risk. For prevalence of behavioral problems, as an example, 14% of facilities were in the top quartile (Q4) for both high and low risk groups, 15% were in the top quartile for high risk but in Q1-3 for low risk, and so on. Depending on the Ql, from 58% to 73% of facilities did not meet our outlier criterion (top quartile) for either high or low risk groups. On the other hand, from 2% to 14% of facilities were in the top quartile for both high and low risk groups. This leaves a substantial proportion with outlier status for one risk group but not for the other. For six of the Ql's, between 10% and 17% of facilities were outliers for the high risk group but non-outliers for the low risk group. Similar percentages were outliers for low risk but non-outliers for high risk. The seventh Ql, prevalence of incontinence, displayed very little interfacility variation for the high risk population. Results from this analysis point to potential interaction effects between characteristics of residents and quality of care provided in the facility. We are investigating this issue, among others, through an ongoing study in which a clinical team from our research project has been assessing validity of Ql's in sampled facilities. The team has been examining patient records and investigating care practices and 764 The Gerontologist

9 Table 5. Percentage of Facilities by Ql Rank (Quartile) Within and Risk Groups Risk Croup (HI, LOW) and Quartile Rank (Q) Quality Indicator HI = Q4 LOW = Q4 HI = Q4 LOW = Q1-Q3 LOW = Q4 HI = Q1-Q3 HI = Q1-Q3 LOW = Q1-Q3 Prevalence of problem behavior toward others. Prevalence of bladder or bowel incontinence. Prevalence of indwelling catheters. Incidence of decline in late loss ADL's. Incidence of contractures. Prevalence of antipsychotic use, in the absence of psychotic and related conditions. Prevalence of stage 1-4 pressure ulcers. 14% 02% 12% 10% 12% 10% 09% 15% 02% 14% 15% 14% 16% 15% 10% 23% 13% 17% 13% 15% 17% 65% 73% 61% 58% 61% 60% 59% Note: "HI" and "LOW" refer to risk groups, and "Q4" and "Q1-3" refer to quartiles where facilities are ranked from low to high according to Ql rates. For example, the first column (HI = Q4; LOW = Q4) denotes the percentage of facilities with Ql rates in the top quartile (outliers) for both high and low risk groups. The second column shows the percentage of facilities in the top quartile (Q4) for the high risk group but in lower quartiles (Q1-3) for low risk group. outcomes for a sample of residents identified through the Ql's. As part of this process, they have looked at resident risk and its relationship to facility characteristics. Preliminary results suggest that some facilities lack the clinical expertise to deal with high risk residents. They are relatively effective at minimizing adverse outcomes when resident risk is low; yet, they have difficulty managing high risk residents having complex medical conditions or severe functional impairment. Conversely, some facilities which specialize in medically complex or post-acute residents may give insufficient attention to low risk residents, i.e., failing to institute preventive practices which could reduce the rate of adverse outcomes. Final results from the study should shed further light on this issue. Discussion In designing a risk adjustment method for the Ql system, we attempted to deal with a number of issues that have been raised in previous studies. First, we took risk into account when assessing both outcome and process indicators of quality. Second, we defined indicator-specific risk factors taken from a particularly rich source of routinely collected data (MDS+). Third, we used both statistical criteria and clinical judgment in deciding on risk elements that we felt would be reasonably independent of facility care practices. We wanted to avoid adjusting for risk elements that in themselves might be reflections of poor quality care. Finally, we adopted a stratification approach (binary risk groups) that would be understandable to individuals responsible for implementing the Ql system, e.g., nursing home regulators or facility staff. We also attempted to capture interaction effects where facilities might provide different types of care or produce different outcomes for high and low risk residents. Results from our analysis of Ql's and risk groups from the Multistate Demonstration showed that high and low risk populations differed significantly in outcomes and processes of care. Moreover, when facilitylevel Ql rates were calculated within risk groups, there were broad differences between facilities. The average Ql rates for facilities in the highest quartile and 90-99th percentile were substantially greater than rates in lower quartiles. Perhaps the most interesting finding was the proportion of facilities with outlier status (top quartile) for one risk group but not the other. These results, plus preliminary evidence from ongoing validation studies of the Ql's, suggest that certain facilities are more likely to experience quality of care problems with high risk residents, whereas other facilities have greater difficulty with the care of low risk residents. Our method of risk adjustment was able to distinguish effectively between facilities according to their Ql prevalence and incidence. Yet, several issues remain in validating this method as a quality assessment technique. First, we need to validate the Ql's as indicators of long term care quality. Simply finding interfacility differences in outcomes or care processes is insufficient to establish problems in resident care. Second, we need to determine if risk adjustment actually leads to more effective comparisons between facilities. For example, are we adjusting for risk elements which are independent of care quality and have we minimized effects of measurement error? Also, does stratifying the population into high and low risk groups (as opposed to either not addressing risk at all, or to using statistical risk adjustment techniques) facilitate application of Ql's in the regulatory or quality assurance process? Are the risk groups meaningful from a clinical standpoint and can they be used to target important differences in care provided to low and high risk populations? Many of these questions are now being addressed through our ongoing validation studies. Initial results from our validation studies suggest that Ql's and associated risk elements are measured accurately. We have discovered quality of care problems, based on independent clinical judgments of project teams, in a high proportion of residents who experienced an adverse care outcome or process as indicated by the Ql's. In addition, care problems Vol. 37, No. 6,

10 have been most pronounced in facilities scoring high on the Ql's, i.e., at or above the 90th percentile. Preliminary evidence about the feasibility and validity of our risk adjustment method also has been promising. State nursing home survey teams and staff of selected facilities in the Multistate Demonstration have been applying the Ql's and the risk adjustment method since late They report an understanding of the method and its use in the quality assessment process. Preliminary reports indicate widespread enthusiasm about the Ql's and their utility in both of these processes. Surveyors find the Ql's particularly helpful in identifying a resident sample for review, an area where risk considerations are especially important. Risk distinctions also are useful in assisting surveyors and facility staff to identify the clinical issues of greatest concern. Although results from our study are encouraging, we do not pretend to have resolved all the issues surrounding risk adjustment in nursing home quality assessment. Selection and application of risk elements is a complex matter requiring considerable judgment which must be informed by empirical analysis and expert opinion. Despite the depth of information available from the MDS+, it is impossible to establish conclusively the causal relationships between risk and quality of care. Unmeasured factors may account for resident or facility differences in outcomes. Moreover, conditions which place residents at risk for adverse outcomes may themselves be a function of care quality. Adjustment based on quality-related risk factors can confound the quality assessment process. These potential problems, which challenge the validity of risk adjustment models, must be addressed as we continue to refine our quality assessment systems. References Berlowitz, D. R., Ash, A. S., Brandeis, G. H., Brand, H. K., Halpern, J. L, & Moskowitz, M. A. (1996). Rating long-term care facilities on pressure ulcer development: Importance of case-mix adjustment. Annals of Internal Medicine, 124, Blumberg, M. S. (1986). Risk adjusting health care outcomes: A methodologic review. Medical Care Review, 43, Davis, M. A. (1991). On nursing home quality: A review and analysis. Medical Care Review, 48, Institute of Medicine. (1986). Improving the quality of care in nursing homes. Washington, DC: National Academy Press. Kane, R. A., & Kane, R. L. (1988). Long-term care: Variations on a quality assurance theme. Inquiry, 25(Spring), Landon, B., lezzoni, L I., Ash, A. S., Shwartz, M., Daley, J., Hughes, J. S., & Mackiernan, Y. D. (19%). Judging hospitals by severity-adjusted mortality rates: The case of CABG surgery. Inquiry, 33(Summer), Lohr, K. N. (1988). Outcome measurement: Concepts and questions. Inquiry, 25(Spring), Morris, j. N., Hawes, C, Fries, B. E., Phillips, C. D., Mor, V., Katz, S., Murphy, K., Drugovich, M. L, & Friedlob, A. S. (1990). Designing the national resident assessment instrument for nursing homes. The Cerontologist, 30, Morris, J. N., Hawes, C, Murphy, K., Nonemaker, S., Phillips, C. D., Fries, B. E., & Mor, V. (Eds.). (1991). Resident assessment instrument training manual and resource guide. Natwick, MA: Eliot Press. Rubin, H. R., & Wu, A. W. (1992). The risk of adjustment. Medical Care, 30, Sainfort, F., Ramsay, J., & Monato, H. (1995). Conceptual and methodological sources of variation in the measurement of nursing facility quality: An evaluation of 24 models and an empirical study. Medical Care Research and Review, 52, Shaughnessy, P. W., Crisler, K. S., Schenkler, R. E., Arnold, A. G., Kramer, A. M., Powell, M. C, & Hittle, D. F. (1994). Measuring and assuring the quality of home health care. Health Care Financing Review, 76, Shaughnessy, P. W., Kramer, A. W. Hittle, D. F., & Steiner, J. F. (1995). Quality of care in nursing homes: Findings and implications. Health Care Financing Review, 76, Thomas, J. W., Holloway, J. L, & Guire, K. E. (1993). Validating risk-adjusted mortality as an indicator for quality of care. Inquiry, 30(Spring), Zimmerman, D. R., Karon, S. L, Arling, G., Ryther-Clark, B., Collins, T., Ross, R., & Sainfort, F. (1995). Development and testing of nursing home quality indicators. Health Care Financing Review, 76, Zinn, J. S., Aaronson, W. E., & Rosko, M. D. (1993). The use of standardized indicators as quality improvement tools: An application in Pennsylvania nursing homes. American Journal of Medical Quality, 8, Received September 30,1996 Accepted July 8,1997 New Publications Director Heather Worley has joined The Gerontological Society of America as its new Director of Publications. Questions concerning the publications may be addressed to her. Bettie Donley retired from the GSA on October 31, following 10 years of service. 766 The Gerontologist