The Melville-Nelson Self-Care Assessment (SCA) is a new tool within a developing

Transcription

1 Interrater Reliability, Concurrent Validity, Resonsiveness, and Predictive Validity of the Melville-Nelson Self-Care Assessment David L. Nelson, Lisa Link Melville, Julie D. Wilkerson, Robyne A. Magness, Jennifer L. Grech, Jamie A. Rosenberg KEY WORDS aged occuational erformance rehabilitation skilled nursing facilities OBJECTIVE. This study examines sychometric characteristics of the Melville-Nelson Self-Care Assessment (SCA), which is designed for occuational theraists working in skilled nursing facilities (SNFs) and subacute rehabilitation. Like the federally mandated Minimum Data Set, the SCA assesses seven self-care occuations in two ways: self-erformance and suort needed. METHOD. Particiants were SNF atients receiving subacute rehabilitation. Primary and secondary diagnoses varied widely; mean age was 76.5 years (SD = 9.8). Using the SCA, the Functional Indeendence Measure (FIM ), and the Klein-Bell Activities of Daily Living Scale, four teams of raters indeendently and simultaneously assessed 54 women and 14 men at admission and discharge; 40 articiants were available for reassessment in their homes after discharge. RESULTS. Total self-erformance scores on the SCA showed (a) interrater reliability (mean ICC =.94); (b) concurrent validity with relevant areas of the FIM (rho =.85) and the Klein-Bell (rho =.86); (c) resonsiveness in detecting change from admission to discharge, t(67) = 6.3, <.001, with a large effect size of 1.10; and (d) moderate redictive validity of caregiving time in the home (rho =.44) and overall function in the home as measured by the total FIM (rho =.55) and total Klein Bell (rho =.54) scores. The self-erformance and suort measures for the seven self-care areas all had accetable interrater reliability and concurrent validity. Eating, ersonal hygiene, and bathing showed little or no resonsiveness to change from admission to discharge, whereas ositive change was substantial in bed mobility, transfers, dressing, and toileting. CONCLUSION. This instrument is recommended for use as art of a comrehensive occuational theray model of ractice in SNFs and subacute rehabilitation. Nelson, D. L., Melville, L. L., Wilkerson, J. D., Magness, R. A., Grech, J. L., & Rosenberg, J. A. (2002). Interrater reliability, concurrent validity, resonsiveness, and redictive validity of the Melville-Nelson Self-Care Assessment. American Journal of Occuational Theray, 56, David L. Nelson, PhD, OTR/L, FAOTA, is Professor of Occuational Theray, Medical College of Ohio, 3015 Arlington Avenue, Toledo, Ohio Lisa Link Melville, MS, OTR/L, is Occuational Theraist, Lake Park, ProMedica Health System, Sylvania, Ohio. Julie D. Wilkerson, MOT, OTR/L, is Occuational Theraist, Bay Park Community Hosital, ProMedica Health System, Oregon, Ohio. Robyne A. Magness, MOT, OTR/L, is Occuational Theraist, Kettering City Schools, Kettering, Ohio. Jennifer L. Grech, MOT, OTR/L, is Clinical Director, Avalon Rehabilitation and HealthCare, Chagrin Falls, Ohio. Jamie A. Rosenberg, MOT, OTR/L, is Director of Occuational Theray, Indian Hills Health and Rehabilitation, Euclid, Ohio. The Melville-Nelson Self-Care Assessment (SCA) is a new tool within a develoing occuational theray model of ractice for skilled nursing facilities (SNFs) and subacute rehabilitation. Basic rinciles of this model of ractice include (a) consistency with the Concetual Framework for Theraeutic Occuation (CFTO; Nelson, 1994, 1996), (b) design for evidence-based develoment of the model through research, and (c) comatibility with the regulatory and administrative constraints of U.S. settings for subacute rehabilitation. Theoretically, within the CFTO are two ways to evaluate the success of a erson s occuation: objectively and subjectively (Nelson, 1994, ). Objective success is defined as occuational erformance and imact that match sociocultural criteria in the occuational form. For examle, the culture rovides normative criteria for aroriate dressing, and a theraist can use a standardized instrument to assess whether a atient s erformance matches those norms. Subjective success, on the other hand, is defined as occuational erformance and imact that match the erson s own uroses. For examle, a atient s subjective urose might involve a unique goal, such as donning a ersonally meaningful brooch or decorative belt. The American Journal of Occuational Theray 51

2 Some occuational theray models of ractice emhasize subjective success; others emhasize objective success; and still others emhasize combinations of the two (Nelson, 1996). In our develoing occuational theray model of ractice for SNFs and subacute rehabilitation, both objective success and subjective success are imortant in the theraeutic rocess. The SCA is designed to rovide an objective measure of self-care for occuational theraists to use in these settings (Nelson & Melville, 2001). SNFs roviding subacute rehabilitation in the United States oerate within a regulatory framework that mandates the use of the Minimum Data Set (MDS; Health Care Financing Administration [HCFA], 1995). The MDS is used by an interdiscilinary team to document many characteristics in addition to self-care. Because the MDS is so broad in scoe, its self-care section does not rovide the deth and detail necessary for occuational theray evaluation and goal setting. As a roject officer of the HCFA (1995) stated, The MDS version 2.0 remains a symbol of comromise robably less information than we would like to have, but clearly an imrovement (reface-. 2). Occuational theraists need a new assessment of self-care that is detailed enough for rofessional ractice yet comatible with the MDS. Prior research confirms the imortance of self-care to occuational theray ractice in these settings. In a survey of randomly samled occuational theraists working in long-term-care facilities, Nelson and Glass (1999) found that occuational theraists contributed to the MDS section on self-care much more than any other asect of the MDS. The goals of the SCA are (a) structural comatibility with the MDS; (b) facilitation of goal writing and documentation of rogress toward goals within the regulatory context; (c) racticality within the time constraints ractitioners face; and (d) sychometric reliability and validity. No other assessment meets these goals. Some well-established measures of self-care are efficient for eidemiological research but lack the necessary detail and resonsiveness that theraists need when writing goals and documenting rogress. For examle, the Katz Index of Activities of Daily Living (Katz, 1983) measures on a 7-oint scale a erson s level of indeendence in six self-care tasks. The erson could show marked imrovement but no indeendence in several areas, but these imrovements would not result in a higher score. In contrast, the Klein-Bell Activities of Daily Living Scale (Klein & Bell, 1982) rovides a much more detailed analysis of self-care and other basic functional abilities, with 170 items relevant to dressing, elimination, mobility, bathing and hygiene, eating, and communication. However, the multiage format and scoring rocedure make the scale difficult to use in daily ractice. The Functional Indeendence Measure (FIM 1 ; Uniform Data System for Medical Rehabilitation, 1992) is widely used in rehabilitation settings. It consists of 18 items that measure self-care, transfers, shincter control, locomotion, and social cognition. Each item is rated on a 7-oint ordinal scale. A great advantage of the FIM is the wealth of research documenting its sychometric characteristics (Granger, Cotter, Hamilton, & Fiedler, 1993; Ottenbacher, Hsu, Granger, & Fiedler, 1996). The major roblem of using the FIM in SNFs and subacute rehabilitation is that self-care items do not transfer easily from the FIM system to the MDS system, which is the regulatory centeriece for these sites of care (Williams, Li, Fries, & Warren, 1997). For examle, the MDS requires that each self-care task be rated on two different scales: self-erformance (how much the erson did) and suort (the degree of assistance needed from others). Each self-care task on the FIM is rated on a single scale. In addition, the tasks are defined differently on the two instruments. For examle, the item for bathing on the MDS includes the ability to transfer, whereas the FIM rates the bath transfer on a searate scale from the bathing scale. These tyes of structural inconsistencies make it difficult for the MDS scorer to translate from the FIM system to the required MDS system. The current study investigates the following sychometric roerties of the SCA: (a) interrater reliability, (b) concurrent validity with the FIM and the Klein-Bell ADL Scale, (c) resonsiveness in measuring change from admission to discharge, and (d) redictive validity in terms of caregiving time and function after discharge in the home. Method Particiants Particiants were atients receiving subacute rehabilitation at a midwestern SNF with 90 beds assigned to subacute care. Inclusion criteria were broad: (a) admission to the facility with a hysician s order for subacute rehabilitation, including occuational theray; (b) informed consent; (c) assignment by nursing staff to the caseload of one of the four occuational theraists articiating in the study, who conducted the rimary testing rocedures; and (d) availability of one of the four occuational theray graduate students, who conducted the interrater reliability rocedures. Potential articiants otherwise meeting the inclusion criteria were excluded if the initial care lan definitely called for discharge to a long-term-care facility as oosed to the ossibility of discharge to the home. Initial testing was con- 1 FIM is a trademark of the Uniform Data System for Medical Rehabilitation, a division of UB Foundation Activities, Inc. 52 January/February 2002, Volume 56, Number 1

3 ducted over a eriod of 214 days. Of the 75 atients roviding informed consent, 68 could be tested at both admission and discharge (droouts occurred because of acute illnesses with unexected discharge to other facilities and because of students schedule conflicts). These 68 articiants constituted the final samle for the tests of interrater reliability, concurrent validity, and resonsiveness. Of these 68, 40 were available for the redictive validity testing conducted in the home after discharge (droouts occurred because of rehositalization, discharge to long-term-care facilities, moving out of state, refusal of home visit or unavailability for telehone calls, and death). See Table 1 for a descrition of the articiants. Instruments Melville-Nelson SCA. As with the MDS 2.0, seven self-care occuations (bed mobility, transfers, dressing, eating, toileting, ersonal hygiene, bathing) are rated in two ways: selferformance (how much the erson did) and suort (the degree of assistance needed from others) (Nelson & Melville, 2001). For self-erformance, each self-care occuation is analyzed in terms of three to nine suboccuations. Table 1. Descrition of the Particiants Interrater, Concurrent Validity, Predictive Variable and Resonsiveness Tests a Validity Tests b Age (years) Range M SD Gender Female Male 14 8 Race Caucasian African-American 4 3 Primary diagnosis Cardiac diseases 10 2 Resiratory diseases 6 3 Osteoarthritis with TKA c 5 5 Osteoarthritis with THA d 5 5 Femur fracture 5 5 Rheumatoid arthritis 2 2 Tibia fracture 3 2 Vertebral fracture 2 2 Cerebrovascular accident 2 2 Infection: rosthesis 2 1 Sinal stenosis 2 1 Other e Number of secondary diagnoses Range M a N = 68. b n = 40. c Total knee arthrolasty. d Total hi arthrolasty. e Singular rimary diagnoses were ankle fracture, humeral fracture, rib fracture, toe fracture, hi relacement, shoulder relacement, osteoarthritis, joint inflammation, diabetes, hernia, ulcer, staholoccocus infection, seticemia, urinary infection, cellulitis, osteomyelitis, hyercholesterolemia, intestinal obstruction, myoneural disorder, gangrene, hallucinations with cardiac disease, neurotic disorder with osteoarthritis, foreign body in soft tissue, and fluid in abdomen. For examle, the occuation ersonal hygiene consists of hair combing and brushing, toothbrushing, denture care, hands washing, face washing, shaving, and alying makeu. Each suboccuation is further analyzed in terms of four sub-suboccuations. For examle, hair combing and brushing erformance is rated searately for front, right, left, and back of the head. The rater observes the articiant while engaged in self-care and judges whether he or she is able to erform each art of the task indeendently. If the articiant is not indeendent in a sub-suboccuation, a score of 1 is tallied for that self-care occuation. For examle, if a articiant can erform all asects of ersonal hygiene excet for one (combing and brushing the back of the head), he or she receives a score of 1 in ersonal hygiene. The ossible range of scoring for self-erformance varies across the seven self-care occuations (from 0 9 for transfers, from 0 36 for dressing). The total score across all seven self-care occuations ranges from 0 to 140, with relatively large numbers indicating dysfunction. If a suboccuation or a sub-suboccuation is not relevant to a articiant, it is not scored (e.g., colostomy care). In this way, articiants were not enalized for failure in irrelevant items. The suort score is drawn directly from the MDS 2.0 with one addition. Each of the seven self-care occuations is rated as 0 (no setu or hysical hel), 1 (setu or standby assistance needed without other assistance), 2a (only contact guard required), 2b (one-erson hysical hel required), or 3 (two-erson hysical hel required). The difference between the SCA suort score and the MDS rating system is the addition of the contact guard category. A 2a or 2b score can be used by the MDS coordinator as a 2, but for statistical uroses in the current study, a 2b is converted into a 3, and a 3 is converted into a 4. FIM. The FIM consists of 18 items rated on a 7-oint ordinal scale. The following items are relevant in the current study: eating; grooming; bathing; dressing uer body; dressing lower body; toileting; transfers involving bed, chair, and wheelchair; transfers involving toilet; and transfers involving tub or shower. Hamilton, Laughlin, Granger, and Kayton (1991) demonstrated the interrater reliability of the FIM. In a samle of 263 rehabilitation inatients, the ICC was.97 for the total score. Kaa for individual items ranged from.61 to.76. Roth, Davidoff, Haughton, and Ardner (1990) studied concurrent validity of the FIM with the Barthel Index and reorted correlations above.90 at admission and at discharge. Kidd et al. (1995) also comared the FIM with the Barthel Index and dichotomized the data so that kaa could be comuted (kaa =.92 at admission and.88 at discharge). Granger et al. (1993) reorted a correlation of.81 between the total FIM score and duration of hel needed er day. The American Journal of Occuational Theray 53

4 Klein-Bell ADL Scale (Klein & Bell, 1979). This scale was chosen as a second criterion measure because it is an established, sensitive measurement of self-care that has been studied for reliability and validity. The Klein-Bell contains 170 items that measure erformance in dressing, elimination, mobility, bathing and hygiene, eating, and communication. Overall, 126 items are relevant to the seven self-care areas assessed by the SCA. Each Klein-Bell item is scored in terms of whether the articiant is able or unable (requiring hysical or verbal assistance). The scorer assigns weights of 1, 2, or 3 to each item. These weights were derived emirically from ratings assigned by a grou of 10 occuational theraists, hysical theraists, and nurses. Similar to the SCA, the Klein-Bell comutes a score for each major area of self-care and a total score across all areas. Interrater reliability of the Klein-Bell was tested among three airs of occuational theraists and three airs of rehabilitation nurses on 20 atients. For all items on all atients, interrater agreement was 92% (Klein & Bell, 1982). It should be noted that kaa would have been a suerior method of testing for interrater agreement because it accounts for chance agreements. The Klein-Bell is one of the few self-care scales to be studied for its redictive validity. Fourteen atients were contacted 5 to 10 months after discharge. The Pearson roduct-moment correlation between reorted number of hours er week of assistance needed in self-care and the Klein-Bell score at discharge was.86 (Klein & Bell, 1982). Caregiving time. An additional variable studied in terms of redictive validity was the duration of time sent by the rimary caregiver in assisting the articiant in self-care. Within 2 weeks after the self-care assessment in the home, a graduate student telehoned the rimary caregiver on 3 different days within a 5-day eriod. The student named the seven self-care areas and asked the caregiver to recall the articiant s self-care on the revious day. Then the student asked, How much time did you, or anyone else, send yesterday heling [the articiant] with these self-care tasks. The final score was the mean duration across the 3 days. Procedure Eight raters worked in four teams, each team consisting of a licensed occuational theraist and an occuational theray graduate student. All raters were trained in accordance with the assessment manuals. Admission and discharge scores on the SCA, FIM, and Klein-Bell were attained within 48 hr of admission or discharge. The four teams collected admission and discharge data on 15, 17, 17, and 19 articiants. One to 3 weeks after discharge, each available articiant was assessed on the three assessments in the home by the same team. This interval was chosen because articiants had enough time to adjust to the home environment but not so much time for the develoment of confounding factors, such as new illnesses. Raters in each team conducted all scoring indeendently, with no discussion of scores, with no visual information about the other s scoring (e.g., backs were turned or searate rooms were used), and with no access to the other s data forms. The SCA was scored first to rovide the best estimate of its interrater reliability and resonsiveness. (The lan of the study did not call for comarable analyses of the FIM and Klein-Bell.) The FIM and Klein-Bell were then scored in a randomly assigned order. Data Analysis The SCA generates 15 variables: 7 self-erformance scores, 1 total self-erformance score, and 7 suort scores. To test for interrater reliability, an ICC was calculated on the admission ratings of each team of raters on the 15 variables. The secific equation used was recommended by Shrout and Fleiss (1979) for evaluating agreement between two raters on an instrument that a single rater will use in future measurement (ICC 1,2); this is a conservative, random effects model. Eliasziw, Young, Woodbury, and Fryday-Field (1994) recommended that the ICC be interreted in terms of reliability as follows: 0 to.20, slight;.21 to.40, fair;.41 to.60, moderate;.61 to.80, substantial; and.81 to 1.00, almost erfect. Commonly, concurrent validity is tested by the correlation between a single rater s scores on two different instruments. However, a rater s memory of scoring the first instrument might well bias the scoring of the second. Therefore, in the current study, concurrent validity was assessed by the occuational theraist s rating of the admission FIM or Klein-Bell scale and the student s rating of the admission SCA. (The same articiant was assessed on two different instruments by two different raters.) This rocedure eliminated the ossible bias of memory and rovided a conservative estimate of concurrent validity. Scores for each of the 15 SCA variables were correlated with relevant items of the Klein-Bell and FIM. For sulementary uroses, the correlations between the SCA total self-erformance and the total 18-item FIM and the total 170-item Klein-Bell also were calculated. We anticiated that the distributions for the SCA would be moderately skewed, with few articiants exhibiting high levels of deendence. Therefore, Searman rank order correlations (rho) were lanned. Guilford and Fruchter (1978) wrote that a criterion validity coefficient (r) may be exected in the range between 0 and.60, with most indices in the lower half of that range (. 87). The current study used Cohen s (1988) suggestions for interreting the effect size of r, where.10 is a slight effect,.30 is a medium effect, and.50 is a large effect ( ). 54 January/February 2002, Volume 56, Number 1

5 Resonsiveness of the SCA from admission to discharge was assessed in three ways: the t test for related measures (admission vs. discharge scores); the effect size for the t test for related measures as calculated by Cohen (1988,. 48); and the effect size recommended for testing resonsiveness by Stratford, Binkley, and Riddle (1996). Only the occuational theraists scores were analyzed. Cohen argued that the t statistic is robust to moderate deartures from the normal distribution when calculating effect sizes (. 19). Cohen s effect size is calculated as the change score divided by the standard deviation, divided by the square root of the difference between 1 and the correlation between the admission score and the discharge score (Cohen, 1988,. 48). Cohen argued that.20 is a small effect,.50 is a moderate effect, and.80 is a large effect. Stratford et al. (1996) recommended calculating the effect size as the change score divided by the standard deviation of the admission score. This rocedure is usually more conservative than Cohen s method because it does not take into account the correlation between the two measures. Sulementary analyses involved the correlations between SCA admission scores and discharge scores to gather information about the within-articiants stability of the instrument. For redictive validity, the Searman rank ordered correlations between SCA total self-erformance at discharge (as measured by the theraist) and the following five variables were calculated: the mean duration of caregiving time er day; the sum of all the relevant FIM items (as measured by the student); the total FIM score (as measured by the student); the sum of all the relevant Klein- Bell items (as measured by the student); and the total Klein-Bell score (as measured by the student). The reason for including total FIM and total Klein-Bell scores was to test the overall effectiveness of the SCA in redicting function, not just self-care. Sulementary redictive validity analyses were conducted for the other 14 SCA variables. Cohen s (1988) rules for interreting r in terms of effect sizes were used. Probability () values must be interreted with caution because of the many tests conducted and because a significant correlation does not necessarily imly a large effect. Because the SCA involves 15 variables, was judged as significant when less than.05/15 =.0033 in the analyses of concurrent validity and resonsiveness. Because five rimary analyses were lanned in the analysis of redictive validity, alha was set at.05/5 =.01. All tests were one-tailed. Negative correlations were exected between the SCA and the other measures of function because a high score on the SCA indicates dysfunction. A ositive correlation was exected between the SCA and caregiving time. Table 2. Interrater Reliability (ICCs) of SCA Admission Scores Across Four Pairs of Raters Variable M Range Self-erformance Total a Bed mobility Transfers Dressing Eating b Toileting Personal hygiene Bathing Suort Bed mobility Transfers Dressing Eating Toileting Personal hygiene Bathing Note. N = 68, with n ranging from 15 to 19 across four airs of raters. SCA = Melville-Nelson Self-Care Assessment. a Sum of seven areas. b ICCs for two airs of raters only because all articiants rated by the other two airs of raters had erfect scores, with no variance to be analyzed. Results Interrater Reliability Table 2 shows that the SCA total self-erformance score has excellent interrater reliability (mean ICC =.94). Six of the seven mean ICCs for secific self-erformance areas are in the almost erfect range identified by Eliasziw et al. (1994), with bathing at the high end of the substantial range (ICC =.77). Interrater reliability for the seven suort scores is somewhat lower, but six are still in the substantial or almost erfect ranges. Bathing suort demonstrates moderate interrater reliability. Some of the four rater teams were able to agree at higher levels than others, which is esecially evident for some suort scores where interrater reliability slied into the fair range for individual teams in the ersonal hygiene and bathing suort areas. In contrast, all 28 ICCs generated by the individual teams in the self-erformance areas were in the substantial or almost erfect range. Concurrent Validity Table 3 shows that total self-erformance on the SCA is highly correlated with relevant areas of the FIM and Klein- Bell. Not included in Table 3 but also large are the correlations between SCA total self-erformance and total scores of the FIM (.85) and Klein-Bell (.85). Therefore, the SCA total self-erformance score rovides a valid measure of function as defined by the FIM. The ossibility of memory bias cannot account for this finding because the correlations are based on indeendent raters. Table 3 also shows that the correlations between secific SCA self-erformance areas and relevant areas of the FIM and Klein-Bell are most- The American Journal of Occuational Theray 55

6 Table 3. Concurrent Validity of SCA Self-Performance With Relevant Areas of the FIM and the Klein-Bell FIM Klein-Bell ADL Scale Variable rho (One-Tailed) rho (One-Tailed) Total.86 a < i <.001 Bed Mobility b < j <.001 Transfers.68 c < k <.001 Dressing.81 d < l <.001 Eating.42 e < m <.001 Toileting.81 f < n <.001 Personal hygiene.45 g < o <.001 Bathing.31 h <.001 Note. N = 68, with n ranging from 15 to 19 across four airs of raters. Correlations are based on the students ratings of the Melville-Nelson Self-Care Assessment (SCA) and the theraists ratings of the Functional Indeendence Measure (FIM) and the Klein-Bell Activities of Daily Living Scale. a Sum of six FIM self-care items and three transfer items. b Not assessed on FIM. c FIM transfers: bed, chair, wheelchair. d Sum of FIM dressing (uer, lower). e FIM eating. f Sum of FIM toileting and transfers: toilet. g FIM grooming. h Sum of FIM bathing and transfers: tub, shower. i Sum of Klein-Bell items 8 66, 68 79, 81 93, 102, , and j Sum of Klein-Bell items k Sum of Klein-Bell items 93 and 102. l Sum of Klein-Bell items m Sum of Klein-Bell items n Sum of Klein-Bell items 66 and and o Sum of Klein-Bell items Sum of Klein-Bell items ly in the large effect range, as defined by Cohen (1988, ). The lowest correlations are in the areas of eating, ersonal hygiene, and bathing, but even in these areas the correlations reflect medium effects or larger. Table 4 shows a similar attern of effect sizes when SCA suort areas are correlated with relevant areas of the FIM and Klein-Bell. All effect sizes between SCA suort variables and Klein-Bell variables are in the large range, and only SCA ersonal hygiene and bathing have correlations of less than.50 with relevant areas of the FIM. Table 4. Concurrent Validity of SCA Suort With Relevant Areas of the FIM and the Klein-Bell FIM Klein-Bell ADL Scale Variable rho (One-Tailed) rho (One-Tailed) Bed mobility a.77 h <.001 Transfers.78 b < i <.001 Dressing.66 c < j <.001 Eating.60 d < k <.001 Toileting.81 e < l <.001 Personal hygiene.41 f < m <.001 Bathing.45 g < n <.001 Note. N = 68, with n ranging from 15 to 19 across four airs of raters. Correlations are based on the students ratings of the Melville-Nelson Self- Care Assessment (SCA) and the theraists ratings of the Functional Indeendence Measure (FIM) and the Klein-Bell Activities of Daily Living Scale. a Not assessed on FIM. b FIM transfers: bed, chair, wheelchair. c Sum of FIM dressing (uer, lower). d FIM eating. e Sum of FIM toileting and transfers: toilet. f FIM grooming. g Sum of FIM bathing and transfers: tub, shower. h Sum of Klein-Bell items i Sum of Klein-Bell items 93 and 102. j Sum of Klein-Bell items k Sum of Klein-Bell items l Sum of Klein- Bell items 66 and and m Sum of Klein-Bell items n Sum of Klein-Bell items Resonsiveness Table 5 shows that SCA total self-erformance is highly resonsive to change from admission to discharge when judged by Cohen s (1988) criteria. The effect size as calculated by Stratford et al. (1996) is lower, however, because it does not take into account the high correlation between admission and discharge scores. Resonsiveness to change tends to be greatest in the areas of transfers, dressing, and toileting whether judged by self-erformance scores or by suort scores. Resonsiveness to change is lowest in eating self-erformance, ersonal hygiene self-erformance, bathing self-erformance, and eating suort. Of the 15 SCA variables, 11 showed significant change from admission to discharge when alha is set at the conservative level of It should be noted that the distribution of the SCA at both admission and discharge tended to be skewed, as anticiated. In many cases, the standard deviations are larger than the means, which indicates that most articiants had relatively low scores, with a few outliers having relatively high scores. Predictive Validity Table 6 shows that the SCA total self-erformance score at discharge is a significant redictor of caregiving time in the home after discharge and of function as measured by the FIM and Klein-Bell. The effect size concerning caregiving time aroaches large as defined by Cohen (1988) and exceeds Cohen s criterion for large with regard to FIM and Klein-Bell scores. The 40 articiants available for follow-u assessment in the home had less dysfunction at discharge (the mean SCA total self-erformance score = 5.6) than the inclusive set of 68 articiants deicted in Table 5 (the mean total self-erformance score = 15.8). Sulementary Analyses The SCA variables that were best at redicting caregiving time in the home were toileting self-erformance (rho =.63) and toileting suort (rho =.52). The SCA variables that were best at redicting overall function in the home, as measured by the total FIM, were transfers self-erformance (rho =.51), toileting suort (rho =.53), and ersonal hygiene suort (rho =.57). The same three variables were the best redictors of the total Klein-Bell score at home (rho =.57,.56,.52, resectively). SCA admission scores tended to be strongly correlated with SCA discharge scores (rho =.73 for total self-erformance, mean rho across the 14 other SCA variables =.54). Eating self-erformance (rho =.26) and bathing self-erformance (rho =.39) were least stable. All correlations were significant at the.0033 level excet eating self-erformance. 56 January/February 2002, Volume 56, Number 1

7 Table 5. Resonsiveness of the SCA in Detecting Change From Admission to Discharge Admission Discharge Change SBR Cohen One-Tailed Effect Effect Variable M SD M SD M SD Paired t () Size a Size b Self-erformance Total (<.001) Bed mobility (.002) Transfers (<.001) Dressing (<.001) Eating (.359) Toileting (<.001) Personal hygiene ( c ) Bathing ( c ) Suort Bed mobility (<.001) Transfers (<.001) Dressing (<.001) Eating (<.042) Toileting (<.001) Personal hygiene (<.001) Bathing (<.001) Note. N = 68, with n ranging from 15 to 19 across four airs of raters. Positive change scores reresent increases in self-care abilities. SCA = Melville-Nelson Self-Care Assessment. a Stratford, Binkley, and Riddle (1996) recommended calculating the effect size as the change score divided by the standard deviation of the admission score. b Cohen s effect size is calculated as the change score divided by the standard deviation, divided by the square root of the difference between 1 and the correlation between the admission score and the discharge score (Cohen, 1988,. 48). c Not meaningful for a one-tailed test. Discussion Reliability and Validity of Total SCA Self-Performance Scores Table 6. Predictive Validity of the SCA Total Self-Performance at Discharge a in Terms of Caregiving and Self-Care After Discharge in the Home b Descritive Correlation With SCA Information at Discharge Variable M SD rho (One-Tailed) Caregiving time in home (min/day) FIM self-care and transfers in homec <.001 FIM total score in home <.001 Klein-Bell self-care in homed <.001 Klein-Bell total score in home <.001 Note. n = 40 excet for caregiving time in home, where n = 39. The mean total erformance score on the Melville-Nelson Self-Care Assessment (SCA) at discharge (n = 40) was 5.6 (SD = 8.1). FIM = Functional Indeendence Measure. a Measured by occuational theraists. b Measured by student. c Sum of six FIM self-care items and three transfer items. d Sum of Klein-Bell items 8 66, 68 79, 81 93, 102, , and This study shows that the total self-erformance scores on the SCA are objective, sensitive measures of self-care that have demonstrated validity when tested against the FIM and Klein-Bell as criterion measures. In addition, SCA total self-erformance score at discharge is a significant redictor of caregiving time and function in the home after discharge. Many articiants, esecially those who were discharged home, attained erfect scores or near-erfect scores on the SCA at discharge, with few showing many roblems in self-care. Given this evidence of a ceiling effect for discharge self-care scores, we believe that resonsiveness and redictive validity would have been even greater if more challenging occuations had been assessed. The roblem of a ceiling effect in the SCA reflects its adherence to the MDS. A measure of self-care as mandated by the MDS might be judged adequate in assessing function in ersons receiving long-term care, but it should be sulemented by measures of relatively advanced occuation in ersons who will return to their homes. Another factor that likely inhibited redictive validity is the fact that many events taking lace after discharge might have influenced scores in the home. For examle, articiants robably received different levels of social suort and health care services. Reliability and Validity of the 14 SCA Self-Performance and Suort Scores First and most imortantly, all 14 scores of the SCA can be judged as objective measures on the basis of conservative tests of interrater reliability. Generally, interrater reliability for SCA self-erformance scores is somewhat higher than for SCA suort scores. This relatively high reliability might be due to more otential variance in the self-erformance measures than in the suort scores, which have a maximum range of only 5. Enhanced variance makes relatively fine discriminations by raters ossible. Concurrent validity estimates are also accetable for The American Journal of Occuational Theray 57

8 both self-erformance and suort. Somewhat lower correlations between the SCA scores on eating, ersonal hygiene, and bathing and the FIM and Klein-Bell scores can be exlained in three ways: (a) the SCA (like the MDS) involves somewhat different rules for measuring these areas; (b) these areas also are somewhat less reliable; and (c) because many articiants were indeendent or almost indeendent in eating and ersonal hygiene at admission, little variance existed in these variables. The same attern of some relatively strong measures and some relatively weak measures is seen in the resonsiveness estimates. Major change occurred from admission to discharge in transfers, dressing, and toileting. Change in bed mobility was significant for the overall samle, but many articiants could show no imrovement because they had no roblems of bed mobility at admission. However, little or no imrovement occurred in the areas of eating, ersonal hygiene, and bathing self-erformance. For eating and ersonal hygiene, the lack of ositive change is robably due to a ceiling effect. Table 5 shows that articiants had few roblems in this area at admission; therefore, there was little room for imrovement. For bathing, the lack of ositive change might have been because occuational theray does not focus on bathing as much as on other areas of self-care in the study facility. Other exlanations are lausible, such as the relatively low interrater reliability for bathing. Overall, one way of interreting the resonsiveness scores is that the SCA is resonsive to change in six of the seven areas of self-care in terms of either self-erformance or suort. Interestingly, the resonsiveness estimates for suort variables tend to be higher than resonsiveness estimates for self-erformance variables. Even though the suort scores have less otential variance and lower interrater reliability than self-erformance scores, suort scores have less of a ceiling effect because the need for setu is considered when measuring suort but not when measuring self-erformance. Persons who need setu hel but no other hel do not receive erfect scores on suort variables, but they might receive erfect scores on self-erformance. This rule of measurement for self-erformance and suort is drawn from the MDS. The lack of resonsiveness in some SCA variables aears to be a consequence of its adherence to rules drawn from the MDS. The MDS mandates the measure of eating as well as mandates what should be measured. Relatively difficult suboccuations, such as oening containers and ouring hot liquids, are not assessed on the MDS, so they are not assessed on the SCA. We believe that the disadvantage of unresonsiveness in a few areas is outweighed by the advantage of SCA consistency with the MDS. Secial Advantages of the SCA Why should an occuational theraist consider using the SCA as a routine art of clinical ractice in SNFs? First, SCA consistency with the MDS is advantageous in terms of interdiscilinary communication. Second, resource utilization guidelines governing reimbursement emhasize selfcare as defined by the MDS. Third, the SCA is efficient, yet detailed because it uses time-saving methods such as documentation by exclusion and rinted forms that are laid out carefully for the theraists convenience. Fourth, the SCA leads directly to goal writing whereby small but meaningful imrovements in self-care can be anticiated. Fifth, the SCA has arallel forms for assessing change in weekly rogress notes and at discharge. Limitations and Directions for Future Research Although samle sizes were adequate to confirm the overall reliability and validity of the SCA, even stronger estimates could be exected with the added ower rovided by a larger samle. A ositive feature of the current study is that the sychometric roerties of the SCA were tested across four teams of raters; however, each team could assess only 15 to 19 articiants. Another limitation involved the airing of graduate students with theraists in the teams. Students without extensive clinical exerience might not have rated comlex areas such as bathing with the same accuracy as theraists, thus lowering interrater reliability somewhat. Additionally, recruitment of the samle deended on student availability, so many otential articiants did not articiate in the study. Future study could address these limitations and include articiants for whom discharge to the community is considered unlikely in the initial care lan. Occuational theraists working in SNFs frequently rovide services for ersons who will be long-term residents. These ersons tend to have greater self-care deficits than those who are exected to return home; therefore, the ceiling effects found in the current study likely will be less roblematic in data collected on long-term residents. Future study also should focus on other asects of reliability and validity. For examle, statistical examination through factor analysis and Rasch analysis could clarify the relationshis of sub-suboccuations to each other and the relationshis of the seven self-care areas to overall self-erformance. Studies of clinical utility, including cost and time, also are needed. Conclusion We view the SCA as a first ste toward the develoment of an integrated system of occuational theray evaluation. It should not be used as the only assessment. 58 January/February 2002, Volume 56, Number 1

9 Occuational theraists are urged to use the SCA in conjunction with relatively challenging measures of objective success (e.g., instrumental occuations of daily living) and with atients self-reorts of subjective success in valued occuations. Future research and ublications are necessary to describe the role of the SCA as art of an overall model of ractice for occuational theray in SNFs and subacute rehabilitation. Acknowledgments We thank the American Occuational Theray Foundation for an Innovation Grant in suort of this research, and Wendi Sargent, OTR/L, Shelby VanBrandt, OTR/L, Amy Henning, OTR/L, and other staff members and administrators of Lake Park, ProMedica Health System, Sylvania, Ohio, for their many contributions to this study. References Cohen, J. (1988). Statistical ower analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum. Eliasziw, M., Young, S. L., Woodbury, M. G., & Fryday-Field, K. (1994). Statistical methodology for the concurrent assessment of interrater and intrarater reliability: Using goniometric measurements as an examle. Physical Theray, 74, Granger, C. V., Cotter, A. C., Hamilton, B. B., & Fiedler, R. C. (1993). Functional assessment scales: A study of ersons after stroke. Archives of Physical Medicine and Rehabilitation, 74, Guilford, J. P., & Fruchter, B. (1978). Fundamental statistics in sychology and education (6th ed.). New York: McGraw-Hill. Hamilton, B. B., Laughlin, J. A., Granger, C. V., & Kayton, R. M. (1991). Interrater agreement of the seven level Functional Indeendence Measure (FIM) [Abstract]. Archives of Physical Medicine and Rehabilitation, 72, 790. Health Care Financing Administration. (1995). Long term care facility resident assessment instrument (RAI) user s manual. Natick, MA: Eliot. Katz, S. (1983). Assessing self-maintenance: Activities of daily living, mobility, and instrumental activities of daily living. Journal of the American Geriatrics Society, 31, Kidd, D., Stewart, G., Baldry, J., Johnson, J., Rossiter, D., Petruckevitch, A., & Thomson, A. J. (1995). The Functional Indeendence Measure: A comarative validity and reliability study. Disability and Rehabilitation, 17, Klein, R. M., & Bell, B. J. (1979). Klein-Bell Activities of Daily Living Scale. Seattle, WA: University of Washington. Klein, R. M., & Bell, B. (1982). Self care skills: Behavioral measurement with Klein-Bell ADL scale. Archives of Physical Medicine and Rehabilitation, 63, Nelson, D. L. (1994). Occuational form, occuational erformance, and theraeutic occuation. In C. B. Royeen (Ed.), AOTA Self-Study Series: The ractice of the future: Putting occuation back into theray (Lesson 2,. 9 48). Rockville, MD: American Occuational Theray Association. Nelson, D. L. (1996). Theraeutic occuation: A definition. American Journal of Occuational Theray, 50, Nelson, D. L., & Glass, L. M. (1999). Occuational theraists involvement with the Minimum Data Set in skilled nursing and intermediate care facilities. American Journal of Occuational Theray, 53, Nelson, D. L., & Melville, L. L. (2001). Melville-Nelson occuational theray evaluation system for skilled nursing homes and subacute rehabilitation [Online]. (Available: htt:// Ottenbacher, K. J., Hsu, Y., Granger, C. V., & Fiedler, R. C. (1996). The reliability of the Functional Indeendence Measure: A quantitative review. Archives of Physical Medicine and Rehabilitation, 77, Roth, E., Davidoff, G., Haughton, J., & Ardner, M. (1990). Functional assessment in sinal cord injury: A comarison of the Modified Barthel Index and the adated Functional Indeendence Measure. Clinical Rehabilitation, 4, Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, Stratford, P. W., Binkley, J. M., & Riddle, D. L. (1996). Health status measures: Strategies and analytic methods for assessing change scores. Physical Theray, 76, Uniform Data System for Medical Rehabilitation, UB Foundation Activities, Inc. (1992). UDS: Theory and ractice. New York: Research Foundation State University of New York. Williams, B. C., Li, Y., Fries, B. E., & Warren, R. L. (1997). Predicting atient scores between the Functional Indeendence Measure and the Minimum Data Set: Develoment and erformance of a FIM MDS crosswalk. Archives of Physical Medicine and Rehabilitation, 78, The American Journal of Occuational Theray 59