A systematic review of interrater reliability of pressure ulcer classification systems

Transcription

1 REVIEW A systematic review of interrater reliability of pressure ulcer classification systems Jan Kottner, Kathrin Raeder, Ruud Halfens and Theo Dassen Aims. To review systematically the interrater reliability of pressure ulcer classification systems to find out which classification should be used in daily practice. Background. Pressure ulcer classification systems are important tools in research and practice. They aim at providing accurate and precise communication, documentation and treatment decisions. Pressure ulcer classifications are criticised for their low degree of interrater reliability. Design. Systematic review. Methods. The data bases MEDLINE, EMBASE, CINAHL and the World Wide Web were searched. Original research studies estimating interrater reliability of pressure ulcer classification systems were included. Study selection, data extraction and quality assessment was conducted independently by two reviewers. Results. Twenty-four out of 339 potentially relevant studies were included in the final data synthesis. Due to the heterogeneity of the studies a meaningful comparison was impossible. Conclusions. There is at present not enough evidence to recommend a specific pressure ulcer classification system for use in daily practice. Interrater reliability studies are required, in which comparable raters apply different pressure ulcer classification systems to comparable samples. Relevance to clinical practice. It is necessary to determine the interrater reliability of pressure ulcer classifications among all users in clinical practice. If interrater reliability is low the use of those systems is questionable. On the basis of this review there are no recommendations as to which system is to be given preference. Key words: classification, diagnosis, nurses, nursing, pressure ulcer, systematic review Accepted for publication: 3 July 2008 Introduction Pressure ulcers (PUs) are serious health problems (Allman 1997). In Europe the prevalence of pressure-related damage to the skin ranges from 10Æ5% in hospital patients (Bours et al. 2002) to 6Æ1% in nursing home residents (Lahmann et al. 2005). PUs are associated with pain and distress for the individuals affected and their treatment causes extensive health care costs (Graves et al. 2005). Furthermore, pressure-related injuries are considered as an important indicator for the quality of care (Calianno 2007). PU classification systems are important tools in PU research and Authors: Jan Kottner, MA, RN, Associate Professor, Department of Nursing Science, Centre for Humanities and Health Sciences, Charité-Universitätsmedizin Berlin, Berlin, Germany; Kathrin Raeder, RN, Master Student, Certified Wound Expert, Department of Nursing Science, Centre for Humanities and Health Sciences, Charité-Universitätsmedizin Berlin, Berlin, Germany; Ruud Halfens, PhD, Faculty of Health, Medicine and Life Sciences, Department of Health Care and Nursing Sciences, Universiteit Maastricht, Maastricht, The Netherlands; Theo Dassen, PhD, RN, Director of the Department of Nursing Science, Centre for Humanities and Health Sciences, Charité-Universitätsmedizin Berlin, Berlin, Germany Correspondence: Jan Kottner, Charité-Universitätsmedizin Berlin, Department of Nursing Science, Centre for Humanities and Health Sciences, Charitéplatz 1, Berlin, Germany. Telephone: jan.kottner@charite.de Ó 2009 The Authors. Journal compilation Ó 2009 Blackwell Publishing Ltd, Journal of Clinical Nursing, 18, doi: /j x

2 J Kottner et al. management. They aim at providing consistent and accurate PU assessment that promotes accurate communication, precise documentation and treatment decisions (Banks 1998, Stotts 2001). The first well documented PU classification system was proposed by Shea in Shea differentiated five categories: the category Closed Pressure Sore indicating a deep tissue injury beneath intact skin and the grades I IV with increasing numbers indicating more severe tissue damage (Table 1). During the following years this classification was modified several times and new classification systems (Table 1) were introduced with varying numbers of categories (Reid & Morison 1994, Healey 1996, Haalboom et al. 1997). The National Pressure Ulcer Advisory Panel of the United States (1989, 1997) and the European Pressure Ulcer Advisory Panel (1998) proposed the two most widely used PU classifications. Apart from some differences both classifications have a lot in common. The National Pressure Ulcer Advisory Panel (2007) has recently published updated definitions of PU grades. PU classification systems are generally criticised for their low level of interrater reliability, therefore questioning their use in practice, research and quality assurance (Haalboom et al. 1997, Russell 2002b, Dealey & Lindholm 2006). Interrater or interobserver reliability indicates the degree to which two or more independently operating raters or observers agree when rating the same subject or object. It is affected by properties of the measurement instrument (e.g. number of categories, operational definitions of items), qualification, knowledge and training of observers, conditions of observation and population characteristics (e.g. prevalence) (Kraemer 1979, Suen 1988, Shrout 1998). Although interrater reliability is a major criterion for assessing the quality and adequacy of an instrument, the criterion of validity is important as well. However, an instrument that is unreliable cannot be valid. A high level of interrater reliability is the prerequisite for validity (Shrout 1998, Polit & Beck 2004). At present there are many studies investigating the interrater reliability of different PU classifications, however, no attempts have been made to systematically review the available research evidence regarding their degree of interrater reliability. Aims The purpose of this study was to systematically review the interrater reliability among health care workers using PU classifications systems to find out which classification can be recommended for daily use. We also aimed at determining the factors affecting interrater reliability. Methods Search The search was conducted by the first investigator (JK). It was supported by a librarian specialised in medical databases. The databases MEDLINE (1965 June 2007), EMBASE (1989 June 2007) and CINAHL (1995 June 2007) were systematically searched. The search included synonyms for PU in combination with different terms for classification. To identify research studies, which examined PU classifications as part of larger studies or trials, we added terms for various cross-sectional or prospective study designs (Fig. 1). The search software WEB EBSPIRS Version 5.12 (Ovid Technologies, NY, USA) was used. Studies from reference lists that seemed to be relevant were obtained and analysed as well. In addition, Google was used to search the World Wide Web combining terms for PUs and classifications. Authors of conference abstracts found on the Internet were contacted for detailed information. No other steps were taken to locate unpublished material or grey literature. Study selection The results from the literature search were screened by the first (JK) and the second investigator (KR) independently by reading the title and/or the abstract. To gain as many relevant studies as possible we determined broad inclusion criteria: 1 Original research studies estimating interrater reliability of PU classifications 2 Language: English or German. Exclusion criteria were: 1 Retrospective study designs, reviews, discussion papers 2 Studies using data from patient records (chart review). Afterwards, the results were compared and any discrepancies discussed until an agreement was achieved. Data extraction Data from relevant studies were selected using a data extraction sheet. It contained: names of authors; year of publication; name of country or region; descriptions of PU classification systems including the number of categories; methods used to conduct the interrater reliability study; numbers, training and qualification of raters; numbers and characteristics of subjects or cases; numbers of observations per rater; inclusion of normal skin (yes/no). Calculated interrater reliability coefficients and 95% CI were taken over from the original study or calculated whenever raw data or 316 Ó 2009 The Authors. Journal compilation Ó 2009 Blackwell Publishing Ltd, Journal of Clinical Nursing, 18,

3 Review Pressure ulcer classifications Table 1 Description of pressure ulcer classifications included in the review Shea (1975) Surrey (1983) in Healey (1995) Torrance (1983) in Edwards & Banks (1999) Grade 1: Most apparent clinical presentation is an irregular, ill-defined area of soft tissue swelling and induration with associated heat and erythema overlying a bony prominence. Extreme Grade I is a moist superficial irregular ulceration limited to epidermis exposing underlying dermis and resembling an abrasion. Anatomic limit: Dermis Grade II: Clinically presented as a shallow full thickness skin ulcer whose edges are more distinct. Anatomic limit: Subcutaneous fat Stage 1: Nonblanching erythema Stage 2: Superficial break in the skin Grade 1: Area of blanching hyperaemia. Light finger pressure will cause blanching of the erythema Grade 2: Nonblanching hyperaemia. Erythema remains when light finger pressure is applied. Superficial damage may be present as blistering, induration or swelling. Epidermal ulceration may expose dermis. Yarkony-Kirk (Yarkony et al. 1990) 1: Red area Present longer than 30 minutes, but less than 24 hours Present longer than 24 hours 2: Epidermis and/ or dermis ulcerated with no subcutaneous fat observed Stirling 2 digit (Reid & Morison 1994) NPUAP* (1989) EPUAP (1998) Stage 0: No clinical evidence of a pressure sore: 0.0 Normal appearance, intact skin 0.1 Healed with scarring 0.2 Tissue damage, but not assessed as a pressure sore Stage I: An observable pressure-related alteration of intact skin the indicators of which as compared to an adjacent or opposite area on the body may include changes in skin colour (red, blue, purple tones), skin temperature (warmth or coolness), skin stiffness and/or sensation (pain) à Grade 1: Nonblanchable erythema of intact skin. Discolouration of the skin, warmth, oedema, induration or hardness may also be used as indicators, particularly on individuals with darker skin Stage 1: Discoloration of intact skin light finger pressure applied to the site does not alter the discolouration: 1.1 Non-blanchable erythema with increased local heat 1.2 Blue/purple/ black discolouration. The sore is at least Stage 1 Stage II: Partialthickness skin loss involving epidermis and/or dermis. The ulcer is superficial and clinically presents itself as an abrasion, blister or shallow crater Grade 2: Partialthickness skin loss involving epidermis, dermis or both. The ulcer is superficial and clinically presents itself as an abrasion or blister Ó 2009 The Authors. Journal compilation Ó 2009 Blackwell Publishing Ltd, Journal of Clinical Nursing, 18,

4 J Kottner et al. Table 1 (Continued) Shea (1975) Grade III: Irregular full thickness skin defect extending into the subcutaneous fat exposing a draining, foul smelling, infected, necrotic base which has undermined the skin for a variable distance. Anatomic limit: Deep fascia Grade IV: Clinical presentation resembles that of a Grade III except that bone can be identified in the base of the ulceration which is more extensively undermined with profuse drainage and necrosis. Anatomic limit: no limit Surrey (1983) in Healey (1995) Stage 3: Destruction of the skin without cavity Stage 4: Destruction of the skin with cavity Torrance (1983) in Edwards & Banks (1999) Grade 3: Ulceration progresses through the dermis to the junction of the subcutaneous tissue. Ulceration has distinct edges, but erythema and induration is present around the wound. Grade 4: Ulceration extends into the relatively avascular subcutaneous fat and underlying muscle becomes swollen and inflamed. Progress is temporarily impeded resulting in lateral extension of the wounding. There is a distinct edge to the ulcer, but deeper areas are distorted. Yarkony-Kirk (Yarkony et al. 1990) 3: Subcutaneous fat observed, no muscle observed 4: Muscle/fascia observed, but no bone observed Stirling 2 digit (Reid & Morison 1994) NPUAP* (1989) EPUAP (1998) Stage 2: Partial thickness skin loss or damage involving epidermis and/or dermis: 2.1 Blister 2.2 Abrasion 2.3 Shallow ulcer, without undermining of adjacent tissue 2.4 Any of these with underlying blue/purple/black discolouration or induration. The sore is at least Stage 2. Stage 3: Full-thickness skin loss involving damage or necrosis of subcutaneous tissue but not extending to underlying bone, tendon or joint capsule: 3.1 Crater, without undermining of adjacent tissue 3.2 Crater, with undermining of adjacent tissue 3.3 Sinus, the full extent of which ins not certain 3.4 Full thickness skin loss, but wound bed covered with necrotic tissue (hard or leathery black/brown/broken tissue or softer or softer yellow/cream/grey slough) which masks the true extent of tissue damage. The sore is at least Stage 3. Until debrided it is not possible to observe whether damage extends into muscle or involves damage to bone or supporting structures Stage III: Full-thickness skin loss involving damage or necrosis of subcutaneous tissue that may extend down to, but not through underlying fascia. The ulcer clinically presents itself as a deep crater with or without undermining of adjacent tissue Stage IV: Full-thickness skin loss with extensive destruction, tissue necrosis or damage to muscle, bone or supporting structures (e.g. tendon, joint capsule, etc.) Grade 3: Fullthickness skin loss involving damage to or necrosis of subcutaneous tissue that may extend down to, but not through underlying fascia Grade 4: Extensive destruction, tissue necrosis, or damage to muscle, bone, or supporting structures with or without fullthickness skin loss 318 Ó 2009 The Authors. Journal compilation Ó 2009 Blackwell Publishing Ltd, Journal of Clinical Nursing, 18,

5 Review Pressure ulcer classifications Table 1 (Continued) Stirling 2 digit (Reid & Morison 1994) NPUAP* (1989) EPUAP (1998) Yarkony-Kirk (Yarkony et al. 1990) Torrance (1983) in Edwards & Banks (1999) Surrey (1983) in Healey (1995) Shea (1975) Stage 4: Full-thickness skin loss with extensive destruction and tissue necrosis extending to underlying bone, tendon or capsule: 4.1 Visible exposure of bone, tendon or joint capsule 4.2 Sinus assessed as extending to bone, tendon or capsule 5: Bone observed, but no involvement of joint space Grade 5: Infective necrosis affects the deeper fascia and muscle. Tissue destruction is rapid, joints, bursae and body cavities can be involved and there is a risk of osteomyelitis. Sores may communicate, resulting in massive tissue destruction. Closed Pressure Sores: Ischemic necrosis in the subcutaneous fat without skin ulceration leading to the development of a bursa-like cavity filled with necrotic debris. Resembling a Grade III sore in extent and depth. Overlying pigmented, thickened and fibrotic skin eventually ruptures, creating a small skin defect draining a large base. Anatomic limit: Deep fascia 6: Involvement of joint space *National Pressure Ulcer Advisory Panel. European Pressure Ulcer Advisory Panel. à Updated contingency tables were presented. Data were extracted by the first (JK) and the second investigator (KR) independently. Afterwards the results were compared. Any discrepancies were discussed until an agreement was achieved. Study quality assessment An intensive search for instruments evaluating the quality of interrater reliability studies provided no results. To our knowledge there are no proposed guidelines or statements presenting any criteria for the quality evaluation of interrater reliability studies. Based on previous studies dealing with the same problem (Audigé et al. 2004, Brorson & Hróbjartsson 2008) and known factors influencing the interrater reliability (Kraemer 1979, Shrout 1998, Dunn 2004) nine criteria were considered as being important: (1) Were the classification system(s) and the number of categories clearly described? (2) Were the repeated assessments conducted independently? (3) Was the number of subjects/targets? (4) Were the assessment conditions clearly described (e.g. skin assessment, use of photographs)? (5) Was the number of raters? (6) Were rater characteristics described? (7) Was the number of observations per rater? (8) Was there a description as to whether normal skin was included? (9) Were the computations of interrater reliability coefficients clearly and comprehensible? With regard to the established quality criteria each study was carefully evaluated as to whether the required information was reported or not. One fulfilled criterion corresponded to one point. After adding up all points the overall quality score ranged from zero to nine points. Only those studies with a quality score of seven or more were included in the final data synthesis. We decided to use this cut-off point because we were unable to validly interpret the results of those studies where more than two criteria were missing. Both reviewers (JK & KR) evaluated each study independently. Any discrepancies were resolved by consensus. Data synthesis and interpretation Studies were divided into two groups depending on whether skin inspection was actually conducted or whether the assessment results were based on images. This classification seemed to be appropriate, because real-life situations (Healy 1996) are hardly comparable to rather artificial ones (Defloor & Schoonhoven 2004, Hart et al. 2006). The use of images limits clinical information which is useful for pressure ulcer classification (Defloor et al. 2006). Results of the original studies were summarised qualitatively. A single summary measure was considered to be Ó 2009 The Authors. Journal compilation Ó 2009 Blackwell Publishing Ltd, Journal of Clinical Nursing, 18,

6 J Kottner et al. #1 Pressure ulcer or pressure sore or decubitus or bed sore or bedsore or decubitus ulcer #2 Classification or grade* or stage* #3 la = German or la = English #4 #1 and #2 and #3 #5 Diagnosis* or diagnostic* or systematic or clinical trial or cohort studies or group or prevalence or incidence or reliability or validity #6 #4 and #5 Figure 1 Search strategy on databases Medline, Embase, Cinahl using ERLWebSPIRS. inappropriate because different statistical approaches were used to calculate the interrater reliability coefficients, such as the overall proportion of agreement (p o ), Cohen s kappa (j), weighted kappa (j w ), proportion of differences in per cent, Spearman s rho, intraclass correlation coefficient (ICC) and non-specified correlation coefficients. Further measures of precision, such as standard errors or confidence intervals, are prerequisites for meta-analysis but were missing in almost all studies. Finally, the way of study design and data analysis varied considerably between the studies. The obtained j-values were interpreted according to the widely used benchmarks of Landis and Koch (1977). They labelled the ranges of j-values as follows: 0Æ00 0Æ20 slight, 0Æ21 0Æ40 fair, 0Æ41 0Æ60 moderate, 0Æ61 0Æ80 substantial, 0Æ81 1Æ00 almost perfect. The interpretation of j w was like that of j (Fleiss et al. 2003). ICC-values were interpreted as proportions of variance explained by within-subject differences. However, j- and ICC-values are highly influenced by the proportion of the trait of the rated sample. That means obtained coefficients could be low when the measures are applied to a very homogenous sample (e.g. high proportion of normal skin in the sample). Values are further influenced by the number of categories. p o -values were regarded as valuable coefficients. However, an interpretation and comparison of p o between the studies was difficult as well. Similar to j, its value is influenced by the presence of the trait and the number of categories. Above all, it has not been corrected for chance agreement. Correlation coefficients, such as Spearman s rho, were regarded as inappropriate measures, because values indicate the degree and direction of association but not of agreement (Bland & Altman 1986). Results Search and selection of studies The search and selection process of included studies is shown in Fig. 2. A total of 1650 references were found in the searched databases. A total of 417 references were duplicate copies and 906 references were excluded based on their titles or abstracts. A total of 327 studies were screened for inclusion criteria. The majority of these studies were pressure ulcer prevalence or incidence studies lacking interrater reliability estimation. Furthermore, seven studies were selected from the World Wide Web search and five studies from the reference lists. Eight studies were not considered for the review, because results were either lacking (n = 4) or the same study was Potentially relevant studies from electronic search (n = 327) Studies identified from reference lists (n = 5) Studies found in the World Wide Web (n = 7) Studies examining interrater reliability but not reporting results (n = 4) Same study published in different journals (n = 3) Studies not meeting inclusion criteria after evaluation of full-text (n = 286) Original study unobtainable (n = 1) Relevant studies included in systematic review (n = 47) Figure 2 Selection process of included studies. 320 Ó 2009 The Authors. Journal compilation Ó 2009 Blackwell Publishing Ltd, Journal of Clinical Nursing, 18,

7 Review published in different sources (n = 3) and one study was unobtainable. In the end, 47 studies were deemed relevant. Study characteristics All 47 studies included in the review were published in English in the period from Most studies were conducted in the USA (n = 16) and the UK (n = 9). Nineteen studies originated from other European countries. One study was identified as being from China and two studies were conducted in Canada. Actual skin assessment in a clinical setting was conducted in 30 studies. In 15 studies the interrater reliability was investigated based on images (e.g. slides, photographs). Both assessment methods were investigated in two studies. Overall, six different classification systems were examined, which included modifications and usage of different numbers of categories. Four studies applied PU classifications that could not be explicitly assigned to one specific system. Two studies examined the interrater reliability for non-blanchable erythema only. Quality assessment Results of the quality assessment are shown in Table 2. A quality score of seven or above was assigned to 24 out of 47 relevant studies (51%) and only those studies were included in the final data synthesis. The most frequently missing details were number of observations per rater (n = 30), information as to whether normal skin was included (n = 21) and a description whether assessments were conducted independently (n = 22). In four cases data of two distinct studies were presented. In Table 2 they are referred to as study 1 and study 2. Final data synthesis Interrater reliability studies based on skin examination Ten studies were included in the final data synthesis describing assessment results of a real skin or wound examination. Study characteristics, methods and results are shown in Table 3. Studies are very heterogeneous regarding applied classification systems, methods, number of categories, rates and rated subjects. Therefore they are hardly comparable. Interrater reliability for the EPUAP classification including the additional category no PU was investigated in three studies. Coefficients ranged from j =0Æ97 (95% CI 0Æ92 1Æ00) (Bours et al. 1999) to j =0Æ31 (Pedley 2004). Corresponding p o -values ranged from 1Æ00 (Bours et al. 1999) to 0Æ49 (Pedley 2004). The study of Bours et al. (1999) was a large scale testing of a data collection form which included Pressure ulcer classifications the EPUAP system. Numerous raters with different background were involved. Only one pair of raters and fewer patients participated in the studies by Halfens et al. (2001) and Pedley (2004). The NPUAP system was examined in three studies. The proportion of agreement ranged from 100% (Gawron 1994) to 58% (Lyder et al. 1999). Even though Allman et al. (1995) applied the complete NPUAP system, agreement was only measured for the category non-blanchable erythema (p o =0Æ79). For the complete NPUAP system no j-values were reported. Interrater reliability coefficients for the Stirling classification (Pedley 2004) were comparably low with j-values between 0Æ37 (p o =0Æ54) for the 1-digit-version and 0Æ48 (p o =0Æ54) for two-digit version indicating fair to moderate agreement. When the adapted Shea scale was applied interrater reliability between all raters was j = 0Æ42 (95% CI 0Æ10 0Æ74) (p o =0Æ67) indicating moderate agreement (Buntinx et al. 1986). Using the adapted Torrance scale agreement among trained nurses ranged from 92 98% (Nixon et al. 1998). After application of an adapted version of the EPUAP/NPUAP classification the interrater reliability between one research nurse team leader and trained, experienced clinical research nurses was j =0Æ97 (95% CI 0Æ93 1Æ00), which could be labelled as almost perfect. The interrater reliability between the research nurses and trained ward nurses was much lower [j = 0Æ63 (95% CI 0Æ61 0Æ66)], which corresponds to substantial agreement. Vanderwee et al. (2006) measured interrater reliability for the distinction between blanchable and non-blanchable erythema. When using a transparent disc j was 0Æ72 (p o =0Æ97). When the finger method was applied j was 0Æ69 (p o =0Æ92). Results indicate that there was no difference between both assessment methods. All investigated raters in these studies were specialised, trained or experienced in pressure ulcer diagnosis. Buntnix et al. (1986), Bours et al. (1999), Pedley (2004), Nixon et al. (2005) and Vanderwee et al. (2006) put their major emphasis of their studies on the investigation of interrater reliability. Remaining studies conducted smaller-scale interrater reliability examinations as part of larger studies. Interrater reliability studies based on images Fourteen studies measuring interrater reliability based on images were included in the final data synthesis (Table 4). Studies differed considerably regarding the applied PU classification system, the version of the classification, the number and qualification or raters, the sample and the computation of results. Ó 2009 The Authors. Journal compilation Ó 2009 Blackwell Publishing Ltd, Journal of Clinical Nursing, 18,

8 J Kottner et al. Table 2 Assessment of methodological quality of reliability studies of pressure ulcer classification systems Quality criteria* Authors (1) Description of classification (2) Independency of assessments (3) Number of subjects / targets (4) Description of assessment conditions (5) Number of raters (6) Description of rater characteristics (7) Observation per rater (8) Inclusion of normal skin (9) Computations of interrater reliability coefficients QS Barbenel et al. (1977) Yes Yes No Yes Yes Yes No No No 5 Warner & Hall (1986) Yes ns à Yes Yes No Yes No Yes No 5 Yarkony et al. (1990) Yes Yes Yes No Yes Yes No Yes No 6 Allcock et al. (1994) Yes ns Yes No No Yes No No No 3 Braden & Bergstrom (1994) Yes ns No No No Yes No No No 2 Gawron (1994) (study 1) Yes ns No Yes Yes Yes No Yes Yes 6 Gawron (1994) (study 2) Yes Yes Yes Yes Yes Yes Yes Yes Yes 9 Allman et al. (1995) Yes Yes Yes Yes Yes Yes Yes No Yes 8 Arnold & Watterworth (1995) (study 1) Arnold & Watterworth (1995) (study 2) Yes ns Yes Yes Yes Yes Yes Yes Yes 8 Yes Yes No Yes Yes Yes No Yes ns 6 Ferrell et al. (1995) No ns Yes Yes Yes No Yes Yes No 5 Healey (1995) Yes Yes Yes Yes Yes Yes Yes No Yes 8 Bergstrom et al. (1996) Yes ns No No No Yes No No Yes 3 Buntinx et al. (1986) Yes Yes Yes Yes Yes Yes Yes Yes Yes 9 Bergstrom et al. (1998) Yes ns No No No Yes No No Yes 3 Nixon et al. (1998) (study 1) Yes ns Yes Yes Yes Yes No Yes Yes 7 Nixon et al. (1998) (study 2) Yes ns Yes Yes Yes Yes No Yes Yes 7 Bergquist & Frantz (1999) Yes No Yes Yes No Yes No Yes Yes 6 Bours et al. (1999) Yes Yes Yes Yes No Yes No Yes Yes 7 Carlson et al. (1999) Yes No No Yes No Yes No No Yes 4 Derre et al. (1999) Yes ns No No No No No Yes No 2 Lyder et al. (1999) Yes Yes Yes Yes Yes Yes No No Yes 7 Halfens et al. (2001) Yes ns Yes Yes Yes Yes Yes No Yes 7 Russell & Reynolds (2001) No Yes Yes Yes Yes Yes Yes No Yes 7 Bours et al. (2002) Yes Yes Yes Yes No Yes No No No 5 Schoonhoven et al. (2002a) (Schoonhoven et al. 2002b) Yes No Yes Yes Yes Yes No No Yes 6 Marrie et al. (2003) No Yes Yes Yes Yes No No Yes No 5 Verdu (2003) Yes Yes Yes Yes Yes Yes Yes Yes Yes 9 Defloor & Schoonhoven (2004) Yes Yes Yes Yes Yes Yes Yes Yes Yes 9 Groeneveld et al. (2004) Yes Yes Yes Yes No Yes Yes No Yes 7 Pedley (2004) Yes Yes Yes Yes Yes Yes Yes Yes Yes Ó 2009 The Authors. Journal compilation Ó 2009 Blackwell Publishing Ltd, Journal of Clinical Nursing, 18,

9 Review Pressure ulcer classifications Table 2 (Continued) Quality criteria* Authors (1) Description of classification (2) Independency of assessments (3) Number of subjects / targets (4) Description of assessment conditions (5) Number of raters (6) Description of rater characteristics (7) Observation per rater (8) Inclusion of normal skin (9) Computations of interrater reliability coefficients QS Buckley et al. (2005) Yes Yes Yes Yes Yes Yes Yes No Yes 8 Kwong et al. (2005) No ns No No No Yes No No Yes 2 Nixon et al. (2005) (study 1) Yes No Yes Yes Yes Yes Yes Yes Yes 8 Nixon et al. (2005) (study 2) Yes No Yes Yes Yes Yes No Yes Yes 7 Vanderwee et al. (2005) No Yes No Yes No Yes No Yes Yes 5 Alves (2006) No Yes Yes Yes Yes No No Yes No 5 Briggs (2006) Yes Yes Yes Yes Yes Yes Yes Yes Yes 9 Defloor et al. (2006) Yes Yes Yes Yes Yes Yes Yes Yes Yes 9 Feuchtinger et al. (2006) No Yes Yes Yes No Yes No No Yes 5 Gunningberg (2006) Yes Yes Yes Yes Yes Yes Yes No Yes 8 Hart et al. (2006) Yes ns Yes Yes Yes Yes Yes Yes Yes 8 Noonan et al. (2006) Yes ns Yes Yes Yes Yes Yes Yes Yes 8 Localio et al. (2006) Yes Yes Yes Yes Yes Yes Yes Yes Yes 9 Vanderwee et al. (2006) Yes Yes Yes Yes Yes Yes No Yes Yes 8 Feuchtinger et al. (2007) No ns Yes No Yes Yes No No Yes 4 Gajewski et al. (2007) Yes ns Yes No Yes No No Yes Yes 5 Schoonhoven et al. (2007) Yes Yes Yes No Yes Yes No No Yes 6 Stausberg et al. (2007) Yes Yes Yes Yes Yes Yes Yes Yes Yes 9 Vanderwee et al. (2007b) Yes Yes No Yes No Yes No No Yes 5 Vanderwee et al. (2007c) Yes Yes Yes Yes No No No Yes Yes 6 *Full description of quality criteria in text. Quality score. à Not. Same study. Bold letters: studies included in final data synthesis. Ó 2009 The Authors. Journal compilation Ó 2009 Blackwell Publishing Ltd, Journal of Clinical Nursing, 18,

10 J Kottner et al. Table 3 Characteristics and findings of interrater reliability studies based on real skin assessment included in final data synthesis Author (year), Country/Region Classification system (number of categories) Methods Raters (k) Subjects/Targets (n) Observations per rater Normal skin Results Notes Gawron (1994), USA Allman et al. (1995), USA Buntinx et al. (1986), Belgium Nixon et al. (1998), UK Bours et al. (1999), Netherlands NPUAP, category necrosis was added (5) Assessment of patients with PUs by principal investigator and nurses NPUAP (4) Two independent assessments of patients within two days Shea, category blanching erythema was added (5) Adapted Torrance scale, category no skin discolouration was added, Grade 2 was divided in Nonblanching redness and superficial skin damage (7) EPUAP, category no PU was added (5) Independent assessments of PUs Paired assessments in pre- (1) and main study (2) Two independent assessments in hospital, nursing home, home health care Principal investigator and trained registered nurses (k = 23) Research nurses (k =2) Nurses (k = 3) and physicians (k = 3) with chronic wound experience, without special training in assessment Study 1: Trained nurses (k = 94) Study 2: Trained nurses (k not ) Hospital and nursing home: trained staff nurses (k not ) and one researcher Home health care: trained primary nurses (k not ), one wound care specialist Hospital patients (n = 2) with PUs Hospital patients (n = 26) PUs (n = 20) among 20 geriatric patients, convenience sample Study 1: Skin sites of hospital patients (n = 664) Study 2: Skin sites of hospital patients (n = 851) Hospital patients (n = 45, 674 observations) Nursing home residents (n = 23, 344 observations) Home health care patients (n =90, 1348 observations) 2 No po =0Æ90 to 1Æ00 Part of prevalence study 26 Not Non-blanchable erythema: p o =0Æ79 20 No No stage 0 PU was observed Agreement for pairs (excluding grade 0): po =0Æ40 to 0Æ80, j=0æ12 to 0Æ65 Average agreement between all raters (excluding grade 0): p o =0Æ67, j =0Æ42 (95% CI 0Æ10 0Æ74) Not Not clearly Yes Study 1: po =0Æ98, disagreement in 15 skin sites affecting 12 patients Study 2: po =0Æ92, disagreement in 72 skin sites Disagreement mainly in no skin discoloration and blanching erythema Yes Hospital: p o =1Æ00, j =0Æ97 (95% CI 0Æ92 1Æ00) Nursing home: p o =0Æ94, j =0Æ81 (95% CI 0Æ73 0Æ90) Home health care agency: agreement between nurses and wound care specialist: p o =0Æ98, j =0Æ49 (95% CI 0Æ35 0Æ63) Most disagreements occurred in diagnosing grade 1 Part of incidence study Part of incidence study po recalculated Part of validation study 95% CIs recalculated 324 Ó 2009 The Authors. Journal compilation Ó 2009 Blackwell Publishing Ltd, Journal of Clinical Nursing, 18,

11 Review Pressure ulcer classifications Table 3 (Continued) Author (year), Country/Region Classification system (number of categories) Methods Raters (k) Lyder et al. (1999), USA Halfens et al. (2001), Netherlands NPUAP (4) Comparison between skin assessment results between research nurse 1 and 2 and principal investigator or wound/skin care clinical nurse specialist EPUAP, category no PU was added (5) Two independent assessments of patients using transparent disc Trained research nurse 1 conducted melanocentric (observing for changes in hue (purple), temperature, induration, use of natural lighting) skin assessment Trained research nurse 2 conducted skin assessment according to current nursing practice Trained staff nurses (k =2) Pedley (2004), UK Stirling 1-digit (5) Stirling 2-digit (15) EPUAP scale, category PU yes/ no was added (5) Nixon et al. (2005), UK Adapted EPUAP/ NPUAP, categories no skin changes, blanching erythema, black eschar were added (7) Simultaneous and independent assessments of PU areas using different scales Simultaneous and independent paired assessments of seven skin sites Trained registered nurses experienced in tissue viability (k =2) Study 1: Clinical research nurse team leader (k = 1) and trained and experienced clinical research nurses (k =4) Study 2: Trained and experienced clinical research nurses (k = 6) and trained ward nurses (k = 109) Subjects/Targets (n) Black and Latino/ Hispanic Elders (n =24) Hospital patients (n = 28) PU areas (n = 35) on 30 adult hospital patients Study 1: hospital patients, 107 skin sites (n = 16) Study 2: hospital patients, 2396 skin sites (n = 362) Observations per rater Normal skin Results Notes Not Not Agreement between research nurse 1 and principal investigator or wound/ skin care clinical nurse specialist: po =0Æ78 Agreement between research nurse 2 and Principal investigator or wound/ skin care clinical nurse specialist: p o =0Æ58 Part of incidence study 28 Not j =0Æ90 Part of prospective study 35 Yes Stirling 1 digit: j =0Æ37, p o =0Æ54 Stirling 2 digit: j =0Æ48, p o =0Æ54 EPUAP scale: j =0Æ31, po =0Æ49; Pressure ulcer (yes/no): j = 0Æ22, p o =0Æ86 Study 1: 16 Study 2: not Yes Study 1: Diagnoses of PU (yes/no): po =1Æ00, j =1Æ00 (all skin sites); Classification across all categories: po =0Æ98, j =0Æ97 (95% CI 0Æ93 1Æ00) (two disagreements concerning normal skin and blanching erythema) Study 2: Diagnoses of PU (yes/no): po =0Æ97, j =0Æ77 (95% CI 0Æ72 0Æ82) (all skin sites); Classification across all categories: p o =0Æ79, j =0Æ63 (95% CI 0Æ61 0Æ66) (508 disagreements ranging from 1 to 3 categories) j and 95% CIs recalculated Ó 2009 The Authors. Journal compilation Ó 2009 Blackwell Publishing Ltd, Journal of Clinical Nursing, 18,

12 J Kottner et al. Table 3 (Continued) Normal skin Results Notes Observations per rater Subjects/Targets (n) Classification system (number of categories) Methods Raters (k) Author (year), Country/Region Finger and transparent disk method were used in random order Sample size calculation was conducted No Finger method: agreement between researcher and nurses: po = 0Æ92 (range: 0Æ83 1Æ00), j = 0Æ69 (range 0Æ48 1Æ00) Transparent disc method: agreement between researcher and nurses: p o =0Æ92 (range: 0Æ83 1Æ00), j = 0Æ72 (range 0Æ44 1Æ00) Agreement between finger and transparent disc method: p o =0Æ97 (95% CI 0Æ95 0Æ98), j =0Æ88 (95% CI 0Æ83 0Æ94) More erythemas were classified as non-blanchable wtih transparent disc method (n = 28) Researcher: 503, Nurses: not Pressure points with erythema of geriatric hospital patients (n = 503) Researcher (k = 1), trained nurses (k = 16) Two independent assessments of six skin sites by researcher and nurses comparing finger and transparent disk method Blanchable and non-blanchable erythema (2) Vanderwee et al. (2006), Belgium p o = proportion of agreement, j = kappa. A four grade NPUAP classification was applied in two studies. Arnold and Watterworth (1995) reported that agreement among seven registered nurses could not be increased by training (p o =0Æ7). Agreement between two PU experts rating 50 photographs was much higher (Noonan et al. 2006). The NPUAP system including a fifths category unstageable was investigated in three studies: j-values ranged from 0Æ56 (Hart et al. 2006) 0Æ75 (Groeneveld et al. 2004) indicating moderate to substantial agreement. Buckley et al. (2005) reported proportions of agreement of home health care nurses with experts ranging from 39% to 100%. Interrater reliability for the classification PU (yes/no) according to the NPUAP among six trained experienced research nurses was 0Æ69 (ICC) which could be labelled as substantial agreement. The EPUAP system was investigated in six studies. Studies including large sample sizes were conducted by Defloor and Schoonhoven (2004) and Defloor et al. (2006). Interrater reliability among PU experts (j =0Æ80 (Defloor & Schoonhoven 2004)) was much higher than among nurses who participated in a congress for wound care. Interrater reliability between trained staff nurses and data collectors was high (j = 0Æ75 (Gunningberg 2006)). Reported proportions of agreement ranged from 85% (Verdu 2003) to 62% (Russell & Reynolds 2001). The Stirling classification was investigated in two studies (Healey 1995, Russell & Reynolds 2001). Interrater reliability for the two-digit version as well as for the one-digit version was low: j =0Æ15 and j =0Æ22. Proportions of agreement for the two-digit version were 30% (Russell & Reynolds 2001) and 39% (Healey 1995). Interrater reliability of the Torrance (j =0Æ29, p o =0Æ60) and Surrey classification (j =0Æ37, p o =0Æ67) was slightly higher indicating fair agreement. The classification applied by Stausberg et al. (2007) was comparable to the NPUAP or EPUAP system, although the descriptions of categories differed slightly. The overall agreement on PU grades was p o =0Æ67 (j =0Æ50) and the agreement for PU diagnosis (yes/no) was p o =0Æ88 (j =0Æ29) indicating fair to moderate agreement. The main issue of the synthesised studies was the investigation of interrater reliability or agreement. In only three studies the examination of interrater reliability was part of larger prevalence studies (Groeneveld et al. 2004, Gunningberg 2006, Noonan et al. 2006). Discussion A main finding of our systematic review was the heterogeneity of studies. It was the reason why a meaningful data 326 Ó 2009 The Authors. Journal compilation Ó 2009 Blackwell Publishing Ltd, Journal of Clinical Nursing, 18,

13 Review Pressure ulcer classifications Table 4 Characteristics and findings of interrater reliability studies based on assessment of images included in final data synthesis Author (year), Country/ Region Classification system (number of categories) Methods Raters (k) Subjects/ targets (n) Observations per rater Normal skin Results Notes Arnold & Watterworth (1995), USA Healey (1995), UK Russell & Reynolds (2001), UK NPUAP (4) Assessments of PU slides before and after training, comparison with expert ratings Torrance (5), Stirling 1-digit (5), Stirling 2-digit (15), Surrey (4) Stirling 2-digit (15), EPUAP (4) Assessment of PU photographs by three groups each using one scale Assessment of PU photographs Comparison with expert panel Registered nurses (k = 7), PU experts (k not ) Surrey: nurses (k = 35) Torrance: nurses (k = 37) Stirling: nurses (k = 37) Tissue viability nurses (k = 27), district nurses (k = 24), acute hospital nurses (k = 25), EPUAP members (k = 21) PU slides (n = 16) PU photographs (n = 10) PU photographs (n = 12) 2 16 No Overall agreement with expert ratings before training: po = 0Æ68; overall agreement with expert ratings after training: p o =0Æ71 10 Not 12 Not Torrance: individual categories agreement j =0Æ17 to 0Æ60; overall j =0Æ29, po =0Æ60 Stirling 2 digit: individual categories agreement j =0Æ02 to 0Æ46; overall j =0Æ15, p o =0Æ39 Stirling 1 digit: individual categories agreement j =0Æ12 to 0Æ50; overall j =0Æ22, p o =0Æ59 Surrey: individual categories j =0Æ18 to 0Æ64; overall j =0Æ37, p o =0Æ67 Highest reliability for rating most severe PUs Mean of absolute values of differences between expert panel and all nurses (precision index): 0Æ36 ± 0Æ15 (Stirling 2-digit), 0Æ49 ± 0Æ15 (EPUAP) Agreement between expert panel and all nurses: p o =0Æ30 (Stirling 2-digit), p o =0Æ62 (EPUAP) Significant lack of consensus in acute hospital nurse group po recalculated Only 79 nurses graded all photographs Ó 2009 The Authors. Journal compilation Ó 2009 Blackwell Publishing Ltd, Journal of Clinical Nursing, 18,

14 J Kottner et al. Table 4 (Continued) Author (year), Country/ Region Classification system (number of categories) Methods Raters (k) Verdu (2003), Spain EPUAP, category not known was added (5) Assessment of PUs including case studies by two groups of nurses Comparison with expert ratings Experimental group: nurses with similar level of experience using a decision tree for grading (k = 32) Control group: nurses with similar level of experience without using decision tree for grading (k = 34) Defloor & Schoon-hoven (2004), Europe EPUAP, categories normal skin, blanchable erythema, incontinence lesion were added (7) Independent assessments of skin alterations by different groups Comparison with ratings from EPUAP trustees EPUAP trustees from UK, Italy, the Netherlands, Denmark, Belgium, Ireland (k =9) PU researchers from the Netherlands, Belgium (k =7) Staff nurses from Belgium (k = 20) PU nurses from the Netherlands, Belgium (k =17) Groeneveld et al. (2004), Canada NPUAP, category unable to stage was added (5) Independent assessments of PUs Trained staff members and third and fourth year nursing students (k not ) Subjects/ targets (n) Observations per rater Normal skin Results Notes PU photographs, cases I, II, III (n =3) Photographs showing different skin alterations, application of transparent disc (n = 56) PU slides (n = 25) Not 3 No Proportions of accurate choices in experimental group: p o =0Æ78 (case I), po =0Æ66 (case II), po =0Æ44 (case III) Proportions of accurate choices in control group: p o =0Æ85 (case I), po =0Æ53 (case II), po =0Æ62 (case III) Using chi-square tests no statically significant differences were found between experimental and control group Overall proportions of accurate choices (both groups): po = 0Æ82 (case I), p o =0Æ59 (case II), po =0Æ53 (case III) 56 Yes EPUAP trustees: po = 0Æ84 PU researchers: j = 0Æ80 (multirater), j range: 0Æ60 0Æ95 Staff nurses: j = 0Æ80 (multirater), j range: 0Æ48 0Æ98 PU nurses: j = 0Æ78 (multirater), j-range: 0Æ42 0Æ92 Total: j = 0Æ80 (multirater) Deviation from ratings from EPUAP trustees: 5,9%, 9 PU photographs wrongly labelled as incontinence lesion, 21 incontinence lesion photographs not detected Not po recalculated j = 0Æ75 (multirater) Part of prevalence study 328 Ó 2009 The Authors. Journal compilation Ó 2009 Blackwell Publishing Ltd, Journal of Clinical Nursing, 18,

15 Review Pressure ulcer classifications Table 4 (Continued) Author (year), Country/ Region Classification system (number of categories) Methods Raters (k) Buckley et al. (2005), USA Briggs (2006), UK NPUAP, category cannot be staged was added (5) EPUAP, categories normal skin, blanchable erythema, moisture lesion, combined lesions, don t know were added (9) Simultaneous and independent assessment of wound photographs including case studies Comparison with WOC nurse ratings Assessment of the same set of PU and skin lesions before and after study of PUCLAS CD-ROM Comparison with correct responses Registered home care nurses (k =33) WOC nurses (k =4) Registered nurses, experience as qualified nurses varied from a few month to five years (k = 52) Defloor et al. (2006), Europe EPUAP, categories normal skin, blanchable erythema, incontinence lesion, don t know were added (8) Phase 1: Assessments of photographs, comparison with ratings of experts Phase 2: Assessments of same set of photographs twice Phase 1: nurses participating at congress for wound care familiar with EPUAP system (k = 473) Phase 2: nurses from university hospital (k = 86) Gunningberg (2006), Sweden EPUAP (4) Assessments of PU photographs Trained staff nurses and data collectors (k = 28) Subjects/ targets (n) Photographs and case studies (n =10) Photographs of wounds and skin lesions on PUCLAS CD-ROM (n = 20) Photographs showing different skin alterations, use of transparent disc (n = 56) PU photographs (n = 10) Observations per rater Normal skin Results Notes 10 Not Complete agreement among WOC nurses except for one PU Agreement of home care nurses with WOC nurses ratings: p o =0Æ39 to p o =1Æ00 (range), p o =0Æ73 (mean) 2 20 Yes Correct responses before education: p o =0Æ02 (16 20 correct answers), po = 0Æ15 (11 15 correct answers), p o =0Æ44 (6 10 correct answers), po = 0Æ39 (1 5 correct answers) Correct responses after education: po = 0Æ08 (16 20 correct answers), p o =0Æ56 (11 15 correct answers), p o =0Æ35 (6 10 correct answers), po = 0Æ02 (1 5 correct answers) Phase 1: 56 Phase 2: Yes Phase 1: interrater agreement: multirater j =0Æ37, averagej = 0Æ50 (95% CI 0Æ49 0Æ52); disagreement with experts: p o recalculated Intrarater reliability study was included po =0Æ55 (grade 1), po =0Æ44 (incontinence lesions), Phase 2 first assessment: interrater agreement: multirater-j =0Æ38, average-j = 0Æ51 (95% CI 0Æ49 0Æ54); second assessment: multirater-j =0Æ43, average-j =0Æ55 (95% CI 0Æ53 0Æ58) 10 No Mean j = 0Æ75 Part of prevalence study Ó 2009 The Authors. Journal compilation Ó 2009 Blackwell Publishing Ltd, Journal of Clinical Nursing, 18,

16 J Kottner et al. Table 4 (Continued) Author (year), Country/ Region Classification system (number of categories) Methods Raters (k) Subjects/ targets (n) Hart et al. (2006), USA NPUAP, category unstageable was added (5) Study 1: Web based assessment of various skin lesions Study 2: Web based assessment of PU photographs according grade Comparison with ratings from expert panel Partly trained staff nurses and wound/skin care nurses from 48 hospitals across the US (k = 256) Study 1: Photographs showing PUs (2), venous (3), arterial (1), diabetic foot ulcer (1) (n =7) Study 2 version 1: Photographs showing PUs with wound descriptors (n = 17); version 2: Photographs showing PUs without wound descriptors (n =17) Noonan et alæ (2006), USA Localio et al. (2006), USA NPUAP (4) Independent assessment of PUs PU according to NPAUP (yes/no) (2) Two assessments of the same set of PU and skin lesions by research nurses PU experts (k = 2) PU photographs (n = 50) Trained experienced research nurses (k =6) PU and various skin lesion photographs (n = 160) Stausberg et al. (2007), Germany Grades 0 (no PU) to 4 (severe damage) (5) Independent assessments of PUs PU experts (k = 7) PU photographs of foot/heel and buttock/hip region (n = 100) po = proportion of agreement, j = kappa, ICC = intraclass correlation coefficient. Observations per rater Normal skin Results Notes Study 1: 7 Study 2: Not No Study 1: overall agreement with expert ratings for wound identification: j =0Æ56 (SD = 0Æ22), overall agreement for PU (yes/no): j =0Æ84 (SD = 0Æ25), Study 2 version 1: overall agreement with expert ratings for PU grading: j =0Æ72 (SD = 0Æ22) Study 2 version 2: overall agreement with expert ratings for PU grading: j =0Æ56 (SD = 0Æ17) Nurses with wound/continence/ ostomy care certification had higher j-values for PU identification and grading po = 0Æ90 Part of prevalence study Yes Agreement among nurses across both assessments: agreement among six raters: 64%, agreement among five raters: 20%, agreement among four raters: 10%, agreement among three raters: 5%; ICC = 0Æ Yes Agreement per photograph: 33% (seven raters), 20% (six raters), 29% (five raters); Agreement grade: p o =0Æ67, mean-j =0Æ50 (95% CI 0Æ45 0Æ54); Agreement PU (yes/no): p o =0Æ88, meanj =0Æ29 (95% CI 0Æ23 0Æ36) Agreement recalculated 330 Ó 2009 The Authors. Journal compilation Ó 2009 Blackwell Publishing Ltd, Journal of Clinical Nursing, 18,