Strategies for Conducting Pressure Ulcer Reliability Studies Sandra Bergquist-Beringer, PhD, RN, CWCN First Annual NDNQI Conference January 30, 2007 Las Vegas, NV What is Reliability? Consistency in Measurement The Extent to Which Measurements Agree with Each Other All Rights Reserved 1
Types of Reliability Inter-rater reliability - the extent to which two or more individuals (coders or raters) agree Intra-rater reliability - the agreement of two or more ratings performed by the same examiner over some specified period of time Example All Rights Reserved 2
Inter-Rater Reliability of Pressure Ulcer Identification and Staging Clinical Examination of Each Patient s Skin (Bedside Observation) Gold Standard Bedside Observation Advantage Contextual Assessment of Wound Disadvantage Time and Labor Patient Inconvenience Types Concurrent Sequential All Rights Reserved 3
Types of Bedside Observation Concurrent All Together Advantage Controls for change in ulcer condition Minimizes loss to follow-up Disadvantage Raters may share information Coordination of staff time Types of Bedside Observation Sequential First One, Then Another Advantage Less difficulty in coordinating staff time Raters less likely to share information Disadvantage Change in ulcer condition Loss to patient follow-up More patient inconvenience All Rights Reserved 4
Inter-Rater Reliability of Pressure Ulcer Identification and Staging Use of Photographs Advantages Minimal patient inconvenience Assessment of many raters Asynchronous assessment of raters Disadvantage Use of Photographs Little context for wound assessment Variations in picture quality Raters may share information All Rights Reserved 5
Display Methods Print Monitor Use of Photographs Why Conduct an Inter-Rater Reliability Study? To determine if reported hospital acquired pressure ulcers are influenced by rater ability to differentiate pressure ulcers from other types of wounds To determine if reported pressure ulcer stages are accurate CURRENT NATIONAL QUALITY FORUM LIST OF NEVER EVENTS Stage 3 or 4 pressure ulcers acquired after admission to a healthcare facility All Rights Reserved 6
Conducting an Inter-Rater Reliability Study Bedside Observation Conducting an Inter-Rater Reliability Study Bedside Observation Identify the Pressure Ulcer Team Larger Hospitals (Many Teams) Expert Individual Team Leaders Team Members Smaller Hospitals (1 Team) Expert Team Leader Team Members Much Smaller Hospitals Expert Leader All Rights Reserved 7
Conducting an Inter-Rater Reliability Study Bedside Observation Review NDNQI Pressure Ulcer Training CD Pressure Ulcers and Staging Pressure Ulcer Staging Pressure Ulcer Definition Pressure Ulcer Locations Other Types of Wounds Arterial Ulcers Venous Ulcers Perineal Dermatitis Diabetic Ulcers Skin Tears Concurrent Bedside Observation Pressure Ulcer Identification and Staging Total Team Rounding Smaller Hospitals (One Team) To all patients Larger Hospitals (Multiple Teams) Divide hospital into units Each team rounds on a select number of units All Rights Reserved 8
Concurrent Bedside Observation Pressure Ulcer Identification and Staging Total Team Rounding Examine the Skin of Each Hospital or Unit Patient Identify the presence of an ulcer Identify type of ulcer Stage all pressure ulcers Identify pressure ulcer location Concurrent Bedside Observation Pressure Ulcer Identification and Staging Total Team Rounding Examine the Skin of Each Hospital or Unit Patient Record Assessment All Rights Reserved 9
Data Collection Form Conducting an Inter-Rater Reliability Study Bedside Observation Team Rules Independent assessments No sharing of information No changes to assessment recorded All Rights Reserved 10
Consecutive Bedside Observation Pressure Ulcer Identification and Staging One Person Rounds Examine the Skin of Each Hospital or Unit Patient Identify the presence of an ulcer Identify type of ulcer Stage all pressure ulcers Identify pressure ulcer location Record Assessment Second Person Rounds Repeat above Team Rules Apply Conducting an Intra-Rater Reliability Study Bedside Observation All Rights Reserved 11
Conducting an Intra-Rater Reliability Study Bedside Observation Much Smaller Hospitals with One Expert Only Round on All Patients Examine the skin of each hospital patient Identify the presence of an ulcer Identify type of ulcer Stage all pressure ulcers Identify pressure ulcer location Record Assessment Re-Round on All Patients Repeat above Team Rules Apply Conducting an Inter-Rater Reliability Study Use of Photographs Note: Prior to Pressure Ulcer Training CD All Rights Reserved 12
Conducting an Inter-Rater Reliability Study Use of Photographs Divide Participants into 2 Groups Group 1 Photographs Only Take Pressure Ulcer Staging and Other Wound Type Tests using Pictures Only Paper and Pencil Answer Sheet with Response Options A. Stage I B. Stage II C. Stage III D. Stage IV E. Unstageable Example Used with permission NPUAP All Rights Reserved 13
Conducting an Inter-Rater Reliability Study Use of Photographs Test Rules No cues or hints from the test giver Independent participant assessments No sharing of information among participants Conducting an Inter-Rater Reliability Study Use of Photographs Group 2 Photos with Wound Description Take Pressure Ulcer Staging and Other Wound Type Tests using Pictures with Wound Descriptions Paper and Pencil Answer Sheet with Response Options A. Stage I B. Stage II C. Stage III D. Stage IV E. Unstageable All Rights Reserved 14
Example Ulcer over heel involves loss of subcutaneous tissue. Wound measure 2.7 cm by 3.5 cm and is 8 mm deep No damage to underlying facia. Used with permission NPUAP Conducting an Inter-Rater Reliability Study Use of Photographs Compare Participant Responses to Known Correct Answer Participant Answer Example Correct Answer A. Stage I Stage III Used with permission NPUAP B. Stage II C. Stage III D. Stage IV E. Unstageable All Rights Reserved 15
Calculating Reliability Percent Agreement Percent Agreement Between Rater and Expert Team Leader Number of Agreements/Total Possible x 100 Does not take into account the agreement due solely to chance. All Rights Reserved 16
Example Inter-Rater Reliability of Pressure Ulcer Staging Bedside Observation by Expert Example Inter-Rater Reliability of Pressure Ulcer Staging Bedside Observation by Rater All Rights Reserved 17
Example Inter-Rater Reliability of Pressure Ulcer Staging Percent Agreement 1/1 x 100 = 100% Agreement Cohen s Kappa (κ) Agreement between 2 or more raters that occurs beyond chance. Κ = {O Ag -E Ag }/ {1-E Ag } O Ag : Observed agreement E Ag : Expected fraction of agreements by chance All Rights Reserved 18
Methods for Calculating Cohen s Kappa Statistical Programs Mathematical http://www.colorado.edu/geography/gcraft/notes/manerror/html/kappa.htm Recommended Levels of Agreement Percent Agreement > 90% Cohen s Kappa > 0.61 0.81 1.00 = Near Perfect Reliability 0.61 0.80 = Substantial Reliability 0.41 0.60 = Moderate Reliability 0.21 0.40 = Fair Reliability < 0.20 Slight Reliability All Rights Reserved 19
Calculating Reliability Intra-Rater Reliability Percent Agreement Calculate percent agreement between 2 or more ratings from the same individual Cohen s Kappa (κ) Calculate agreement between 2 or more ratings from the same individual Calculating Reliability Use of Photographs All Rights Reserved 20
Calculating Reliability - Use of Photographs Overall Agreement Percent Agreement Calculate percent agreement between participant responses to known correct answer for each rater Overall Percent Agreement = Sum of % Agreements/Total # Raters Cohen s Kappa Example Percent Agreement 1 person 8/10x100% = 80% 8 All Rights Reserved 21
Calculating Reliability Use of Photographs Group 1 Pictures Only Calculate overall percent agreement or Kappa agreement between participant responses and correct answer Group 2 Pictures with Wound Description Calculate overall percent agreement or Kappa agreement between participant responses and correct answer Calculating Reliability Use of Photographs True Percent Agreement is Theoretically Somewhere Between These Two Results All Rights Reserved 22
Sandra Bergquist-Beringer, RN, PhD, CWCN School of Nursing University of Kansas Medical Center 3901 Rainbow Blvd. Mail Stop 4043 Kansas City, KS 66160 sbergquist-beringer@kumc.edu All Rights Reserved 23