Replication analysis of the validity and comparability of Patient Safety Indicators (PSI): the impact of AHRQ exclusions

Transcription

1 Replication analysis of the validity and comparability of Patient Safety Indicators (PSI): the impact of AHRQ exclusions by Vladimir Stevanovic and Lihan Wei The OECD HCQI Expert Group Meeting Paris, 3 June 2010

2 Background HCQI Expert Group meeting in June 2009 Concerns were raised that the data may reflect more coding and registration practices than actual differences in patient safety Several countries expressed reservations about the publication of PSIs in Health at a Glance 2009 due to perceived risk of misinterpretation

3 Background PSI Subgroup meeting in October 2009 The Secretariat presented a technical analysis on the impact of the AHRQ exclusions The findings implied that the exclusions may have varying impacts across all indicators apart from the obstetric ones It was proposed further analysis to be undertaken through a voluntary subsample of countries

4 Objectives To improve the comparability of PSIs by: Assessing the impact of AHRQ exclusions on the PSI rates, and Exploring whether these exclusions account for any undesired or increased variation across countries The scope of this analysis did NOT include exclusions that are inherent to the concept of an indicator by their nature (e.g. children, pregnant women)

5 Patient safety indicators PSI05 - Foreign body left in during procedure PSI07 - Catheter-related bloodstream infection PSI12 - Post-operative pulmonary embolism (PE) or deep vein thrombosis (DVT) PSI13 - Postoperative sepsis PSI15 - Accidental puncture or laceration

6 Countries Canada* Denmark Finland Israel New Zealand + Singapore Spain Sweden Switzerland United States* Norway * both POA and non-poa data + previous analysis

7 Methods The assessment of impact of each individual exclusion in the AHRQ algorithm Rate ratio of the difference between post- and pre-exclusion rate and the pre-exclusion rate Allows the code criteria to be met at any dg field Negative value = the rate-lowering effect Positive value = the rate-increasing effect

8 Example PSI05 LOS exclusion PreExcl 88 / 865,955 = LOS 1+ LOS 0 PostExcl 61 / 544,002 = Excl 27 / 321,953 = Impact = (PostExcl - PreExcl) / PreExcl = ( ) / * 100= +11%

9 Example PSI05 LOS exclusion Foreign body left in during procedure Case is a surgical or a medical discharge LOS < 24 hours or 0 days? yes Exclude no Age =18 y or >18 y? yes no yes Case is assigned to MDC 14 or the PDx is listed in table M3? no Exclude the impact of each individual exclusion PDx is identical to the numerator definition? yes Exclude no Add case to denominator population Count denominator population and report to OECD secretariat SDx is identical to the numerator definition? yes Count numerator population and report to OECD secretariat Add case to numerator population

10 Data collection

11 Results The results indicate that the exclusions within the AHRQ algorithms have varying impact While clinical exclusions are considered inherent to the construction of PSIs, length of stay and non-elective admission type exclusions are believed to introduce bias

12 % impact PSI12 DVT/PE length of stay <2 days

13 Events (num) PSI12 DVT/PE NZL LOS excl LOS (days)

14 LOS exclusion rate PSI12 DVT/PE NZL Initial rate rate lowering effect rate increasing effect LOS (days)

15 % impact PSI13 Postop.sepsis acute adm.types Non-elective admission types

16 PSI13 Postop.sepsis SPA & SWE Non-elective admission Exclusion rate Impact SPAIN 39.5% % SWEDEN 62.5% %

17 PSI13 Postop.sepsis initial vs. final rate Initial Final SPA SWE

18 Results cont. As a result of multiple exclusions, some indicators are calculated from very small samples and are therefore especially sensitive to variations

19 PSI13 Postoperative sepsis Numerator sample sizes Initial 2dx Final sample size Canada 3, Denmark 2, Finland Israel New Zealand 1, Norway Singapore 2, Spain 12,794 1, Sweden 3,555 1, United States 86,892 4, Total 116,093 8,

20 Results cont. Comparisons between the patient and discharge level data show relatively consistent ratios for the PSI05 - Foreign body left in during procedure PSI15 - Accidental puncture or laceration across the following rates: - patient (qualifying event in any dg field based on patient level data), - discharge (qualifying event in any dg field based on discharge data) - discharge sdx (qualifying event in secondary dg field based on discharge data as calculated from the AHRQ algorithms)

21 PSI05 Foreign body left in during proc patient discharge discharge sdx Canada Denmark Finland Israel New Zealand Singapore Spain Sweden United States

22 PSI15 Accidental puncture or laceration patient discharge discharge sdx Canada Denmark Finland Israel New Zealand Spain Sweden United States

23 Present on admission flag Data provided by Canada and the United States show considerable differences in terms of percent change due to POA coding for PSI12 DVT/PE and PSI13 Postoperative sepsis The effect is larger in general for the US than Canada across PSIs

24 Canada POA vs. non-poa

25 United States POA vs. non-poa

26 PSI Subgroup s recommendations Length of stay The AHRQ algorithms are built on several exclusions that are intended to affect the bias in comparison across US states Patient safety indicators with short length of stay (< 24h) exclusion have an effect of reducing bias, while longer length of stay exclusions tend to increase bias Recommendation 1: Collecting data for events by day breakdowns for length of stay may give greater understanding of the effect of this exclusion and inform possible revisions

27 PSI Subgroup s recommendations Non-elective admission type The post-operative sepsis indicator shows large and varying effects from the non-elective admission type exclusion This is due to varying definitions and coding practices across countries Recommendation 2: Collect additional information to understand how acute and elective admissions are defined in each country and consider possible revision of this exclusion

28 PSI Subgroup s recommendations Patient level data Patient safety indicators rely heavily on the quality of principal and secondary diagnoses data coded in hospital records Discharge level data does not provide the necessary information to detect under-reporting, hence those countries with UPI may be able to provide additional data for patient-level events Recommendation 3: Collect additional information on qualifying events in PDX field only based on UPI

29 PSI Subgroup s recommendations Coding practice Coding may be performed by various healthcare professionals or dedicated clinical coders Clinical coding practice could be also affected if the medical records determine financial reimbursement Recommendation 4: Collect additional information on coding and registry practices in each country

30 Members of the HCQI Expert Group are invited to: Comment on the findings of this analysis Decide on whether a limited number of PSIs - PSI05 Foreign body left in during procedure - PSI15 Accidental puncture or laceration - PSI18 Obstetric trauma due to vaginal delivery with instrument - PSI19 Obstetric trauma due to vaginal delivery without instrument is mature enough for the publication in the OECD Quality of Care document in preparation for the Ministerial meeting in October 2010 Decide on whether the collection of additional data in the year 2010/2011 is warranted to inform the future development work