Reliability of Evaluating Hospital Quality by Surgical Site Infection Type ACS NSQIP Conference July, 01
Surgical Site Infection Common cause of patient morbidity 5%-6% for colorectal procedures Significant associated consequences Longer length of stay, reduced health-related quality of life, higher mortality Cost of SSI: $10,443-$5,546 Policy focus on reducing SSI Public reporting of hospital SSI rates Value-based purchasing and pay-for-performance initiatives
Classification of SSI Superficial incisional SSI Deep incisional SSI Superficial SSI often easily treated Deep/organ-space SSI often req IV abx, perc drainage, and/or reoperation Organ-space SSI * CDC, ACS-NSQIP
Objective To determine the validity and reliability of measuring hospital quality based on risk-adjusted rates of: 1. Superficial SSI. Deep/organ-space SSI 3. Any SSI (superficial, deep, organ-space) Compared how each model ranks hospital-level quality for SSI (validity) Estimated the likelihood that differences in quality measurements reflect true differences in performance between hospitals (reliability)
Data Source and Study Sample ACS-NSQIP public use file, 009 Hospital-based surgical registry of patient risk factors and 30-day postoperative outcomes Colorectal procedures identified by CPT code Commonly performed, relatively high SSI rates 5,455 cases from 36 hospitals
Determination of hospital quality Developed a hierarchical multivariate logistic model for each outcome of interest: 1. Superficial SSI. Deep/organ-space SSI 3. Any SSI 30 covariates included for risk-adjustment Procedure type, diagnosis group, demographics, preoperative risk factors
Determination of hospital quality Hospital treated as a random effect in models Each hospital has a different random intercept Accounts for clustering of patients within hospitals Hospital quality for SSI determined by the random intercept odds ratio: Estimates the odds of SSI at the specified hospital vs. the odds at an average hospital Odds ratio >1 = Worse performance Odds ratio <1 = Better performance Compared how each model ranked hospital-level riskadjusted performance
Reliability of hospital quality measurement Reliability is the likelihood that differences in quality measurements reflect true differences in performance between hospitals Reliability = hospital specific error
Reliability of hospital quality measurement Reliability is the likelihood that differences in quality measurements reflect true differences in performance between hospitals Reliability = hospital specific error Hospital-to-hospital variance measures observed variation in risk-adjusted infection rates between hospitals in the cohort
Reliability of hospital quality measurement Reliability is the likelihood that differences in quality measurements reflect true differences in performance between hospitals Reliability = hospital specific error Hospital-to-hospital variance measures observed variation in risk-adjusted infection rates between hospitals in the cohort Hospital-specific-error is the sampling error or measurement error for an individual hospital
Reliability of hospital quality measurement Reliability = hospital specific error Measured on a scale from 0 1
Reliability of hospital quality measurement Reliability = hospital specific error Measured on a scale from 0 1 0 indicates that apparent variation between a hospital s quality measurement and the average hospital is statistically unreliable
Reliability of hospital quality measurement Reliability = hospital specific error Measured on a scale from 0 1 0 indicates that apparent variation between a hospital s quality measurement and the average hospital is statistically unreliable 1 indicates that any observed variation is due to a real difference in performance for that hospital
Reliability of hospital quality measurement Reliability = hospital specific error Measured on a scale from 0 1 0 indicates that apparent variation between a hospital s quality measurement and the average hospital is statistically unreliable 1 indicates that any observed variation is due to a real difference in performance for that hospital 0.7 = threshold for determining if a measure adequately discriminates real differences in quality
Results Poor agreement on hospital quality between superficial SSI and deep/organ-space SSI models (weighted kappa 0.05) Model Mean reliability of hospital quality measurement Superficial SSI 0.650 Deep/organ-space SSI 0.404 Any SSI 0.586
Reliability increases with increased sample size Reliability Reliability 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0. 0.1 0 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0. 0.1 0 A. Superficial SSI Model B. Deep/Organ-Space SSI Model 0 100 00 300 400 Number of Patients C. Any SSI Model 0 100 00 300 400 Number of Patients Reliability 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0. 0.1 0 0 100 00 300 400 Number of Patients 0.7 = threshold for determining if a measure adequately discriminates real differences in quality
Limited # of hospitals able to achieve reliability=0.7 Model Number of colorectal cases needed to achieve reliability = 0.7 % of hospitals nationwide that performed required caseload in 009* Superficial SSI 11 % Deep/organ-space SSI 458 1% Any SSI 156 16% * Nationwide Inpatient Sample, 009
Summary Risk-adjusted hospital quality is significantly different for superficial SSI and deep/organ-space SSI Aggregating SSI types (i.e. the any SSI model) does not confer an obvious advantage for improving the reliability of quality comparisons Limited number of hospitals nationwide perform sufficient colorectal cases to achieve reliability threshold of 0.7
Limitations Superficial SSI s may be undercoded if hospitals only record the more severe SSI type in pts with multiple SSI s May not be accurate to group deep and organspace SSI s ACS-NSQIP hospitals are predominantly medium and large academic hospitals, thus findings may not be generalizable
Implications Essential that measures by valid and reliable for high-stakes quality comparisons such as public reporting and pay-for-performance. High and low quality should be accurately and consistently identified Policies and initiatives aimed at reducing SSI should consider treating superficial and deep/organ-space SSI s as two distinct entities. Implications for other performance measures with aggregated outcome events
Acknowledgements Collaborators: John L. Adams Warren B. Chow Clifford Ko Bruce Lee Hall American College of Surgeons Veteran s Affairs Administration RWJF Clinical Scholars Program UCLA Department of Surgery