ACS NSQIP Modeling and Data, July 14, 2013 Mark E. Cohen, PhD Continuous Quality Improvement American College of Surgeons
Today s presentation on ACS NSQIP statistics 1. An intuitive explanation of our: (1) patient risk adjusted; (2) procedure mix adjusted; and (3) shrinkage adjusted modeling 2. What do profiling results tell us 3. How are cases assigned to different models 4. How to interpret the site summary report and bar plots, particular with respect to the hospital odds ratio Hope to address some of the most recurring questions we get about ACS NSQIP statistics.
What is a statistical model and why is it needed? ACS NSQIP needs to provide fair comparisons of surgical quality across hospitals. Since every patient, and every hospital s patient pool and procedure mix, are different, we need to compensate for those differences. Hospital A: Smaller operations on healthier patients Hospital B: Bigger operations on sicker patients
What is a statistical model and why is it needed? A statistical model can be used as a mathematical recipe that gives direction on how to apply compensation. = expectations for adverse outcomes decreased by X amount, the observed event rate bar for being average is lowered = expectations for adverse outcomes increased by Y amount, the observed event rate bar for being average is raised There are many different, useful, recipes and none of them will be perfect in the way it compensates. Nevertheless, there is no justification for not compensating and for not using the best recipe available.
Three separate adjustments 1. Risk adjustment to control for differences in patient characteristics 2. Risk adjustment to control for differences in the types of surgeries undertaken (procedure mix) 3. Shrinkage adjustment to stabilize odds ratio estimates when sample sizes are small All three adjustments are implemented during the same modeling process.
Three separate adjustments 1. Patients Hospitals patients differ in age, general health, comorbidities, laboratory values, etc. Some patients are sicker than others this is the classic focus of risk adjustment. versus Common important variables are: Age, ASA Class, Functional Status, Albumin, Emergent, (pre-operative) Sepsis If we build predictive models that include these variables, their effects are controlled for what remains is a purer measure of the hospital contribution to patients outcomes.
Three separate adjustments 2. Procedures Hospitals differ in the complexity/risk profile of surgeries that they perform. versus versus Our two surgery-specific variables are: RVU and CPT based risk. If we build predictive models which include information about the specific surgery performed, their effects are controlled for what remains is a purer measure of the hospital contribution to patients outcomes.
Three separate adjustments 2. Procedures Adjustment for procedure (via CPT) has evolved through 4 stages 1. No Adjustment (reliance on RVU) 2. Adjustment for 10 organ system-based ranges of CPT codes 3. Adjustment based on approximately 300 groups of similar CPT codes. Groups constructed by our clinical experts based on surgical similarity In a preliminary model, CPT groups are used to predict an outcome A linear risk score is assigned to each CPT group (this turns the 300 categories into 1 continuous risk variable) Use this continuous risk variable in the follow on models 4. Adjustment for each CPT code
Three separate adjustments 2. Procedures As of this SAR - 4 years of SAR data, about 1.5 million surgical records, to assign a linear surgical risk score, for each of our adverse outcomes, to every primary CPT in ACS NSQIP. Yields finer-grained estimates, compared to estimates for 300 CPT categories. In aggregate, individual CPT risk yields slightly better predictions of outcomes more accurate models - than 300 CPT categories.
Three separate adjustments 3. Shrinkage In our models, we combine information we have for the hospital with what we know about all hospitals. We Shrink the hospital s estimate toward the grand mean the smaller the hospital s N, the greater the shrinkage toward the grand mean. With shrinkage adjustment, fewer hospitals are assigned extreme/unreasonable values. The estimates are stabilized. Imposing shrinkage is mathematically complicated but it is also very intuitive, and we commonly do it. Forced to make a decision with limited specific information, we include other general knowledge.
Three separate adjustments 3. Shrinkage The 1-record scenario the patient either lived or died (of course, no one would advocate drawing conclusions from 1- record, but the problem applies generally to small samples) The best estimate (naively) applying classical statistical methods (un-shrunken, ignoring ancillary information) is that either every surgical patient lives at that hospital or every patient dies. Is this reasonable? Is a wide confidence interval a sufficient solution? Is a required sample size (say, N 200), an optimal solution?
Three separate adjustments 3. Shrinkage The 1-record scenario the patient either lived or died The best shrunken (again, this is towards the grand mean) estimate will be that this hospital does slightly better (if the patient lived) or slightly worse (if the patient died) than the average hospital. Again, what we re doing is pooling information from the single record at hand, with what we know about all hospitals in general.
Three separate adjustments 3. Shrinkage The 1-record scenario the patient either lived or died The average error (the difference between reality and our estimates ) will tend to be larger for un-shrunken estimates than shrunken estimates.
Three separate adjustments 3. Shrinkage Colorectal Morbidity for 61 Hospitals Providing <= 50 cases 2.0 1.5 O/E or OR 1.0 0.5 0.0 Logistic O/E Modeling Method and Metric Hierarchical OR
The profiling result After all of this, the profiling result we get is the: (1) patient risk-adjusted; (2) procedure-mix risk adjusted; (3) shrinkage-adjusted hospital odds ratio where - odds = number of patients with an event/number of patients without an event. The greater the odds the more likely the event. odds ratio = odds at your hospital/odds at the average NSQIP hospital odds ratios < 1.0 indicate performance better than average, odds ratios > 1.0 indicate performance worse than average The next talk will address how odds ratios are reported
What does The profiling result tell us? Given our types of patients and procedure mix, are we doing better or worse than the average ACS NSQIP hospital doing the same procedures on the same patients. This is valuable information; given what we do, how are we doing? But, a patient s perspective might be which hospital offers me the best outcomes for my operation. This requires an extra-modeling stratification - limiting comparisons of eligible hospitals to those judged to do the required procedure on patients with this risk profile. Also valuable information, but different; given what needs to be done, who should do it?
What does The profiling result tell us? Public reporting can be inconsistent in differentiating between these perspectives, lay persons may not understand the issue Highly targeted models help to ameliorate the problem As ACS NSQIP models become finer-grained (from ALL CASES to Subspecialty to Targeted), we are comparing more similar procedures across hospitals - so there will be less concern about the for what we do restriction. Targeted models move us closer to answering the given what we do, how are we doing and given what needs to be done, who should do it questions simultaneously.
Assignment of cases to models new this this SAR SCRs select cases based on sampling protocols that are constructed specifically for the program in which the hospital participates GV, Multispecialty, Small/Rural, Targeted. However, the program-related sampling protocol does not determine the assignment of cases to models. Once the case is an official case, then will be included in one or more models, provided the case meets the inclusion criteria for those models. From least to most restrictive, these criteria are:
Assignment of data to models All Cases (AllCASES): All cases are included General (GV): Cases assigned to General or Vascular surgery* General (GEN): Cases assigned to General surgery* Vascular (VASC): Cases assigned to Vascular surgery* Colorectal (COLORECT): Cases with required colorectal CPT codes Surgical Subspecialty (SS): Cases assigned to the subspecialty* Measure DSM, Deep/OS SSI: All cases are included Measure Elderly: All cases where patient age is 65 Measure Colon, LEB: Cases with required CPT codes Measure UTI: All cases except those with certain CPTs Targeted: Cases with required CPT codes from Targeted hospitals that are participating in the particular surgical target
Assignment of data to models * Prior to this SAR, assignment to GV, GEN, VASC, and SS had been based on self-declared surgeon specialty While this supports a surgeon (or surgical-service analysis), these assignments may not always coincide with the actual type of surgery being done. With ACS NSQIP s recruitment of smaller hospitals, where general surgeons often do operations across many nominal specialties, the impact of this issue increases. For modeling, is it better to have a hodgepodge of surgical types or true surgical groupings? For this SAR we ve adopted CPT, rather than surgeon selfdeclared, specialty.
Assignment of data to models Approximately 2000 CPTs were assigned to GV, GEN, VASC, and Subspecialties 1. Some CPTs are easily assigned to a single surgical specialty. Self-declared surgeon specialty is ignored. 2. For other CPTs, two or more two or more surgical specialties are represented (overlapping turf). When that happens, we assign the case based on self-declared surgeon specialty. But, if the self-declared surgeon specialty is not among the acceptable specialties for that CPT, the case is assigned to the specialty with the greatest number of cases. 3. Finally, some surgeries can be assigned to any surgical specialty. For these, assignment is by self-declared surgeon specialty. Case assignments to models are slightly different using the two systems. Modeling results are slightly different; this is for the better.
Assignment of data to models Two consequences of this: 1. You might see models that you ve never seen before. Compared to past SARs, a hospital participating in GV may now see odds ratios reported for many subspecialties, if their general and vascular surgeons are doing surgeries with CPTs classified as those belonging to those subspecialties. 2. Your platform based surgeon-specialty counts may not match these CPT-based counts You will need to depend on posted SAR case occurrence reports to determine which of your cases are used in the models.
Resources These posted presentations The main SAR document The ACS NSQIP statistical staff Published reports and bibliographies Recently published paper in JACS
ACS NSQIP Statistical Staff Mark Cohen, PhD markcohen@facs.org Lynn Zhou, PhD lzhou@facs.org Kristopher Huffman, MS khuffman@facs.org Yaoming Liu, PhD yliu@facs.org Kari Kraemer, PhD kkreamer@facs.org Xiangju Meng, MS xmeng@facs.org Brian Matel, MA bmatel@facs.org Shenglin Chen, PhD schen@facs.org Associate Director and principal statistical/clinical consultant Bruce Hall, MD, PhD, MBA